# Technical Documentation: Imbad0202/academic-research-skills > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/Imbad0202/academic-research-skills) ### Overview (`README.md`) # Academic Research Skills for Claude Code [](https://github.com/Imbad0202/academic-research-skills/releases/tag/v3.23.0) [](https://doi.org/10.5281/zenodo.20696614) [](https://creativecommons.org/licenses/by-nc/4.0/) [](https://buymeacoffee.com/crucify020v) [简体中文版](README.zh-CN.md) | [繁體中文版](README.zh-TW.md) | [日本語版](README.ja-JP.md) | [한국어](README.ko-KR.md) | [Español](README.es-ES.md) A comprehensive suite of Claude Code skills for academic research, covering the full pipeline from research to publication. **Install in 30 seconds** (Claude Code CLI / VS Code / JetBrains, v3.7.0+): ```text /plugin marketplace add Imbad0202/academic-research-skills /plugin install academic-research-skills ``` Then try `/ars-plan` to walk through your paper structure via Socratic dialogue, or jump to [Quick install](#quick-install) for prerequisites and the traditional symlink flow. > **AI is your copilot, not the pilot.** It can draft text, including a whole paper in full mode, but the decisions stay yours, and the pipeline stops for your confirmation at every stage. It handles the grunt work (hunting down references, formatting citations, verifying data, checking logical consistency) so you can focus on the parts that actually require your brain: defining the question, choosing the method, interpreting what the data means, and deciding what comes after "I argue that." You remain the author, and you answer for every claim you submit. > > Unlike a humanizer, this tool doesn't help you hide the fact that you used AI. It helps you write better. Style Calibration learns your voice from past work. Writing Quality Check catches the patterns that make prose feel machine-generated. The goal is quality, not cheating. ### Why human-in-the-loop, not full automation? Lu et al. (2026, *Nature* 651:914-919) built **The AI Scientist** — the first fully autonomous AI research system to publish a paper through blind peer review at a top-tier ML venue (ICLR 2025 workshop, score 6.33/10 vs workshop average 4.87). Their Limitations section enumerates the failure modes that any fully-autonomous AI research pipeline inherits: implementation bugs, hallucinated results, shortcut reliance, bug-as-insight reframing, methodology fabrication, frame-lock, citation hallucinations. ARS is built on the premise that **a human researcher augmented by AI avoids these failure modes better than either alone**. Stage 2.5 and Stage 4.5 integrity gates run a 7-mode blocking checklist (see [`academic-pipeline/references/ai_research_failure_modes.md`](academic-pipeline/references/ai_research_failure_modes.md)); the reviewer offers an opt-in calibration mode that measures its own FNR/FPR against a user-supplied gold set. [**Zhao et al.**](https://arxiv.org/abs/2605.07723) (2026-05) audited 111M references across 2.5M papers on arXiv, bioRxiv, SSRN, and PMC. Their conservative estimate is 146,932 hallucinated citations for 2025 alone, with an observed mid-2024 inflection; for the bioRxiv-to-PMC pairing they report 85.3% preprint-to-published persistence. The paper describes "real citations deployed to support claims the cited references do not actually make" as an open challenge. ARS v3.7.1 added trust-chain frontmatter for source provenance; v3.7.3 added locator infrastructure (three-layer citation anchors) for future claim-level audits and surfaces advisory risk signals at cite time (ARS labels the claim-faithfulness gap internally as "L3"; this is ARS terminology, not the paper's). v3.7.x is motivated by Zhao et al.'s corpus-scale findings; corpus-scale evaluation of ARS itself remains future work. v3.8 closes the second half of the L3 gap. v3.7.3 made every citation carry a locator anchor; v3.8 adds an opt-in audit pass (`ARS_CLAIM_AUDIT=1`) that fetches the cited source against each anchor and judges whether the claim is actually supported. Five new HIGH-WARN classes (claim-not-supported, negative-constraint-violation, fabricated-reference, anchorless, constraint-violation-uncited) gate-refuse output through the formatter terminal hard gate. A calibration runner ships with a 25-tuple synthetic gold set and FNR<0.15 + FPR<0.10 acceptance thresholds. Its shipped test drives the runner with a stub judge that returns the gold labels, so it checks the tooling, not a live judge; no live-judge calibration result is recorded yet, and the ramp-on plan waits for one (v3.8 spec §5). [**Ren et al.**](https://arxiv.org/abs/2607.13104) (2026, *Self-Improvements in Modern Agentic Systems: A Survey*) supplies a third, survey-level anchor. Its scientific-discovery synthesis (§7.4) concludes that discovery agents cannot easily verify novelty, correctness, or reproducibility on their own and may exploit weak proxies instead, must manage evidence across heterogeneous tools and literature, and raise governance issues — "scientific writing can also amplify misinformation when the evidence is weak." Its generation-loop chapters (§5.1–§5.2) list human auditing and retained human anchors among the practical safeguards for self-generated evaluation loops, and its historical chapter (§2.2) records the oldest form of the same lesson: the practical success of Lenat's EURISKO depended heavily on the user serving as the external evaluation signal, pruning unproductive heuristic drift — a limitation the survey notes persists in modern agentic systems. ARS cites the survey as design rationale for its human-in-the-loop stance, not as empirical proof that human-in-the-loop pipelines outperform autonomous ones; the survey's actionable deltas for ARS are tracked in #539–#541 and #547–#550. [**Gartenberg et al.**](https://doi.org/10.1287/orsc.2026.ed.v37.n3) (2026, *Organization Science* 37(3):795-812, "More versus better") supplies a fourth anchor, and the first from the journal side. The *Organization Science* AI Task Force scored every first submission (6,957) and every text-format review (10,389) the journal received between January 2021 and February 2026 with a commercial AI-writing classifier and standard readability indices. Manuscripts scored as heavily AI-written read worse on those indices and were desk-rejected more often; reviews scored as more AI-written leaned toward theory and away from data; and the editors conclude that current AI tools, amplified by publish-or-perish incentives, "appear to be pushing the system toward an equilibrium of more rather than better research." Their §5 contrasts "cognitive surrender" (Shaw & Nave, 2026, as cited there) with human-first use and asks authors to disclose how a manuscript was produced. The evidence is observational, aggregate, and from one journal, and the classifier is a proprietary instrument. ARS cites the editorial as design rationale for recording volume as a non-goal (see `POSITIONING.md`) and for the Collaboration Depth Observer and the claim-strength ladder, not as evidence about ARS output; the actionable deltas are tracked in #829–#833. [**Wang, Li et al.**](https://arxiv.org/abs/2609.07713) (2026-09, *The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishing*, a survey of 230 sources) supplies a fifth anchor, and the first that treats research production and peer review as one coupled system. Its evaluative-authority ladder (§4.1) runs from author-facing feedback through reviewer assistance and official AI reviews to scoring and decision support, with the survey's point that capability at one rung does not justify use at the next; ARS's simulated panel sits on the lowest rung by design (see `POSITIONING.md`). Two of its findings shape the reviewer roadmap. First, as the survey summarizes Dycke & Gurevych (2026, §4.5), 391 edits that break a paper's scientific support relations produced no statistically significant difference in the tested automated reviewers' aspects, sentiment, or scores compared with 540 soundness-neutral controls, while presentation-only rewrites with the science held fixed moved AI-review scores (§5.2); the survey's §9.2 conclusion is that a static evaluation can overstate an AI reviewer's reliability once authors can observe and adapt to it. ARS tracks the corresponding Round-1 paired controls in #871 and the author-identity-cue controls (§7.2) in #872; both are measurements, not new mechanisms. Second, its §9.1 cites Brodeur et al. (2026, *PNAS* 123(22):e2524747123), a randomized study in which 288 researchers in 103 teams reproduced published quantitative social-science results under three conditions: human-only, AI-assisted (ChatGPT as a collaborative tool), and AI-led (ChatGPT with minimal human oversight). Human-only and AI-assisted teams reproduced 94% and 91%, AI-led teams 37%, and the AI-assisted teams detected fewer major coding errors than the human-only teams. In that study, then, AI assistance did not verify better than humans alone and AI-led verification did much worse; ARS reads this as a reason to keep verification human-led at every checkpoint, not as evidence that its own checkpoints or integrity gates are effective. The survey is a synthesis rather than an experiment, its structured search stops at 2026-07-01 and the later targeted update did not rerun every query (§10), its deployment evidence is concentrated in a small number of AI/CS conferences, OpenReview-based settings, and selected journals, and its "arms race" framing is a lens, not a finding; ARS cites it as design rationale, not as evidence about ARS output. v3.3 was inspired by [**PaperOrchestra**](https://arxiv.org/abs/2604.05018) (Song, Song, Pfister & Yoon, 2026, Google): Semantic Scholar API verification, anti-leakage protocol, VLM figure verification, and revision-trajectory tracking. ARS now implements that last idea through categorical, evidence-anchored criterion trajectories rather than score deltas. --- ## Architecture & pipeline **👉 [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)** — the full pipeline view: flow diagram, stage-by-stage matrix, data-access flow, skill dependency graph, quality gates, and mode list. The architecture doc supersedes the sprawling pipeline description that used to live here. Everything about *what runs in which stage* now lives in one place. ## Quick install **Prerequisites** - [Claude Code](https://docs.claude.com/en/docs/claude-code/setup) (latest; plugin packaging requires recent versions) - `ANTHROPIC_API_KEY` exported, or set on first `claude` run - *Optional:* Pandoc for DOCX, tectonic + Source Han Serif TC for APA 7.0 PDF (Markdown output works without either) - *Optional (real Python):* needed only for the write-scope guard and a few opt-in commands; the core skills are prompt-driven. Details, including the Windows notes on Git Bash and the Microsoft Store Python stub, are in [docs/SETUP.md § Python (optional)](docs/SETUP.md#python-optional). > **Which controls are active in *your* install channel?** Availability varies by install channel. See the per-channel map: [docs/CONTROL_AVAILABILITY.md](docs/CONTROL_AVAILABILITY.md). **Plugin install (v3.7.0+, recommended):** ```text /plugin marketplace add Imbad0202/academic-research-skills /plugin install academic-research-skills ``` **Verify it works:** run `/ars-plan` and describe a paper you're working on — ARS will start a Socratic dialogue to map out chapter structure. For a single-shot test instead, try `/ars-lit-review "your topic"`. **👉 [docs/SETUP.md](docs/SETUP.md)** — full guide: install Claude Code, set up API keys, optional Pandoc/tectonic for DOCX/PDF, cross-model verification (`ARS_CROSS_MODEL`), and six installation methods (Plugin, project skills, global skills, claude.ai Project, repo-cloned, Claude Science import). **👉 [docs/DATA_FLOWS.md](docs/DATA_FLOWS.md)** — what leaves your machine (bibliographic resolvers, optional consent-gated cross-model calls, the plugin update check), what is cached locally, for how long, and how to turn each path off. **👉 [docs/RISK_REGISTER.md](docs/RISK_REGISTER.md)** — the standing risks the suite knows about, which existing controls address each one, the evidence status behind those controls, and what remains open. **Using Claude Science?** The five skills import directly: **Skills → Import from GitHub**, paste `https://github.com/Imbad0202/academic-research-skills`, **Preview**, then **Import** (requires v3.14.0+ of this repo — the importer reads the explicit skill paths in the marketplace manifest). Imports are point-in-time snapshots: re-import after ARS updates. Imported skills carry the ARS methodology (research / writing / review protocols); Claude Code-specific machinery — slash commands, hooks, subagent orchestration — does not transfer. See [docs/SETUP.md](docs/SETUP.md) Method 5 for details. **Using Pi?** Install the in-tree, community-maintained wrapper with `pi install git:github.com/Imbad0202/academic-research-skills`. It keeps the original ARS content authoritative and documents Pi-specific orchestration and hook limitations. See [`pi/README.md`](pi/README.md). **Using Codex CLI?** Install the sibling distribution instead: [`Imbad0202/academic-research-skills-codex`](https://github.com/Imbad0202/academic-research-skills-codex) — same workflow content, Codex-native packaging as a single `$academic-research-suite` skill with `ars-*` aliases. **Third-party platforms and integrations** that wrap or host ARS are listed in [THIRD_PARTY.md](THIRD_PARTY.md) — community-submitted and not reviewed or endorsed by the maintainer. **Governance:** who decides, what cross-model review does and does not provide, and the project's end-of-life posture are stated in [GOVERNANCE.md](GOVERNANCE.md); security reporting and triage in [SECURITY.md](SECURITY.md). ## Performance & cost **👉 [docs/PERFORMANCE.md](docs/PERFORMANCE.md)** — per-mode token budgets, full-pipeline estimate (about US$3–7 for a 15k-word paper at 2026-09 list prices, before cache discounts), and recommended Claude Code settings (Auto mode; Agent Team optional). ## Guides & articles - [Academic Writing Shouldn't Be a Solo Act](https://open.substack.com/pub/edwardwu223235/p/academic-writing-shouldnt-be-a-solo?r=4dczl&utm_medium=ios) — full pipeline walkthrough (English) - [學術寫作不該是一個人的事：一套開源 AI 協作工具如何改變研究者的工作流](https://open.substack.com/pub/edwardwu223235/p/ai?r=4dczl&utm_medium=ios) — 完整使用指南（繁體中文） --- ## Features at a glance - **Deep Research** — 13-agent research team with Socratic guided mode, PRISMA systematic review, intent detection, dialogue health monitoring, optional cross-model DA, Semantic Scholar API verification. - **Academic Paper** — 12-agent paper writing with Style Calibration, Writing Quality Check, LaTeX hardening, visualization, revision coaching, citation conversion, anti-leakage protocol, and VLM figure verification. - **Academic Paper Reviewer** — 7-agent multi-perspective peer review with criterion-bound, evidence-anchored narrative judgements (Journal-Fit Reviewer + 3 dynamic reviewers + Devil's Advocate), concession threshold protocol, attack intensity preservation, optional cross-model DA critique / calibration, R&R traceability matrix, read-only constraint. Current live reviews remain `NOT_CALIBRATED`; full calibration produces a bounded candidate profile, while live-profile application is not yet wired. - **Academic Pipeline** — 10-stage pipeline orchestrator with adaptive checkpoints, claim verification, Material Passport, optional `repro_lock`, optional cross-model integrity verification, mid-conversation reinforcement, and narrative criterion-by-criterion regression checks (the typed trajectory carrier is deferred). - **SR-Screener** — protocol-driven study screening for systematic, scoping and rapid reviews: two blinded AI reviewers plus a third-reviewer adjudicator, ordered exclusion codes, no silent defaults, resumable batch runs, QC (seed studies, near-miss rechecks, kappa and PABAK), PRISMA 2020 counts, EndNote/Zotero RIS groups, and a `literature_corpus[]` handoff to `academic-paper`. AI decisions are decision support; the review team verifies them. - **Data Access Level Metadata** (v3.3.2+) — every skill declares `data_access_level` (`raw` / `redacted` / `verified_only`); enforced by `scripts/check_data_access_level.py`. Pattern adapted from Anthropic's automated-w2s-researcher (2026). See [`shared/ground_truth_isolation_pattern.md`](shared/ground_truth_isolation_pattern.md). - **Task Type Annotation** (v3.3.2+) — every skill declares `task_type` (`open-ended` or `outcome-gradable`). All current ARS skills are `open-ended`. - **Benchmark Report Schema** (v3.3.5+) — JSON Schema + lint for honest benchmark comparisons. See [`shared/benchmark_report_pattern.md`](shared/benchmark_report_pattern.md). - **Artifact Reproducibility Lockfile** (v3.3.5+) — optional `repro_lock` sub-block on Material Passport. **Configuration documentation, not replay guarantee** — LLM outputs are not byte-reproducible. See [`shared/artifact_reproducibility_pattern.md`](shared/artifact_reproducibility_pattern.md). - **Model Tiering** (#517, v3.16+) — optional `ARS_MODEL_TIERING` switch with two directions: `economy` (execution-type agents dispatch one tier below the session model, floor Opus-class) and `quality-boost` (judgment-type agents at integrity gates and final review step up to the frontier tier). Default unset = byte-equivalent to pre-#517 behavior. See [`shared/model_tiering.md`](shared/model_tiering.md). - **Canonical Cross-Model Handoff Envelope** (#527, v3.17+) — the owner→dispatcher→owner blind-checkpoint transport path (#523) now has a machine-stable `[CROSS-MODEL-HANDOFF v1]` envelope with a normative Python grammar (`scripts/cross_model_handoff.py`) instead of prose-only enforcement, pinning agreement/divergence/malformed-result routing across all three checkpoint owners. See [`shared/cross_model_verification.md`](shared/cross_model_verification.md) §"Cross-model handoff envelope". - **Experiment Provenance Intake** (#260) — optional `experiment_provenance[]` on the Material Passport records experiments the scholar ran **externally** (ARS never runs experiments), and manuscript claims join to them via `claim_intent_manifest.planned_experiment_ids[]`. The integrity gate (Stage 2.5/4.5) audits each experiment-backed claim against declared provenance — `ALIGNED` / `OVERSTATED` / `NOT_SUPPORTED_BY_PROVENANCE` / `PROVENANCE_INSUFFICIENT` — **without judging whether the experiment itself was correct**. A fail-closed `experiment_intake_declaration` makes "did you run experiments?" an explicit decision the scholar answers after Stage 1, or on entry when a run starts later (even literature-only runs declare `no_experiments_declared`). See [`shared/handoff_schemas.md`](shared/handoff_schemas.md) §"Experiment Provenance Intake (#260)". **Integrity and verification boundary:** ARS checks the manuscript and the reported process—including citation existence, claim–source alignment, reported methodology, declared experiment–result alignment, figure/table fidelity, and reporting/process/package conformance. Some checks are sampled or LLM-mediated. ARS does **not** establish that procedures were actually performed, raw data are authentic, or results reproduce; a consistently reported fabrication can pass these checks. See [POSITIONING.md § Integrity checks and the empirical-work boundary](POSITIONING.md#integrity-checks-and-the-empirical-work-boundary). --- ## Showcase: real pipeline output See the complete artifacts from a real pipeline run, including peer review reports, integrity verification reports, and the final paper: > **A March 2026 record, not current performance.** This run (2026-03-07 to 03-08) used academic-pipeline v2.3, before ARS v3.3 added the Semantic Scholar check and v3.11 added the deterministic four-index citation gate. Its numbers describe that version; the current gates have not been measured on this paper. The byline names Claude (Anthropic) as author because the researcher asked for that during the experiment. The paper is not an Anthropic publication, and ARS's positioning is that the tool does not replace the researcher and does not claim authorship ([POSITIONING.md](POSITIONING.md#what-this-is-not)). **[Browse all pipeline artifacts →](examples/showcase/)** | Artifact | Description | |---|---| | [Final Paper (EN)](examples/showcase/full_paper_apa7.pdf) | APA 7.0 formatted, LaTeX-compiled | | [Final Paper (ZH)](examples/showcase/full_paper_zh_apa7.pdf) | Chinese version, APA 7.0 | | [Integrity Report — Pre-Review](examples/showcase/integrity_report_stage2.5.pdf) | Stage 2.5: flagged 15 problem references (8 with bibliographic errors, 6–8 likely fabricated) + 3 statistical errors | | [Integrity Report — Final](examples/showcase/integrity_report_stage4.5.pdf) | Stage 4.5: zero regressions confirmed | | [Peer Review Round 1](examples/showcase/stage3_review_report.pdf) | Journal-Fit Reviewer + 3 Reviewers + Devil's Advocate | | [Re-Review](examples/showcase/stage3prime_rereview_report.pdf) | Verification after revisions | | [Peer Review Round 2](examples/showcase/stage3_review_report_r2.pdf) | Follow-up review | | [Response to Reviewers](examples/showcase/response_to_reviewers_r2.pdf) | Point-by-point author response | | [Post-Publication Audit Report](examples/showcase/post_publication_audit_2026-03-09.pdf) | Full-reference audit run separately with Claude Code + WebSearch: 21 of the 68 final references still had problems after 3 rounds of integrity checks | --- ## Companion: Experiment Agent If your research involves running experiments (code or human studies) before writing, the [Experiment Agent](https://github.com/Imbad0202/experiment-agent) skill fills the gap between ARS Stage 1 (RESEARCH) and Stage 2 (WRITE). ``` ARS Stage 1 RESEARCH → RQ Brief + Methodology Blueprint ↓ experiment-agent → run/manage experiments → validate results ↓ ARS Stage 2 WRITE → write paper with verified experiment results ``` **What it does**: executes code experiments (Python, R, etc.) with real-time monitoring, manages human study protocols with IRB ethics checklist, interprets statistics with 11-type fallacy detection, and verifies reproducibility. **How to use together**: pause the ARS pipeline after Stage 1, run experiments in a separate experiment-agent session, then bring the results (with Material Passport) back to ARS Stage 2. ARS requires zero modification. See the [experiment-agent README](https://github.com/Imbad0202/experiment-agent) for setup instructions. **Stage 1 intake declaration (#260, #925)**: at the checkpoint after Stage 1, or on entry when a run starts after Stage 1, ARS asks you whether the paper reports experiments or data analyses you ran yourself, and records your answer, in your words, in a fail-closed `experiment_intake_declaration` on the Material Passport. ARS does not infer the answer from the manuscript. If you ran experiments externally, the scholar enters one `experiment_provenance[]` entry per experiment (`experiment_id`, nested `repro_lock`, `planned_vs_executed[]`, `negative_results[]`, `known_limitations[]`) and the declaration is set to `experiments_declared`; if not, it is set to `no_experiments_declared`. The declaration is **required on every post-#260 passport** — a run that touches no experiments still declares `no_experiments_declared`, so the integrity gate can never be silently bypassed by a forgotten provenance block. The `experiment_id`s are frozen at this intake point, which closes before the first Stage 2 writer is dispatched (or before the first integrity gate when no Stage 2 lies ahead), so you can pause after Stage 1, run the experiments, and record them when you return; the writers later reference them via `planned_experiment_ids[]`. **Teaching-side companion**: [Teaching Skills](https://github.com/YujxZJCN/teaching-skills) applies the ARS architecture (skill ensembles, shared contracts, staged gates, a Course Passport) to the teaching side of academic life — course design → lessons → assessment → delivery → reflection; its `sotl` mode hands classroom-inquiry projects off to ARS deep-research / academic-paper for the publication phase. --- ## Usage ### Quick Start ``` # Start a full research pipeline You: "I want to write a research paper on AI's impact on higher education QA" # Start with Socratic guidance You: "Guide my research on AI in educational evaluation" # Write a paper with guided planning You: "Guide me through writing a paper on demographic decline" # Review an existing paper You: "Review this paper" (then provide the paper) # Check pipeline status You: "status" ``` ### Individual Skills #### Deep Research (8 modes) ``` "Research the impact of AI on higher education" → full mode "Give me a quick brief on X" → quick mode "Do a systematic review on X with PRISMA" → systematic-review mode "Guide my research on X" → socratic mode (guided) "Fact-check these claims" → fact-check mode "Do a literature review on X" → lit-review mode "Compare these papers in WHY/HOW/WHAT format" → three-way-scan mode "Review this paper's research quality" → review mode ``` #### Academic Paper (11 modes) ``` "Write a paper on X" → full mode "Guide me through writing a paper" → plan mode (guided) "Build a paper outline" → outline-only mode "I have a draft, here are reviewer comments" → revision mode "Parse these reviewer comments into a roadmap" → revision-coach mode "Write an abstract for this paper" → abstract-only mode "Turn this into a literature review paper" → lit-review mode "Convert to LaTeX" / "Convert citations to IEEE" → format-convert mode "Check citations" → citation-check mode "Generate an AI disclosure statement for NeurIPS" → disclosure mode "Audit my rebuttal draft against the reviews" → rebuttal-audit mode ``` #### Academic Paper Reviewer (6 modes) ``` "Review this paper" → full mode (Journal-Fit Reviewer + R1/R2/R3 + Devil's Advocate) "Quick assessment of this paper" → quick mode "Guide me to improve this paper" → guided mode "Check the methodology" → methodology-focus mode "Verify the revisions" → re-review mode "Calibrate this reviewer against my gold set" → calibration mode ``` #### Academic Pipeline (Orchestrator) ``` "I want to write a complete research paper" → full pipeline from Stage 1 "I already have a paper, review it" → mid-entry at Stage 2.5 (integrity first) "I received reviewer comments" → mid-entry at Stage 4 ``` > Pipeline ends with **Stage 6: Process Summary** — auto-generates a paper creation process record with 6-dimension Collaboration Quality Evaluation (1–100 scoring). #### SR-Screener (8 modes) ``` "Turn my proposal into a screening protocol" → protocol mode "Is this abstract eligible for my review?" → quick mode (single-reviewer triage) "Pilot the screening with my seed studies" → pilot mode "Screen these database exports" → ta-screen mode "Screen the full texts of the advanced records" → ft-screen mode "Adjudicate the conflicts in my Rayyan export" → adjudicate mode "Double-check my exclusions" → audit mode "Give me the PRISMA numbers for the screening" → report mode ``` ### Supported Languages - **Traditional Chinese** (繁體中文) — default when user writes in Chinese - **English** — default when user writes in English - Bilingual abstracts (Chinese + English) for academic papers > **Using a different language?** Socratic mode (deep-research) and Plan mode (academic-paper) use **intent-based activation** — they detect the meaning of your request, not specific keywords. This means they work in **any language** without modification. > > However, the general `Trigger Keywords` section (which determines whether the skill is activated at all) still lists English and Traditional Chinese keywords. If you find the skill isn't activating reliably in your language, you can add your language's keywords to the `### Trigger Keywords` section in each `SKILL.md` file to improve matching confidence. ### Supported Citation Formats - APA 7.0 (default, including Chinese citation rules) - Chicago (Notes & Author-Date) - MLA - IEEE - Vancouver ### Supported Paper Structures - IMRaD (empirical research) - Thematic Literature Review - Theoretical Analysis - Case Study - Policy Brief - Conference Paper --- ## Skill Details Per-agent responsibilities and per-stage artifacts now live in [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md). Version numbers are anchored here so release metadata stays in one place. ### Deep Research (v2.12.1) 13-agent research team. Modes: full, quick, review, lit-review, three-way-scan, fact-check, socratic, systematic-review. Full agent roster and artifacts: see ARCHITECTURE.md §3. ### Academic Paper (v3.3.1) 12-agent paper writing pipeline. Modes: full, plan, outline-only, revision, revision-coach, abstract-only, lit-review, format-convert, citation-check, disclosure, rebuttal-audit. Output: MD + DOCX (via Pandoc when available) + LaTeX (APA 7.0 `apa7` class / IEEE / Chicago) → PDF via tectonic. Full agent roster and per-phase responsibilities: see ARCHITECTURE.md §3. ### Academic Paper Reviewer (v1.11.1) 7-agent multi-perspective review with **criterion-bound narrative judgements**. Modes: full, re-review, quick, methodology-focus, guided, calibration. Current live reviews and Schema 6 packages remain `NOT_CALIBRATED`; full calibration can produce a bounded candidate profile, but application to a live review is not wired. No numerical total is mapped to Accept, Minor Revision, Major Revision, or Reject. First-round review panel vs. contract-governed re-review dispatch boundary: see ARCHITECTURE.md §3 Stage 3 / Stage 3'. ### Academic Pipeline (v3.23.0) 10-stage orchestrator with integrity verification, two-stage review, Socratic coaching, and collaboration evaluation. Pipeline rules (protocol the agents follow, not runtime guarantees): every stage requires user confirmation checkpoint; integrity verification (Stage 2.5 + 4.5) is MANDATORY with no unrecorded bypass (every override requires user reasoning recorded for Stage 6); R&R Traceability Matrix (Schema 11) maps each reviewer concern to the author's revision claim and records whether the re-review verified it. v3.4 added the Compliance Agent (PRISMA-trAIce + RAISE) at Stage 2.5 / 4.5. v3.5 adds the **Collaboration Depth Observer** (`collaboration_depth_agent`, advisory only — never blocks) at every FULL/SLIM checkpoint and at pipeline completion. MANDATORY integrity gates (2.5 / 4.5) explicitly skip the observer so compliance checks are not diluted. Based on Wang & Zhang (2026), IJETHE 23:11. Stage-by-stage matrix with agents, artifacts, and gates: see ARCHITECTURE.md §3. ### SR-Screener (v1.0.0) 4-agent study screening between `deep-research` (question, protocol, search) and `academic-paper` (writing the review). Modes: protocol, quick, pilot, ta-screen, ft-screen, adjudicate, audit, report. Two blinded reviewer subagents (Read and Grep only) screen every record against the user-confirmed protocol, a third reviewer adjudicates advance-vs-exclude conflicts, and standard-library Python scripts parse RIS / PubMed .nbib / Web of Science / CSV exports, de-duplicate, batch, merge, and build the screening log, RIS groups, PRISMA 2020 counts, a methods draft with `[TO COMPLETE]` slots, and a `literature_corpus[]` file. Rules the agents follow (not runtime guarantees): no record is screened before the user confirms the protocol, no failed call becomes a default "exclude", and the review team verifies the decisions before the numbers are reported. See [`sr-screener/SKILL.md`](sr-screener/SKILL.md). --- ## v3.0 Optimizations: What We Discovered About AI's Structural Limits ### What happened While using ARS to write a reflection article about AI in higher education, I ran into three structural problems that no amount of prompt engineering could fix: 1. **Frame-lock**: I asked the AI to run a devil's advocate debate against its own thesis. It did — four rounds, each more refined than the last. But every round stayed inside the frame I'd set. The DA attacked arguments, never premises. It never asked "are we even discussing the right question?" This is the same pattern that caused the 31% citation error rate in v2.7's stress test: the verifying AI and the generating AI share the same cognitive frame. 2. **Sycophancy under pushback**: Every time I challenged the DA's attacks, it conceded too quickly. It retracted findings faster than it launched them. The model's training rewards conversational harmony — so "the user pushed back" was treated as evidence that the attack was wrong, when often it just meant the user was persistent. 3. **Intent misdetection**: The Socratic Mentor kept trying to converge and produce deliverables ("Want me to write this up?") when I was still exploring. It couldn't distinguish "the user wants a deep philosophical discussion" from "the user wants an RQ brief." Both look like engagement, but they need opposite AI behaviors. ### What we changed (v3.0) **Devil's Advocate — Concession Threshold Protocol** (`deep-research` + `academic-paper-reviewer`) - DA must now score every rebuttal on a 1-5 scale before responding - Concession only allowed at score ≥4 (rebuttal directly addresses core attack with evidence) - Score ≤3: hold position and restate the original attack - Anti-sycophancy rules: no consecutive concessions, concession rate tracking, frame-lock detection after each checkpoint **Socratic Mentor — Intent Detection Layer** (`deep-research`) - Classifies user intent as exploratory vs. goal-oriented at dialogue start and every 3 turns - Exploratory mode: disables auto-convergence, raises max rounds to 60, prohibits "want me to summarize?" prompts - Goal-oriented mode: standard convergence behavior - Anti-premature-closure rules: in exploratory mode, the user decides when to stop **Socratic Mentor — Dialogue Health Indicator** (`deep-research`) - Silent self-assessment every 5 turns on three dimensions: persistent agreement, conflict avoidance, premature convergence - Auto-injects challenging questions when agreement pattern detected - Invisible to user (to prevent gaming), but log available for post-session review ### Why this matters These optimizations don't solve AI's structural limits — they make the limits visible and manageable. The DA will still eventually concede if pushed hard enough. The Socratic Mentor will still have some convergence bias. But now there are explicit checkpoints that slow down the sycophancy, force the DA to justify concessions, and prevent the Mentor from wrapping up before the user is ready. The deeper lesson: AI literacy isn't about learning to use AI as a tool, following ethics rules, or fearing AI risks. It's about engaging AI deeply enough to discover its structural limits yourself — and your own thinking limits in the process. --- ## License This work is licensed under [CC-BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). **You are free to:** - Share — copy and redistribute the material - Adapt — remix, transform, and build upon the material **Under the following terms:** - **Attribution** — You must give appropriate credit - **NonCommercial** — You may not use the material for commercial purposes **Attribution format:** ``` Based on Academic Research Skills by Cheng-I Wu https://github.com/Imbad0202/academic-research-skills ``` --- ## Contributors **Cheng-I Wu** (吳政宜) — Author and maintainer **[aspi6246](https://github.com/aspi6246)** — Contributor. The v3.1 optimization was inspired by patterns from [Claude-Code-Skills-for-Academics](https://github.com/aspi6246/Claude-Code-Skills-for-Academics): read-only constraint pattern, anti-pattern codification as first-class design, cognitive framework approach (teaching "how to think" not just procedures), and lean skill size philosophy. **[mchesbro1](https://github.com/mchesbro1)** — Contributor. Originally proposed and drafted the IS Basket of 8 journals for `academic-paper-reviewer/references/top_journals_by_field.md` ([Issue #5](https://github.com/Imbad0202/academic-research-skills/issues/5)). **[cloudenochcsis](https://github.com/cloudenochcsis)** — Contributor. Extended the IS section from the *Basket of 8* to the full *Senior Scholars' Basket of 11* — adding *Decision Support Systems*, *Information & Management*, and *Information and Organization* ([Issue #7](https://github.com/Imbad0202/academic-research-skills/issues/7), [PR #8](https://github.com/Imbad0202/academic-research-skills/pull/8)). Sourced from the [AIS Senior Scholars' List of Premier Journals](https://aisnet.org/research/seniorscholarsbasket/). **[eltociear](https://github.com/eltociear)** (Ikko Eltociear Ashimine) — Contributor. Translated the Japanese README ([`README.ja-JP.md`](README.ja-JP.md)) ([PR #161](https://github.com/Imbad0202/academic-research-skills/pull/161)). **[xpfo-go](https://github.com/xpfo-go)** (xpfo) — Contributor. Translated the Simplified Chinese README ([`README.zh-CN.md`](README.zh-CN.md)) ([PR #181](https://github.com/Imbad0202/academic-research-skills/pull/181)). **[devCharlotte](https://github.com/devCharlotte)** — Contributor. Translated the Korean README ([`README.ko-KR.md`](README.ko-KR.md)) ([PR #469](https://github.com/Imbad0202/academic-research-skills/pull/469)). **[Yaobin29](https://github.com/Yaobin29)** — Contributor. Proposed reviewer-response tooling in [PR #433](https://github.com/Imbad0202/academic-research-skills/pull/433); the `deep-research three-way-scan` mode and the `academic-paper rebuttal-audit` mode (rescued from the PR's `audit` concept) were integrated from that contribution in v3.12.1. **[ktao732084-arch](https://github.com/ktao732084-arch)** — Contributor. Expanded the `academic-paper` disclosure system with nine medical-publishing policy targets, target-specific required-fact intake, and fail-closed standalone rendering ([Issue #596](https://github.com/Imbad0202/academic-research-skills/issues/596), [PR #599](https://github.com/Imbad0202/academic-research-skills/pull/599)); expanded the EQUATOR clinical-reporting reference with condensed CARE, STARD and TRIPOD+AI guidance plus a fail-closed study-design routing sequence ([Issue #594](https://github.com/Imbad0202/academic-research-skills/issues/594), [PR #601](https://github.com/Imbad0202/academic-research-skills/pull/601)); and designed and contributed the standalone Chinese-literature resolver, API protocol, and synthetic transport-fixture suite ([Issue #595](https://github.com/Imbad0202/academic-research-skills/issues/595), [PR #600](https://github.com/Imbad0202/academic-research-skills/pull/600)). **[didacrios](https://github.com/didacrios)** — Contributor. Translated the Spanish README ([`README.es-ES.md`](README.es-ES.md)). --- ## Changelog Only the three most recent releases are summarized here. The full release history, including the one-paragraph summaries that used to live on this page, is in [CHANGELOG.md](CHANGELOG.md). ### v3.23.0 (2026-10-03) — `sr-screener` as a fifth skill, repairs from a pipeline walk-through, and evidence and ledger hardening > **One new skill and behaviour repairs found by reading a full pipeline run end to end; the prompt-level changes are unmeasured, and `sr-screener` makes no screening-accuracy claim:** v3.23.0 adds `sr-screener` (#919, contributed by @erfanz97), a fifth skill that turns a review protocol into eligibility rules the user confirms and screens titles/abstracts and full texts with two blinded reviewer subagents and an adjudicator; its tests use synthetic records, and its spreadsheet exports neutralise formula text (#951). A walk-through of a default pipeline run on paper (#925-#929) changed what such a run does: the v3.6.7 Audit Artifact Gate is opt-in, the scholar is asked once whether the paper reports their own experiments, constraints set for a run are kept in the user's words and quoted to each later dispatch they apply to (at review stages they act at the checkpoints), the integrity gates follow one rule set and list paywalled sources as notes, Stages 5 and 6 produce only the files asked for, and opt-in switches surface Stage 5 refusals at Stage 4.5. The Stage 2.5 and 4.5 checkpoints replay evidence rows from a folder the orchestrator names (#933, #947, #948), a source the gate finds fabricated stays out of later revisions (#936), and the ledger readers report parse failures without quoting the file (#898, #945). Smaller changes: per-source method weaknesses in reading outputs (#916), a review-form note at a fixed point (#921), Schema 1 aligned with its producer (#938), and `/ars-citation-check` inheriting the session model (#912). ### v3.22.2 (2026-09-25) — Run ledger and handoff check, acronym check, a wider instruction/data boundary, and routing and front-page repairs > **Two deterministic checks pinned by synthetic tests; the prompt-level changes are unmeasured:** v3.22.2 adds a run ledger (#887). When a pipeline run has a passport file, the orchestrator appends the user's initial instructions, each checkpoint's question and the user's answer in their exact words, step receipts, and file hashes to a local ledger beside the passport. After compaction, resume, or a subagent return, `scripts/run_ledger.py report` compares the ledger with what the summary or report claims and lists the differences; it now prints that handoff check itself, in English or Traditional Chinese, and hashes the files an entry names when the entry is written (#898). The ledger holds the user's exact words, and `docs/DATA_FLOWS.md` lists it and how to delete it. The release also adds `scripts/check_acronyms.py` (#849, proposed by @reiropke), which reports acronyms that are undefined, defined after first use, or defined twice, without calling a model; the prompts have the calling session run it on saved drafts and abstracts, and a review appends its report to the Editorial Decision Letter as an advisory attachment that the decision, the revision roadmap, and re-review criteria do not draw on. Synthetic tests pin both scripts; whether runs write the ledger entries or call the check is unmeasured. The instruction/data boundary now reaches third-party text in dispatches and passport imports (#890), text a receiver reads through its own tool calls, and each skill's main session (#894); a lint pins every copy, and the effect is unmeasured (the opt-in claim-audit judge prompt changes with it, so cached verdicts from the old prompt are not reused). Repairs: the routing core reaches plugin and skills-copy installs (#892); an explicit request stays explicit when the mode's usual input is missing (#889); `/ars-lit-review` no longer points a run under way to another workflow (#897); the revision coach keeps peer review out of the committee-correspondence variant (#854); the orchestrator scopes "authoritative" skill output to deliverable ownership (#888); and the front page and showcase now match their sources (#908). The routing results come from one session per fixture, a smoke test rather than a rate. One new schema describes the ledger; no existing schema, command model, or effort setting changes. ### v3.22.1 (2026-09-23) — Model currency for Claude Opus 5.5, citation-check loading and Chinese APA 7 repairs, and a Pi wrapper fix > **Model currency and repairs; the new prompt-level guard is unmeasured:** v3.22.1 names Claude Opus 5.5 beside Claude Fable 5.1 as a supported session model, after a two-reader audit of the Opus 5.5 system card that retires no guardrail (#883). The docs add effort guidance (Claude Code starts Opus 5.5 at `medium`; heavy runs should use `high` or above), one list-price re-derivation for both models, and tiering guidance that reads the ladder as the vendor's lineup order, not a capability order. Because the card reports that Opus 5.5 acts more often than earlier models on instructions inside pasted text, the revision coach now treats pasted reviewer and committee text as data, pinned by a lint; the effect of that prompt-level guard is unmeasured. The release also repairs mode loading and citation checks: the 13 plugin mode commands invoke their namespaced core skill and resolve bundled references from the plugin root, which restores citation-check loading (#857); Chinese APA 7 checks catch a missing in-text author abbreviation, keep ambiguity exceptions and complete reference-list author fields, and propose a reorder only with evidence of a stroke-order inversion (#882); citation checks now distinguish visible syntax errors from unverified resolution or source claims (#882); and new English, Traditional Chinese, and Korean trigger phrases route citation-check requests, with CI bounding each skill description at 1,024 code points (#858, #864). The Pi wrapper accepts string-array system prompts (#880). No schema, command model, or effort setting changes. --- ### Overview (`evals/README.md`) # evals/ — gold-set corpora for ARS measurement targets This directory holds the v3.10 #184 generalized gold sets. Each subdirectory under `gold/` is a self-contained gold set for one measurement target. The structure mirrors the v3.8 `scripts/fixtures/claim_audit_calibration/` pattern but generalizes to multiple targets. ## Layout ``` evals/ ├── README.md # this file ├── gold/ │ ├── citation_extraction/ # Phase 1a/1b — baseline for #182 │ │ ├── README.md │ │ ├── manifest.yaml │ │ ├── tuples/ │ │ │ └── NNN-{kind-slug}-{discriminator}.json │ │ └── expected_outcomes.json │ ├── rq_framing_patterns/ # #257 Socratic wording advisory calibration │ ├── status_classification/ # Phase 2 (lands post-#183) │ └── summarization_adequacy/ # Phase 2 (lands post-#183) ``` ## Running the harness (#184 Phase 1b) The multi-task harness `scripts/run_evals.py` discovers every `gold//manifest.yaml`, measures each task, and emits a report shaped by `shared/evals_lift_report.schema.json`: ``` PYTHONPATH=. python -m scripts.run_evals # all tasks PYTHONPATH=. python -m scripts.run_evals --task citation_extraction --output report.json PYTHONPATH=. python -m scripts.run_evals --baseline before.json --compare after.json ``` `--baseline` + `--compare` produce a side-by-side report (`lift_pre` / `lift_post`). The ranking-lift gate `scripts/check_ranking_lift.py` reads those reports and blocks on un-acknowledged regressions; CI wires both via `.github/workflows/eval-harness.yml` (Delta 3 path filter). Tasks whose entrypoint module or gold set is not yet present are reported as `pending`/`skipped` rather than failing. ## Authoring conventions See each task's `README.md` for task-specific conventions (tuple naming, kind distributions, expected outcomes shape). ## Validator Run `python -m scripts.check_evals_gold_set evals/gold/` to validate any gold set against its manifest. The same validator runs in CI on every PR that touches `evals/gold/**`. ## Provenance - Phase 1a (citation-extraction gold set) + Phase 1b (`run_evals.py` harness + lift gate): v3.10 #184, spec `docs/design/2026-05-21-v3.10-184-extend-eval-harness-spec.md` - RQ framing patterns: Kong #257 idea-diversity advisory, spec `docs/design/2026-05-28-kong-257-idea-diversity-coverage-gap-advisory.md` - Phase 2 (status + summarization): scheduled post-#183 ship --- ### Overview (`pi/README.md`) # Pi wrapper Thin, community-maintained compatibility wrapper for running the original Claude Code Academic Research Skills (ARS) in [Pi](https://pi.dev/). The wrapper does not copy or modify ARS skill, agent, reference, schema, script, or command content. Pi loads the four original `SKILL.md` trees and exposes the original `commands/ars-*.md` files as prompt templates. A small input adapter reads the selected original command at invocation time and preserves trailing user arguments. ## Load-bearing runtime differences from Claude Code These differences affect how ARS evidence should be interpreted: 1. **The wrapper does not provide agent isolation or orchestration.** If the Pi setup has no matching subagent, workflow, or parallel-agent capability, ARS specialist roles run sequentially in the current context. That is degraded execution and must be disclosed; it is not independent multi-agent review. 2. **Claude hooks do not run in Pi.** In particular, ARS write-scope enforcement remains prompt-level unless the user's Pi environment supplies a separate enforcement mechanism. Do not represent the Claude `PreToolUse` hook boundary as active. Run `/ars-pi-doctor` to inspect the current environment. The curated [degraded-mode end-to-end evidence](../examples/pi/README.md) shows both boundaries in a completed run. ## Requirements - Pi with package support (`pi install` / `pi -e`) - The complete repository package, loaded from GitHub or from a local checkout; `pi/` is only a wrapper around its parent files and cannot be copied by itself - Optional capabilities depend on the user's Pi setup: - a subagent, workflow, or parallel-agent skill/tool for true multi-agent execution - a web-search or page-retrieval skill/tool for literature search and verification - Python, Pandoc, and tectonic for the optional ARS features that already require them There are no required Pi orchestration or web-search dependencies. When a capability is unavailable, the wrapper tells Pi to use an installed equivalent or disclose degraded execution instead of pretending the operation ran. ## Try without installing Load the package temporarily from GitHub for the current Pi run: ```bash pi -e git:github.com/Imbad0202/academic-research-skills ``` To try a local checkout instead, clone the repository first: ```bash git clone https://github.com/Imbad0202/academic-research-skills.git cd academic-research-skills pi -e ./pi ``` Then inspect optional capabilities and try a mode: ```text /ars-pi-doctor /ars-plan /ars-lit-review AI-assisted systematic reviews /ars-reviewer /ars-full ``` The original skills are also directly available: ```text /skill:deep-research /skill:academic-paper /skill:academic-paper-reviewer /skill:academic-pipeline ``` For automatic ARS skill selection from subsequent natural-language prompts, explicitly enable ARS mode first: ```text /ars-pi-start ``` ## Install from GitHub Pi can install the canonical repository directly because the root `package.json` points to this wrapper and the original ARS resources: ```bash pi install git:github.com/Imbad0202/academic-research-skills ``` Pin a branch, tag, or commit that contains the Pi wrapper when needed: ```bash pi install git:github.com/Imbad0202/academic-research-skills@REF ``` Update installed Git packages with `pi update --extensions`. Remove this package with: ```bash pi remove git:github.com/Imbad0202/academic-research-skills ``` ## Install from this checkout Keep the checkout at a stable path, then run from the repository root: ```bash pi install . ``` The nested manifest also supports local installation from the wrapper directory: ```bash pi install ./pi ``` Pi stores a local path rather than copying the repository. Update ARS with normal Git operations; the wrapper continues to load the original files. If an optional sandbox restricts reads to the current working directory, allow read access to the repository checkout or run Pi from the repository root. Supporting files are intentionally not copied into the wrapper. Remove using the same source form used during installation: ```bash pi remove . # or: pi remove ./pi ``` ## Capability doctor `/ars-pi-doctor` runs without an LLM call. It reports discovered orchestration and web-retrieval capabilities plus Python, PyYAML, Pandoc, tectonic, sandbox, and Claude-hook status. Missing optional dependencies remain the user's choice; the wrapper does not install them. ## System-prompt scope For prompts submitted while Pi is idle, installing the package does not add ARS text to ordinary Pi prompts. While ARS mode is inactive, the wrapper removes only this package's four ARS skill entries from the newly started run's system prompt, preventing automatic model invocation without modifying the original `SKILL.md` files. When Pi is idle, an `/ars-*` command or one of the four direct `/skill:*` entries above activates ARS mode before the new agent run starts, so that request receives both the original skill and the compatibility note. `/ars-pi-start` explicitly enables ARS mode and automatic skill selection for subsequent natural-language prompts. The state survives resuming the same session, follows the selected branch during `/tree` navigation, and resets in a new session. `/ars-pi-doctor` does not activate it. Pi 0.83.0 does not rebuild the system prompt when a prompt is queued into an agent run that is already streaming. A mid-stream `/ars-*` or direct `/skill:*` prompt can therefore execute under that run's existing system prompt without the wrapper-injected compatibility note; similarly, `/ars-pi-stop` changes the persisted mode immediately but cannot remove ARS text from the in-flight run. The RPC `steer` and `follow_up` methods also bypass Pi's `input` event, so a direct ARS `/skill:*` sent through either method does not activate the wrapper. To receive the scoped system-prompt guarantee, wait for Pi to become idle and submit the request through the normal interactive or RPC `prompt` path. To hide the ARS skills again and continue unrelated work without the compatibility note, run: ```text /ars-pi-stop ``` ## Wrapper regression test The wrapper accepts both upstream Pi's single-string system prompt and a string-array prompt (as observed in OMP 18.2.6). Array inputs remain arrays: the inactive XML filter runs per block, and activation appends the compatibility note as a separate block without mutating the host's array. This is prompt-container compatibility, not a full OMP port. Skill hiding still recognizes Pi's XML `` entries; it does not hide OMP's different `` text listing. Package discovery and complete research workflows in other hosts are outside this change. ```bash node --test pi/wrapper.test.mjs ``` The test covers idle-prompt skill hiding through canonical and symlink-spelled load paths, XML-escaped locations, preservation of missing and adjacent unrelated skills across alternate block formatting, same-request `/ars-*` and direct `/skill:*` activation while idle, manual start/stop toggling, `/tree` state restoration, argument-safe script-path rewriting, and single-pass argument-placeholder substitution. The array regressions additionally cover selective XML filtering, preservation of block boundaries and frozen input arrays, activation, repeated preparation, and deactivation. ## What the wrapper translates The wrapper reads `/ars-*` invocations from the original Claude command files, strips their frontmatter, appends trailing arguments when the command has no argument placeholder, converts executable `python scripts/...` paths to checkout-absolute paths, and expands the original target `SKILL.md` through Pi's native `/skill:*` mechanism. While ARS is active in the session, it also adds a short compatibility note to Pi's system prompt: - repository-root ARS paths resolve against this checkout - Claude tool names mean “use the equivalent available Pi capability” - multi-agent work searches available tools and configured Pi skill locations for an installed orchestration capability, otherwise it uses sequential execution with a disclosure - `WebSearch`, `WebFetch`, and `/websearch` search available tools and configured Pi skill locations for an installed web capability; no capability means no verification claim - Claude-specific command model hints are ignored, so the active Pi model is inherited The `pi/package.json` manifest performs the remaining mapping directly: | Claude distribution resource | Pi resource | | --- | --- | | four original skill directories | four Pi skills | | `commands/ars-*.md` | `/ars-*` Pi prompt templates with argument-preserving native skill expansion and absolute utility-script paths | | Claude tool/runtime assumptions | short capability-based compatibility note | ## Scope This is intentionally a basic wrapper, not a reimplementation of Claude Code or ARS orchestration. The original ARS content remains authoritative and unmodified. The project license remains CC BY-NC 4.0. Attribution and noncommercial restrictions apply. --- ### Overview (`plugin-evals-citation-check/README.md`) # plugin-evals-citation-check — `academic-paper` citation-check flow Eval suite for `claude plugin eval`. One flow per suite: **citation-check** (manuscript excerpt + reference list + a user-supplied "source pack" → citation error report). Sibling of `plugin-evals/` (revision-coach); the two suites do not share cases. Quality spec (author-defined, 2026-09-13): the four citation failures that matter are 無中生有 (a reference the user has no source for), 張冠李戴 (right paper, wrong authors), 小題大作 (a hedged or minor finding cited as an established result), and 以訛傳訛 (citing a retracted or concern-flagged paper as live evidence). Every fire case supplies a complete source pack so these are detectable **offline**; the sandbox has no network, so "無中生有" is graded as "flags it as unverifiable and does not claim it exists", never as a real lookup. Secondary axes: mechanical errors (orphans, numbering, et al., & vs and), no false positives on clean entries, no claim of online verification, no rewriting of the manuscript. All inputs are synthetic (fictional papers, journals, authors; DOIs use the reserved `10.5555` example prefix and graders explicitly allow the model to say so). ## Run ```bash claude plugin eval . --eval-dir plugin-evals-citation-check --ablation with-without --judge-model opus ``` Add `--no-publish` to keep the HTML report local. Headline number is Δ (with-plugin score − without-plugin score). `runs: 3` per case. Cases request `model: sonnet`, an alias that moves to each new Sonnet release, so compare runs only when the resolved agent and judge versions match (the `/ars-citation-check` command itself inherits the session model). The judge must be a different, larger model — the runner's default judge is haiku, which is both too small and never to be used here. ## Cases | Case | Fires? | Style × language | Planted | Primary graders | |---|---|---|---|---| | 01-apa-en-misattribution | yes | APA 7 × en | wrong authors; ref not in pack; orphan in-text; uncited entry; & vs and | content-caught (llm w1.5) | | 02-apa-zh-mixed-overclaim | yes | APA 7 × zh-TW mixed | claim strength exceeds source; Expression of Concern in pack; 三人未用「等」 | content-caught (llm w1.5) | | 03-ieee-en-prose | yes | IEEE × en | wrong authors; simulation cited as production; [7] with no entry; numbering not in order of appearance | content-caught (llm w1.5) | | 04-vancouver-en-terse-retraction | yes | Vancouver × en, style not named | retraction in pack; ref not in pack; year mismatch on one entry | content-caught (llm w1.5) | | 05-apa-es-locale | yes | APA 7 × es (#850 scenario) | wrong authors; orphan in-text; uncited entry | content-caught (llm w1.5) + locale-respected (llm w1) + no-chinese-chars regex | | 06-chicago-nb-en-footnotes | yes | Chicago NB × en | book not in pack; hypothesis cited as established; note without bibliography entry; pinpoint outside page range | content-caught (llm w1.5) | | 07-neg-convert-apa-to-ieee | no (format-convert shape) | — | — | is-conversion (llm) + numbered-in-order regex + not-audit-report regex | | 08-neg-python-unused-imports | no | — | — | regex on the two unused imports co-occurring with an "unused" statement + `tool_used: Skill` with `min: 0, max: 0, arm: both` (scored in both arms, unlike the display-only `skill-fired`) | Shared graders on 01–06: `content-caught` (w1.5), `format-caught` (w0.5, spec- level mechanics), `no-false-positive` (w1), `honest-unverified` (w1), `no-overreach` (w0.5), plus `skill-fired` (`tool_used: Skill`, matching `academic-research-skills:academic-paper`, display-only under ablation, never moves Δ). The exact core-skill match excludes command-stub-only invocation. With `--ablation none`, this grader contributes to the score; report loading coverage separately from output-quality grades. Every llm rubric is written as "work through the checks one at a time and quote the evidence"; keep that style when adding graders. ## Routing verification (2026-09-21) Claude Code 2.1.278, `--ablation none --model sonnet --judge-model opus --runs 1 --no-publish`, from a working directory outside the plugin checkout: - All six positive cases called `academic-research-skills:academic-paper`. Cases 07 (format conversion) and 08 (unused Python imports) passed their negative-case rubrics; 08 made no Skill call. - The full eight-case run passed six cases. Cases 01 and 02 hit the 600-second timeout **after** calling the core skill; their output-quality failures remain failures, not successful end-to-end runs. - Isolated reruns: case 01 passed all rubrics; case 02 completed with the core skill invoked but failed `format-caught` and `no-false-positive` (score 0.75). It omitted the three-author 「等」 correction and incorrectly called the correctly stroke-ordered Chinese references misordered. These output-quality failures were the motivation for #882 below; the routing acceptance was not a suite-wide pass. - A separate manual `/academic-research-skills:ars-citation-check` smoke, using case 03's synthetic material, called the core skill and read `citation_compliance_agent.md`. This is runtime evidence for citation-check, not a runtime coverage claim for all 13 mode commands. These are loading/routing checks, not a with/without ablation or a measured quality uplift. Earlier candidates are not pooled with this run: simply adding nested Skill instructions still allowed command-only execution; hiding mode commands without updating the SessionStart announcement led to attempted calls to user-only commands. The announcement now names the core Skill-tool targets for natural-language requests, including resume/compact events. ## Chinese format repair verification (#882, 2026-09-21) The same case-02 prompt and **all original quality rubrics remain unchanged**. Additional `agent-loaded` / `locale-guide-loaded` indicators record Read calls; trace inspection checks that those reads succeeded. They are not quality scores. The first repair candidate caught the missing three-author abbreviation and accepted the correct stroke order in all three outputs. One output still failed `no-false-positive` by promoting a DOI-prefix heuristic into a required reference correction. The final candidate adds a general evidence boundary: visible DOI syntax can be checked, but resolution/source claims require actual resolver/source evidence. No fixture prefix or fixture author names are added to the operative rules. Final candidate, Claude Code 2.1.278, observed agent `claude-sonnet-5`, requested judge `opus`, `--ablation none --no-publish`: | Run | Result | |---|---| | Case 02, three attempts | Two completed outputs passed every original quality rubric and both loading indicators; one timed out after loading both files. | | Full citation suite, one attempt per case | Cases 01 and 03–08 passed; case 02 timed out after loading both files (7/8 overall). | | One isolated case-02 follow-up after other runs finished | Completed and passed every original quality rubric and both loading indicators. | Across those final-candidate case-02 attempts: **3 passed, 2 timed out**. Timeouts remain failures and are not omitted from denominators. No completed final-candidate output reproduced either reported defect or the DOI-prefix false positive. This is bounded synthetic evidence, not a claim of universal accuracy, a clean full-suite run, or measured with/without uplift. The separate [Chinese boundary suite](../plugin-evals-citation-locale/README.md) checks that the repair still detects real ordering errors, preserves two-author and disambiguation cases, and honors an explicit venue romanization override. All three boundary cases passed on the final candidate (one run each). ## Side channels and ceilings (pilot 3, 2026-09-13, 1 run × 2 arms, sonnet agents) | Channel | Ceiling | Observed max | |---|---|---| | wall-clock per run | 600 s (`timeout_seconds`; over = score 0) | 142 s | | turns per run | 20 (`max_turns`) | 7 | | agent cost per run | none enforced | $0.45 | | full pilot (8 cases × 2 arms × 1 run, incl. opus judge) | — | $4.65 | Pilot 1 (before calibration, opus agents) cost $6.89 and peaked at 155 s / 7 turns / $0.83 per run; pilot 2 (sonnet agents) cost $3.99. Pilot 3 followed a cross-model review of the suite (13 findings, 12 applied: disputable plantings replaced, presence regexes tied to an "unused" statement, metadata preservation required on the conversion negative, real journal names replaced). ## Known caveats - **Historical loading failure (2026-09-13 pilot).** The with-plugin arm invoked a command stub but could not find the mode prompt outside its cwd; `citation_compliance_agent.md` never loaded. Its near-zero Δ measured the stub plus base model, not the complete mode. #857 adds an explicit core-skill invocation and plugin-root references, and makes mode command stubs user-invocable only so automatic routing selects the core skill directly. - **Historical trigger rate was 3 of 6 fire cases.** The old loose `input_match: academic` counted a command-stub invocation as firing. That number is not comparable to the stronger core-skill indicator now used. The Spanish trigger `verificar citas` subsequently arrived with #856; #858 adds English, Traditional Chinese, and Korean citation-check phrases. - **03 was a `/ars-citation-check` slash-command case in pilot 1.** The without-plugin arm answered "Unknown command" and Δ was +0.70 for the wrong reason (command existence). Switched to a prose trigger. - **Clean entries must be clean at the claim level too.** Pilot 1 failed `no-false-positive` in 5 of 6 fire cases because the planted "clean" sentences overstated their own abstracts and the models (correctly) said so. Inputs were tightened and the grader now defines a false positive narrowly (entry-level error, listed under corrections, or proposed change); claim-wording comments, DOI-prefix remarks, and conditional house-style notes are allowed. - **02 order of scripts is not graded.** The reference list puts English entries before Chinese ones; the author accepts either order for Taiwan journals, so `format-caught` only checks the 「等」 rule. - **02 bold vs italic on Chinese journal names** is house style, not an error; entries are plain and the grader treats either remark as neutral. - **no-overreach** allows per-sentence replacement wording for a misrepresenting claim and an *offer* to redraft; a rewritten whole excerpt or design-level critique fails it. - **Regex presence checks are secondary.** `orphan-named` (w0.5) only proves the orphan key was mentioned; the paired llm `format-caught` decides whether it was flagged. In 08 the regexes require the import name within 120 characters of an "unused" statement (headings such as 「沒用到的引用：」 on the line above count). - **04 does not require the style to be named.** Both arms fixed the planted year mismatch without ever writing "Vancouver"; the grader only fails a response that applies author-date rules to the numbered list. - **07 and 08 show Δ 0** — the base model already handles them. They stay as regression guards (07 must not become an audit report; 08 must not fire). --- ### Overview (`plugin-evals-citation-locale/README.md`) # Chinese citation-check boundary cases (#882) Companion to `plugin-evals-citation-check`. All papers and people in these inputs are fictional. This suite checks the scope of the abbreviation and ordering repair, rather than changing the original case-02 acceptance rules. ```bash claude plugin eval . --eval-dir plugin-evals-citation-locale --ablation none --model sonnet --judge-model opus --runs 1 --no-publish ``` | Case | Required behavior | |---|---| | 09 — stroke inversion and two authors | Report a genuine out-of-order Chinese list using the supplied surname stroke counts; preserve both names in a two-author citation. | | 10 — same-year disambiguation | Preserve all names when two three-author works differ only at the final author; do not shorten the reference-list entries. | | 11 — venue romanization override | Accept the supplied alphabetical romanized order despite its differing from stroke order. | The original mixed-overclaim case remains in the main citation suite and is repeated three times for the two reported defects. Its existing quality rubrics are unchanged. Additional `tool_used` graders record attempted reads of the agent and Chinese guide; successful loading is checked in the traces. Loading metrics are distinct from the output-quality judgements. These small synthetic tests do not establish universal accuracy or a with/without uplift. The supplied stroke counts in case 09 isolate applying the ordering rule from looking up character data. They do not test a general-purpose stroke-count lookup implementation. No such implementation is added by this prompt fix. ## Verification (2026-09-21) On the final #882 candidate, Claude Code 2.1.278 with `--ablation none --model sonnet --judge-model opus --runs 1 --no-publish` passed all three cases and all their graders (3/3). The observed agent was `claude-sonnet-5`. Case 09 uses different personal names from the operative guide examples. The result covers these three supplied-input boundaries; it does not certify all Chinese ordering/author-name cases. --- ### Overview (`plugin-evals/README.md`) # plugin-evals — `academic-paper` revision-coach flow Eval suite for `claude plugin eval`. One flow per suite: **revision-coach** (reviewer comments → Revision Roadmap + Response Letter skeleton; committee-correspondence variant when the user names a real committee). Quality spec (author-defined, 2026-09-12): the primary axis is **no unauthorised rewriting** — the response must not draft manuscript prose, must not change things no reviewer asked for, and must not assert results or changes that have not happened. Secondary axes: no comment dropped, push-back allowed on wrong comments, committee letters get a tracker with no peer-review grading. All inputs are synthetic (fictional studies, no real names or institutions). ## Run ```bash claude plugin eval . --eval-dir plugin-evals --ablation with-without --judge-model sonnet ``` Add `--no-publish` to keep the HTML report local. Headline number is Δ (with-plugin score − without-plugin score). `runs: 3` per case. ## Cases | Case | Fires? | Shape | Primary grader | |---|---|---|---| | 01-journal-mixed-format-zh | yes | 3 reviewers, mixed numbered/paragraph/bullet, abstract + methods excerpt attached | no-rewrite (llm, w1.5) | | 02-decision-letter-email-en | yes | editor email, unnumbered paragraphs, one factually wrong reviewer point | no-rewrite (llm, w1.5) | | 03-iclr-rebuttal-en | yes | OpenReview scores + "should we push back" | no-fabrication (llm, w1.5) | | 04-ethics-committee-letter-zh | yes (committee variant) | formal REC letter, user names the committee | tracker-covers-5 (llm, w1.5) + regex on grading labels (w0.5) | | 05-terse-contradictory-zh | yes | terse opener, contradictory reviewers, scope-changing request | scope-creep-flagged (llm, w1.5) | | 06-neg-existing-rebuttal-draft-zh | no (rebuttal-audit shape) | comments + existing draft, one point missed | no-new-skeleton (llm, w1.5) + regex | | 07-neg-landlord-letter-zh | no | non-academic letter | regex + llm + Skill not called | `skill-fired` (`tool_used: Skill`) on 01–05 is display-only under ablation and never moves Δ. So is `no-committee-branch` on 03 (`arm: with-only`, #854). ## Routing retest (2026-09-21, #854) After the #857 command-loading and SessionStart routing changes, case 03 was run three times with Claude Code 2.1.278, `--ablation none --model opus --judge-model opus --runs 3 --case '03-*' --no-publish` from outside the plugin checkout. All three called `academic-research-skills:academic-paper`, read `revision_coach_agent.md`, and passed the existing no-fabrication and pushback rubrics. Trace/output inspection found **0 of 3** applying the committee variant. That branch assessment is manual, not an assertion made by the existing graders; the same-family judge is not independent validation. The observed model ID was `claude-opus-5`. This small synthetic retest does not rule out the intermittent failure or close #854. Earlier intermediate candidates included timeouts and are not counted as completed passing runs here. No committee guard or agent definition changed. ## Committee-branch guard (2026-09-24, #854) The counts and costs in this section are maintainer-reported: they come from local runs whose outputs are not committed (`plugin-evals/results/` is ignored by git). Case 03 gained `no-committee-branch` with the #854 guard (see `CHANGELOG.md`): a with-only regex that fails when the answer emits a line of the committee variant's output, that is, a concern-tracker or preserved-source heading or bold label, a concern-ID heading or comment, the status line, or the blockquoted human-subjects footer that `committee_correspondence_protocol.md` requires. A heading or bold-label line with "no", "not", "none", "without", or "n/a" in it does not count, and an answer that declines the variant in prose passes even when it names the variant's parts. `scripts/test_check_committee_correspondence.py` runs examples of both through a JavaScript `RegExp` with the grader's flags, because the eval runner is a JavaScript program. Against the maintainer's 2026-09-12 results it flags the two runs named under Known caveats and none of the other nine, the without-plugin arms included. Neither flagged run used the protocol's required strings; the heading pattern caught both. The grader reads only the answer text, so it misses a run that only announces the branch, or only writes the bundle's files, without printing their lines. It would flag a peer-review answer that titles a section "concern tracker" with no negating word, and a refusal that quotes one of those lines on a line of its own. Retest with Claude Code 2.1.281, `--ablation none --model claude-opus-5-5 --judge-model sonnet --runs 7 --case '03-*' --no-publish`, from outside the plugin checkout: with the guard, **0 of 7** runs took the branch; on unchanged `main` with only the new grader added, also **0 of 7** (5 of those 7 read `revision_coach_agent.md`). The misroute was seen on `claude-opus-5`, and Opus 5.5 did not reproduce it without the guard, so this retest shows no effect of the guard; the guard is prompt-level and its effect is unmeasured. None of the fourteen runs built the Revision Roadmap or the response skeleton; each answered the push-back question directly. Cost: US$2.46 with the guard, US$2.70 on `main`. The retest ran on an earlier text of the guard: the sentence in `committee_correspondence_protocol.md` came later, and the mode table then gave the variant its own row, which the final change dropped. It was also graded with an earlier text of `no-committee-branch`; re-graded with the final text, the counts are the same. Case 04 (the ethics-committee letter), 7 runs on the final text with the same settings: the skill fired in 2 runs, and both took the committee variant (concern IDs and a segmentation for the user to confirm); `tracker-covers-5` passed in 1 of 7. In the other 5 runs Opus 5.5 answered without invoking the skill. The guard sits in files that load only after the skill fires, and the skill descriptions and the SessionStart hook are unchanged from `main`, so those 5 runs do not test it. The stored 2026-09-12 runs on `claude-opus-5` fired the skill in 2 of 2. Cost: US$1.58. ## Side channels and ceilings (pilot 2026-09-12, 1 run × 2 arms) | Channel | Ceiling | Observed max | |---|---|---| | wall-clock per run | 600 s (`timeout_seconds`; over = score 0) | 322 s | | turns per run | 15 (`max_turns`) | 10 | | agent cost per run | none enforced | $0.80 | | full pilot (7 cases × 2 arms × 1 run) | — | $4.63 | ## Known caveats - **03 fires since #851.** Before the #851 description fix the with-plugin arm answered the ICLR "should we push back" prompt without invoking the skill (0 of 2 pilots). After the fix: 7 of 7 verification runs invoked it. One of those seven said it was applying the #668 committee-correspondence branch to the ICLR peer reviews (the skill forbids inferring committee authority), and a second produced that branch's shape without naming it; four said the reviewers are peer review and one did not say. Tracked as #854 (see Committee-branch guard above). - **Judge style matters.** With the runner's judge, rubrics phrased as a bare list of claims produced repeated 3-vote FAILs on outputs that a reasoning judge (same model, asked to quote evidence first) passed. Every llm rubric in this suite that showed that pattern (02 all-covered, 03 no-fabrication, 03 pushback-per-reviewer) is written as "work through the checks one at a time and quote the evidence"; keep that style when adding graders. - **no-rewrite is judged, not regex-checked.** Manuscript prose vs. quoted reviewer text cannot be told apart lexically. A single suggested sentence of manuscript text is borderline and judges have passed it. - **05 and 07 show Δ 0** — the base model already handles them. They stay as regression guards. --- ### Overview (`examples/pi/README.md`) # Pi degraded-mode end-to-end evidence This directory is a privacy-minimized evidence bundle from one completed `/ars-full` run through the thin Pi wrapper. It demonstrates checkpoint enforcement, source-read provenance gating, Stage 5 finalization, Stage 6 process recording, and persisted terminal completion when no Pi orchestration capability was installed. This is evidence from one synthetic test scenario, not a general conformance certificate or a claim that degraded execution is equivalent to independent multi-agent review. ## Run identity | Field | Value | | --- | --- | | Run ID | `ars-pi-e2e-gpt-5.6-sol-20260803T051847Z` | | Pi | `0.83.0` | | Provider/model | `openai-codex/gpt-5.6-sol` | | ARS pipeline | `3.19.0` | | Repository HEAD used for curation | `682b158ea02b6bdcc1f70b1ba577f237645bf7f5` | | Execution | sequential, same recovered Pi/model context | | Terminal state | Stage 6 `completed`, checkpoint confirmed, acknowledgement received, 36/36 round trips | The run crossed a disclosed transport recovery and automatic context compaction. Continuity was established from persisted artifacts, state, and hashes; it was not represented as a new academic run. After the run, the wrapper's system-prompt injection was scoped to ARS-active sessions and inactive prompts were hardened to hide this package's ARS skill entries. That guarantee and its probes cover prompts submitted while Pi is idle. Pi 0.83.0 does not rebuild the system prompt for prompts queued into an in-flight run, and its direct RPC `steer` and `follow_up` methods bypass the wrapper's `input` handler; those residuals are documented rather than represented as covered. The compatibility text used after `/ars-full` is unchanged; the final wrapper received fresh idle-path activation, automatic-invocation, and package-load regression tests, but these small post-run changes did not receive another full pipeline execution. See [`model-portability.md`](model-portability.md). ## Load-bearing Pi boundaries 1. **No wrapper-provided agent isolation.** [`runtime/ars-pi-doctor.txt`](runtime/ars-pi-doctor.txt) reported `Orchestration: none; sequential/degraded mode`. Specialist roles therefore ran sequentially in one recovered Pi/model context. No independent reviewer, writer, verifier, observer, formatter, or cross-model judgment is claimed. The external RPC driver delivered synthetic test-operator checkpoint responses only; it was not available to the inner Pi session as an orchestration tool or skill. 2. **No Claude hook enforcement.** The same doctor record reported that Claude hooks are unavailable in Pi. ARS write-scope enforcement was prompt-level in this run; no deterministic Claude `PreToolUse` hook boundary is claimed. The installed `/skill:websearch` capability was visible to Pi. Source verification remained bounded as disclosed in the provenance and process records. [`runtime/package-load-validation.json`](runtime/package-load-validation.json) records fresh offline root/nested package loads that found all three wrapper commands, all 16 `/ars-*` prompt templates, and all four ARS skills, plus nine passing unit tests (including filesystem-backed symlink/XML-location, block-boundary, and single-pass argument-substitution probes) and five real Pi idle-prompt-scope probes. ## Evidence chain ### Full stage chain [`run-log.md`](run-log.md) provides the chronological checkpoint record. Representative byte-for-byte outputs make each executed stage inspectable: - Stage 1 Research: [`stage-01/revised-research-report.md`](stage-01/revised-research-report.md) - Stage 2 Write: [`stage-02/course-paper-draft.md`](stage-02/course-paper-draft.md) - Stage 2.5 Integrity: [`stage-02.5/focused-reverification-report.md`](stage-02.5/focused-reverification-report.md) - Stage 3 Review: [`stage-03/editorial-decision-and-revision-roadmap.md`](stage-03/editorial-decision-and-revision-roadmap.md) - Stage 4 Revise: [`stage-04/response-to-reviewers.md`](stage-04/response-to-reviewers.md) - Stage 3′ Re-review: [`stage-03p/verification-review-report.md`](stage-03p/verification-review-report.md) - Stage 4′ Re-revise: [`stage-04p/validation-report.md`](stage-04p/validation-report.md) - Stage 4.5 Final integrity: [`stage-04.5/final-integrity-verification-report.md`](stage-04.5/final-integrity-verification-report.md) - Stage 5 Finalize and Stage 6 Process summary: final package, process records, checkpoints, and terminal state below ### Source-read hard gate 1. [`stage-05/cite-time-gate-refusal-report.md`](stage-05/cite-time-gate-refusal-report.md) and [`stage-05/cite-time-gate-refusal.json`](stage-05/cite-time-gate-refusal.json) show that Stage 5 refused converted output while all 79 citation occurrences remained unacknowledged `LOW-WARN`. 2. After the required human attestation was consumed, [`stage-05/cite-time-gate-pass.json`](stage-05/cite-time-gate-pass.json) recorded 21 attested sources, 79/79 markers resolved to `OK`, and a hard-gate `PASS` before clean outputs were emitted. 3. Exact attestation wording, Pi session identifiers, event identifiers, and human-read ledger were intentionally omitted from the public bundle. The aggregate gate state is sufficient for port evidence and is not independent proof that reading occurred. ### Stage 5 - Retained public evidence: final [`paper.md`](stage-05/package/paper.md), [format validation](stage-05/package/format-validation.md), cite-time refusal/PASS records, and [FULL checkpoint](stage-05/full-checkpoint.md). - The run also generated a DOCX and advisory submission-verifier artifacts. Those redundant binary/snapshot files are intentionally omitted; their hashes and outcomes remain in the checkpoint and persisted state. - No paper PDF exists because the test operator declined paper LaTeX. HTML-to-PDF was not used. - The submission-package policy was advisory. B1–B5 and C1–C2 remained `NOT-CHECKED`; this is not a clean universal submission-verifier pass. ### Stage 6 and terminal state - The concise public process evidence is the [English Markdown record](stage-06/process-record-en.md). Redundant PDF, LaTeX, translated copy, and build-validation files are omitted. - [Terminal checkpoint](stage-06/terminal-checkpoint.md) was correctly nonterminal when presented. - [`state/pipeline-state.json`](state/pipeline-state.json) is the controlling proof: `pipeline_state: completed`, Stage 6 `completed`, checkpoint confirmed, terminal acknowledgement received, and completion timestamp persisted. Exact acknowledgement wording and transport identifiers are omitted. ## Model portability [`model-portability.md`](model-portability.md) compares observed `openai-codex/gpt-5.6-sol` behavior with the repository's illustrative Claude-oriented example. It separates runtime-established divergences from output differences that cannot be causally attributed to the model. ## Privacy boundary - No maintainer name, email, account, home-directory path, credential, API key, authorization token, or Git remote is included. - “Scholar” decisions in generated artifacts are synthetic test-operator inputs for the canonical example. Funding, conflict, affiliation, authorship, and placeholder statements are not biographical declarations about the Pi port maintainer. - The public bundle records only aggregate human-read gate state; exact attestation wording and internal session/event identifiers are excluded. - Public-source UUIDs appearing inside government document URLs are source locators, not Pi or user identifiers. ## Preserved limitations - Sequential same-context role simulation is weaker than independent agents or context windows. - Source verification and the scholar's source-read attestation have distinct evidentiary meanings. - The paper is a non-systematic evidence synthesis and retains the nonblocking PRISMA-trAIce/RAISE reporting warning. - The submission verifier was advisory and incomplete for checks requiring absent venue-profile or corpus inputs. - Four title-page placeholders remain. - The run experienced model-stream stalls, recovery, and compaction. See [`inventory.json`](inventory.json) for source-to-curated path and hash provenance. `SHA256SUMS` covers every curated file except itself. Raw RPC streams, full session logs, internal identifiers, downloaded source files, redundant binaries/translations, and intermediate caches are intentionally excluded. --- ### Overview (`examples/showcase/README.md`) # Showcase: Full Academic Pipeline Output Complete artifacts from a real academic pipeline run, demonstrating the end-to-end quality assurance process. **Paper**: *From Snapshots to Trajectories: How Agentic AI Will Redefine Student Learning Outcomes and Transform Student Success Measurement — Implications for Taiwan's Next Cycle of Institutional Accreditation* > **Version note.** This run (2026-03-07 to 03-08) used academic-pipeline v2.3, before ARS v3.3 added the Semantic Scholar check and v3.11 added the deterministic four-index citation gate. The numbers below describe that version; the current gates have not been measured on this paper. The byline names Claude (Anthropic) as author because the researcher asked for that during the experiment. The paper is not an Anthropic publication, and ARS's positioning is that the tool does not replace the researcher and does not claim authorship ([POSITIONING.md](../../POSITIONING.md#what-this-is-not)). ## Paper Outputs | File | Description | |------|-------------| | [full_paper_apa7.pdf](full_paper_apa7.pdf) | Final paper (English, APA 7.0 format) | | [full_paper_zh_apa7.pdf](full_paper_zh_apa7.pdf) | Final paper (Chinese, APA 7.0 format) | ## Integrity Verification Reports | File | Stage | Verdict | |------|-------|---------| | [integrity_report_stage2.5.pdf](integrity_report_stage2.5.pdf) | Pre-Review (Stage 2.5) | FAIL — flagged 15 problem references (8 with bibliographic errors, 6–8 likely fabricated) + 3 statistical errors | | [integrity_reverification_stage2.5.pdf](integrity_reverification_stage2.5.pdf) | Re-verification (Stage 2.5) | PASS — all 22 issues fixed | | [integrity_report_stage4.5.pdf](integrity_report_stage4.5.pdf) | Post-Revision (Stage 4.5) | PASS — zero regressions, 3 new refs verified | | [post_publication_audit_2026-03-09.pdf](post_publication_audit_2026-03-09.pdf) | Post-Publication Audit | 21/68 refs had problems, 19 removed or corrected — Claude Code + WebSearch audit, run separately from paper generation | ## Peer Review Reports | File | Description | |------|-------------| | [stage3_review_report.pdf](stage3_review_report.pdf) | Round 1: Journal-Fit Reviewer + 3 Reviewers + Devil's Advocate (5 role-separated reviews, all generated by Claude; not independent reviewers) | | [stage3prime_rereview_report.pdf](stage3prime_rereview_report.pdf) | Re-review after Round 1 revisions | | [stage3_review_report_r2.pdf](stage3_review_report_r2.pdf) | Round 2: Follow-up review | | [response_to_reviewers_r2.pdf](response_to_reviewers_r2.pdf) | Author's point-by-point response to reviewers | ## Process Documentation | File | Description | |------|-------------| | [paper_creation_process_en.pdf](paper_creation_process_en.pdf) | Pipeline process documentation (English) | | [paper_creation_process_zh.pdf](paper_creation_process_zh.pdf) | Pipeline process documentation (Chinese) | These PDFs predate the public role rename. Any `EIC` text inside them is the legacy serialized source ID now displayed as **Journal-Fit Reviewer**; the archived binaries are intentionally unchanged. ## Pipeline Stages Covered ``` Stage 1: Deep Research (13-agent research pipeline) Stage 2: Paper Writing (12-agent writing pipeline) Stage 2.5: Integrity Verification ← MANDATORY, caught 22 issues Stage 3: Peer Review Round 1 (Journal-Fit Reviewer + R1/R2/R3 + Devil's Advocate) Stage 3': Author Revision + Re-review Stage 3-R2: Peer Review Round 2 Stage 4: Final Revision Stage 4.5: Final Integrity Verification ← MANDATORY, confirmed 0 regressions Stage 5: Finalization (LaTeX → PDF, bilingual) Stage 6: Process Summary Post-Publication: Claude Code + WebSearch audit of all 68 refs → 21 issues found, 19 removed or corrected ``` ## Post-Publication Audit (2026-03-09) After the pipeline completed, Claude Code with WebSearch, run separately from paper generation, audited all 68 references and found problems in 21 of them (31% of the reference list) that had survived three rounds of automated integrity checking: 4 not found (likely fabricated), 6 with wrong authors, 7 with wrong metadata (year, title, pages, journal, or DOI), and 4 minor format or title issues. Afterward, the 4 were removed from the paper and 15 of the other 17 were corrected. The audit PDF calls the 31% a false-negative rate; it is the share of final references with a problem, not a miss rate over all problem references. At least seven of the 21 (cited as Banihashem, El-Banna, Gandara, Kestin, Stanford SCALE, Tao, and Temper) had been flagged at Stage 2.5 and marked FIXED at re-verification, yet the audit still found errors in them, such as wrong authors or an altered title. This stress test led to the integrity verification agent v2.0 overhaul (Anti-Hallucination Mandate, gray-zone elimination, known hallucination pattern library). See [post_publication_audit_2026-03-09.pdf](post_publication_audit_2026-03-09.pdf) for the full report. --- ### Overview (`scripts/adapters/README.md`) # ARS Literature Corpus Adapters Reference adapters that produce `literature_corpus[]` entries for the ARS Material Passport. Defined by the adapter contract at [`academic-pipeline/references/adapters/overview.md`](../../academic-pipeline/references/adapters/overview.md). ## What these are Three starting-point adapters: - **`folder_scan.py`** — scans a directory of files, parses citation metadata from filenames. - **`zotero.py`** — reads a Better BibTeX JSON export. - **`obsidian.py`** — reads an Obsidian vault (frontmatter Convention A or Karpathy-style Convention B). Each writes two files: a `passport.yaml` with `literature_corpus[]` entries, and a `rejection_log.yaml` listing whatever the adapter could not map onto a valid entry. **These are references, not products.** If your corpus is in Zotero, Obsidian, or a plain folder, you can run these directly; if it is in Notion, Readwise, Airtable, a custom SQLite, etc., copy a reference adapter and adapt it to your source. ## Running a reference adapter ### folder_scan ```bash python scripts/adapters/folder_scan.py \ --input /path/to/your/pdf/folder \ --passport /tmp/passport.yaml \ --rejection-log /tmp/rejection_log.yaml ``` Filenames are parsed with two conventions plus a fallback: `{Family}_{Year}_{title}.{ext}`, `{Family}{Year}{title}.{ext}`, or `... {Year} ... {Family} ...`. Files whose filename cannot yield both a family and a year are rejected. Non-ASCII filenames are rejected (Unicode support is a deferred extension). Symlinks are followed but the rejection log records the un-resolved path so collisions across subdirectories remain distinguishable. ### zotero Export your Zotero library as **Better BibTeX JSON** (Zotero → File → Export Library → Format: Better CSL JSON, with the Better BibTeX extension installed). Then: ```bash python scripts/adapters/zotero.py \ --input ~/zotero_export.json \ --passport /tmp/passport.yaml \ --rejection-log /tmp/rejection_log.yaml ``` This adapter does NOT call the Zotero Web API. If you want live sync, write your own adapter using this file as a starting point. Items missing required fields (`title`, `authors`, parseable `year`, `source_pointer`) are rejected with a categorical reason; seasonal dates like `Spring 2024` are rejected as `year_unparseable`. ### obsidian ```bash python scripts/adapters/obsidian.py \ --input ~/ObsidianVault \ --passport /tmp/passport.yaml \ --rejection-log /tmp/rejection_log.yaml ``` Files under `_templates/` and `.obsidian/` are skipped. Wikilinks (`[[other note]]`) are NOT resolved; they appear as literal text in `user_notes`. Notes with valid YAML frontmatter use Convention A (BibTeX-style: `citekey:`, `authors:`, `year:`, `title:`); notes without frontmatter fall through to Convention B (filename stem as citation_key, body as user_notes). Notes with malformed YAML frontmatter are rejected as `invalid_field_format` rather than silently re-classified as Convention B. ## Validating adapter output Before using the passport in any ARS workflow: ```bash python scripts/check_literature_corpus_schema.py \ --passport /tmp/passport.yaml \ --rejection-log /tmp/rejection_log.yaml ``` ## How to write your own adapter 1. Read the adapter contract at [`academic-pipeline/references/adapters/overview.md`](../../academic-pipeline/references/adapters/overview.md). 2. Read the JSON schemas: - [`shared/contracts/passport/literature_corpus_entry.schema.json`](../../shared/contracts/passport/literature_corpus_entry.schema.json) - [`shared/contracts/passport/rejection_log.schema.json`](../../shared/contracts/passport/rejection_log.schema.json) 3. Copy one of `folder_scan.py` / `zotero.py` / `obsidian.py` as a starting point. 4. Change the input-reading and field-mapping code for your source. 5. Keep the output shape exactly (sorted `literature_corpus[]` by `citation_key`, always emit `rejection_log.yaml`, schema-clean entries only). 6. Set `obtained_via: "other"` and `adapter_name: ""` on every entry you produce. 7. Reuse helpers in [`_common.py`](_common.py) where possible — `make_citation_key`, `ensure_unique_citekey`, `parse_csl_name`, `parse_semicolon_names`, `path_to_file_uri`, `write_passport`, `write_rejection_log`, `now_iso`. They encode the contract details so you do not have to re-derive them. 8. Validate: `python scripts/check_literature_corpus_schema.py --passport `. 9. Write tests modeled on `scripts/adapters/tests/`. The conftest fixtures `clean_timestamps` and `load_yaml` make golden-output testing easy. ## Privacy reminder `abstract` and `user_notes` can contain publisher-copyrighted material. Before sharing a passport that contains these fields publicly (e.g., in a public repo or over the web), make sure you have the right to publish that text. ARS does not enforce this — it only warns you in the schema description. ## Tests ```bash cd pytest scripts/adapters/tests/ -v ``` CI runs these on every push / PR that touches `scripts/adapters/**`, `shared/contracts/passport/**`, the adapter overview, or either of the two lint scripts. --- ### Overview (`shared/contracts/README.md`) # ARS Shared Contracts Schema files for cross-skill contracts: reviewer sprint contracts, Material Passport ports, and (v3.6.7+) cross-model audit artifact pipelines. ## Stage capability / evidence matrix (#745) - `capability/stage_capability_matrix.json` (`stage-capability-matrix/1.0`) is the single machine-readable source for per-stage mechanism status, deterministic conformance, behavioral-evidence provenance, transport limits, and the maximum claim that evidence licenses. Behavioral statuses (`DESIGNED` / `NOT_RUN` / `MEASURED` / `MIXED` / `OUT_OF_SCOPE`) cannot collapse: an unrun eval can never carry a result, a measured row must carry full provenance (in-repo `eval_ref`, model, population, ISO date, result summary), and stale measurements require a visible staleness note. A measured row whose suite publishes `measurement-*.json` reports must bind the current one (date-equal, sibling supersession detected), `CI_GATED`/`TESTED` conformance must name existing lints/tests (`conformance_pinned_by`, D4-style), and `claim_anchors` bind top-level capability sentences verbatim so rewording a claim without touching the matrix fails CI. - `docs/STAGE_CAPABILITY_MATRIX.md` is GENERATED from the matrix (`scripts/check_stage_capability_matrix.py --render`) and byte-pinned by the same lint. The matrix indexes evidence, it does not create it: a row licenses at most its recorded `max_licensed_claim`, never more. ## Sealed Promotion Bakeoff preregistration (#789) - `cross_model/promotion_bakeoff_sealed_commitment.schema.json` is the closed public pre-fleet carrier: campaign id, LF-normalized probe-set SHA-256, and the fixed 30-row aggregate composition only. It has no row, label, path, or free-text field in which the answer key can be exposed. - `cross_model/promotion_bakeoff_sealed_reveal.schema.json` binds that immutable commitment to the post-fleet `probe_set.json` reveal. The standard-library checker `scripts/check_promotion_bakeoff_preregistration.py` validates the closed probe shape, strict Git introduction order, immutable history, canonical paths, worktree/index state, and detectable reuse across all local refs and every historical probe version. It rejects shallow history and unreadable historical objects; the sole 2026-08-19 unsealed grandfather is pinned to its canonical path, normalized digest, blob history, and file mode. Verification receipts enumerate every bound squash/cherry-pick reveal copy whose remote publication time must be witnessed after the fleet. The checker proves repository evidence only. Remote publication time and fleet start/end time cannot be derived from local Git, so its receipt explicitly requires the public permalink, passing CI witness, and fleet-time bounds in the run report before a result is gate-eligible. The full operator lifecycle and claims boundary live in `../cross_model_verification.md` § Promotion Bakeoff. ## Research-family workflow profiles (#742) - `research_workflow/research_workflow_profile.schema.json` (`research-workflow-profile/1.0`) is the closed, manuscript-blind declaration of stage applicability and vocabulary. Omitted task families resolve to `unresolved_fit`, never to applicable. The profile embeds a SHA-256 over its JSON Canonical Form with `content_sha256` replaced by 64 zeroes; persisted profile files must themselves be exact canonical bytes. - `research_workflow/research_workflow_profile_selection_receipt.schema.json` (`research-workflow-profile-selection-receipt/1.0`) records an append-only user selection/confirmation/fallback chain. A correction appends a new binding and one visible `profile_context_changed` stale mark per supplied prior-profile stage output. It never rewrites or removes an artifact. Newly introduced authority requirements remain attached to each mark with an explicit `authority_sensitive_reuse_gate: unmet`, so a later consumer can block that reuse until the applicable check passes. - `../research_workflow_profiles/field_general.json` is the mandatory shipped fallback: every task family is unresolved except the field-general integrity gates, alternatives are unresolved, the live-branch budget is 3, and an empty authority list means “unknown; ask the user,” never “not required.” It ships with explicit English and zh-TW display names. The standard-library runtime is `scripts/research_workflow_profile.py`. Its `select` command treats an omitted `--profile` as a visibly recorded automatic fallback; an explicit profile requires `--selected-by`. `correct` consumes an explicit prior-profile binding and stage-output inventory, then emits the new receipt to stdout only. No command reads manuscript content, infers a family, changes the default workflow, or writes scholar-owned artifacts. The human usability protocol and every outcome claim remain `NOT_RUN`. ## Opt-in inquiry branch ledger (#743) - `research_workflow/inquiry_branch_ledger.schema.json` (`inquiry-branch-ledger/1.0`) is the closed event-source contract for author-originated branches, parked AI-surfaced facets, explicit adoption, disposition, reopen signals, profile rebound, and first-degree artifact staleness. Dense event ids plus a canonical previous-event hash chain detect interior rewrite/reorder; the separate passport digest is the required trusted head for truncation detection. - `passport/inquiry_ledger_ref.schema.json` closes the optional Material Passport pointer to `{ledger_path, ledger_version, content_sha256}`. It never embeds a branch or profile and never widens a filesystem path outside the caller-supplied workspace root. The deterministic offline runtime is `scripts/inquiry_branch_ledger.py`. Replay requires every exact profile bound by the ledger so the live-branch budget is never inferred from a current default. Reopen appends contiguous deterministic stale-cause events; clearing one cause cannot clear another. Ledger/passport publication uses a stable sidecar lock and durable recovery journal. The feature is OFF unless `ARS_INQUIRY_LEDGER=1`, does not materialize before a second branch, and shows compact summaries only at the frozen Stage 1, 2.5, 4.5, or still-actionable recorded-signal moments. Behavioral and usability evidence is `NOT_RUN`. ## PDF read-integrity and optional content advisory (#512 follow-up) - `pdf/pdf_read_preflight.schema.json` accepts the unchanged legacy structural sidecar or the all-or-nothing opt-in content extension. In that extension, `verdict` is explicitly `verdict_scope: STRUCTURE_ONLY`; `OCR_RECOMMENDED` never rewrites that structural value into a content claim. The schema binds the legacy shape to tool version 1.0.0 and the extension shape to 1.1.0. - `pdf/pdf_content_classifier_worker.schema.json` closes the stdout of the fixed isolated worker to two classifications, three unavailable reasons, finite bounded confidence, and bounded non-negative page indexes. Runtime additionally binds every page to the structural page count. - `pdf/pdf_content_classifier_diagnostic.schema.json` is the separate POSIX-only, local mode-`0600` operator artifact. Platforms without `fchmod` reject its CLI option before path creation. Its explicitly untrusted detail is capped and never copied into or referenced by the sidecar. File-writing CLI invocations use conservative NFC/casefold keys, path resolution, and existing-inode checks to reject aliases before worker launch; the stdout-only legacy path adds no such precondition. POSIX sidecar output pre-binds the parent dirfd/inode, then uses a private `0700` fixed-name staging directory and anchored dirfd-relative publication. Open-inode checks reject staging entry swaps; atomic final-entry replacement does not follow post-check links. Non-POSIX output fails closed; stdout classification remains available. The diagnostic parent is likewise dirfd-bound before the worker. These are instantaneous inode postconditions, not a general same-UID sandbox; callers control the output parent. A failed diagnostic unlinks only the no-follow leaf still matching its created fd inode, preserving the primary error and any attacker replacement while keeping its exclusive path retryable. Runtime: `scripts/pdf_read_preflight.py` and `scripts/pdf_content_classifier_worker.py`. Frozen opt-in scope and residual risk: `docs/design/2026-08-13-512-pdf-content-classification-sandbox-spec.md`. ## Claim-standing candidate ledger (#655 Track A) - `claim_standing/query_plan.schema.json` (`claim-standing-query-plan/1.0`) binds one exact high-impact checkpoint claim, at most three accepted queries, at most four discovery-index identities, filters, authorized content class, frozen caps, and retrieval-only consent through a closed consentable-plan projection (version 1.1 below adds the stance-authorizable consent). - `claim_standing/retrieval_input.schema.json` (`claim-standing-retrieval-input/1.0`) carries already-retained, adapter-neutral attempts, closed retry-authorization receipts, raw hits, explicit version relations, and caller-supplied relevance success/failure evidence bound to exact claim/candidate inputs and canonical prompt bytes. - `claim_standing/candidate_ledger.schema.json` (`claim-standing-candidate-ledger/1.0`) preserves every attempt and raw-hit terminal state while recording deterministic work-family selection. - `claim_standing/query_plan_v1_1.schema.json` (`claim-standing-query-plan/1.1`) is identical to 1.0 except that the consent decision may be `retrieval_plus_stance`, in which case a top-level `stance_plan` names the exact stance provider/model surface, the consent binds its hash, and the authorized content classes extend by exactly `claim_and_selected_evidence_to_stance_provider`. A 1.0 plan stays valid under its own schema; the runtime validator accepts both versions and enforces the stance bindings only on 1.1. - `claim_standing/transmission_ledger.schema.json` (`claim-standing-transmission-ledger/1.0`) accounts every event that left the session during one probe: retrieval-query events derived one-to-one from the retained attempts and stance-classification events copied verbatim from the stance runner's transmission records, each carrying recipient, purpose, exact content classes, byte count, local hash, time, consent receipt, retention disclosure, and result state. `scripts/check_claim_standing_transmissions.py` is the normative builder/validator: an event outside the plan's authorized content classes or consented recipient roster fails closed (design §9 gate 14), and validation is exact replay. - `claim_standing/stance_record.schema.json` (`claim-standing-stance-record/1.0`) is the stance-classification output for one probe run: full §7 probe-identity hashes, one row per selected work family under the closed §5.1 vocabulary and cross-field rules (a performed row requires at least one `EVR-CS-` evidence-row reference; metadata-only coverage can never be performed), the all-selected distribution whose seven buckets must sum to `selected_total`, a mandatory `STANCE CLASSIFICATION UNMEASURED` banner, and no scalar credibility/confidence/trust field anywhere (test-pinned). Provider retention disclosure is also closed: `known` requires a non-empty reference and `unknown` requires null. The schema-level `\\S` checks are portable first screens. The runtime applies a single NFKC visible-semantic-text predicate to the claim/query, disclosure, identity, available-abstract, successful-assessment, and failure-detail surfaces; it rejects surrogates and text made only of Unicode control/format, separator, combining, whitespace, or punctuation characters. Failed malformed assessment raw output remains exact evidence and may itself be whitespace- or format-only. DOI text is stable identity only when strict NFKC/prefix-trimmed `10.<4-9 digits>/` validation succeeds. The pure local finalizer is `scripts/build_claim_standing_candidate_ledger.py`; the authoritative boundary is `shared/references/claim_standing_candidate_ledger_protocol.md`. This slice has no discovery adapter, network/model call, stance classification, rendering, evidence-row extension, pipeline hook, or held-out dispatch. It is an offline substrate only, remains unmeasured, and does not close #655. Its CLI never creates an output for a `session_only` plan. `build --output` requires the existing, hash-bound `explicit_local_export` consent state; there is no command-line override. Authorized output must exactly match the hash-bound absolute `authorized_output_path`, is exclusive/no-follow where the host supports it, mode `0600` from creation, file- and directory-fsynced, and truthfully carries the consented persistence/export/path state at the ledger root and in each work family's sharing scope. Its separate rights basis remains `not_assessed`; local persistence consent is not a rights claim. ## Codex subscription citation transport (#630) - `cross_model/codex_citation_request.schema.json` — closed, bounded data-only request for one citation; no path or caller-authored prompt is representable. - `cross_model/codex_citation_receipt.schema.json` — closed verdict plus exact app-server search-result bindings and a fixed containment receipt. Runtime: `scripts/cross_model_codex_transport.py`. Protocol and limitations: `shared/cross_model_verification.md`. Frozen design: `docs/design/2026-08-11-630-codex-subscription-citation-transport-spec.md`. The adapter is scoped to Stage 2.5 / 4.5 citation-integrity calls and is not a general DA, reviewer, judgment, or handoff transport. ## Sprint contracts (v3.6.2+) Sprint contract templates for reviewer hard-gate orchestration. Schema: `shared/sprint_contract.schema.json` (Schema 13). Validator: `scripts/check_sprint_contract.py`. Spec: `docs/design/2026-04-23-ars-v3.6.2-sprint-contract-design.md`. Protocol: `academic-paper-reviewer/references/sprint_contract_protocol.md`. ### Shipped templates **v3.6.2 (reviewer family)**: - `reviewer/full.json` — panel 5, 5 dimensions, 4 failure conditions - `reviewer/methodology_focus.json` — panel 2, 2 dimensions, 3 failure conditions **v3.6.6 / suite v3.6.8 (generator-evaluator family)**: - `writer/full.json` — single-agent writer, 7 dimensions (D1 section_completeness / D2 citation_density / D3 argument_blueprint_fidelity / D4 total_word_count / D5 per_section_word_count / D6 acknowledged_limitations / D7 register_consistency), 5 failure conditions (F1 / F4 / F2 / F3 / F0). No `scoring_plan` field. - `evaluator/full.json` — single-agent evaluator, 5 dimensions (D1 originality / D2 methodological_rigor / D3 evidence_sufficiency / D4 argument_coherence / D5 writing_quality), 7 failure conditions (F1 / F2 / F3 / F6 / F4 / F5 / F0). Carries full `scoring_plan` + `disagreement_handling`. Both writer + evaluator templates ship under Schema 13.1 (allOf branches 11/12 require `pre_commitment_artifacts` for `writer_full` and `disagreement_handling` for `evaluator_full`; branches 5/6 pin `failure_conditions[].action` to mode-specific enums; branches 8/9 pin F0 contains to the mode's accept variant). Orchestration block lives in `academic-paper/SKILL.md` § "v3.6.6 Generator-Evaluator Contract Protocol" + the writer/evaluator agent files. ### Reserved reviewer modes without shipped templates `reviewer_calibration` and `reviewer_guided` are in the schema enum but ship without templates. Those modes continue to operate in their existing form (no contract, no hard-gate) until a follow-up patch release adds their templates. `reviewer_re_review` left the Schema 13 enum with #576 Spec B: re-review is governed by the dedicated contract family under `re_review/` (four schemas + `scripts/check_re_review_synthesis.py`), not by a Schema 13 template — a contract claiming `mode: reviewer_re_review` no longer validates. ### How to add a new template 1. Add the file under `shared/contracts//.json`. 2. Run `python scripts/check_sprint_contract.py --ars-version vX.Y.Z`; expect zero errors and zero soft warnings. 3. If `expression` strings use new phrasing, update `sprint_contract_protocol.md` and the synthesizer prompt's recognised-pattern list in the same PR. ## Passport contracts (v3.6.4+) Schemas for Material Passport input ports. - `passport/literature_corpus_entry.schema.json` (v3.6.4) — Schema 9 `literature_corpus[]` entries produced by user-written adapters. - `passport/bibliographic_integrity_signal.schema.json` (#678/#651/#660) — v1.0 additive signal carrier, v1.1 authoritative retraction-status rows, and the v1.2 title/abstract tortured-phrase advisory profile, including resolver disagreement/reinstatement, judgment context, freshness, and opt-in finalizer policy eligibility. The separate `retraction_status_cache_v1` namespace and pure resolver live in `scripts/retraction_status.py`. - `passport/rejection_log.schema.json` (v3.6.4) — adapter output companion logging entries that could not be included in the corpus. - `passport/reset_ledger_entry.schema.json` (v3.6.3) — `reset_boundary[]` ledger entries for the opt-in passport reset boundary protocol. - `passport/inquiry_ledger_ref.schema.json` (#743) — optional digest-bound pointer to the separate canonical inquiry branch ledger; missing or mismatched targets fail visibly and unpointed candidates are ignored. - `passport/standing_constraint_entry.schema.json` (#927) — `standing_constraints[]` entries: a run-wide constraint the user stated, in their words. Schema tests: `scripts/test_standing_constraint_entry_schema.py`. - `passport/excluded_source_entry.schema.json` (#936) — `excluded_sources[]` entries: a reference an integrity gate judged `NOT_FOUND`, kept off later writer dispatches. Schema tests: `scripts/test_excluded_source_entry_schema.py`. - `passport/audit_artifact_entry.schema.json` (v3.6.7 Step 6) — `audit_artifact[]` entries recording one cross-model audit run per downstream-agent deliverable. Two lifecycle states (proposal / persisted) share the schema via `oneOf`. Cross-artifact invariants are enforced by `scripts/check_audit_artifact_consistency.py`. Spec: `docs/design/2026-04-30-ars-v3.6.7-step-6-orchestrator-hooks-spec.md` §3.1-§3.2. - `passport/version_records.schema.json` (Kong #258) — optional `phase2_investigation/version_records.yaml` sidecar for academic citation version families (preprint -> proceedings -> journal extension). This is deliberately a sidecar: `literature_corpus_entry.schema.json` stays adapter-owned and unmodified. - `passport/human_read_log.schema.json` (#513/#738) — the user-owned `USER_ATTESTED_READ` ledger (`_human_read_log.yaml`, written by `scripts/ars_mark_read.py`), including required scope on new marks and legacy-compatible scope absence. Declaration-only: it is not verified reading or comprehension. Missing/unknown scope remains `coverage_unknown` and never promotes to `ok`. - `passport/user_attested_read_resolution.schema.json` (#738) — the closed output of `scripts/human_read_attestation_resolver.py`; only `state: covered` is `ok_eligible`, while absence, rescind, partial/unknown coverage, unresolved anchors, and invalid ledgers remain explicitly non-promoting. This output is a `transient_routing_decision`, not a persisted audit receipt: it carries no input digest and must be recomputed from the current ledger and exact anchor on each finalizer pass. Its closed `finalizer_disposition` keeps absent or rescinded marks as unacknowledged LOW-WARN, partial/unknown active marks as acknowledged-partial, anchor failures on the locator-precedence route, and invalid ledgers on a blocking route. ## Shared evidence rows (#656) `shared/contracts/evidence/claim_registry.schema.json` (#737) is the closed E1 population contract: exact raw-draft hash plus exact UTF-8 byte span and equal text for every registered claim. `claim_registry_coverage_report.schema.json` is the deterministic, replay-bound gap report emitted by `scripts/claim_registry_coverage.py`; it binds exact raw draft and serialized registry hashes and joins only those exact spans. It covers citation-bearing and quantitative candidate sentences only. Its finite grammar includes common Markdown/numeric/author-year/Pandoc/inline-reference citations plus unit-bearing numbers, p-values, `N=...`, and common effect-size/ratio notation; unrecognized scholarly syntax remains possible. The report always records `semantic_extraction_coverage: not_machine_detectable`; it cannot certify that every substantive claim was extracted into E1. Consumers replay the report against both inputs; absence, stale bindings, or validation failure are unresolved execution states, never a zero-gap result. ## Review-panel provenance (#740) `reviewer/review_panel_provenance_input.schema.json` records actual seat-level observations; `reviewer/review_panel_provenance.schema.json` is the closed, replay-derived artifact built by `scripts/review_panel_provenance.py`. Both are closed to `reviewer_full`, bind the exact raw-byte digest of `reviewer/full.json`, and require the ordered `EIC`, `R1`, `R2`, `R3`, `DA` roster. Actual execution observations remain nullable and are never filled from those seat labels. The artifact keeps role separation, within-panel invocation context separation, peer-output blinding, model-family diversity, provider diversity, and accountable-human diversity as six separate `true` / `false` / `unknown` axes. `fresh_context_scope: within_panel_attempt_only` makes explicit that no cross-attempt history is checked. Missing facts remain unknown, same-family or family-unknown execution carries a fixed correlated-error disclosure, and no persona-derived binary independence field is admitted. `reviewer/review_panel_provenance_carrier.schema.json` is the exact closed Schema 6 valid/invalid union. The valid branch is accepted only after the runtime hashes the referenced artifact's exact raw bytes, validates and replays it, and compares the normalized-manifest digest, execution-topology digest, fresh-context scope, and six axes. The invalid branch admits only `absent|unreachable|digest_mismatch|schema_invalid|replay_invalid`, forces all axes to `unknown`, and carries no path or digest. `reviewer_full` must emit one branch; the other closed modes must omit the field. Use `scripts/review_panel_provenance.py validate-schema6` for mode-scoped presence and replay validation. `shared/contracts/evidence/evidence_row.schema.json` defines the closed `evidence-row/1.0` carrier for evidence shown at human-adjudication checkpoints. V1 has one consumer discriminator, `surface: phase_e_claim_verification`: the Stage 2.5/4.5 Phase E Claim Verification Report persists one row per `(claim_id, ref_slug, anchor)` at Integrity Report `phases.E_claims.evidence_rows[]`. The array is additive: a positively identified pre-#656 report may omit it and, only with `--allow-legacy-absence`, renders exactly `LEGACY — EVIDENCE ROWS UNAVAILABLE`; missing shape alone is not legacy proof and never means that evidence was checked. The opt-in Stage 4→5 `claim_audit_results[]` contract is a separate lifecycle and is unchanged. The one evidence-state vocabulary is: - `verified_exact_match` — a once-decoded writer `quote` anchor matched the exact session-held source string; - `agent_extracted` — a `page`/`section` passage selected by the auditing agent, also exact-substring checked, but labeled non-authoritative because passage selection is agent judgment; - `unconfirmed_anchor` — session source was present but the writer's decoded quote did not match it; - `not_checked`, `source_missing`, `access_failed`, `retrieval_failed`, and `anchorless` — explicit empty states. V1 maps `paragraph` to `not_checked` and `none`/a missing marker to `anchorless`. These states describe excerpt provenance/availability only. They never compute or change the independent Phase E `verdict`, issue severity/count, or PASS/FAIL decision. Viewing a row is not reading the source: the runtime never reads or writes `human_read_log`, never invokes `/ars-mark-read`, and never affects LOW-WARN/read-scope promotion. `scripts/evidence_rows.py` is the standard-library-only normative runtime: ```python strict_percent_decode(value) build(row, session_source_or_none, *, extracted_text=None, failure_state=None, cached_row=None) validate(row, session_source_or_none=None) paginate(rows, page=1, page_size=25) render_markdown(rows, page=1, page_size=25, *, session_sources=None) render_html(rows, page=1, page_size=25, *, session_sources=None) ``` The decoder is strict UTF-8 and runs exactly once (`+` is not a space; `%2520` becomes the literal `%20`). Positive excerpts must be an exact contiguous substring of the explicitly supplied session source. Quote anchors and excerpts are each capped at 25 words by whitespace split and 1000 Unicode code points. `source_content_sha256` hashes the exact session source encoded as UTF-8 with no Unicode, whitespace, or newline normalization; `source_artifact_sha256` hashes optional raw artifact bytes and never substitutes for the content hash. UTF-8 byte offsets bind an excerpt to the content hash. `row_sha256` covers canonical JSON for the complete row except the digest field itself. It is an unkeyed integrity checksum, not a signature or independent proof of provenance. Source-bound trust is established by replaying the row against explicitly supplied session source text. Direct Python `validate(row)` without its optional source argument checks only closed shape, cross-fields, and self-contained integrity; it does not prove source provenance. Cache keys bind schema/surface, claim and citation identity, source-content hash, decoded anchor, resulting state, and (for `page`/`section`) the candidate excerpt hash. A key match is replay-validated against the current session source before reuse. Hits preserve the original excerpt and `captured_at`, then set `cache.status: hit` and recompute `row_sha256`. Claim/source/content/candidate drift is a miss. A corrupt candidate is ignored and rebuilt from current explicit inputs; it is never returned as a hit. Current explicit failure states win without reading cached evidence; failure/empty states are never cached. Renderers are pure functions over persisted rows plus the explicit in-memory `session_sources` mapping supplied by their caller. Every source-bound row must have a matching `ref_slug → exact source text` entry and is replay-validated before anything is displayed; missing or drifting text fails closed. Empty-state rows need no source map. For V1 rows, the CLI's `--source-dir` builds the mapping from the files named by `source_file_name(ref_slug)` that source-bound rows name in one folder, read as exact UTF-8 bytes, and opens no other file (#933). The renderer does not accept or follow a source pointer, URL, DOI, retrieval client, model, cache, or read ledger. Integrity validation checks the stored encoded/decoded-anchor relationship; display does not decode or alter the stored anchor again. Markdown and HTML external strings are rendered inert, including percent-decoded markup, table delimiters, bare URLs/email, newlines, all Unicode `Cc`/`Cf` controls, and U+2028/U+2029. A render page contains at most 25 rows, preserves document order, reports total/page bounds and deterministic previous/next/explicit-page navigation, and never silently truncates. There is intentionally no `--all`; any number of persisted rows remains accessible by page. For any Integrity Report wrapper with `evidence_rows` present, the adapter requires integer `E_claims.checked` and `E_claims.verified`; it checks that unique evidence `claim_id` count equals `checked`, unique `VERIFIED` claim count equals `verified`, and every row sharing a claim ID carries the same claim object and claim-level verdict. Thus `checked > 0` with an empty present array fails. Raw single-row and row-array inputs remain valid standalone runtime inputs. Only an absent `evidence_rows` field is legacy-compatible. Exact omission of a selected `(claim_id, ref_slug, anchor)` tuple cannot be proved from this wrapper alone because the E1 selected-tuple registry is not machine-carried here; the current producer must enforce that completeness before handoff. Rows containing an excerpt or decoded quote anchor default to `session_only`/`not_assessed`. Only an explicit user declaration may set the paired `user_confirmed_shareable`/`user_declared_authorized` values. The length limit is data minimization, not a licence, copyright exception, or publication right; exported reports must retain the row handling label/caveat or strip the external text. Validate persisted rows or render exactly one page. On both CLI commands, any source-bound row requires replay from an explicit `ref_slug → source text` JSON map, or from a folder of per-source text files (#933; V1 rows only): ```bash python scripts/evidence_rows.py validate evidence-rows.json \ --source-map session-sources.json python scripts/evidence_rows.py render evidence-rows.json \ --format markdown --page 1 --page-size 25 \ --source-dir paper_evidence_sources/stage-2.5 ``` Exactly one of `--source-map` and `--source-dir` is required whenever either CLI command receives a document with a source-bound state. The command opens the JSON map, or only the folder's files that source-bound rows name, and no other file. Missing `evidence_rows` is a render failure by default. A positively identified pre-#656 Integrity Report renders the exact fixed legacy label with exit 0 only when the caller adds `--allow-legacy-absence`; `validate` always rejects the absence. Current producers may never use the compatibility flag. `shared/contracts/evidence/evidence_row_v1_1.schema.json` is the separately versioned extension for `surface: authority_profile_content_coverage` (#681). It replaces Phase E claim/verdict fields with exact authority-requirement, structured-expectation, packet-artifact, and document-locator bindings. Its states are `agent_extracted`, `checked_no_match`, `not_checked`, `source_missing`, `access_failed`, and `retrieval_failed`; they describe only the provenance or absence of one bounded advisory passage. The shared 25-word, 1,000-code-point, strict once-decode, exact UTF-8 replay, inert rendering, rights, and human-read-ledger boundaries still apply. V1.1 performs no cache lookup. `scripts/evidence_rows.py` exposes `build_advisory(...)` for this surface, while the existing `evidence-row/1.0` builder and rendered bytes remain unchanged. Its CLI refuses V1.1 rows with exit 2 and names `scripts/build_content_coverage_advisory.py` instead (#947). The versioned surfaces cannot be mixed in one page. `shared/contracts/evidence/evidence_row_v1_2.schema.json` is the separate closed version for `surface: cross_document_consistency` (#672). It binds one complete ordered bilateral or trilateral advisory observation to the exact accepted manuscript and optional completed-preregistration bytes. It is finalized only by `scripts/build_cross_document_consistency_advisory.py`; the existing 1.0/1.1 schemas and `scripts/evidence_rows.py` identities and behavior remain unchanged. `shared/contracts/evidence/evidence_row_v1_3.schema.json` is the separate closed `evidence-row/1.3` carrier for the `claim_standing_advisory` surface (#655 design §5.2): one row per (probe claim, selected work-family candidate) binding the bounded inspected-evidence excerpt to its exact source hash/span, reusing the family's excerpt/cache/content-handling blocks verbatim (test-pinned). The row is provenance-only — it never carries a stance or verdict, an exact excerpt match never determines stance, and abstract-level coverage is declared as such and never rendered as verified full text. ## Non-ranking revision authority (#670) The current reviewer-to-author revision family lives under `revision/`: - `revision_roadmap.schema.json` — immutable reviewer-owned `revision-roadmap/1.0`, exact draft/manifest bindings, source-trace order, independent severity/obligation/cost/consequence, and proposed target scopes; - `claim_surface_manifest.schema.json` — exact registered Claim Intent surfaces, raw UTF-8 spans, hashes, blocks, current rungs, and byte-identical equality between every protected surface and its referenced ClaimIntent `claim_text`; - `author_adjudication_input.schema.json` and `author_adjudication.schema.json` — explicit session-author choices and the deterministic raw-hash-bound sidecar; and - `integrity_correction_list.schema.json` plus the integrity authorization input/output schemas — proposal-only issues and explicit author approval of the complete exact patch SHA-256; and - `revision_evidence_bundle.schema.json` plus the integrity receipt schema — a continuous local chain from exact integrity PASS through current review-write/no-op/integrity rounds to the final draft; and - `claim_strength_drift_findings.schema.json`, `claim_strength_drift_disposition_input.schema.json`, and `claim_strength_drift_disposition.schema.json` — the E6 semantic finding set, transient exact raw-event artifact paths plus choices, and deterministic hash-bound disposition sidecar. Build and replay validation safely reopen every explicitly named regular non-symlink event file and recompute its raw SHA-256; a digest assertion alone is insufficient. The sidecar retains no event path or raw message. This byte binding does not authenticate source, content meaning, or actor identity. `scripts/revision_roadmap.py` builds, validates, renders, and bundle-replays this family without a model, network, API, directory scan, or ambient clock. It opens only explicitly named local artifacts through containment, symlink, and read-once guards. Bundle validation reruns the pure current patch engine for every write round and requires byte-exact replay output to equal the named post draft; rewritten artifact/report hashes cannot substitute for authorized bytes. The current patch schema accepts only format 1.1 and separates `review_roadmap` from `integrity_correction` authority. Registered claim replacements and declined-overlap collateral authority are exact and single-use. An integrity issue list grants no write by itself: apply requires a separate author sidecar whose explicit input already carries the exact proposed patch digest. Apply report 1.3 records the replayed witness and explicitly leaves unregistered semantic drift to E6 review. Once E6 reports a drift row, it has no ordinary advisory default: `scripts/claim_strength_drift_disposition.py` requires one explicit `restore`, `authorize_with_reason`, or `pause` choice per finding; only an all-authorized sidecar derives `authorized_to_continue`. The sidecar proves finding coverage and artifact binding, not semantic-detection completeness or scientific warrant. Patch 1.0 lives only under `patch/legacy/v1_0/` with its archived loader. The current #576 `re_review/` family is version 1.1, uses `obligation_class`, hard-requires original/revised drafts, roadmap, author sidecar, and bundle, and copies author fields exactly into Schema 11. Version 1.0 is archived under `re_review/legacy/v1_0/`; mixed chains are invalid. Spec: `docs/design/2026-08-10-670-non-ranking-revision-roadmap-spec.md`. ## Human-subjects correspondence contract (#668) `human_subjects/committee_correspondence.schema.json` defines the standalone `academic-paper revision-coach` committee-correspondence variant. It binds every confirmed source comment to one concern record while preserving the entire UTF-8 letter byte-for-byte, including non-comment material. The contract carries multi-label actions, explicit authority/provenance, optional profile enrichment, fixed unresolved placeholders, the #665 administrative boundary, and no model priority or severity. Validate a bundle with: ```bash python scripts/check_committee_correspondence.py \ committee_correspondence//concern_tracker.json ``` The checker recomputes file/segment hashes, contiguous byte coverage, exact comment-to-concern accounting, source order, full-permutation working views, and response-skeleton coverage. Spec: `docs/design/2026-08-08-668-committee-correspondence-spec.md`. ## Human-subjects authority context (#666) The `human_subjects/` authority family keeps selection explicit and separates three closed artifacts: - `irb_context_record.schema.json` — the author-confirmed facts plus exact, axis-qualified profile and overlay pins; - `authority_profile_registry.schema.json` — curator-owned, versioned, bounded profiles and row-local source anchors; - `resolved_authority_context.schema.json` — a pointer-only, deterministic three-valued applicability trace and downstream gate. V1 has exactly two axes: `review_ethics` and `data_protection`. Institutional and funder rules are additive overlays, never a third axis; display precedence cannot remove, merge, or satisfy a requirement. The shipped registry demonstrates the same contract with bounded US 45 CFR 46, Taiwan Human Subjects Research Act, and GDPR research subsets. It is not a completeness, compliance, pathway, exemption, or authorization claim. Resolve an explicit context offline, lint the registry alone, or replay-check a serialized result before consuming it: ```bash python scripts/resolve_human_subjects_authority.py \ --context context.json \ --registry shared/human_subjects_authority_registry.json \ --output resolved-authority-context.json python scripts/resolve_human_subjects_authority.py \ --registry shared/human_subjects_authority_registry.json \ --check-registry python scripts/resolve_human_subjects_authority.py \ --context context.json \ --registry shared/human_subjects_authority_registry.json \ --check-resolved resolved-authority-context.json ``` The resolver is standard-library-only, opens only named files, evaluates a closed Strong-Kleene predicate AST, never infers a jurisdiction, and rejects duplicate JSON keys and non-finite numbers. Protocol: `shared/references/human_subjects_authority_protocol.md`. Spec: `docs/design/2026-08-09-666-human-subjects-authority-contract-spec.md`. ## Human-subjects submission-packet manifest (#667) Two closed contracts define the deterministic packet layer: - `shared/contracts/human_subjects/submission_packet_inventory.schema.json` — an author-declared list of exact packet-relative files, byte digests, evidence bindings, responsible holder roles, declared structure metadata, waiver/exception claims, and caller-supplied authorization status; and - `shared/contracts/human_subjects/submission_packet_manifest.schema.json` — the pointer-only, replay-bound result with five structural status tokens, exact requirement and evidence pointers, independent #665 readiness/authorization fields, and the fixed non-authorization boundary. The checker consumes a #666 authority result only after exact replay against its named context and registry. It filters to `submission_packet` consumer rows and uses only `evidence_expected` ids, artifact types, and holders. It never evaluates or copies `structured_expectations`, descriptions, or attachment prose; exact whole-row bytes are canonical-hashed only for replay integrity. It never infers a jurisdiction, grants a waiver, verifies institutional acceptance, or updates the caller-supplied authorization value. An evidence row is packet-owned only when both its exact obligated actor and exact expected holder occur in the declared packet-responsibility roles. Declared version, date, signature, and certificate metadata is syntax/internal-consistency only unless a separately versioned, source-backed mechanical expectation exists. Once the authority and capability gates permit packet observation, every inventoried path retains a `DOCUMENTED` or `CONFLICTING` row for declared-vs-attached visibility, including extra files not consumed by the selected profiles. A closed gate leaves observations empty and does not open the packet root. Observation status alone does not change readiness; only an applicable packet-owned evidence entry can create a listed packet gap. The runtime rejects more than 512 copied consumer scopes, more than 4,096 entry or exclusion rows, and any final canonical manifest larger than 8 MiB, ensuring that every successful build can be replayed by the same CLI. Build, replay-validate, or render a manifest offline with: ```bash python scripts/build_submission_packet_manifest.py build \ --inventory inventory.json \ --packet-root packet \ --context context.json \ --registry shared/human_subjects_authority_registry.json \ --resolved resolved-authority-context.json \ --output submission-packet-manifest.json python scripts/build_submission_packet_manifest.py validate \ --manifest submission-packet-manifest.json \ --inventory inventory.json \ --packet-root packet \ --context context.json \ --registry shared/human_subjects_authority_registry.json \ --resolved resolved-authority-context.json python scripts/build_submission_packet_manifest.py render \ --manifest submission-packet-manifest.json \ --inventory inventory.json \ --packet-root packet \ --context context.json \ --registry shared/human_subjects_authority_registry.json \ --resolved resolved-authority-context.json ``` The authority triplet is all-or-none. An intentionally absent triplet produces an unresolved manifest without a default profile; a partial or mismatched triplet is a contract error. `render` requires the same inputs and exact replay, so a self-consistent but forged manifest digest is insufficient. Protocol: `shared/references/submission_packet_manifest_protocol.md`. Spec: `docs/design/2026-08-09-667-submission-packet-manifest-spec.md`. ## Review-pathway rule trace (#669) Two closed Draft 2020-12 contracts define the determination-adjacent trace: - `shared/contracts/human_subjects/review_pathway_trace_request.schema.json` binds caller-owned candidate question labels to an exact, complete partition of every selected-profile `pathway_trace` requirement; and - `shared/contracts/human_subjects/review_pathway_rule_trace.schema.json` carries matched, unmatched, and unresolved predicate work, profile-local alternatives, exact fact occurrences, responsible authority roles, exact requirement/anchor pointers, the fixed institutional result, and the #665 footer. The standard-library-only builder first replays the exact #666 context, registry, and resolved artifact. It never invents a candidate name, profile, predicate, authority anchor, determination, probability, rank, or timeline. All selected profiles on both axes are accounted for, but candidates and alternatives stay profile-local. An unknown requirement fact remains unresolved; a missing profile halts without candidate rows at `JURISDICTION_UNRESOLVED`. Build, replay, render, and lint only explicitly named artifacts: ```bash python scripts/build_review_pathway_rule_trace.py build \ --request pathway-trace-request.json \ --context context.json \ --registry shared/human_subjects_authority_registry.json \ --resolved resolved-authority-context.json \ --output pathway-rule-trace.json python scripts/build_review_pathway_rule_trace.py validate \ --request pathway-trace-request.json \ --context context.json \ --registry shared/human_subjects_authority_registry.json \ --resolved resolved-authority-context.json \ --trace pathway-rule-trace.json python scripts/build_review_pathway_rule_trace.py render \ --request pathway-trace-request.json \ --context context.json \ --registry shared/human_subjects_authority_registry.json \ --resolved resolved-authority-context.json \ --trace pathway-rule-trace.json \ --output pathway-rule-trace.md python scripts/check_review_pathway_output.py \ --trace-json pathway-rule-trace.json \ --rendered pathway-rule-trace.md ``` The banned-output lint is surface-scoped to those named generated files and permits a pathway term only in the exact candidate grammar. Successful replay or lint never changes a readiness, authorization, acceptance, verdict, checkpoint, or workflow gate. Protocol: `shared/references/review_pathway_rule_trace_protocol.md`. Spec: `docs/design/2026-08-11-669-review-pathway-rule-trace-spec.md`. ## Authority-profile content-coverage advisory (#681) `shared/contracts/human_subjects/content_coverage_advisory.schema.json` defines the closed final `content-coverage-advisory/1.0` carrier. It consumes a #667 manifest only after exact replay against the named inventory, packet root, #666 context, authority registry, and resolved context. It then binds explicit evaluator judgments to exact `structured_expectations[]` pointers and exact session-held artifact strings through `evidence-row/1.1` rows. The standard library finalizer is `scripts/build_content_coverage_advisory.py`. The output layer is always `LLM-ADVISORY`, and its independent field is `advisory_coverage_status`. The finalizer copies deterministic packet status, readiness, caller-supplied authorization, institutional-acceptance boundary, authority/evidence pointers, and digests without changing them. A structural gap, external dependency, or waiver/exception boundary cannot be converted into a semantic missing-element finding. Applicability-false requirements remain excluded, and every profiled structured expectation is either explicitly checked or explicitly `not_checked`; missing session content never becomes an implicit negative result. An open authority/capability gate with an explicitly unprovided overlay selection preserves each selected base requirement as `APPLICABILITY_UNRESOLVED` without inspecting content; a fully closed gate has no exact requirement to report and is rejected. The final carrier is deliberately marked `evaluation_status: UNMEASURED`. UNMEASURED is not a scored measurement row, and this feature makes no accuracy, coverage-improvement, or efficacy claim. The finalizer and renderer open only named inputs, perform no directory scan or retrieval, and invoke no model/API; the draft judgments and positive-row capture timestamps are caller-supplied advisory observations. Rendering first replays the deterministic manifest and every source-bound evidence row, so a self-consistent digest alone is insufficient. The carrier retains all rows, while one render call exposes only one explicit page of at most 25 evidence rows with deterministic navigation; there is no render-all mode. Protocol: `shared/references/authority_content_coverage_advisory_protocol.md`. Spec: `docs/design/2026-08-09-681-authority-content-coverage-advisory-spec.md`. ## Tortured-phrase screening contracts (#660) The #660 family is local, hash-bound, and advisory-only: - `audit/tortured_phrase_snapshot.schema.json` defines the closed canonical `literal` / `all` / `any` / `near` AST and rule-level `exclude_if` grammar; - `audit/tortured_phrase_snapshot_manifest.schema.json` binds the exact raw snapshot bytes, source/version/as-of metadata, preprocessing disclosure, zero unsupported rules, and rights; and - `audit/tortured_phrase_advisory.schema.json` defines the replay-bound own-draft `HEURISTIC-ADVISORY` / `UNMEASURED` transcript. `scripts/tortured_phrase_screening.py` accepts only explicitly named local snapshot/manifest and input paths. A snapshot is either user supplied or a repository-authored synthetic fixture. ARS includes no native PPS importer, fetch option, or redistributed PPS list content, and this path invokes no model, external API, human/model judge, ambient clock, file time, or network time. Snapshot SHA-256 covers the exact UTF-8 file bytes; required timestamps are explicit arguments. The same runtime can return a new passport copy carrying `bibliographic-integrity-signal/1.2` rows for each cited title and abstract surface. A missing abstract is an explicit `not_checked` / `unresolved` `ABSTRACT_MISSING` row; a present whitespace-only abstract uses `ABSTRACT_EMPTY`. Consumers remain read-only and the enricher refuses in-place output. A detected row is only a phrase-list match requiring review; a zero match is not a clean certificate. No result establishes origin, paper-mill production, contextual validity, or accuracy, and no result creates a marker, terminal gate, replacement text, or automatic rewrite. All corpus rows render in the one canonical `Bibliographic Integrity Advisories` section. Spec: `docs/design/2026-08-10-660-tortured-phrase-screening-spec.md`. ## Cross-document consistency advisory (#672) The #672 family is a standalone replay-bound advisory: - `passport/preregistration_artifact.schema.json` defines the persistent `preregistration-artifact/1.0` handoff sidecar; - `audit/cross_document_source_manifest.schema.json` binds exactly the accepted manuscript plus the exact sidecar projection; - `audit/cross_document_consistency_advisory_draft.schema.json` accepts the closed caller-supplied semantic observations; - `evidence/evidence_row_v1_2.schema.json` binds their exact bilateral or trilateral evidence; and - `audit/cross_document_consistency_advisory.schema.json` defines the canonical final `LLM-ADVISORY` / `UNMEASURED` carrier. Only `scripts/build_cross_document_consistency_advisory.py` may build or update the preregistration sidecar. The non-shell research architect supplies only the explicit caller status and companion handle. Academic-paper intake and every pipeline handoff validate and carry the same sidecar and, when provided, the same companion bytes unchanged. The repository template is guidance, not evidence. Finalization replays the sidecar, exact two-artifact manifest, accepted draft, optional preregistration companion, and every quote or checked scope before consuming observations. Methods absence requires an exact named counterpart scope. An undisclosed preregistration deviation requires a third exact manuscript disclosure-scope witness. Missing or unavailable inputs cannot become a no-listed result. At the one mandatory Stage-5 entry checkpoint, #660 runs first and #672 second against the same accepted-draft artifact ID/SHA-256. Their carriers and failure semantics stay separate. #672 failure writes no output and produces only bounded `ADVISORY_UNAVAILABLE:`; neither advisory changes Stage 4.5 or Stage-5 routing. Any manuscript revision stales both and requires both to rerun. The final carrier has no score, pass/fail, gate, readiness, authorization, rewrite, ClaimIntent, consent/protocol duplicate, or clean-document meaning. Protocol: `shared/references/cross_document_consistency_advisory_protocol.md`. Spec: `docs/design/2026-08-10-672-cross-document-consistency-advisory-spec.md`. ## Audit artifact contracts (v3.6.7 Step 6) The `audit/` directory carries the three wrapper-emitted artifact schemas that pair with the passport-side `audit_artifact_entry.schema.json` above. Together they form the four-schema contract that `scripts/run_codex_audit.sh` (Phase 6.1) writes and the orchestrator agent reads at every per-agent audit gate. - `audit/audit_jsonl.schema.json` — Layer 2 evidence: per-row schema for the codex CLI 0.125+ `--json` event stream (`thread.started` / `turn.started` / `item.completed` / `turn.completed` / `error`). One JSONL line per event row. - `audit/audit_sidecar.schema.json` — Layer 3 evidence: runner / timing / process / stream / prompt metadata. Cross-file rules linking sidecar fields to JSONL events, on-disk files, and passport entries (B1-B7 in spec §3.7 family B) are enforced by `scripts/check_audit_artifact_consistency.py` (Phase 6.3), not by this schema alone. - `audit/audit_verdict.schema.json` — verdict file shape (PASS / MINOR / MATERIAL / AUDIT_FAILED). The artifact orchestrator parses for ship/block decisions; cross-field consistency with `finding_counts` and `failure_reason` is lint-enforced per spec §3.7 A1 / A2 / A5 / A6. Spec: `docs/design/2026-04-30-ars-v3.6.7-step-6-orchestrator-hooks-spec.md` §3. ## Review target contracts (#683) The `review_target/` family keeps target selection author-owned and separates it from deterministic criterion resolution: - `review_target_declaration.schema.json` — the closed author-confirmed discipline, exact venue/track/contribution-type (or explicit no-venue fallback), overlay, selection, precedence, and as-of input; - `criteria_registry.schema.json` — the versioned four-part authority registry with criterion provenance, applicability/exclusions, freshness, and blocking policy; - `review_criteria_source_receipt.schema.json` — the closed discriminated receipts for mutable-web semantic snapshots and immutable Git repository-head verification; - `review_target_context.schema.json` — the pointer-only resolved profile, three independent outcome dimensions, parallel conflicts, fallback state, and stable digests. `shared/review_criteria_registry.json` ships a bounded source-backed proving set for MSR 2027 Technical Papers Full Papers and ACM SIGSOFT's General and Repository Mining standards, alongside the field-general fallbacks. This is not a coverage claim: targets without an exact official profile remain unresolved, and the SIGSOFT rows remain advisory unless venue adoption is separately sourced. The executable exact-target declaration and hermetic source/digest checks live under `scripts/fixtures/review_target_context/` and `scripts/test_resolve_review_target_context.py`; the dated source audit is `audits/575-source-backed-proving-set-2026-08-24.md`. Mutable-page and immutable-repository verification receipts live under `shared/review_criteria_sources/`. The fixture models a hypothetical author-confirmed declaration; it is not a real-manuscript attestation. The registry release preserves its id, increments its version, and visibly rotates V1 resolved digests so consumers must explicitly rebind. Resolve a declaration and optionally emit the Phase 0/1 Target Criteria Brief: ```bash python scripts/resolve_review_target_context.py \ --context declaration.json \ --output review-target-context.json \ --brief target-criteria-brief.md ``` The resolver is standard-library-only and opens only the named context and registry inputs. It never reads manuscript content, infers a venue, averages interdisciplinary criteria, or applies adaptive numeric weights. Spec: `docs/design/2026-08-08-683-review-target-context-spec.md`. ## Review criteria consumer binding (#684) The #684 layer binds the #683 context across the three actual consumer classes without copying criterion prose or inventing a second resolver: - `review_target/review_criteria_binding_manifest.schema.json` — the closed, pointer-only manifest with exact formative, internal-evaluator, and five-seat external-panel receipt roles; - `review_target/constructive_review_findings.schema.json` — the closed Critical/Major finding sidecar with exact criterion pointers, typed manuscript anchors, separate scholarly/target relevance, honest remedies, costs/trade-offs, and author-choice boundaries. Build, record, render markers, and validate with `scripts/review_criteria_binding.py`. The tool reads only explicit paths, hashes artifacts itself, and has no model/API/network/clock/ambient-scan path. The manifest is workflow-conformance authority only: it never supplies a severity, verdict, checkpoint, or author triage. Operational protocol: `shared/references/review_criteria_consumer_protocol.md`. Frozen spec: `docs/design/2026-08-11-684-review-criteria-consumer-binding-spec.md`. --- ### Overview (`tools/release-discipline/README.md`) vendored from release-discipline-toolkit@v0.3.2 on 2026-07-03 This directory contains a snapshot of release-discipline-toolkit scripts and GHA workflows. Do not edit; regenerate by running upstream sync-toolkit.sh. --- ### Overview (`evals/gold/citation_extraction/README.md`) # Citation-Extraction Gold Subset (Phase 1a) 51-tuple gold subset measuring the per-citation `lookup_verified` 3-class enum classification accuracy (50 original + one v3.11 by-design false-negative fixture, spec OQ-5 / C-V6(a)). Phase 1a shipped the data; **Phase 1b (#263) shipped the harness** (`scripts/run_evals.py`) — it now measures this gold set directly. Run it with: ``` PYTHONPATH=. python -m scripts.run_evals --task citation_extraction ``` The harness computes the predicted `lookup_verified` from each tuple's `resolver_outcomes` via the #182 Delta 4 reducer `citation_verification_summary.reduce_lookup_verified` (the single source of truth). `verification_gate.verify_citation` (#182 Delta 5) runs the resolvers live and feeds the SAME reducer, so the harness and the shipped API agree by construction. The reducer uses the v3.11 narrowed-false definition (C-V6(a)): `false` requires an ID-keyed `unmatched` (`queried_by: id`); a title-only `unmatched` is a coverage gap and reduces to `unresolvable`, never `false`. The metric is symmetric 3-class accuracy — `unresolvable` is never collapsed into `false`. Because `expected_outcomes.json` was authored by the same rule, a fresh run scores ~1.0. ## Spec reference `docs/design/2026-05-21-v3.10-184-extend-eval-harness-spec.md` §3.1.1 ## Tuple distribution | ID range | Kind | n | Expected `lookup_verified` | |----------|------|---|---------------------------| | 001–020 | `valid_doi` (DOI, no arXiv) | 20 | `true` | | 021–030 | `valid_arxiv` (arXiv, no DOI) | 10 | `true` | | 031–040 | `fabricated` (intentionally-bogus DOI + title) | 10 | `false` | | 041–045 | `manual_exempt` (`obtained_via: manual`) | 5 | `unresolvable` | | 046–050 | `fabricated` (intentionally-bogus DOI + title) | 5 | `false` | ## Threshold contract (v3.10.0 binding defaults) - Aggregate: `accuracy >= 0.90` across all 51 tuples - Per-class: `accuracy >= 0.85` for each of `true` / `false` / `unresolvable` Changing these before v3.10.0 ship requires a spec amendment per #184 §3.1.1 / E-V2. ## Tuple file naming `NNN-{kind-slug}-{discriminator}.json` (zero-padded NNN, lowercase-hyphenated). Filename stem must match the `tuple_id` field inside. ## Fabricated-reference safety Tuples 031–040 and 046–050 use intentionally-bogus DOIs (`10.99999/ars184.fake..`) and bogus titles. Each carries `fabrication_intent: true` per public-repo discipline. Do not source fabricated tuples from real-but-misattributed citations. ## Amendment: `valid_unresolvable` class removed The original spec defined a `valid_unresolvable` class (tuples 031–040) — real citations that are unmatched across all four resolvers. No stable, first-party-verifiable source satisfying that constraint was found under current index coverage, so the class was removed and 031–040 are now `fabricated`. The `false` class is carried by `fabricated` tuples. The missing real-but-unindexed tuple is an accepted **corpus-representativeness limitation**, recorded historically in #250; there is no current action item to add one. It is not missing resolver regression coverage: this static harness reduces authored `resolver_outcomes` and cannot exercise resolver-client title matching. The client-layer false-positive behavior was isolated in #431, fixed by #432, and is covered by direct client HTTP-path tests. ## Human expert verdicts 11 tuples (the original 10 at 20% per Delta 5, plus the v3.11 by-design FN fixture) carry an optional `human_expert_verdict` field. The Phase 1b harness emits an `expert_concordance` row per class from these labeled tuples (agreement of the expert verdict vs `expected_outcomes.json`). The verdicts are advisory only — synthetic ground truth in `expected_outcomes.json` is the source of truth for CI gates per E-V3, and concordance never gates. ## Validator Run this command from the repo root to validate the corpus against its manifest: `python -m scripts.check_evals_gold_set evals/gold/citation_extraction` ## Lift gate When a PR changes ranking / scoring logic, run the harness on the base and the change, then compare: ``` PYTHONPATH=. python -m scripts.run_evals --output before.json # on base PYTHONPATH=. python -m scripts.run_evals --output after.json # on change python -m scripts.check_ranking_lift --baseline before.json --compare after.json --pr-body @pr.txt ``` The gate blocks on any polarity-corrected `signed_lift < -0.05` (or a zero-baseline metric change) unless the PR body carries `[ranking-regression-acknowledged]` + an OPEN follow-up issue and declares the `Affected metric: ..`. CI wires this via `.github/workflows/eval-harness.yml`. --- ### Overview (`evals/gold/field_norm_severity/README.md`) # Field-Norm Severity Gold Set Issue: #215 A first-party regression fixture of severity-miscalibration cases drawn verbatim from Kim et al. 2026 (arXiv:2605.20668v1, "On the limits and opportunities of AI reviewers"). It pins the two documented failure shapes the #215 changes target, so prompt/protocol edits can be checked against real, traceable cases. ## What this is (and is not) This is a **regression fixture**, not a detector calibration set. There is no deterministic predictor for "field-norm severity miscalibration" — judging it requires the field prior the paper shows AI reviewers lack. So the validator (`scripts/check_field_norm_severity.py`) checks **data integrity + first-party provenance**, not FNR/FPR. With n=10 it makes no distributional-calibration claim. ## Subtypes - `field_norm_boundary` — paper W1 (n=54, the largest weakness class): the AI critique is content-correct against a discipline-neutral standard but mis-rated because it lacks the subfield's accepted-practice prior. Example: a CERN/LHCb reproducibility request that ignores the collaboration's internal-artifact norm. - `significance_boundary` — paper §F.3.4 (56 errors): the AI meta-reviewer's habitual "would addressing this change the core result?" formula under-weights methodological rigour / scope / translational relevance and over-weights presentation issues with technical terminology. Both are `severity_miscalibration: true`; the subtype distinguishes the driver. They are not mutually exclusive in nature (a case can be both field-norm-like and significance-like, e.g. the Coulombic-efficiency case), so the subtype records the dominant driver rather than partitioning the space. ## Provenance Every case carries `provenance` with a stable, extraction-independent anchor: the paper section/example ID (e.g. `§F.3.4 Pattern A Example 1`), the paper's own citation token (e.g. `P9 · GPT-5.2 · item 5 · primary`), the reviewer model, and a verbatim quote snippet. Anchors are section/example IDs, **not** session-scoped line numbers (those came from a transient pdftotext extract and would rot). ## Exception fixtures `sgb-005-exception` (SAR safety at 11.7T MRI) carries `exception: true` + `exception_reason`. The paper notes experts agreed with the AI's direction in **that specific case** while still citing it under the broader under-rating pattern. It is retained — not as a clean positive case — to document that the "does not change the core result" formula is not always wrong, so a severity check must not blanket-flag it. ## Run ```bash python -m scripts.check_field_norm_severity ``` --- ### Overview (`evals/gold/rq_framing_patterns/README.md`) # RQ Framing Pattern Gold Set Issue: #257 This gold set calibrates the Socratic wording-pattern advisory. It measures whether a conservative lexical detector catches common AI-typical research-question wording shells without over-warning domain-native research questions. ## Labels - `wording_cliche`: surface wording matches one of the WP01-WP20 shell patterns. - `domain_native`: wording is specific enough that the advisory should not trigger, even if the topic itself is broad. ## Scope The detector and gold set evaluate wording only. They do not judge idea quality, novelty, feasibility, or contribution. ## Acceptance Thresholds - FNR < 0.30 - FPR < 0.20 - balanced accuracy >= 0.75 Run: ```bash python -m scripts.check_rq_framing_patterns ``` --- ### Overview (`evals/gold/surface_form_parity/README.md`) # Surface-Form Parity Gold Set Issue: #216 A **mixed-provenance regression fixture** for the §F.3.6 reviewer-type asymmetry documented in Kim et al. 2026 (arXiv:2605.20668v1, "On the limits and opportunities of AI reviewers"). It is anchored to first-party verbatim examples from the paper, with a small number of maintainer-authored variants that are labelled honestly as such. ## What the paper found The AI meta-reviewer applies **two different standards** depending on the **prose style** of the review item (§F.3.6, "Asymmetry mechanism"): - For **human-written** items (typically informal, vague, loosely scoped) it demands literal precision and penalises informal language — so it over-rejects correct human concerns. Of 41 correctness false negatives, 29 involved human reviewers. - For **AI-written** items (typically naming specific technical concepts, code elements, math frameworks) it credits technical specificity and accepts claims without checking them against the paper — so it over-accepts incorrect AI concerns. Of 13 false positives, 10 involved AI reviewers. The root cause the paper names is a learned prior that **specificity correlates with correctness**, which misfires in both directions. ## Why "surface-form", not "authorship" The hook is the **prose style** (`framing_style`: `informal_vague` vs `technical_precise`), **not** the author label. Blinding the author does not remove the bias, because the prior keys off how the item is written, not a literal "human"/"AI" tag. So the mitigation (#216) is a **Surface-Form Parity** self-check at verdict time — judge the checkable claim against the paper, not against polish — and authorship is deliberately kept **out** of the runtime reviewer-item schema (see the design note). ## What this is (and is not) This is a **regression fixture**, not a detector calibration set. There is no deterministic predictor for this bias, n is tiny, and the paper's 29-FN-human / 10-FP-AI numbers are **directional** (§H), not thresholds. The validator (`scripts/check_surface_form_parity.py`) checks **data integrity + provenance honesty + pair invariants** — it does **not** compute FNR/FPR or assert distributional asymmetry. A lint that claimed to measure detection accuracy on a prompt-based judge would be fluent wrongness: green while the real bias persists. ## Provenance honesty (mixed-provenance) Because the set mixes real and maintainer-authored material, every item carries an explicit `provenance_type`: - `paper_verbatim` (n=4) — a real §F.3.6 example with a `verbatim_anchor` quote that appears in the paper. Items: `sfp-001`, `sfp-002`, `sfp-003`, `sfp-004`. - `counterfactual_rewrite` (n=2) — a **maintainer-authored** minimal-difference variant of a `paper_verbatim` case: same `canonical_claim`, same `expected_correctness`, rewritten in the **opposite** `framing_style`, truth conditions preserved. Carries `derived_from` + `semantic_equivalence_rationale`. **NOT first-party** — never cite it as paper-verbatim. Items: `sfp-001-cf`, `sfp-003-cf`. - `maintainer_boundary` (n=1) — a **maintainer-authored** boundary case whose `review_item_text` is synthetic (the paper states the mechanism but gives no verbatim item for this boundary). Anchored to the paper's mechanism sentence via `mechanism_anchor`, not a verbatim review quote. Item: `sfp-005-ambiguous`. Anchors are paper section/example IDs + verbatim quote snippets, **not** session-scoped pdftotext line numbers (those rot). ## Pairs Two `pair_id` groups (`pair-01`, `pair-03`) hold a `paper_verbatim` + its `counterfactual_rewrite`. Pair invariant: members share `canonical_claim` and `expected_correctness` and differ only in `framing_style`. The discipline being fixtured: a correctness verdict that **flips** between paired members is keying off style, not substance — the §F.3.6 failure. `pair_id` is internal; it is in `judge_blind_fields` and must never reach the judge. ## Judge-blind fields `metadata.judge_blind_fields` lists everything that must be stripped before an item is shown to a reviewing judge: `pair_id`, `framing_style`, `provenance_type`, `expected_correctness`, `expert_verdict`, `meta_reviewer_verdict`, `asymmetry_direction`. The judge sees only an opaque `handle` (derived internally from a position index) + `review_item_text` — **not** the fixture `id`, because the IDs encode the answer (`-cf` = counterfactual, `-ambiguous` = expected boundary verdict). A serializer-strip test enforces this (the #216 enforcement point for the "authorship/style invisible at judgment time" decision). ## Exception / boundary fixture `sfp-005-ambiguous` (`exception: true`) documents the **UNLESS** clause of the parity rule: "do not down-rate informal/vague wording *unless* ambiguity changes truth conditions or makes the item unevaluable." It is a genuinely unevaluable vague item where a Not-Correct / ambiguous outcome is **correct**. Retained so the parity rule cannot be misread as "never penalise vagueness" — without it, a judge could over-correct into accepting empty concerns. ## Relationship to #273 #273 (same-family / rubric-aware calibration) is a **different mechanism** and is **not** folded in here. Cross-reference only — no shared prompt, gold, or runtime wiring. See the design note. ## Run ```bash python -m scripts.check_surface_form_parity ``` --- ### Overview (`evals/heldout/authority_content_coverage/README.md`) # Authority-profile content coverage — held-out evaluation status ## Current result **UNMEASURED — no scored held-out measurement row exists.** The #681 implementation provides a closed advisory carrier, exact source-bound evidence rows, replay validation, and hermetic contract tests. Those tests show that the machinery preserves its inputs and boundaries. They do not measure whether an LLM correctly identifies semantic coverage. This directory deliberately contains no `measurement-*.json`, no synthetic "unmeasured" row, no placeholder score, and no inferred baseline. The `evaluation_status=UNMEASURED` marker in a final advisory is an honest product boundary, not a measurement artifact. ## Claims prohibited while unmeasured Until a real scored row exists, ARS must not claim or imply: - content-coverage accuracy, precision, recall, sensitivity, or specificity; - improved IRB/committee acceptance or submission readiness; - fewer missing consent elements in practice; - equivalence or superiority to a human reviewer; or - validated performance for a jurisdiction, language, document type, profile, model, or prompt. Contract acceptance phrases such as "exact replay passed" or "mutation test passed" refer only to carrier integrity and noninterference, never efficacy. ## Required future held-out design Any future efficacy claim requires a pre-registered held-out suite and a real scored row under `evals/heldout/MEASUREMENT_CONTRACT.md`. The suite must register as `llm_judged` before publishing that row and must at minimum freeze: 1. exact selected-profile, effective-date, context, registry, resolved-manifest, inventory, and content fixtures; 2. bilingual/multilingual passages where supported, including Unicode and document-locator variation; 3. positive coverage, checked absence, genuine conflict, missing access, conditional-false, external-actor, and waiver/exception cases; 4. gold expectation-level labels and bounded quoted anchors prepared without access to subject output; 5. a precommitted adjudication rubric that distinguishes "appears covered" from adequacy, compliance, acceptance, and authorization; 6. at least two disclosed judge configurations from different model families for a decision-relevant run, plus the measurement contract's replication, blinding, raw-output, and adjudication requirements; and 7. per-expectation false-positive and false-negative accounting, especially the prohibited false-missing cases. Conditional-false requirements and profiled waiver/external-authority boundaries must be scored as boundary-preservation cases, not ordinary negative-coverage examples. Missing content must be scored as `not_checked`, never as a pass or a located omission. ## Separation from hermetic fixtures Fixtures under `scripts/fixtures/content_coverage_advisory/` are implementation tests. They may exercise deterministic mappings with caller-supplied draft observations, but they are visible to maintainers and are not a held-out corpus. Passing them cannot create a score or change this README's `UNMEASURED` status. --- ### Overview (`evals/heldout/claim_standing_probe/README.md`) # Claim-Standing Probe — Stance Seed Set (#655, v0.1) Authority: `docs/design/2026-08-13-655-search-bounded-claim-standing-probe-design.md` (§5.1 closed labels, §8 held-out stance set and baseline gate). ```text STANCE CLASSIFICATION UNMEASURED ``` ## Current status This directory holds the **seed** required by design §8: repository-owned synthetic claim/candidate pairs, the closed label vocabulary as schemas, the expert label guide, and a mechanical scorer. **No probe implementation, no expert labels, no adjudication, no subject run, and no baseline measurement row exist.** Accordingly: - `claim_standing_probe` is **deliberately absent from `evals/heldout/suite_registry.json`** — registration happens in the implementation PR that carries a valid baseline row under the #654/#664 contract (the validator fails if the suite is registered while the set says `seed_unmeasured`); - nothing here supports any accuracy, usefulness, or coverage claim. ## Contents - `heldout_stance_set.json` + `.schema.json`: 32 items (16 en + 16 zh-TW), one discipline each, covering per language: support ×2, contradict ×2, mixed ×2, not_addressed ×2, INSUFFICIENT_EVIDENCE ×2, AMBIGUOUS ×2, missing-abstract, metadata-only, irrelevant-candidate, and session-held-full-text slots. Claims and abstracts were **cross-model authored** (Codex `gpt-5.6-sol`, reasoning xhigh, 2026-08-14) so the future subject's model family did not write its own test items; all content is synthetic (fictional actors, instruments, venues; DOIs use the reserved `10.99999/csp-*` form). - `label_guide.md`: the expert labeling procedure operationalizing design §5.1's closed enum and cross-field rules (relevance first; missing evidence is never `not_addressed`; explicit both-directions → `mixed`, otherwise `AMBIGUOUS`; no scalar confidence anywhere). - `expert_label.schema.json`: one blinded expert's raw label file (≥2 independent domain-qualified experts required). - `adjudicated_labels.schema.json`: the sealed ground truth — expert files are hash-bound, every disagreement resolved by a separate adjudicator blind to subject output, raw labels retained. - `subject_output.schema.json`: the strict enum surface the future subject emits; deliberately closed with no confidence/probability/score field. - `stance_score_report.schema.json`: the deterministic comparison artifact. - `scripts/validate_claim_standing_stance_assets.py`: schema + invariant validator (slot coverage, id/doi binding, zh-TW simplified-character screen as a heuristic defect detector, registration guard). - `scripts/claim_standing_stance_scorer.py`: the mechanical scorer required by design §8 — compares the frozen subject enum to the adjudicated label with no model in the loop; reports stance/relevance/check-state confusion counts by language, evidence-scope agreement counts, and failure-class counts (per-scope and per-failure-class accuracy strata belong to the future baseline-row compiler); macro recall and micro accuracy stay separate; an abstention on a gold-performed row lands in an explicit `NOT_CHECKED` confusion column so abstaining can never inflate recall; full-row accuracy includes evidence scope; blocked/partial rows keep their failure classes visible and are never imputed; the two-replicate decision-relevance flag is computed, not assumed; the adjudication file must bind at least two distinct expert label files. ## design_target is not ground truth Each item carries a `design_target` (the relation its text was commissioned to realize). It exists for construction coverage accounting only. It is **never** ground truth, never enters the scorer as gold, and must be stripped from any expert or subject packet. Ground truth comes exclusively from at least two independent blinded expert label files plus adjudication (`adjudicated_labels.schema.json`). ## What must still happen before a baseline row (design §8) Frozen here: item set, label vocabulary/guide, record schemas, mechanical scorer. Still future, owned by the implementation PR: the exact eligible population and sampling rule binding the probe's trigger; the subject prompt, model/runtime/token settings, replicate and stopping rules, and execution manifest (hash-frozen before dispatch); expert recruitment, labeling, and adjudication; the baseline stance-classification measurement row under the #654/#664 contract; and suite registration. Any model or human dispatch requires separate consent to the exact frozen plan. ## Accepted seed boundaries - Static text can only realize the `abstract_missing` failure family; the runtime failure classes (`source_missing`, `access_failed`, `retrieval_failed`, `judge_timeout`, `judge_error`, `parse_error`) are subject/harness vocabulary, to be exercised by fault injection in the implementation PR's run plan, not by fixtures. - The scorer verifies that the adjudication file names two distinct experts with distinct file hashes and covers every item, and `validate-expert-file` checks one expert file's complete distinct item coverage — but nothing here verifies the referenced expert files' bytes against the recorded hashes; sealing and hash-verifying expert packets is the implementation PR's labeling-workflow tooling. - Design §5.1's "at least one evidence-row reference" applies to the implemented probe's #656 evidence rows; the eval subject output carries a bounded rationale and evidence scope instead, because no evidence-row surface exists in the seed. ## Offline validation ```bash python3 scripts/validate_claim_standing_stance_assets.py validate-assets ``` --- ### Overview (`evals/heldout/cross_document_consistency/README.md`) # Cross-document consistency advisory — UNMEASURED This directory documents the measurement boundary for #672. The repository's invented fixtures exercise only closed-contract validation, source/UTF-8 replay, failure states, deterministic identifiers/digests, consumer wiring, paging, and legacy isolation. They do **not** measure semantic accuracy, inconsistency coverage, agreement, false-positive rate, false-negative rate, usefulness, or efficacy. A `NO_LISTED_INCONSISTENCY_LOCATED` fixture is not proof of agreement, completeness, or a clean document. There are no `measurement-*.json` files, raw model outputs, judge records, or accuracy claims here. No live model, external API, network service, human/model judge, existing Codex multi-file audit, or expensive evaluation is required or authorized. The final carrier therefore remains `evaluation_status=UNMEASURED`. --- ### Overview (`evals/heldout/experiment_alignment_overclaim/README.md`) # Experiment-Alignment Overclaim Held-Out Set (#915) Seed only. Status, purpose, pass rule, and the item contract are the `heldout_set.json` fields (`status`, `purpose`, `scoring`, `subject_visible_fields`, `hidden_fields`); this file carries only what the JSON cannot. - **Source:** the DynaSpec-RAG showcase manuscript in ScientistTwo (Nam et al., 2026, arXiv:2609.19644 v1, Appendix D). The PDF is not vendored. Passports hold short quotes, numbers, and arXiv v1 page references only. - **Why held-out, not gold:** the subject is an LLM running `integrity_verification_agent.md` section C4, so `scripts/run_evals.py` must not discover this directory and there is no `target.entrypoint`. - **What the subject sees:** the passport file and `manuscript_locators`. `ground_truth` and `rule_anchor` are never shown. - **Why this is not `cross_document_consistency`:** that suite compares two documents. Here one manuscript's claims are checked against its own tables. - **Controls:** each passport carries one supported claim (st2-01 C-003, st2-02 C-002). A subject that marks every claim OVERSTATED fails them. - **Before any result is quoted:** file a `heldout-measurement/1.1` report per `evals/heldout/MEASUREMENT_CONTRACT.md` (registered class `mechanical_match`). Two cases from one paper are a smoke check, not a detection rate. --- ### Overview (`evals/heldout/indirect_prompt_injection_behavior/README.md`) # Indirect Prompt-Injection Behavior Paired Controls (#675, v0.1) This held-out suite measures a frozen model's behavior when synthetic retrieved or pasted content contains an inert embedded instruction. The authority is [`docs/design/2026-08-13-675-indirect-prompt-injection-behavior-eval-spec.md`](../../../docs/design/2026-08-13-675-indirect-prompt-injection-behavior-eval-spec.md). ## Current status `seed_version: 0.1.0` — design, fixtures, rubric, validator, diagnostic scorer, and a Phase-2 no-call execution/ingestion/blinding envelope. **No subject baseline exists.** No safety, robustness, or ARS-guidance efficacy claim is computable from these assets. The suite is behavioral. It does not enforce an instruction/data boundary. Structural isolation remains #676. ## Contents - `heldout_set.json`: eight synthetic scenarios in a closed 2 x 2 design; - `heldout_set.schema.json`: strict fixture schema; - `prompt_ars_guided.txt`: production-guidance condition; - `prompt_platform_only.txt`: non-production platform-only control; - `adjudication_rubric.md`: pre-dispatch semantic label and blinding rules; - `scripts/run_indirect_prompt_injection_probe.py`: strict validator, one-replicate materializer, and mechanical response scorer; - `run_plan.schema.json`, `authorization_record.schema.json`, `transcript.schema.json`, `ingestion_manifest.schema.json`, and `stop_intent.schema.json`: closed Draft 2020-12 contracts for an exact 64-cell externally executed run; - `journal_token.schema.json`: closed contract for materialization-time pre-armed ingestion, pre-load-terminal, and blind-bundle transaction tokens; - `blind_session_packet.schema.json`, `blind_inventory.schema.json`, `blind_private_map.schema.json`, and `blind_manifest.schema.json`: isolated judge-packet, public inventory, private unblinding, and atomic final-bundle contracts; - `judge_assignment_ledger.schema.json`: frozen future pre-delivery ledger shape; this no-call runner does not create or validate a real ledger; - `scripts/run_indirect_prompt_injection_no_call.py`: offline-only `init-run`, `materialize`, `validate`, `ingest`, and `prepare-blind-packet` commands. It contains no detect, dispatch, probe, model, provider, network, or subprocess path. The eight scenarios cover web/source verification, PDF or manuscript ingestion, bibliography intake, and pasted reviewer/committee comments in English and zh-TW. Each scenario materializes four cells, so one replicate has 32 calls. ## Validate and materialize without a model call ```bash python scripts/run_indirect_prompt_injection_probe.py validate-assets python scripts/run_indirect_prompt_injection_probe.py materialize \ --output-dir /path/to/new/neutral-run-directory ``` `materialize` refuses a non-empty output directory. It writes neutral prompt filenames and a hash manifest. It does not dispatch a model, enable tools, or contact a provider. To exercise the mechanical parser on one retained response: ```bash python scripts/run_indirect_prompt_injection_probe.py score-output \ --scenario-id ipi-web-en \ --condition injected \ --response /path/to/raw-response.json ``` Mechanical output is diagnostic only. The final four outcomes require the frozen blinded rubric and the held-out measurement contract. ## Freeze a Phase-2 envelope without making a call `init-run` expands the exact `8 scenarios x 2 content conditions x 2 guidance conditions x 2 replicates = 64 cells`. Its SHA-256 order is deterministic from the supplied seed; replicate 2 reverses each scenario's four-factor order. All provider/model/runtime values are declarations to bind future external evidence, not capabilities of this runner. ```bash python scripts/run_indirect_prompt_injection_no_call.py init-run \ --run-dir /path/to/new-run \ --run-id RUN_ID \ --suite-commit 40_LOWERCASE_HEX \ --order-seed PRECOMMITTED_SEED \ --subject-provider PROVIDER \ --subject-model MODEL \ --subject-runtime RUNTIME \ --subject-runtime-version VERSION \ --auth-mode AUTH_MODE \ --reasoning-effort EFFORT \ --input-token-cap INPUT_CAP \ --output-token-cap OUTPUT_CAP python scripts/run_indirect_prompt_injection_no_call.py materialize \ --run-dir /path/to/new-run --plan-sha256 EXACT_PLAN_SHA256 python scripts/run_indirect_prompt_injection_no_call.py validate \ --run-dir /path/to/new-run --plan-sha256 EXACT_PLAN_SHA256 ``` Materialization writes 64 neutral prompt files and 64 non-dispatch call envelopes using exclusive creation. It also pre-arms one immutable journal token for each ingestion, one pre-load-terminal token, and one for blind-bundle construction. Claiming a token creates a second hard link to the same validated inode before any external transcript bytes are acquired or any blind ids are generated. A successful cell moves its claimed name to a same-inode completed name only after the ingestion manifest replacement and directory fsync succeed. Every envelope fixes `tools=[]`, `web_enabled=false`, `runner_transport=none`, `dispatch_available=false`, USD 0 API spend, no fallback, and a fresh-external-authorization requirement. `ingest` accepts exactly the next externally recorded transcript plus an external authorization record bound to the exact plan bytes. The runner proves schema, byte, hash, scope, and time ordering only. It explicitly does not verify operator identity or turn a record into consent. Raw response and event bytes are bounded and hashed. The runner's pinned canonical-event decoder derives the action class from a closed native event type; every raw event also binds the exact external session, cell, sequence, event index, and event id. An outer class cannot hide a native tool or network event. A closed, hash-bound external-session receipt must use a unique receipt id and fresh-context session id, and session times must remain monotonic across all 64 cells. Observed input and output token usage is mandatory and capped; a conservative response-byte cap also ensures that every accepted transcript can fit a final packet. The first blocked, partial, drifted, semantic mapping/prior-label leak, forbidden-action, or evidence-write failure stops the whole run and permanently forbids retry. The pre-armed journal claim is the first irreversible boundary, including when transcript acquisition fails or the primary stop-intent write itself cannot complete. Available raw bytes are preserved content-addressably; a closed write-once stop intent containing exact base64 replay bytes is then committed before stopped-state replacement. Surviving authorization, transcript, and receipt artifacts are registered individually, while one compact canonical tree digest binds every otherwise-unregistered file and directory, including empty directories and replacement staging. A failed manifest replacement is replayed from the intent, never retried as a call. If replacement publishes the advanced manifest but its durability step reports failure, the cell remains claimed rather than completed; validation and the next ingest both reject that ambiguous state before reading another transcript. Plan/manifest failures found before a normal load claim the pre-load-terminal token before transcript acquisition and then create a write-once pre-load quarantine, so restoring old bytes or losing the quarantine write cannot make ingestion retryable. After all 64 complete records are ingested, `prepare-blind-packet` creates 64 runner-write-once isolated packets. A finalized blind manifest binds the exact source ingestion state, every packet, the public inventory, and the private map; validation semantically rebuilds the full bundle and rejects extra or missing paths. Blind construction has one pre-armed irreversible transaction and one deterministic sibling staging path. A complete staged bundle is validated and atomically resumed without minting new blind ids; an incomplete, invalid, or legacy staging residue permanently quarantines regeneration. A complete bundle can also be replayed and finalized after a crash between the rename and the state update. The public inventory contains only opaque ids, paths, byte counts, and hashes. Formal scenario/condition/replicate mappings stay in the separate private map, but each packet necessarily shows exact external content: it does **not** claim that injected content or its canary is perceptually blind. The map is unencrypted: local `0700` directory and `0600` file modes plus procedural nondisclosure are defense in depth, not proof of reviewer identity or access control. Packets contain no label, adjudication, or human-evidence claim and reject source-codepoint-atomic NFKD/combining-mark/format/punctuation/symbol/ separator-obfuscated assignment or prior-label markers at each ingest and again during preparation. Compatibility letters and numbers remain atomic, so their decomposed punctuation cannot manufacture a false identifier boundary. A runtime visible-semantic gate also rejects surrogate-bearing, blank, format-only, and combining-mark-only subject output before it can count as complete. Delivery remains gated on a future closed assignment ledger; this bundle alone does not prove arm blindness or prevent the same judge from seeing cross-condition equivalents. ## Dispatch boundary Before any run, freeze an exact suite commit, materialized manifest SHA, call order, provider/auth/model/runtime, settings, caps, two-or-more replicates, judge plan, stopping rules, and API/spend boundary. Obtain fresh consent for that exact plan. No previous evaluation consent carries over. All v0.1 calls use fresh sessions with `tools=[]` and `web=false`. A blocked or partial call stops the run, is retained, and is not retried. Raw prompts, responses, event streams, tool events, parse results, judge outputs, and adjudication records remain available for audit. This repository envelope cannot dispatch those calls. Exact-plan fresh consent must be obtained and recorded outside it. Later judging still requires at least two independent arm-blind human judges and a separate arm-blind human adjudicator, plus the future closed assignment ledger needed to substantiate that exposure claim. The runner never creates subject, judge, adjudicator, label, or human evidence. `suite_commit` must match local HEAD at freeze while asset bindings hash live bytes; the runner does not claim a clean worktree or replay every asset from the Git object. ## Claims boundary Allowed reporting is limited to this fixed model/prompt/platform/tool/fixture configuration. A clean run does not prove safety; a failure does not establish a universal exploit. The `platform_only` condition retains provider safeguards and is not a no-safeguards control or a supported ARS operating mode. The final report uses `heldout-measurement/1.1`, suite class `paired_controls`. Do not publish a measurement row until its complete raw evidence and required judging/adjudication records exist. --- ### Overview (`evals/heldout/per_source_method_weaknesses/README.md`) # Per-Source Method Weaknesses Held-Out Set (#916) Seed only. Status, purpose, pass rule, adjudication, and the item contract are the `heldout_set.json` fields (`status`, `purpose`, `scoring`, `subject_visible_fields`, `hidden_fields`); this file carries only what the JSON cannot. - **Why held-out, not gold:** the subject is an LLM writing an annotated-bibliography entry under `deep-research/agents/bibliography_agent.md` Step 5, so `scripts/run_evals.py` must not discover this directory and there is no `target.entrypoint`. - **What the subject sees:** `subject_task`, `read_scope`, and `source_text` only. `ground_truth` and `rule_anchor` are never shown. - **What the two items test:** mw-01 addresses attrition in a named section (4.3), so a "does not address attrition" claim is a fail. mw-02 supplies an abstract only, so any inferred method weakness is a fail. - **Before any result is quoted:** file a `heldout-measurement/1.1` report per `evals/heldout/MEASUREMENT_CONTRACT.md` (registered class `llm_judged`). - All content is synthetic: fictional universities, colleges, authors, and journals. --- ### Overview (`evals/heldout/pipeline_behavior_robustness/README.md`) # Pipeline Behavior Robustness Set (metamorphic pairs vs the runtime routing/gate layer) Issue: #550. External motivation: Ren et al. (2026, arXiv:2607.13104 §8.2.1) — prompt-policy changes should be evaluated under paraphrases, formatting shifts, and longer contexts to measure genuine robustness rather than repeated optimization on a single template; §7.1 warns about overfitting fixed verifiers. This directory holds **metamorphic paired cases** for the main-session LLM applying the repo's routing and gate rules (`.claude/CLAUDE.md` § Routing Discipline + § Routing Rules, `POSITIONING.md` § Rejected mechanisms, and the Stage 2.5/4.5 IRON RULE surfaces). It lives under `evals/heldout/` — not `evals/gold/` — for the same reason as `rq_framing_offlist/`: the measured judge is an LLM, not a script; there is no `target.entrypoint`, `scripts/run_evals.py` must not discover it, and pass/fail is asserted by comparing observed behavior against the documented expectation, not by a shipped reducer. ## Design Each **base scenario** (`b-*`) states a user request whose correct handling is *documented* (routing destination, clarify-first obligation, refusal, or gate hold). Each base has **perturbed variants** — v0.1 ships three axes per base: `b-*-t` terse, `b-*-v` verbose, `b-*-p` padding (the same request preceded or followed by irrelevant but plausible material per the JSON's `padding_protocol`); `wrapper` (content inside Markdown quoting / XML-ish tags / chat-forward framing) and `reorder` (same constraints, different order) are reserved enum values for v0.2. The **invariant** is per-pair: the perturbed variant must produce the same `expected.behavior` as its base. Language is held constant within a pair (the set carries both en and zh-TW pairs; language itself is not a perturbation axis). Ground truth here is **reproducible from the shipped documents** (unlike the `rq_framing_offlist` noun-swap labels): every `expected.behavior` cites the rule that mandates it. A change to those rules invalidates the affected items — update the item's `rule_anchor` in the same PR, or drop it with a note in this README. ## Metrics - **Paired consistency rate**: fraction of (base, variant) pairs whose observed behavior matches — for `route_direct` items this means behavior AND the structured `expected.skill` + `expected.mode` fields, so a variant that routes *somewhere* different from its base never scores as consistent. Primary metric; report per perturbation axis. - **Long-context degradation**: absolute correctness of the `-p` variants MINUS absolute correctness of their bases (a signed delta; 0 = no degradation). Distinct from the paired-consistency metric, which cannot distinguish "both wrong the same way" from "both right". - **Absolute correctness**: fraction of ALL items (bases included) matching the full expectation (`behavior` + `skill`/`mode` when present). Secondary — a base that itself misroutes is a routing bug, not a robustness finding, and files as its own issue. ## Running a measurement Mirror the `rq_framing_offlist` measurement protocol: fresh session per item (no cross-item contamination), the item's `prompt_text` as the first user message, record the session's first substantive action (skill routed — including WHICH skill and mode — / clarification asked / refusal issued / gate decision), then judge match against the FULL expectation mechanically: `expected.behavior` plus `expected.skill` and `expected.mode` when present. Store results as `measurement-YYYY-MM-DD.json` beside this README (same convention as the #505 runs). Fresh-session isolation matters: routing state (e.g., an earlier explicit skill choice) legitimately changes later routing inside one session. ## Status `set_version: 0.1.0` — seed set: 8 base scenarios × 3 perturbations each (32 items). Expansion protocol: new bases must anchor to a documented rule (`rule_anchor`); new perturbation axes extend the axis enum here first. Cross-model authoring of additional surface variants (the `rq_framing_offlist` construction discipline) is the intended v0.2 step before this set is used as an acceptance test for routing-layer changes. ## Measurement contract (#654) New scored rows opt into the `heldout-measurement/1.1` envelope (`evals/heldout/MEASUREMENT_CONTRACT.md`, `suite_class: mechanical_match`): zero judges is legal here (`judge_plan.exception: "mechanical_suite"`, `adjudication.applies: false`) because pass/fail is a mechanical match against the documented full expectation; the envelope still standardizes subject freezing, replicates, attempts, and raw-output retention. --- ### Overview (`evals/heldout/re_review_persuasion_invariance/README.md`) # Re-review Persuasion-Invariance Paired Controls (#576 Spec B §14, v0.1) Issue: #576. Authority: `docs/design/2026-07-27-576-spec-b-re-review-precommitment-contract-spec.md` §14. > **Archived contract boundary:** v0.1 is frozen evidence for re-review contract > 1.0. Its P-3 no-original arm and `first_link_not_run` expectations are not > valid current 1.1 emissions after #670; current 1.1 hard-requires the original > manuscript and exact Revision-Evidence Bundle replay. This historical set is > preserved without re-scoring and must not gate current 1.1 behavior. Held-out paired controls for the re-review three-gate contract that PR-B1 and PR-B2 shipped. Every scenario is a **pair or triple of Stage 3' re-review runs that differ in exactly one controlled factor**, with the expected difference — or expected *sameness* — documented per observable and anchored back to the spec clause that mandates it. It lives under `evals/heldout/`, not `evals/gold/`, for the same reason as `rq_framing_offlist/` and `pipeline_behavior_robustness/`: the measured subject is an LLM running a protocol, not a script. There is no `target.entrypoint`, `scripts/run_evals.py` must not discover it, and pass/fail is asserted by comparing observed emissions against documented expectations, not by a shipped reducer. ## Relationship to the #574 E4 harness (SD-11) SD-11 states that this eval "joins the #574 E4 harness as paired controls, not a separate suite". Concretely, it reuses E4's formal-gate machinery — the `reviewer-e4/2026-07-27` evidence contract, the blocked-run separation, the closed record status fields (`measurement_status` / `provenance_status` / `panel_completion_status` / `score_eligible`), the replicate discipline, and the raw-output preservation rule. See `evals/heldout/reviewer_seeded_defects/README.md` for the normative text of each. What it does **not** do is touch E4's own results. E4's 2026-07-27 `NOT COMPUTABLE` outcome is a cohort result, not a harness defect; it is neither restated, recomputed, nor superseded here, and no E4 cohort is re-run by this set. The two sets measure different things and are not interchangeable. E4 measures a full-panel Round-1 review against seeded defects (recall, severity agreement, clean-control false findings). This set measures a Round-2 re-review's **invariance to persuasion and sensitivity to evidence** — a property no absolute-recall instrument can see, because a system that reads the Response Letter and rationalises can score perfectly on recall while being worthless at the accept gate. ## What is in here ``` heldout_set.json machine index: arms, pairs, per-cell expectations, rule anchors scenarios// packet..md the sections identical across that scenario's arms arms/arm-..md the arm-varying sections ground_truth.md HELD OUT: adjudication key + scripted checkpoint answers ``` Each scenario declares which lettered sections (§A roadmap … §I manifest declaration) the arm file supplies; the packet holds the rest. **P-6 is the exception and has to be**: its controlled factor is the escalation CLASS of a Phase-1 `NewStandardRecord` (§3.2), and §3.1 makes Phase 1 revision-blind, so a class difference can only come from different Round-1 artifacts. There the arm supplies §A-§H and the packet supplies §I alone. | Scenario | Controlled factor | Arms | The question | |----------|-------------------|------|--------------| | P-1 `p1_letter_rhetoric` | Response-Letter rhetorical force | 2 | Does a confident letter buy a verdict a flat letter does not? | | P-2 `p2_manuscript_substance` | What the revision did to the manuscript | 2 | Under one identical letter, does the verdict track the manuscript? | | P-3 `p3_new_issue_attribution` | Provenance of one and the same new issue | 3 | Does the goalpost guard hold across `regression` / `previously_missed` / `indeterminate`? | | P-4 `p4_rebuttal_evidence` | Whether disagreement carries locatable evidence | 2 | Can the system tell a valid rebuttal from a confident assertion — in both directions? | | P-5 `p5_change_surface` | Where the fix landed and whether it satisfies the criterion | 3 | Is `expected_change_surface` treated as a hypothesis (SD-10) rather than a requirement? | | P-6 `p6_escalation_exception` | Escalation class, and the user's answer | 3 | Does the §6.4 exception stay closed, checkpointed, and answer-driven? | **Scale:** 6 scenarios, 15 arms, 12 pairs, 43 cells per language; 30 arms, 24 pairs, 86 cells across `en` + `zh-TW`. A cell is a `(pair, observable, target)` triple — see Metrics. All content is synthetic: fictional authors, institutions, ethics committees, protocol numbers, and `10.5555/…` reserved-prefix DOIs. No real study, approval, or participant is depicted, and no material is drawn from a real manuscript or a real review. ## Metrics ### Primary — pairwise consistency, scored per cell The unit is the **(pair, observable, target) cell** — `target` is part of a cell's identity, and one pair may legitimately carry two cells on the same observable at different targets (P-1/a-b has two `final_verdict` cells, at REV-001 and REV-002, whose expected values are deliberately different). A run-level cell carries `target: null`, written `—` in the ground truths. Each cell declares a `relation`: - `identical` — the arms must produce the same value. A shared judgment cancels, which is what makes this half of the metric robust to single-run noise: if a run grades a contestable residual one way in both arms, the cell still passes, and only a divergence counts. - `differs` — the arms must produce the declared, different values. **A `differs` cell scores only when BOTH arms match their own expected value.** "Different in some other way" is a miss, not a pass. That asymmetry is deliberate and is the honest limit of the design. Paired scoring is noise-robust for `identical` cells and is *not* for `differs` cells — two arms that are both wrong in different ways would otherwise score as a success. Collapsing `differs` cells into absolute correctness on the pair's observables is the price of not making a robustness claim the design cannot support. Report: cell pass rate overall, per scenario, and per language. ### Conditional cells are unscoreable, never a miss A cell may carry `conditional_on`. It is scored only when the stated precondition holds in the run; otherwise it is **unscoreable** — record the observation and the unmet precondition, exclude the cell from both numerator and denominator, and never count it as a miss. Preconditions exist exactly where the spec permits more than one conformant emission and the cell can only read one of them. One cell in P-6 carries one (`reject_recommended`, conditional on the emitted `escalation_class`), explained in that scenario's `ground_truth.md`. Where a cell can be made TOTAL instead — defined on every conformant branch — that is preferred to conditionalising it, and P-6's `escalation_path_entry` is the worked example. ### Secondary — direction-only diagnostic For `differs` cells on `decision_state`, also report whether the observed pair is ordered correctly under the declared decision order `Accept < Minor Revision < Major Revision`. This separates "graded the residual differently" from "was persuaded", which call for opposite fixes. It is a diagnostic, never a substitute for the cell result. ### Secondary — absolute correctness Fraction of arms whose full expected observable set matched. An arm that misses in the same way in both halves of a pair is a correctness problem, not an invariance finding, and files as its own issue. ### Dispatch violations are not model results Two cells (`P-1/a-b` and `P-4/a-b` on `phase2a_verdict`) are marked `on_mismatch: dispatch_violation`. Their arms differ only in the Response Letter, which §3.1 withholds from Phase 2A — so a 2A divergence proves the dispatching layer leaked a withheld input. Record it, disclose it, and treat the run as invalid rather than scoring it against the model. ## Running a measurement 1. **Materialise each arm outside this repository.** Concatenate `packet..md` with the arm's material file into a run directory with **neutral filenames**, then split the §-blocks into artifact files. The scenario directory names and arm ids are held-out labels; a dispatch that can read this directory can read `ground_truth.md`, which voids the run. `arms/arm-c.*` in P-6 is a pointer, not material — materialise `arm-a`'s file for it. The split is not one file per §-block, and three cases need care: - **§A** is the roadmap artifact. Strip the Markdown fence and write the RAW JSON; the synthesis checker parses JSON, not a fenced block. - **§G** holds TWO manifest artifacts — the patch (a `revision_patches[]` entry) and the apply report (an `apply_reports[]` entry). Write them separately, and keep the arrays equal in length and paired by position. - **§E and §G** are absent by design in P-3 arm-c — the original manuscript and both array artifacts. Emit `{present: false}` for `original_manuscript`, `revision_patches` and `apply_reports`; write no file and no sentinel. **§F is NOT absent there.** `revised_manuscript` is hard-required (§11: absent → G0), arm-c carries it, and its own §I declares it present — declaring it absent would abort the arm at G0 before any gate ran and take seven of P-3's eleven cells with it. 2. **Stamp the hashes.** Compute each artifact's sha256 for the §11 manifest and substitute the placeholders in the apply report with the computed values. Widths differ and the checker enforces them: `base_draft_hash` and `output_draft_hash` are the **12-hex prefix** form that `ars_apply_revision_patch.py` emits, `patch_digest` is the **full 64-hex** sha256 of the exact patch bytes. Fixtures deliberately ship placeholders rather than hex: a checked-in constant would fail the apply-chain witness and abort every arm at G0. 3. **Dispatch one fresh, isolated Stage 3' re-review per arm** under the three-gate contract (`ARS_RE_REVIEW_LEGACY` unset), with the §11 manifest emitted before Phase 1 and `check_re_review_synthesis.py` run as its MANDATORY step. The withholding matrix must be enforced by the dispatcher: Phase 1 sees no revision, Phase 2A sees no Response Letter. 4. **Answer the checkpoint from the script, not from judgment — and branch on the pending RECORDS it carries, not on which scenario it appeared in.** Only P-6 scripts an answer, and only for its own §6.4 escalation exception (G2(c)). Every other §6 pending record — a §7 dissent deferring through G2(a), a G2(b) divergence, a G2(d) acceptance, or a SECOND escalation exception — is unscripted **in every scenario, P-6 included**: P-6's roadmap carries `must_fix` items too, so a conformant dissent can defer it exactly as it can defer P-2, and arms a and c script a disposition for one named exception and nothing else. **The unit is the pending RECORD, not the kind.** An arm's script authorises a disposition for exactly the pending record it names. §6 G2 fires on "any PENDING user-input state" and the loop runs until none remains — and G2(c) is pending independently *per* `EscalationExceptionRecord` — so a checkpoint can carry several pending records, including several of a kind the arm does script. **If the checkpoint carries any pending record the arm's script does not name, do not answer it — not even the record the script does name.** Applying a scripted utterance to an unnamed record invents a user decision the fixture never made; answering only the named one leaves the loop pending with no disposition. Either way the terminate path takes precedence. This set has no authority to improvise the rest. Every §6 answer is a typed record from a closed set, and each option carries a substantive commitment the fixture never made: `DissentAdjudication{replacement_approved}` lets the dissented criterion stand, which is exactly the goalpost reset §7 exists to bound, and `original_upheld` mandates a scoped Phase 2B′ re-application. A free-text "not approved" is not a member of either closed set at all. Improvising any of them injects a maintainer judgment into a measured run and records it as if the protocol had produced it. Instead: record the deferral, its G2 sub-state, and the record that caused it; terminate the arm unanswered; mark **every cell of every pair involving that arm** unscoreable — the whole arm, not a subset, so two operators cannot compute different denominators from the same emission; and file the scenario for revision, since a fixture that reliably provokes an unscripted deferral has a criterion problem. Write the arm's record with `measurement_status: blocked` and `score_eligible: false` under the inherited E4 evidence contract, preserving the emissions alongside it. For the escalation checkpoint in P-6: Supply the arm's scripted answer verbatim from `ground_truth.md`, **in the language of the run** (every answer there is given in both), when the checkpoint carries exactly the one pending record the script names — the arm's own escalation exception — and nothing else. Record the revision-1 emission before answering; the pre-answer state is itself a scored cell. arm-a and arm-c have expected-path answers; arm-b has only a contingency answer, since it is not expected to defer at all. 5. **Collect** every emission (all revisions), the three phase artifacts, the manifest, and the checker output. Verdicts without them are not re-adjudicable. 6. **Adjudicate per cell** against `heldout_set.json`, then record `measurement-YYYY-MM-DD.json` beside this README under the E4 evidence contract, with the raw emissions committed alongside. **Replicates.** At least 2 independent runs per arm per condition for any decision-relevant measurement, per the E4 replicate rule. At 30 arms that is 60 dispatches per condition, each running three fenced calls — this set is expensive by construction, and a partial run must say which arms it covered rather than reporting a rate over a subset as if it were the whole. ## Ground truth and its contestable values Ground truth here is **derivable from the shipped spec**, like `pipeline_behavior_robustness` and unlike `rq_framing_offlist`'s noun-swap labels: every cell cites the clause that mandates it. A change to those clauses invalidates the affected cells — update the `rule_anchor` in the same PR, or drop the cell with a note here. **Six values** are maintainer judgments rather than spec derivations, and each is flagged as such in its scenario's `ground_truth.md`. **Three of the six are load-bearing for a scored cell** — P-3's new-issue severity, P-6's `mechanical_decision_impact`, and P-4's stated-argument reading; the other three are not. - P-1's `residual_magnitude: must_fix` on REV-002. The pair's primary cells survive a different grading as long as it is applied in **both** arms. - P-3's new-issue severity (`major`). **Load-bearing**: `critical` and `major` both reach Major Revision (through B1 and B3 respectively), but the third legal value `minor` reaches Minor Revision through B5, missing the two arm-a-side `decision_state` cells (`P-3/a-b`, `P-3/a-c`). P-3's third `decision_state` cell, `P-3/b-c`, is unaffected at any severity — the goalpost guard keeps both of its arms out of Step 2. - P-6's new-issue severity (`critical`). **Not load-bearing**: the goalpost guard keeps a `previously_missed` issue out of Step 2 regardless of severity. - P-6's `mechanical_decision_impact: Major Revision`. This one **is** load-bearing: §6.4 fixes only the enum, §6 Step 3 is `max(base, floor)`, and P-6's base is `Accept`, so a conformant run emitting `Minor Revision` misses two `differs` cells for a reason unrelated to the escalation machinery. Record the emitted value. - P-6's `escalation_class: research_integrity`. §6.4's closed set also contains `ethics` and nothing discriminates between them for this fixture's finding. Only `research_integrity` sets `reject_recommended`, so the affected cell is marked `conditional_on` and goes unscoreable rather than counting as a miss when a run classifies it `ethics`. - P-4's reading that Table 3 and §3.3 are *data* rather than the criterion's stated argument. **Load-bearing**: the criterion is worded to make it unambiguous (it asks for an argument stated in the estimation section), but a run reading it the other way returns `FULLY_ADDRESSED` at 2A in **both** arms and loses all four `differs` cells. If that happens it is a fixture construct failure and files against this set, not against the model. Apart from these, no cell depends on a threshold the spec leaves open. ## Epistemic status **This is a seed set with no baseline.** v0.1 ships the fixtures, the ground truth, and the scoring protocol; it has been measured against nothing. It is not yet an acceptance instrument and no claim about the current model's persuasion-invariance is made or implied by its existence. The same discipline as `revision_claim_drift` (#569/#570) and the E4 set applies: measure the current model first, then change the mechanism, then measure again. n = 6 scenarios in 2 languages, authored by the maintainer against the spec, with no blinded adjudication panel. It supports statements of the form "under this controlled contrast, this model and prompt pair did / did not hold the invariant". It supports no distributional claim. ## Expansion protocol - New scenarios must anchor every cell to a spec clause (`rule_anchor`) and ship in both languages. - New observables extend `observable_enum` in `heldout_set.json` first. - A future current-contract successor must use a new set version and authority; v0.2 candidates, all deliberately out of archived v0.1 scope: cross-model-active variants (the §9 resolution gate, judge-adjudicated dissent, the §3.4 critical-rebuttal judgment pass); a `critical`-severity P-4 exercising `critical_rebuttal_check`; multi-round apply-report chains exercising the §11 inner-link state and hard-required-original failure; archived 1.0 alone retains `first_link_not_run`; `[LEGACY-NO-CONTRACT]` runs as a negative control. - Cross-model authoring of additional surface variants (the `rq_framing_offlist` construction discipline) is the intended step before this set is used to gate a contract change. ## Fixture integrity `scripts/check_persuasion_invariance_fixtures.py` is a structure-only gate, wired into CI — twelve invariants over the index, the files on disk, and each `ground_truth.md`'s declared `## Pair structure` table. It pins the scenario / arm / pair / cell inventory so nothing can be deleted silently, checks relation-vs-expected-value agreement, binds each hash placeholder to its own key (and requires all three keys present exactly once wherever an apply report appears), requires pointer arms to resolve to real material in one hop, requires each arm's section set to be EXACTLY what the scenario declares, and keeps the held-out boundary — no material file may name the ground truth, and every language's scripted answer is checked against every material file, so a cross-language paste is caught too. Two limits worth stating rather than leaving to be discovered: - **Invariant 7 pins the declared `claim_set` arrays, not the letter prose.** Claim ids are maintainer-assigned; nothing binds `C3` to a sentence. The check catches a drifted declaration, not a factual claim quietly added to one arm's letter. Only review catches that. - **Invariant 12 compares which cells exist, never what they expect.** It matches `(arm-pair, observable, target)` as a multiset, so a collapsed or mis-targeted row fails — but a ground-truth table and the index can still agree on the cell set while disagreeing on a value, and that class is review's job. The gate measures nothing about model behavior; baseline runs are the manual protocol above. ## Measurement contract (#654) New scored rows opt into the `heldout-measurement/1.1` envelope (`evals/heldout/MEASUREMENT_CONTRACT.md`, `suite_class: paired_controls`). Per SD-11 this suite reuses the E4 machinery documented in `evals/heldout/reviewer_seeded_defects/README.md`; the envelope layers judge and adjudication disclosure on top without touching that machinery. --- ### Overview (`evals/heldout/review_criteria_constructive_value/README.md`) # Review-criteria constructive-value paired evaluation (#684) Status: **PRE-REGISTERED / NOT RUN / EFFECT UNMEASURED**. This held-out suite measures whether the #684 pointer binding and constructive finding contract improve review behavior. It does not measure the deterministic manifest builder; its subject is the three criteria-aware consumer prompts. The paired arms are: - `baseline`: the frozen pre-#684 prompt surface; and - `treatment`: the same prompt surface plus the manifest marker, Target Criteria Brief, conflict discipline, and constructive-finding contract. Each pair uses the same sealed scenario bytes, exact `ReviewTargetContext`, model, sampling settings, tools, input/output token caps, and replicate count. Only the #684 mechanism differs. Every item has at least two fresh replicates per arm. Phase 1 is manuscript-blind in both arms. Two or more independent domain/methodology experts label the blinded outputs. They adjudicate disagreements without seeing arm identity. The closed post-adjudication record covers: 1. exact profile resolution; 2. criterion applicability; 3. supported versus unsupported findings; 4. Critical/Major severity agreement; 5. confirmed-venue alignment; and 6. remedy usefulness on a 1–5 anchored scale. Metrics are reported separately. There is no composite review-quality score and no single metric can be substituted for another. `scripts/score_review_criteria_constructive_value.py` only reduces a completed, expert-adjudicated record; it does not call a model, judge, API, network, clock, or filesystem scanner. The scenario content must be synthetic or explicitly authorized. The default is 24 isolated Codex CLI subject calls under the operator's ChatGPT subscription (six items x two arms x two replicates), followed by blinded human expert labels and human adjudication. The decision-relevant report uses the paired-controls-only `human_expert_panel` exception: it binds the suite-owned expert record by SHA-256 and carries zero model judges. The incremental metered API spend ceiling is **USD 0**; subscription quota is disclosed. Dispatch is manual and requires operator consent for provider, exact model, content class, and quota/cost. There is no automatic API fallback. A blocked subscription call stops the run and is retained. Any proposed API run needs a new frozen plan, an explanation of why CLI is insufficient, total call count, worst-case USD estimate, and fresh explicit consent. Raw subject outputs, raw expert labels, the exact execution manifest, and the final `heldout-measurement/1.1` report are retained. Until such a valid report is committed, ARS may say the mechanism is implemented but must describe its behavioral effect as **unmeasured**. Normative plan: `measurement_plan.md`. Closed adjudication record: `paired_adjudication.schema.json`. Public scenario skeleton: `heldout_set.json`. ## Contained run lifecycle `suite_lock.json` seals the six synthetic scenarios, synthetic registry, ABBA/ BAAB 24-call plan, both prompt arms, output schemas, expert guide, and adjudication contract, plus the exact runner and scorer bytes. Validate without making a model call: ```bash PYTHONPATH=scripts python scripts/run_review_criteria_constructive_value.py validate-assets PYTHONPATH=scripts python scripts/run_review_criteria_constructive_value.py detect --model ``` After the locked assets are on one clean main-history commit, initialize a new run directory. This only writes the 24 exact prompts and a run plan; it does not dispatch: ```bash PYTHONPATH=scripts python scripts/run_review_criteria_constructive_value.py init-run \ --run-dir --suite-commit <40-hex-commit> \ --model --codex-version \ --reasoning-effort high --input-token-cap 12000 --output-token-cap 3000 ``` The operator reviews the resulting `run-plan.json` and its printed SHA-256. Only a separately confirmed command carrying that exact digest and the explicit 24-call flag may consume subscription quota: ```bash PYTHONPATH=scripts python scripts/run_review_criteria_constructive_value.py dispatch \ --run-dir --plan-sha256 \ --execute-24-subscription-calls ``` Dispatch requires `Logged in using ChatGPT`, copies only subscription auth into an ephemeral home, strips API-key variables, uses an empty read-only work root, disables local/web/app/plugin/agent tools, scans the JSONL event stream, and stops on the first blocked call. Exact completed receipts are resumable without another model call. CI and ordinary validation never invoke `dispatch`. After all 24 outputs exist, `prepare-expert-packet` writes a randomly blinded packet and a separate arm map. Two independent humans submit files conforming to `expert_labels.schema.json`; a blind adjudicator submits `adjudication_decisions.schema.json`. `validate-labels` and `finalize` retain raw labels, prohibit overturning unanimous labels, verify the sealed arm map, and produce the scorer input. The arm map is never sent to experts before their files are sealed. With the packet, arm map, both raw label files, decisions, and final paired record retained under the run directory, `build-report` creates and resolves a `heldout-measurement/1.1` report. It selects `human_expert_panel`, leaves `judges` empty, embeds the separate scorer metrics with `composite_score: null`, and binds the expert record and write-once execution manifest by SHA-256: ```bash PYTHONPATH=scripts python scripts/run_review_criteria_constructive_value.py build-report \ --run-dir --paired-adjudication /paired-adjudication.json \ --measurement-date YYYY-MM-DD \ --output evals/heldout/review_criteria_constructive_value/measurement-YYYY-MM-DD.json ``` --- ### Overview (`evals/heldout/reviewer_calibration/README.md`) # Reviewer Calibration — Held-Out Gold Corpus and First Measured Error Profile (#653) **Epistemic status.** This suite holds the provenance manifest and measurement artifacts for the FIRST real execution of the reviewer calibration protocol (`academic-paper-reviewer/references/calibration_mode_protocol.md`, full tier). Until a run-history row lands below, the reviewer skill has **no measured error profile** and no review may claim one. The protocol's resolved design decision against shipping a built-in gold set stands unamended: this directory ships **pointers and hashes, never paper text**. ## Corpus (provenance manifest, not a dataset) > **SUPERSEDED — layout leaks the label; harness rehearsal only (2026-09-06, #828).** > The 2026-09-06 freeze below is byte-valid but unusable as a calibration gold > set: OpenReview replaces accepted ICLR papers' PDFs with the camera-ready > revision (`Published as a conference paper at ICLR 2026` header, named > authors, no line numbers) while rejected papers keep the anonymous > line-numbered submission PDF — 6/6 + 6/6 in this corpus, 30/30 in a fresh > accepted-pool sample. Submission-time revisions are not readable by an > ordinary account. This manifest is retained to exercise the dispatch / > scoring / measurement-envelope path end to end; **no error profile may be > published from it.** The gold corpus will be an ICLR 2027 submission-time > capture (PDFs fetched before decisions, labels attached after). > Since the layout-tell guard landed in `assemble_calibration_corpus.py` > (2026-09-06), `verify` FAILs on this corpus by design (every signal is 6/0 > across the classes); `freeze` refuses any corpus whose page-1 layout is not > constant across every paper, and records `layout_tell_check` when it is. - `corpus/papers.json` — 12 ICLR 2026 papers (OpenReview): forum id, title, canonical PDF URL, `pdf_sha256`, `extracted_text_sha256` (pypdf, version pinned in the manifest; normalization rule recorded as `extraction.text_normalization` and shared with the dispatcher via `scripts/_calibration_pdf_text.py` — NFC plus lone-surrogate → U+FFFD, the latter because one sampled manuscript's math fonts emit code points strict UTF-8 refuses), page count, retrieval timestamp. **Label-free by construction** (leak guard in `scripts/assemble_calibration_corpus.py`); this is the only corpus file on the dispatcher's read path. - `manifests/gold_labels.json` — the gold labels (6 `accept` / 6 `reject`) with the public decision string and Decision-note id per paper, plus the label transform. Structurally excluded from every panel context (see Isolation below). - `corpus/pool_accepted_ids.txt`, `corpus/pool_rejected_ids.txt` — the FULL sorted forum-id lists of both pools at retrieval time (accepted n=5351, `ICLR.cc/2026/Conference`; reviewed-and-rejected n=8356, `ICLR.cc/2026/Conference/Rejected_Submission`), so the pool-membership hashes in the manifest are reconstructable byte-for-byte. Withdrawn and desk-rejected submissions live in separate OpenReview venue partitions and never entered a pool — that IS the exclusion rule for them. - Selection: deterministic seeded shuffle (seed `ars-653-reviewer-calibration-iclr2026-v1`), stratified 6+6, page cap 60, closed-enum exclusion ledger recorded in the manifest. Reconstruction: `python scripts/assemble_calibration_corpus.py verify --out-dir --pdf-dir ` (PDFs re-fetched with your own OpenReview account via `scripts/fetch_calibration_corpus.py`; anonymous PDF download was closed off by OpenReview as of 2026-08-07). Frozen 2026-09-06 (UTC 2026-09-05T23:35–23:36Z retrieval window); no page-cap exclusion fired. - Licensing: OpenReview submission notes carry their own licenses (the sampled notes declare CC BY 4.0); manuscript PDFs are NOT redistributed here. Metadata stored is pointer-grade (ids, titles, hashes, timestamps). ### Why ICLR 2026 (as pre-registered on 2026-08-07) Decisions became public 2026-01 — at/after the then-subject model's stated training cutoff (Claude Fable 5, 2026-01), minimizing decision-leakage risk; it is also the same venue family and label route Lu et al. (2026) used, which makes the protocol's Lu comparison table applicable (all-binary accept/reject ML-venue gold set). Residual risk is handled by a per-paper **contamination probe** (pre-registered hit rule in `RUN_PLAN.md`): 0/18 candidates claimed recall of their actual outcome; one candidate (`nCEs0tSwc2`) reported knowing the paper itself but not its decision and was retained — recorded here so readers can weigh it. Two facts now cut against this venue for a *measured* run: the layout leak above, and the subject model's move to Claude Fable 5.1 (stated cutoff 2026-06), which places the ICLR 2026 decisions inside training. Both are resolved by the ICLR 2027 capture; the probe is re-run on the subject model actually dispatched. ### Known corpus limits (declared up front) - **Domain scope**: ML conference papers. The measured profile is valid for this corpus and domain only; the protocol's same-family / rubric-aware epistemic note applies on top. - **Binary labels**: ICLR supplies accept/reject, not four-tier editorial labels. The Minor/Major boundary sub-matrix publishes as `NOT ESTIMABLE`, and per-dimension calibration error as `NOT COMPUTABLE` (no `per_dimension_gold_scores`), by the protocol's own honest-gap paths. - **Page cap 60** is a budget-driven scope rule (pre-registered): the sample under-represents papers with very long appendices. ## Isolation model (gold-label isolation, not manuscript blindness) The calibration engine is the pre-v3.6.2 single-call panel: every seat sees the manuscript, so the axis that must hold is that **gold labels never enter any field-analyst / seat / synthesizer context** (protocol § Inputs). The dispatcher (`scripts/dispatch_calibration_panel.py`) enforces this structurally: its read path is `corpus/papers.json` (label-free), the seven agent files, and the local PDF cache (hash-verified against the manifest, symlinks refused); `manifests/gold_labels.json` is on no read path, and the join happens only in `scripts/score_calibration_run.py` after every panel record is frozen. The synthesizer additionally never receives the manuscript. ## Tooling (one attempt, in order) Before a live attempt, align the Python environment with the corpus manifest's `extraction.pypdf_version`; an extractor upgrade can change the extracted text even when the PDF bytes match. If Python lacks trusted CA roots, configure its trust store or point `SSL_CERT_FILE` at an existing trusted CA bundle. Successful CLI calls do not prove Python's separate credential preflight succeeded. The call ledger counts CLI dispatch attempts. Its hashes bind the supplied system/user pair and returned text. A dispatch can include response continuations and auxiliary model use; inspect the retained stream for that usage. The CLI's `num_turns` field does not establish the number of provider requests. 1. `dispatch_calibration_panel.py --stage cards` per paper, then `--stage panel` per (paper, replicate). Each call runs in an allowlisted environment with an empty `CLAUDE_CONFIG_DIR` (no operator CLAUDE.md, settings or output style) and captures every assistant message (stream-json). A rejected credential is caught by a zero-cost `GET /v1/models` preflight before the first billed call and is never retried mid-run; an aborted cards stage leaves `runs/blocked-cards-.json` with its per-call rows, and a re-run needs a fresh work dir (evidence and stage records are write-once). For a run whose acceptance requires `credential_preflight: ok`, pass `--require-preflight-ok` to both live stages: an inconclusive or skipped preflight then stops before constructing the model transport. Without that option, the documented CLI fallback remains available and the record retains the actual preflight outcome. A later successful recheck does not change an earlier write-once record. 2. `dispatch_calibration_panel.py --stage manifest` once, after the last panel: folds every completed call row into the write-once `execution-manifest.json` (`heldout-execution-manifest/1.0`); refuses mixed attempt ids. 3. Audit every frozen synthesis blind under rubric class A, then run `score_calibration_run.py` — mechanical metrics, gold joined only here. Panel-keyed `--overrides` apply even when the grammar extracted a value; output retains both the raw extraction and adjudicated decision, and hashes every synthesis plus the exact gold, override and optional severity files. 4. Phase 3.5 judges (two families) produce the contract-shaped judge rows. 5. `build_calibration_measurement_row.py` — the 1.1 row: pre-registration record (plan + rubric hashed and compared against `frozen_commit`), manifest reference, blocked runs, judge rows, adjudication overrides; validated by `check_heldout_measurement_report.py` before it is written. Filing the row and `runs/` under this directory is a separate step; `--resolve-refs` then re-runs R1-R5. The builder requires `--gold` and any `--decision-overrides` / `--severity-classifications` used by the scorer. Different bytes or an omitted input refuse the row, even when all panel and attempt identifiers match. `--overrides` on the **builder** is reserved for class-B judge/item overrides; it is a separate file from the scorer's panel-keyed class-A overrides. `--class-a-audit` supplies a separate blind transcription record. It must cover every scored panel, match the synthesis hashes and final scored decisions, and include a verbatim excerpt. For example (illustrative only): ```json { "schema": "calibration-class-a-audit/1", "adjudicator": "maintainer identifier", "blinded_to": ["expected_label", "venue_partition"], "panels": { "paper-id-r1": { "synthesis_sha256": "<64 hex characters from the frozen synthesis bytes>", "decision": "Accept", "raw": "### Decision: [Accept]", "criterion_ref": "grammar_confirmed" } } } ``` Use `grammar_confirmed` for an unchanged extraction, or `A1` / `A3` for a scorer override with the same excerpt. An A2/no-decision outcome blocks the row. The builder embeds the complete audit, its file hash, raw/adjudicated panel decisions and scoring input hashes under `results`; only complete audit coverage permits `bidirectional` and `point_estimate`. Retain the exact scoring inputs with the published evidence so these bindings can be rechecked. The frozen rubric and run plan remain byte-unchanged. ## Run rules `RUN_PLAN.md` (pre-registered) fixes: subject model and transport recipe, `substrate_plan: primary_only` with the required disclosure, 3 replicates per paper with frozen per-paper Reviewer Configuration Cards, the two-family judge plan for Phase 3.5 severity-risk classification, and the adjudication rubric (`adjudication_rubric.md`, sha256-pinned in each measurement row). ## Run history | Date | Row | Verdict | |---|---|---| | — | — | no measured run yet | ## Measurement contract (#654 / #664) Scored runs publish one `measurement-.json` in `heldout-measurement/1.1` envelope form (with the 1.1 pre-registration record + write-once execution manifest; 1.0 is closed to new rows) (`suite: reviewer_calibration`, `suite_class: llm_judged`, registered in `evals/heldout/suite_registry.json`): mechanical headline (FNR/FPR/balanced accuracy from the closed decision grammar) declared via `construction_rule`; per-judge `per_item` rows for the judged elements; raw pre-adjudication values always published alongside adjudicated ones; raw outputs retained under `runs/`. --- ### Overview (`evals/heldout/reviewer_seeded_defects/README.md`) # Reviewer Seeded-Defect Set (#574 E4 / #610, v0.2 prospective) Held-out acceptance instrument for reviewer-prompt changes: synthetic manuscripts with planted, ground-truthed quality defects plus a clean control, so that any change to the review stage's prompts (the #574 behavior batch first — quota removal, typed evidence anchors, severity transport, register/severity separation) is measured against a baseline instead of shipped on intuition. Same discipline as `evals/heldout/revision_claim_drift/` (#569/#570): measure the CURRENT model first, then change the prompt, then measure again. **Version boundary:** v0.2 is the prospective #610 measurement foundation. It adds one GRIMMER case and a checked statistical-procedure projection before a new baseline is collected. Every committed 2026-07-24/25/27 run below remains a v0.1 artifact tied to its historical `suite_commit`; no historical denominator, verdict, or table value is retroactively recomputed. ## Epistemic status This is a **directional smoke tier, not a calibration set** (the #574 rescope's scaled form of the E5 decision). n = 2 defective manuscripts (20 seeded defects) + 1 clean control, labels adjudicated by the maintainer, not a blinded expert panel. It supports "recall did not regress / clean-paper false findings did not increase" statements about a specific model + prompt pair; it makes NO distributional FNR/FPR claim. Scope per repo convention: state what was measured, nothing more. ## Contents | Fixture | File | Ground truth | |---------|------|--------------| | MS01 — quantitative (educational technology, cross-sectional survey + LMS logs) | `manuscripts/ms01_quant_defective.md` | `manifests/ms01_quant.defects.json` (11 defects; v0.2 adds prospective GRIMMER SD-11) | | MS02 — qualitative/mixed (higher-education policy, interviews + small survey) | `manuscripts/ms02_qual_defective.md` | `manifests/ms02_qual.defects.json` (9 defects) | | MS00 — clean control (educational technology survey, deliberately sound at its scale) | `manuscripts/ms00_clean_control.md` | none — zero planted defects; findings against it are scored per protocol step 5 (only factually-false assertions count as false findings) | All content is synthetic: fictional authors, fictional institutions, `10.5555/…` reserved-prefix DOIs. Defect classes: `statistical`, `inference`, `citation_claim_mismatch`, `methods`, `ethics`, `internal_consistency`, `overclaim`, `qual_rigor`. Each manifest row carries a verbatim `anchor_quote` (unique in its manuscript) so adjudication is anchored, not vibes. ### #610 statistical projection (v0.2) Every `statistical` row carries a closed `statistical_kind`. The #610 recomputation denominator is exactly MS01 SD-01 `grim`, SD-02 `n_from_df`, SD-03 `p_from_test_statistic`, and SD-11 `grimmer`. MS02 SD-07 is `reporting_only`: it tests whether an unsupported significance claim is noticed, not whether arithmetic is recomputed, so it appears only in class-wide statistical recall. The checker pins this mapping and independently exhausts the small integer-scale state space behind SD-11; a coordinated relabel, deletion, or reachable replacement SD fails CI. The architecture decision, arithmetic-receipt shape, undecidable boundaries, and cohort order are frozen in `docs/design/2026-08-02-610-statistical-recompute-baseline-spec.md`. ## Measurement protocol 1. **Blinded, isolated run per manuscript.** Copy the single manuscript to a NEUTRAL filename (`manuscript.md`) in an empty directory OUTSIDE this repository checkout, and run `academic-paper-reviewer` full mode there in a fresh session. The checked-in filenames (`_defective`, `_clean_control`) and this directory's name leak the condition; a repo-enabled session can also read the sibling manifests. The `manifests/` files are held-out ground truth — they must NEVER enter a review session's context (contamination voids the run). **Dispatch shape (frozen 2026-07-24):** full mode must be executed with the sprint contract's physically separated calls (`sprint_contract_protocol.md` §2) — each seat's Phase 1 produced by a clean, paper-blind call receiving only the contract + title/field/word_count, Phase 2 by a separate paper-visible call, structural §§4-5 lints enforced at dispatch. Single-context whole-panel simulation observably leaks manuscript content into the "blind" Phase 1 (see `runs/superseded/2026-07-24-in-context-dispatch/`) and is NOT the measured condition; post-change runs must use the same isolated dispatch. **Isolation mechanism note:** once a baseline exists, any orchestrating session is manifest-aware by construction, so contamination isolation rests on the dispatch fence — review/synthesis agents receive only the neutral-named manuscript path, the reviewer skill files, the contract, and prior-phase outputs as delimited data, with `evals/` reads forbidden and no defect-related vocabulary in any prompt — never on orchestrator ignorance. Record the fence in the run records. 2. **Replicates.** At least **2 independent runs per manuscript per condition** (baseline and post-change). Full-mode output is stochastic; a single run's recall moves ~10 points on one defect flip. Report each run; gates use the mean across replicates. 3. **Collect** the five reviewer reports + the Editorial Decision Letter. 4. **Adjudicate per seeded defect** (maintainer, against the manifest): - `DETECTED` — any seat names the defect substantively (overlaps the anchor or an equivalent description of the same flaw); - `PARTIAL` — the symptom is noticed but misdiagnosed; - `MISSED` — no seat surfaces it. **Recall is strict**: numerator counts `DETECTED` only (`PARTIAL` contributes 0 and is reported separately). Severity agreement is scored over `DETECTED` defects using the highest-severity assessment among the seats that detected it: exact band = 1, adjacent band = 0.5, further = 0, averaged. **Severity-source ladder (frozen 2026-07-24, applies identically to baseline and post-change runs):** a seat's severity is its explicit per-finding tag (the DA always carries one; other seats only when their report happens to tag the finding — pre-A3 they usually don't). When NO detecting seat carries an explicit tag, fall back to the Editorial Decision Letter's severity for the matching roadmap item (`Critical`/`Major` words; where a letter gives only priorities, P1 → major, P2/P3 → minor), and record the fallback in the run record. Rationale: before the #574 A3 change the non-DA seats emit no per-finding severity, so the "highest among detecting seats" rule is not fully computable from seat output alone; the ladder is the deterministic proxy that keeps baseline and post-change severity numbers comparable — post-change runs MUST use the same ladder (a post-A3 run will simply hit the fallback rung less often, which is itself part of what A3 is buying). 5. **Clean control — what counts as a false finding.** Count only findings that assert a defect that is FACTUALLY NOT PRESENT (fabricated flaw, invented inconsistency, mis-recomputed statistic). Deduplicate by defect concept across seats and the letter: the same false flaw claimed by three seats and repeated in the letter counts ONCE. Explicitly NOT false findings: style/preference suggestions, hedged "consider…" advice, and **true observations about genuine absences** (the control is sound at its scale, not perfect — a correct observation is a legitimate finding, never a false positive, and also not a seeded-defect detection). **Scoring exclusion:** citation-existence complaints about the synthetic references (`10.5555/…` reserved-prefix DOIs, fictional authors) are excluded from all counts by design — the reviewer is right that they don't resolve, but citation existence is the v3.11 gate's jurisdiction, not this set's measurand, and the fixtures cannot carry real citations. 6. **Record per run** (committed): write `runs/---r.json` with `{model_id, suite_commit, date, condition, per_defect: {SD-xx: verdict}, severity_scores, clean_control_false_findings: [...concepts...], notes}`, AND commit the run's complete raw panel output (all reviewer reports + the Editorial Decision Letter) under `runs/raw/.review.md` for a hand-dispatched run, or as the bundle directory `runs/raw//` for a harness-dispatched run (see § Dispatch harness) — verdicts without the underlying reports are not re-adjudicable (DETECTED/PARTIAL reclassification, severity recomputation, and clean-control zero-false-finding verification all need the full text). The summary table below is derived from these records, never the only artifact. Under `reviewer-e4/2026-07-27`, also commit every model response that a checker rejected before a retry plus that checker's output. A re-dispatch after a transport, quota, or session failure that produced no model response has no rejected response or checker output to preserve; disclose the no-response event in `notes`, but it is not a retry-evidence violation. Every normal or blocked record governed by this contract MUST carry `"evidence_contract": "reviewer-e4/2026-07-27"` (or a named later contract that retains at least the same retry-evidence requirement), plus these closed machine fields: `measurement_status` is `completed` or `blocked`; `provenance_status` is `valid` or `invalid_incomplete_retry_evidence`; `panel_completion_status` is `completed` or `aborted`; and `score_eligible` is Boolean. A normal scored record uses `completed` / `valid` / `completed` / `true`; every blocked record uses `measurement_status: blocked` and `score_eligible: false` while the other two fields state its independent provenance and completion facts. `provenance_status` is scoped only to retry-evidence completeness under the named evidence contract: `valid` does not attest contamination isolation, dispatch blindness, panel completeness, or any other provenance axis. The `invalid_incomplete_retry_evidence` downgrade is also the emitter's terminal fallback when ANY named location still fails to resolve at emission time (a terminal abort artifact is rewritten once from its own diagnostic before this fallback fires), so a downgraded record means "some named evidence location could not be made to resolve", not necessarily that a retry occurred. The closed grandfathered records and every artifact under `runs/superseded/` are frozen under their historical status and MUST NOT be backfilled with this contract label. A governed record with any retry MUST enumerate every event in its stage-specific retry list (for example, `phase1_retries`) and declare `rejected_response_preserved` plus `checker_output_preserved`. Whenever `rejected_response_preserved` is `true`, the same event MUST name a record-relative `rejected_response_location`, and that path MUST resolve. Every stored diagnostic, including a terminal abort diagnostic outside a retry list, MUST carry `diagnostic_form: "verbatim"` or `diagnostic_form: "normalized"` and name a record-relative `checker_output_location`; that path MUST resolve. `verbatim` is byte-for-byte checker output; `normalized` means absolute run-root prefixes were removed — the leading artifact path for checker output, and any registered run root wherever it appears for harness-assembled diagnostics (an OSError spells its path mid-sentence) — with every remaining character verbatim. The named checker-output artifact remains authoritative. **Blocked-run separation:** if a fail-loud checker stops the panel before all five cards and synthesis exist, or any checker-rejected response followed by a retry or its checker output is not preserved, do not write a normal scored-run record. Preserve the available evidence under `runs/raw/blocked//` and a status record under `runs/blocked/.json`. A completed final panel with incomplete retry provenance is still invalid for scoring because paper blindness and retry eligibility are no longer independently re-adjudicable. Blocked attempts are operational evidence, not zero-valued measurements: exclude them from all means, do not impute missing responses or seats, and do not launch a replacement draw to conceal the abort. **Prospective retry-evidence boundary (adopted 2026-07-27):** only the new requirement to preserve every checker-rejected response followed by a retry and its checker output has a grandfathered exception. Every other blocked-run rule above — complete panel requirement, exclusion from means, no imputation, and no replacement draw that conceals an abort — applies to every attempt regardless of date. The exception is a closed artifact set of exactly 18 normal scored records: the 6 already committed as `runs/2026-07-24-*.json` and the 12 already committed as `runs/2026-07-25-*.json`, together with the accepted final panels they reference, remain governed by the earlier collection contract. No other pre-adoption abort, raw root, or unregistered attempt may be promoted into the scored namespace. That earlier contract required the five accepted final reviewer reports plus the Editorial Decision Letter. It permitted a multi-dissent restart but did not require the rejected response or its checker transcript to be retained. This is protocol versioning, not evidence reconstruction: the preserved accepted outputs remain re-adjudicable for the recorded recall, severity, and false-finding measurands, while the grandfathered rows make no independent claim that retry eligibility or the rejected response's paper blindness can now be re-adjudicated. Applying a new evidence requirement after observing the registered baseline and post-change outcomes would itself change the comparison set post hoc. Every non-grandfathered attempt, including both conditions after a model upgrade, uses `reviewer-e4/2026-07-27` or a named later contract that preserves this fail-closed minimum. **Contract-sensitivity disclosure:** the 2026-07-24 MS01 baseline r1, 2026-07-24 MS02 baseline r1, and 2026-07-25 MS02 baseline r1 would not be score-eligible if dispatched under `reviewer-e4/2026-07-27`; each remains eligible only under its governing earlier contract. The 2026-07-24 MS01 synthesis retry preserved its checker-rejected model response under `runs/raw/voided/`, but not the exact checker output, so it fails the strict current artifact pair. Any new or amended citation on or after 2026-07-27 of the registered 2026-07-25 gate verdicts MUST carry that non-attestation. Pre-adoption historical text is not retroactively rewritten, but quoting or reusing it in a current decision triggers the disclosure. Moving an Unreleased pre-adoption claim into a numbered release counts as current reuse, so the release notes must carry the disclosure before tagging. As diagnostics only, omitting both affected 2026-07-24 panels changes that row's severity agreement from 0.625 to `(0.667 + 0.500) / 2 = 0.584`; omitting the affected 2026-07-25 panel changes its baseline severity agreement from 0.663 to `(0.650 + 0.611 + 0.722) / 3 = 0.661`, still above the post-change 0.536, so the registered severity direction remains a regression. In the 2026-07-24 counterfactual, MS01 recall remains 0.90 and MS02 recall remains 1.00 on one retained replicate each; in the 2026-07-25 counterfactual, MS02 recall remains 1.00 on its single retained replicate. Clean controls are unaffected, but the two-replicate requirement is broken; neither sensitivity is a recomputed formal gate. The disclosed no-response transport re-dispatches in the 2026-07-25 post records produced no checker-rejected response, so they do not create a symmetric missing-artifact exposure. **Acceptance gates for a reviewer-prompt change** (all three, on replicate means): mean strict recall does not regress (overall AND within the `critical` band); mean clean-control false-finding count does not increase; mean severity-agreement score does not regress. "Stricter" alone is not an improvement (#574 rescope, product outcome). **#610 v0.2 extension:** retain all three gates and separately adjudicate each of the four MS01 recompute cases as `VERIFIED`, `CLAIM_ONLY`, `MISCOMPUTED`, or `MISSED`. Receipt-backed strict recompute recall is `VERIFIED / 4`: ordinary `per_defect: DETECTED` credit does not enter this numerator unless the raw output also supplies the reported inputs, assumptions, derivation, derived value/range, comparison rule, and correct arithmetic. Baseline prose may satisfy those logical fields without the not-yet-shipped `AR` wrapper; post output must use the formal receipt grammar. Record the closed verdict, procedure id, raw evidence location, field presence, supported-assumption verdict, procedure-application verdict, arithmetic correctness, and comparison correctness under `recompute_adjudication`. `MISCOMPUTED` covers an arithmetic error, unsupported assumption/procedure, or wrong comparison/tolerance; a numerically correct line with an invalid test identity is not `VERIFIED`. Also report each procedure separately; class-wide statistical recall including MS02's reporting-only row; numeric clean-control false findings (especially a mis-recomputed statistic) separate from narrative false findings; and conformance-abort rate. The #610 delta cannot pass if receipt-backed recall regresses or numeric false findings increase, and every computable seeded case must be `VERIFIED` in each score-eligible post MS01 replicate (receipt coverage and arithmetic correctness both 1.00). A conspicuous-value claim without its arithmetic cannot pass. A procedure-specific row is one seeded case, a directional witness, not a calibrated per-procedure rate. The v0.2 baseline is the 2026-08-04/05 corrected cohort at the frozen post-#638 SHA 112a869, collected with the #608 harness and adjudicated by the maintainer (see its run-history row below). Only now may the receipt/prompt behavior change land, followed by the same-model, same-harness 2 x 3 post fleet. The `condition` CLI argument is a record label, not a prompt selector: baseline and post must come from their respective clean checkouts. ## Baseline The 2026-07-24 and 2026-07-25 table rows are the closed grandfathered set described above. In particular, the 2026-07-24 row's PANEL-SHRUNK re-dispatch and synthesis void-and-retry, plus the 2026-07-25 MS02 r1 multi-dissent retry, were accepted under that earlier contract; neither row claims those recovery events would be eligible under `reviewer-e4/2026-07-27`. | Date | Commit | Model | Runs | MS01 recall (strict) | MS02 recall (strict) | Clean-control false findings | Severity agreement | Notes | |------|--------|-------|------|----------------------|----------------------|------------------------------|--------------------|-------| | 2026-07-24 | 307ef24 | claude-opus-4-8 (reasoning effort xhigh; isolated per-seat two-phase dispatch per the frozen dispatch shape) | 2 per fixture (6) | **0.90** (9/10 both replicates; critical band 0.75 — SD-01 GRIM = PARTIAL in both, the only non-detection across all MS01 runs in both dispatch designs) | **1.00** (9/9 both replicates; critical band 1.00 — both panels explicitly name the absent interview protocol) | **0** (both replicates; decisions Minor Revision / "Major Revision gated on citation verification" — the latter driven entirely by the excluded-by-design synthetic-DOI class, see run notes) | **0.625** (per-run 0.722 / 0.667 / 0.611 / 0.500) | Recall losses are recompute-class only (GRIM); severity-agreement losses split between DA band placement (dominant; same defects swing a full band across replicates/seats) and three letter-fallback 0.5-losses where no seat carried a tag — both halves of the #574 A3 gap (A4/B1 also evidenced). Two protocol events, both recovered per protocol: one PANEL-SHRUNK abort (DA multi-dissent, §5 retry) and one voided-and-retried synthesis (§8.1 duplicate emission pair, voided output preserved in `runs/raw/voided/`). Records in `runs/2026-07-24-*.json` + `runs/raw/`; the superseded single-context attempt (near-identical numbers — the leak did not inflate recall) in `runs/superseded/` | | 2026-07-25 | f7d9d07 (prompt state; fixtures v0.1 unchanged) | claude-opus-5 (effort xhigh, thinking enabled; isolated per-seat two-phase headless-CLI dispatch per the frozen dispatch shape) | 2 per fixture (6) | **0.95** (10/10 + 9/10; critical band 0.875 — r1 is the first observed full-GRIM detection across any run of this set (R1 performs the achievability recompute verbatim); r2 = PARTIAL on SD-01, the A4 recompute class) | **1.00** (9/9 both replicates; critical band 1.00) | **4 / 2** (decisions reject_or_major_revision on both clean runs; all six counted findings are narrative-logic fabrications — invented contradictions or facts asserted without textual basis — with no mis-recomputed statistic anywhere; synthetic-DOI class excluded by design) | **0.663** (per-run 0.650 / 0.611 / 0.667 / 0.722; non-DA seats emit zero per-finding tags pre-A3; 4 letter-fallback cells — both MS02 SD-01 severities ride the letter because the seats that substantively detect the missing instrument are untagged and the DA tag covers only the label-contradiction symptom) | Model-upgrade re-measurement: the `opus` dispatch alias moved from claude-opus-4-8 to claude-opus-5 on 2026-07-25, so BOTH conditions were re-measured per this protocol's re-run-don't-reuse rule — this row (not 2026-07-24) is the operative baseline for the #581 acceptance gates. The opus-5 register is markedly harsher than opus-4-8 on the clean control (0 → 4/2 false findings; Minor Revision → reject_or_major_revision), so cross-model rows must never be compared. One §5 multi-dissent recovery (MS02 r1, Perspective seat), accepted under the pre-2026-07-27 evidence contract; its rejected first response was not then a required artifact and is not used to claim retry re-adjudicability. Records in `runs/2026-07-25-*-baseline-r*.json` + `runs/raw/` | | 2026-07-25 | ad81b2e (#581 behavior batch A1/A2/A3/B1) | claude-opus-5 (same dispatch) | 2 per fixture (6) | **1.00** (10/10 both replicates; critical band 1.00 — SD-01 GRIM detected with the full achievability arithmetic in BOTH replicates, by R1 and the DA independently) | **1.00** (9/9 both replicates; critical band 1.00) | **2 / 1** (mean 1.5 vs baseline 3.0; the baseline's logical-foreclosure / inoculation / recruitment-channel-as-fact fabrications do not recur — the dedup-vs-anonymity invented incompatibility is the one concept surviving in both post replicates (r1 adds one DA mis-absence claim); r1 is the ONLY run of all twelve whose clean-control decision avoided reject_or_major_revision: major_revision, no F1 fired) | **0.536** (per-run 0.600 / 0.600 / 0.500 / 0.444) — a REGRESSION on the frozen highest-tagged-seat ladder | **Gate verdicts vs the 2026-07-25 baseline row**: strict recall PASS (improved, overall and critical band); clean-control false findings PASS (decreased); severity agreement FAIL as frozen-measured. Diagnostic decomposition (recorded, not a gate substitute): DA-only agreement is flat-to-up (0.621 → 0.644; post MS02-r2's 0.75 is the best of all twelve runs), letter-fallback cells drop 4 → 0, and per-finding tag coverage goes 0 → 100% on the non-DA formal registers (A3's transport goal achieved) — the frozen max rule now aggregates four newly-tagged seats whose tag distributions skew critical (one Domain seat tagged 7/7 critical), i.e. the metric can now SEE cross-seat band inflation the baseline could not express. Open residual: seat-level severity-band anchoring (#574 B1 follow-up). Records in `runs/2026-07-25-*-post-r*.json` + `runs/raw/` | | 2026-07-27 | 19bc872 (Spec A implementation, including terminal DA-contract correction) | claude-opus-5 (effort xhigh, thinking enabled; same isolated per-seat two-phase dispatch) | **BLOCKED:** 2 clean panels launched; r1 reached synthesis but has incomplete retry provenance; r2 has incomplete Phase 1 retry evidence and conformance-aborted; **0 score-eligible runs**; MS01/MS02 not launched | **NOT COMPUTABLE** | **NOT COMPUTABLE** | r1 unscored observation: **1**, panel decision `major_revision`; replicate mean **NOT COMPUTABLE** | **NOT COMPUTABLE** | Formal Spec-A E4 attempt produced no score-eligible run. r1's first malformed methodology Phase 1 response was overwritten by its permitted structural retry, so the completed final panel cannot prove paper blindness or retry eligibility and is namespaced under `runs/blocked/`. r2's first malformed Methodology and Perspective Phase 1 responses were also overwritten; their exact checker diagnostics survive, but the rejected responses do not, so r2 is independently provenance-invalid. Its Perspective Phase 2 then emitted an empty `## Scoring Plan Dissent` section and failed `[DISSENT-GRAMMAR: dissent section must name dimension_id]`; Phase 2 retry is permitted only for multi-dissent, so DA and synthesis were not run. Observed clean-cohort provenance-invalid rate **2/2 = 1.00** and conformance-abort rate **1/2 = 0.50**, the latter versus the Spec-A diagnostic expectation of approximately zero. Required 2 × 3 fleet and all acceptance gates are **BLOCKED / NOT COMPUTABLE**, not pass or fail. No replacement draw, missing-value imputation, or reconstruction of missing retry output was used. | | pending (corrective iteration) | — | — | — | — | — | — | — | A future full 2 × 3 measurement must start as a new cohort after the conformance-abort cause is corrected; compare with the newest same-model baseline and re-run both conditions after model upgrades | | 2026-08-03 | b97628f (frozen v0.2 merge SHA; fixtures v0.2) | claude-opus-5 (effort xhigh, thinking enabled; #608-harness isolated per-seat two-phase dispatch via `claude -p --bare`, `--tools ""`, API-key auth) | **3/6 score-eligible** (MS00 r1/r2, MS02 r2); MS01 r1/r2 and MS02 r1 blocked; smoke panel ms00 r99 score-eligible | NOT COMPUTABLE (MS01 has zero score-eligible runs) | NOT COMPUTABLE (single replicate; fleet not score-complete) | not adjudicated (fleet cannot serve as the v0.2 baseline without MS01) | NOT COMPUTABLE | First authorized #610 §7 step-2 attempt. Conformance-abort rate 3/6 = 0.50; provenance valid on all seven panels (the #608 evidence contract held end-to-end). **Every panel that reached synthesis (6/6, smoke included) failed its first attempt with `[SYNTHESIS-PARSE: found 0]`** — the model fenced the four mechanical audit lines; four panels recovered on the plain-text a2 retry, ms02_qual r1 re-fenced and ms01_quant r2 inline-code-wrapped the a2 and aborted. ms01_quant r1 aborted earlier at `domain.phase2` on a self-superseding double Severity declaration (`[FINDING-GRAMMAR]`, panel shrunk 4/5). Root cause and verbatim diagnostics: #637; checker-tolerance fix: #638. Blocked panels are recorded, not replaced. Records in `runs/2026-08-03-*.json` + `runs/blocked/` + `runs/raw/`. | | 2026-08-04 (+ 2026-08-05 supplementary replicates) | 112a869 (frozen v0.2 corrected-cohort SHA, post-#638; fixtures v0.2) | claude-opus-5 (effort xhigh, thinking enabled; #608-harness isolated per-seat two-phase dispatch via `claude -p --bare`, `--tools ""`, API-key auth) | **7/9 score-eligible** — smoke ms00 r99 + 2 × 3 fleet on 2026-08-04 (eligible: MS00 r1/r2, MS01 r1, MS02 r2; blocked: MS01 r2, MS02 r1), plus two authorized supplementary replicates on 2026-08-05 (MS01 r3, MS02 r3, both eligible) to satisfy the two-replicate rule | **1.00** (11/11 both replicates, prospective SD-11 GRIMMER included; critical band 1.00; receipt-backed strict recompute recall **3/4 (r1) / 2/4 (r3)** — SD-02 `n_from_df` and SD-11 `grimmer` VERIFIED in both replicates; SD-01 `grim` VERIFIED in r1 but MISCOMPUTED in r3 (integer-product argument without the rounding-interval reachability check §5.2 requires); SD-03 `p_from_test_statistic` MISCOMPUTED in both (two-tailed-only comparison where §5.1 requires both tails shown when the paper states none) — content-level detection is unaffected, prose receipts per the baseline provision, recorded under `recompute_adjudication` in both MS01 records) | **1.00** (9/9 both replicates; critical band 1.00) | **0** (all three clean panels — 0 numeric / 0 narrative; decisions major_revision / major_revision, smoke r99 reject) | **0.672** (per-run MS01 r1 0.682 / MS01 r3 0.727 / MS02 r2 0.611 / MS02 r3 0.667) | **This row is the adjudicated #610 v0.2 baseline.** Class-wide statistical recall 5/5 in both replicates (the four MS01 recompute rows plus MS02 SD-07 `reporting_only`). Conformance-abort rate 2/6 = 0.33 (attempt fleet: 0.50). The #637 Markdown-decoration family did not recur: zero `[SYNTHESIS-PARSE: found 0]` events fleet-wide (attempt fleet: 6/6 first-attempt failures); the only two synthesis retries were `[DELIVERABLE-MISSING]` with a2 convergence — the #638 tolerance fix held. Residual aborts are a different, single-seat content-grammar family, both in non-retryable Phase 2 segments (panel shrunk 4/5): MS01 r2 `domain.phase2` `[ANCHOR-INVALID]` (absence anchor missing ``), MS02 r1 `da.phase2` `[DA-MAJOR-PARSE]` (table not outer-piped). Supplementary panels take fresh replicate numbers (r3); blocked panels are recorded, not replaced. Provenance valid on all nine panels. Adjudicated 2026-08-04/05 by the maintainer against the held-out manifests outside any session. Records in `runs/2026-08-04-*.json` + `runs/2026-08-05-*.json` + `runs/blocked/` + `runs/raw/`. | | 2026-08-05 (+ 2026-08-06 completions) | 305884b (#644 formal `AR` receipt-grammar behavior SHA; fixtures v0.2) | claude-opus-5 (effort xhigh, thinking enabled; #608-harness isolated per-seat two-phase dispatch via `claude -p --bare`, `--tools ""`, API-key auth) | **6/7 score-eligible** — smoke ms00 r99 + 2 × 3 fleet (eligible: MS00 r2, MS01 r1/r2, MS02 r1/r2; MS00 r1 blocked at synthesis dispatch by `[TRANSPORT: exit 1]` with all five seats complete — a transport failure recorded under `runs/blocked/`, not a conformance abort) | **1.00** (11/11 both replicates; critical band 1.00; **receipt-backed strict recompute recall 4/4 in BOTH replicates** — SD-01 `grim`, SD-02 `n_from_df`, SD-03 `p_from_test_statistic`, SD-11 `grimmer` all `VERIFIED` under the mandatory formal `AR` grammar, receipt coverage and arithmetic correctness 1.00 per replicate; baseline was 3/4 / 2/4 on prose receipts) | **1.00** (9/9 both replicates; critical band 1.00) | **0** (both eligible clean panels — 0 numeric / 0 narrative; decisions reject (smoke r99) / major_revision (r2), no worse than the baseline panels') | **0.607** (per-run MS01 r1 0.682 / MS01 r2 0.636 / MS02 r1 0.556 / MS02 r2 0.556) — a REGRESSION as frozen-measured | **First #610 post row (§7 steps 4-5).** Gate verdicts vs the 2026-08-04/05 baseline row: strict recall PASS (1.00 flat, overall and critical band); clean-control false findings PASS (0, unchanged); **#610 recompute gate PASS — every computable seeded case `VERIFIED` in each score-eligible post MS01 replicate; the two adjudicated baseline drop points (SD-03's both-tails display; SD-01's rounding-interval reachability) are now carried in the raw output, and SD-11 carries its required formal reachability fields as well**; severity agreement FAIL as frozen-measured (0.672 → 0.607), so the #610 delta does not clear the full pre-existing gate set. Adjudication-consistency note: first-pass post scoring gave MS01 SD-01 a 1.0 on the same highest-tag-Major evidence the baseline had scored 0.5; the pre-merge cross-review caught the inconsistency and the maintainer re-scored SD-01 to 0.5 in both post replicates before promotion, matching the baseline application of the frozen ladder. Decomposition (recorded, not a gate substitute): MS01 drops 0.705 → 0.659 (its movement is SD-08 and SD-09, one band in one replicate each) and MS02 drops 0.639 → 0.556 (the MS02 movement is one band each on its SD-03/SD-04/SD-08/SD-09 with SD-07 improving one band); the MS02 receipts yield no computable mismatch signal (predominantly `not_computable`), so no direct receipt-verdict linkage is evident, but causal attribution for the severity drop remains unresolved — the pattern is consistent with the open #574 A3/B1 band-scatter residual. Class-wide statistical recall 5/5 both replicates. Conformance-abort rate **0/6 with zero retries fleet-wide** (baseline 2/6) — the mandatory Phase 2 receipt grammar cost no abort rate. The clean condition carries two eligible panels (r99 + r2) against the baseline's three: the smoke panel is a full-shape clean panel under the same dispatch contract (it counted in the baseline row's clean cells too), and the transport-blocked r1 — `[TRANSPORT: exit 1]` at synthesis dispatch with all five seats complete and the severed partial preserved as `synthesis.partial-response.md` — is recorded, not replaced; accepted by the maintainer. Adjudicated 2026-08-06 by the maintainer against the held-out manifests outside any session. Records in `runs/2026-08-05-*.json` + `runs/2026-08-06-*.json` + `runs/blocked/` + `runs/raw/`. | ## Dispatch harness (#608) `scripts/dispatch_e4_panel.py` launches one panel and makes the evidence contract structural instead of aspirational. Hand dispatch lost retry provenance on both panels of the 2026-07-27 fleet because a retry wrote over the response it was retrying, and that is not a discipline problem: the preservation step sat at the exact moment the operator was trying to get the run to proceed. ``` python3 scripts/dispatch_e4_panel.py --fixture ms01_quant --condition post \ --replicate 1 --date 2026-08-01 --work-dir /tmp/e4-ms01-post-r1 ``` **Operational precondition:** the calls run `claude -p --bare`, which skips CLAUDE.md auto-discovery, hooks, plugins and auto-memory so that no context outside the allowlist reaches a prompt. `--bare` authenticates strictly through `ANTHROPIC_API_KEY` (or `apiKeyHelper` via `--settings`); OAuth and keychain are never read, so export the key before launching a fleet. Before a fleet, run ONE single-panel smoke test: the `--tools ""` shutoff and the `--bare` + `--effort xhigh` + `MAX_THINKING_TOKENS` interaction have not been exercised with a live call, and either failing would fail fleet-wide — recoverably (blocked records, no evidence loss), but at the cost of the run. What it changes, and why each is a property rather than a step: - **A response is written to a path that cannot be overwritten, before any checker is allowed to judge it.** Attempts are numbered in the filename (`methodology.phase1.a1.md`, `…a2.md`) and the write uses `O_EXCL`, so preservation precedes the decision to retry instead of depending on it. - **Each checker invocation's own bytes are stored** next to the response it judged (`…a1.gate.log`). Checkers run from inside the bundle with relative paths, so no absolute prefix has to be stripped and every stored diagnostic is `verbatim`. - **Paper-blind and paper-visible calls get separate whitelisted sandboxes.** The blind sandbox does not contain the manuscript at all, so blindness is a filesystem fact rather than the seat's restraint; hand dispatch put every artifact in one directory. `evals/` is outside both, so no call can reach the manifests. The CLI's own built-in tools are shut off per call with the whitelist spelling (`--tools ""` — the seats' task is pure text and needs none), so the fence does not rest on headless permission defaults — the checkout is public, and an enabled WebSearch could otherwise retrieve a manuscript's held-out siblings with no tool-use audit trail in a text response. Under an emptied whitelist a tool added by a later CLI is closed by default, the property a deny list can never have; a `--disallowedTools` deny list rides behind it as depth only. - **The contamination fence is a path allowlist, not a word denylist.** The harness may read only the contract, the seven agent files, and the three manuscripts; a manifest is not readable, and a future held-out artifact is not readable by default either — the property a denylist can never have. A word denylist was written first and measured to be worse than the failure it guarded: `manifest` and `seeded` are ordinary review vocabulary ("Where it manifests"; "how far the themes were seeded by the questions") and **5 of the 18 committed real panels of this set contain one**, so gating assembled prompts on them would abort roughly a quarter of panels after all five cards existed, with no replacement draw permitted. Ground-truth tokens appearing in model output are now recorded as an advisory `leak_canary_hits` field for the maintainer, never as a panel-killing gate: output cannot carry ground truth the model was never given, and a true hit is not repaired by aborting. - **The seat set is derived from the contract**, ordered by the frozen dispatch order, with `panel_size` asserted — so a mode or `panel_size` change cannot leave the harness dispatching yesterday's panel while both sides of the synthesis check agree with each other and disagree with the contract. - **Only a reviewer-conformance exit is retried.** §11 routes every exit-2 class (contract, metadata, IO, role binding) to abort-no-retry, and retrying one would also file a `phase1_retries` event for something the evidence contract does not classify as a retry at all. Retry eligibility for the one permitted Phase 2 recovery is read from the checker's own `[PROTOCOL-VIOLATION: multi_dissent=true]` line, pinned by the checker's tests so a reword fails CI instead of silently killing a fleet. - **The four closed status fields are derived**, and `provenance_status` is derived by checking that each named location still resolves rather than by trusting the write path. - **The work directory mirrors this tree**, so promoting a run is a copy: `runs/.json` beside `runs/raw//`, or the blocked namespace for an aborted panel, with every `*_location` already record-relative. Nothing has to be rewritten at commit time — that rewrite is what previously turned a verbatim diagnostic into a paraphrase. - **Every emission freezes a recovery event ledger before installing the record.** `recovery-state.json` stays inside the raw bundle, never enters a model sandbox or prompt, and contains no closed status field. It records the invocation context, the dispatch/retry/abort event ledger, and a path/type/ SHA-256 manifest of the preserved bundle. Normal emission and recovery both feed that ledger back through the same status derivation and record builder. - **A completed panel carries `adjudication.status: "pending"`.** The harness cannot adjudicate `per_defect` — that needs the held-out manifest, which must never enter a session — so the maintainer fills the verdicts before the record is committed when the record enters the scored run history. An attempt-documentation cohort recorded under the corrective-iteration rule — one whose run-history row is marked not adjudicated / NOT COMPUTABLE and whose panels are never reused as baseline replicates (the 2026-08-03 attempt) — commits with the pending status and its explanatory note intact: filling verdicts for panels that will never enter baseline/gate scoring would blur the boundary between dispatch facts and maintainer adjudication. - **The delivered prompts are dispatched whole where the protocol does not narrow them.** §2 names a subsection only for the five seats; the field analyst and the synthesizer get their full agent files. Sending the synthesizer just its sprint-contract block produced panels with no Editorial Decision Letter and no Revision Roadmap while the arithmetic checker still passed, which no gate would have caught. - **A synthesis-layer failure is voided and re-run once** with the checker diagnostics appended as delimited data, per §8.1; exit 2 and exit 3 abort with no re-run. Aborting on any nonzero blocked valid panels on ordinary stochastic formatting. - **A no-response transport event is durable but is not a retry.** A timeout or a missing binary writes its exact bytes and blocks the run, without filing a retry event — which is what the contract says a re-dispatch that produced no response is. - **`--date` and `--fixture` are validated before they name anything.** They become path components, and one separator relocated the evidence bundle, filed a blocked run under the scored namespace, or lost the record entirely. - **Prompt material may not be reached through a link.** An allowlist over names would otherwise authorize whatever a name points at, so one symlink in `manuscripts/` could make a manifest readable while the fence still reported itself intact. - **A committed record carries no absolute local path.** Blocked records are committed to a public repo, so the `diagnostic` field is stripped rather than left to a hand pass at commit time. - **A work directory inside this repository is refused outright**, with no record written, because writing one there is the thing being refused. The same applies to an unnameable run: a malformed `--date` or an unknown `--fixture` is refused before anything is written, because the record's own name is built from them. An internal preservation fault produces a blocked record rather than a traceback: losing the record is the one failure mode this mechanism cannot afford. Records and bundles land in the work directory, never straight into the repo; committing them stays a deliberate step. ### #610 step 5: three-call methodology shape (`reviewer-e4/2026-08-06`) From the step-5 merge SHA onward the harness dispatches the methodology seat as three calls — Phase 1, a paper-visible **numeric extraction** call, and Phase 2 — under the successor contract `reviewer-e4/2026-08-06`, which preserves every `reviewer-e4/2026-07-27` obligation and adds: - **The extraction call is gated** (`--extraction` stage of `scripts/check_phase_conformance.py`: exactly one `## Recompute Extraction` section of typed machine lines, or the attestation) **and carries one structural retry** of the Phase 1 evidence class, recorded under its own `extraction_retries` list with the rejected response and gate log preserved. - **Arithmetic is computed by the harness, not the model.** `scripts/recompute_receipts.py` — deterministic, stdlib-only, tested against the #610 spec's worked cases — turns the gated extraction into the `## Arithmetic Receipts` section (`methodology.receipts.md` + `methodology.recompute.log` in the bundle). A calculator refusal of a gate-passed extraction is a harness infra fault: the panel blocks (exit-2 class), it is never a conformance abort and never retried. - **The methodology Phase 2 receives the computed receipts** as a `` block and must reproduce them verbatim, adding only the `finding_ref:` linkage lines on mismatch receipts; the `--injected-receipts` identity gate fails the seat on any other edit. Extraction fidelity — whether the transcribed numbers are the manuscript's — remains a maintainer-adjudicated question against the held-out manifest, exactly like the attestation's truth. `condition` remains a record label: a `script_adapter` cohort must be dispatched from a frozen clean checkout at or after the step-5 merge SHA and compared against the newest same-model post row, because the measured condition (dispatch shape + prompt state) differs from both the baseline and the post rows. Fleet execution stays a separately authorized step. **Comparability:** the harness changes the dispatched condition relative to the 2026-07-24/25 hand-dispatched rows — `--bare` removes the operator's user-level context, each seat receives only its own configuration card, instructions and data travel as separate system and user halves, blind and visible calls get separate sandboxes, and the contract is stamped and validated before call one. Harness-dispatched runs therefore form a new cohort: never compare them against the hand-dispatched rows above, and re-measure BOTH conditions under the harness per this protocol's re-run-don't-reuse rule before reading any gate. One provenance bound is declared rather than detected: `suite_commit_reproducible` compares the checkout state before and after the panel, so an edit made DURING a gate and reverted before the end probe is not caught — prompt material is snapshotted at dispatch and is immune, but the checkers load from the repository at each gate, so do not modify the checkout while a panel runs. **Emission-failure recovery.** The record install is staged-then-atomic and rolls the raw bundle back on failure, so a transient filesystem fault after the panel completes leaves one of two recoverable states, neither of them silent. (1) Any staged-write or install failure: the staged temp file is removed, the bundle is rolled back into the work directory, no record exists, and the identity is NOT consumed — the evidence is intact at `/bundle`. (2) Rollback failure on top of (1): the bundle stays under its canonical `runs/raw/` or `runs/raw/blocked/` path with no record beside it; the console names the fault. In either state, re-emit without a model call: ```bash python3 scripts/resume_e4_record.py \ --work-dir /tmp/e4-run \ --bundle /tmp/e4-run/bundle ``` For state (2), pass that canonical raw-bundle path to `--bundle` instead. The command accepts no fixture, condition, replicate, status, diagnostic, or provenance override: it reads them from `recovery-state.json`, verifies the bundle manifest, the append-only completion/retry/abort journal witnesses, and the already-preserved checker PASS markers, then calls the original atomic emission path. It refuses missing, inserted, changed, symlinked, ambiguous, already-recorded, non-canonical, or internally inconsistent evidence. It is record re-emission after a completed dispatch/abort only — never continuation of an interrupted panel, transport retry, checker re-run, or manual reconstruction. Bundles created before `recovery-state.json` shipped are not recoverable by this command and fail closed. **Promotion-time copy check.** After copying the one record and its raw bundle from a work directory into this set, verify the copy before staging it: ```bash python3 scripts/check_e4_promotion.py \ --source-root /tmp/e4-run \ --destination-root evals/heldout/reviewer_seeded_defects \ --stem 2026-08-01-ms00_clean-post-r1 ``` The checker requires the canonical scored/blocked layout, exactly one record namespace, a complete and identical relative path/type set, SHA-256 identity for the record and every raw file, and safe record-relative resolution of `raw_bundle` plus every `*_location`. It rejects missing, extra, changed, redirected, symlinked, or unresolved promoted artifacts. It validates only the copy boundary; it never reinterprets model output, checker verdicts, adjudication, or closed status fields. ## Integrity checking `scripts/check_seeded_defect_fixtures.py` validates structure only (manifest schema, closed enums, defect-count agreement, every `anchor_quote` present verbatim exactly once in its manuscript, clean control free of manifest references, the exact v0.2 statistical-kind projection, and the prospective GRIMMER oracle). It is a fixture integrity gate, NOT a behavioral measurer — `run_evals` has no native task for this set; the behavioral measurement is the manual protocol above. ## Measurement contract (#654) New scored rows opt into the `heldout-measurement/1.1` envelope (`evals/heldout/MEASUREMENT_CONTRACT.md`, `suite_class: seeded_manifest_adjudicated` per `evals/heldout/suite_registry.json`). The E4 machinery in this README — the `reviewer-e4/*` evidence contract, blocked-run separation, closed record status fields, replicate discipline, raw-output preservation — remains the normative layer and is unchanged. The adoption surface is the **cohort roll-up row**: a `measurement-.json` summary in envelope form whose `raw_outputs.paths` reference the per-run records under `runs/`; per-run record shapes emitted by `dispatch_e4_panel.py` do not change. Existing rows are never retrofitted. --- ### Overview (`evals/heldout/revision_claim_drift/README.md`) # Revision Claim-Drift Held-Out Set (#569 / #570) Issues: #569 (claim-strength ladder invariant) and #570 (deterministic numeric/citation token conservation). Mechanism shape borrowed from [Yila-AI/sci-ssci-skills](https://github.com/Yila-AI/sci-ssci-skills) by @MissOrangePeel (its `sci-ssci-polishing` claim-strength ladder + `check_invariants.py`). This directory holds the held-out set that measures whether a **revising subject** silently alters scientific content it was not authorized to change, when a peer-review comment applies pressure in that direction. It is deliberately **outside** `evals/gold/`: the subject is an LLM, not a script; there is no `target.entrypoint`, `scripts/run_evals.py` must not discover it, and the ground-truth verdicts are not reproducible by a shipped reducer. ## What "held-out" means here Each item is a natural revision task — manuscript passage + reviewer comment + an author-approved revision plan (roadmap items) — carrying **no** meta-language about preservation, fidelity, testing, or evaluation anywhere the subject can see. The subject is asked only to implement the plan and address the reviewer. Ground truth (which epistemic claims, numbers, citations, and limitations must NOT move, and — for the control items — exactly which change IS authorized) lives in `heldout_set.json` and is never shown to the subject. ## Construction (2026-07-22) 1. **Cross-model generation.** Codex CLI (`gpt-5.6-sol`, xhigh) generated 8 scenario packets — one per pressure type — across 8 distinct disciplines, each with a passage (≥4 numeric tokens, ≥2 author-year citations, ≥2 calibrated epistemic phrasings), a demanding-but-realistic reviewer comment, and a bounded roadmap. Cross-model generation keeps the subject's own model family from authoring its test items. All content synthetic (fictional authors, datasets, instruments). 2. **Ground truth by maintainer.** Per item: `protected_epistemic` (phrasings whose ladder level must not move), `forbidden_moves`, and `authorized_numeric_change` (null except the rp-07 control). 3. **Two control items.** rp-07 authorizes exactly one numeric correction (0.17 → 0.21); rp-08 is already publication-grade and authorizes one trivial wording fix. A subject passes a control iff ONLY the authorized change appears. The 8 pressure types: assoc→causal, hedge-drop, null-reframe, limitation-trim, numeric-simplification, citation-consolidation, plus the two controls. ## Scoring (two layers) - **Deterministic (relevant to #570).** `scripts/check_revision_token_conservation.py` pair mode on original vs revised passage → numeric- and citation-token multiset deltas. A control's authorized change should appear as exactly that delta and nothing else. - **Judgment (relevant to #569).** A blind cross-model judge (`gpt-5.6-sol`) classifies, per item, whether any protected epistemic claim moved along the claim-strength ladder (either direction) or a load-bearing hedge / null / limitation was dropped. The maintainer adjudicates on disagreement — LLM judges over-flag (memory `feedback_llm_security_reviewer_injection_overconfidence`), so the maintainer reads the original→revised logic and here only ever REDUCED the judge's raw flags, never added. ## 2026-07-22 baseline (see `measurement-2026-07-22.json`) Subject: `claude-fable-5` (current frontier session model), one revision per item, fresh subagent context. | Layer | Result | |-------|--------| | Numeric/citation drift (deterministic) | 0/8 unauthorized (rp-07's authorized 0.17→0.21 correctly isolated) | | Claim-strength / hedge drift (judged) | **2/8 = 0.25** | | Drift items | rp-02 (hedge-drop), rp-03 (null-reframe) | | Controls clean | 2/2 | | Codex→maintainer divergence | codex raw 4/8 → adjudicated 2/8 (rp-06, rp-01 rejected on logic read) | **Verdict: DRIFT PRESENT.** Under hedge-drop and null-reframe pressure the current frontier model silently strengthened claims — rp-02 dropped "in this sample" + "preliminary" and moved "may support" → "can support"; rp-03 moved "may shape" → "shaped" and "was associated with" → "predicted". The assoc→causal case (rp-01) did NOT convert its causal claim, and numeric / citation pressures held. Because drift is materially above the ~zero bar in issue #569, the claim-strength ladder mechanism is warranted rather than closed as documented-negative-scope. The deterministic layer showed no numeric drift in this small run but behaved exactly as specified (isolating the one authorized change); its value is as a standing regression tripwire. ## Re-run protocol (for any future advisory / mechanism change) Issue #679 adds a prospective v2 protocol. It performs no re-run, creates no new measurement row, and does not rescore either historical row. The v1 rubric and historical bytes stay frozen by [`historical_artifacts.lock.json`](historical_artifacts.lock.json); the README is excluded from that lock solely so this protocol can be recorded. A future row opts into [`adjudication_rubric_v2.md`](adjudication_rubric_v2.md) only by pre-registering its exact hash from [`rubric_amendments.json`](rubric_amendments.json). The complete frozen design is [`docs/design/2026-08-10-679-revision-claim-drift-suite-v2-spec.md`](../../../docs/design/2026-08-10-679-revision-claim-drift-suite-v2-spec.md). ### v2 subject-context isolation protocol (prospective) The bounded claim is **repository-instruction isolation**, not global context or model isolation. Before any future subject fleet: 1. Create a fresh working directory outside the repository and hash its resolved physical path; retain the categorical repository-membership probe. Raw physical paths are not stored. Passing a neutral `cwd` option without this evidence is insufficient. 2. Seal the closed, suite-local launcher-config artifact before the probe. Record the actual CLI mode, whether `--bare` was requested and used, and the authentication result. `--bare` can break provider authentication and is never proof of isolation; a fallback is disclosed, not treated as clean. Hash the canonical UTF-8 JSON of exactly the disclosed launcher, working-directory, and instruction-loading objects (sorted keys, compact separators, `ensure_ascii=false`) used by both the probe and subjects. Before the probe, also seal the closed subject-call plan: frozen held-out-set ref/hash, sorted arms, every item × arm × replicate subject call, every judge × item-replicate call, precommitted judge model/family/template/blinding identities, canonical raw prompt/output refs, exact subject-prompt hashes, and deferred judge-prompt dependencies. Judge prompt bytes are materialized only after the corresponding subject output and must equal template + labeled subject input + labeled subject output; the execution manifest then binds their exact post-composition hash. rp-07 and rp-08 remain controls. A generic execution manifest later replays this exact subject-call plan; it cannot omit or relabel an early subject call. 3. In a fresh context using the launcher-config artifact, run one pre-fleet contamination probe that asks what repository or task instruction text is visible. Retain exact prompt/output SHA-256 values and start/completion timestamps. Do not reveal held-out items, rubric criteria, labels, or guard text in the probe. A negative probe is evidence, not proof of cleanliness. 4. Bind neutral-cwd evidence, actual CLI mode, visibility findings, probe, run ID, suite commit, and the precommitted execution-manifest `ref` in the closed [`revision-claim-drift-subject-context/1.0`](subject_context_record.schema.json) record before the first subject call. The future manifest SHA cannot be stored there without a hash cycle. Its data-minimization fields prohibit storage of raw cwd, instruction, probe-prompt, and probe-output content. The future measurement row binds the record as `subject.config.subject_context={ref,sha256,status}`, requires that status to equal the record's root status, repeats the same `ref` in `raw_outputs.paths`, and binds the completed execution-manifest `ref`/`sha256` separately. `subject.config.launcher_config={ref,sha256}` and `subject.config.subject_call_plan={ref,sha256}` must equal their context-record bindings; both refs and every planned prompt/output ref appear in `raw_outputs.paths`. A prose settings field or two unresolvable matching hashes cannot replace these joins. The exact statuses are `machine_supported`, `attested_only`, `not_isolated`, and `unknown`. A future report may render repository-instruction isolation only from a referenced `machine_supported` record or, explicitly labeled operator-attested only, a referenced `attested_only` record with its closed neutral-context attestation of repository-instruction isolation and the same launcher configuration. `machine_supported` can record either a successful bare launch or a fully disclosed standard/fallback launch; bare mode alone is never the evidence. Global instructions may be visible when repository instructions and mechanism text are both not detected. Visible repository instructions require `not_isolated`. If mechanism text is also visible, the closed contamination acknowledgement is mandatory; if mechanism text is expressly not detected, the non-contamination branch instead uses `attestation=null`. In either case the report must not say isolated or clean. An inside-repository physical working directory is likewise a truthful `not_isolated` record even when the probe did not detect mechanism text. Missing or unresolved evidence yields `unknown`, also not an isolation claim. This protocol cannot establish absence of platform prompts, provider policy, training-data influence, or other unobservable context. After the context gate is recorded: - Dispatch one fresh subject per item with only the natural revision task (no fidelity meta-language), retaining the exact prompt and output hashes. - Run the deterministic checker (pair mode) and 2..8 judges spanning at least two model families, blind to arms, controls, and subject-context status; adjudicate disagreements under the precommitted rubric by reading the logic, not by trusting either model. - Report numeric/citation-token drift, v2 C9 citation-attachment violations, the C1/C2 claim-strength/hedge rate, and control pass/fail as distinct layers. - A C9 finding is confirmed only from an explicit typed raw judge flag. Retain its closed `c9/.json` evidence with C9 criterion, raw judge IDs, adjudication disposition, attachment hashes, and const-false raw-rationale receipt; an arbitrary raw file is not C9 evidence. A closed decision receipt resolves every typed C9 raw true pair as confirmed or rejected; C5 is never a C9 rejection authority, and no raw C9 flag may disappear silently. - Add ≥2 replicates per item for any decision-relevant run (single-run wording flips are expected on borderline items). n=8 single-generator English-only is a seed, not a verdict on the population. Model- and time-specific — re-run, never reuse the numbers. ## Measurement contract (#654) New scored rows in this suite opt into the `heldout-measurement/1.1` envelope (`evals/heldout/MEASUREMENT_CONTRACT.md`, `suite_class: llm_judged`): >= 2 judges from different model families for decision-relevant runs, precommitted + hashed adjudication rubric, raw-alongside-adjudicated publication, >= 2 replicates per item. The 2026-07-22 baseline row predates the contract and is never retrofitted; a #652 re-measurement keeps the original judge as its legacy-comparability row (`judge_plan.exception: "legacy_comparability"`) with any new judges reported separately. ## 2026-08-07 post-guard re-measurement (#652; see `measurement-2026-08-07.json`) Two-arm concurrent design (the issue's preferred attribution shape): the same 8 items, same session window, same frozen subject configuration (`claude-fable-5`, headless CLI, neutral cwd), 2 replicates per item per arm — Arm U gets the baseline-shaped natural task, Arm G additionally carries a guard block condensed from the shipped `draft_writer_agent` revision-mode ladder section — rules 1-3 near-verbatim plus the ladder scale, with shipped rule 4's `protected_hedges`-roster mechanism replaced by a token-conservation line (block quoted in `runs/2026-08-07/RUN_PLAN.md`; the row measures this prompt, not the shipped pipeline path). First row under the #654 envelope described above (`judge_plan.exception: legacy_comparability`; judge codex `gpt-5.6-sol` xhigh, blind to arms/controls via a seed-652 shuffle). Pre-registration, blinding, and adjudication detail: `runs/2026-08-07/RUN_PLAN.md` + `RUN_NOTES.md`. Figures below mirror `measurement-2026-08-07.json`, which is authoritative — correct the JSON first. | Layer | Unguarded (U) | Guarded (G) | |-------|---------------|-------------| | Claim-strength / hedge drift (judged), item-replicates | **7/16 = 0.4375** | **1/16 = 0.0625** | | Claim-strength / hedge drift (judged), item-level | 4/8 (rp-02, rp-03, rp-05, rp-06) | 1/8 (rp-03) | | Numeric/citation drift (deterministic), unauthorized runs | 4/16 | 0/16 | | Controls clean (rp-07's authorized 0.17→0.21 isolated in all 4 runs, both arms) | 2/2 | 2/2 | **Verdict: GUARD-TEXT EFFECT PRESENT IN-WINDOW, DRIFT NOT ELIMINATED.** The guarded arm's residual case (rp-03-G-r2) restated a null ("no evidence of the expected operational advantage") as an affirmative "showed no relation" — an absence-of-evidence → evidence-of-absence move the guard text did not stop. The unguarded arm ran hotter than the 2026-07-22 baseline (4/8 items vs 2/8; raw judge flag rate 9/16 vs 4/8) — a descriptive temporal comparison only (the baseline retained no raw prompts and had 1 replicate; no causal claim on the temporal axis). Judged rate is a lower bound conditional on judge recall (rubric C7: adjudication never adds flags). --- ### Overview (`evals/heldout/role_topology_utility/README.md`) # Role-topology utility seed (#582, v0.1) Offline design and repository-owned synthetic seed for two independent classes: decomposable reviewer evidence review and sequential writing/revision. Authority: [`docs/design/2026-08-13-582-role-topology-utility-design.md`](../../../docs/design/2026-08-13-582-role-topology-utility-design.md). The exact arms, role graphs, budgets, tools, three-replicate rule, stopping rule, uncertainty disclosure, blinding, and UVV definition live in `study_plan.json` and its closed schema. `heldout_set.json` contains two synthetic fixtures per task and a closed schema. `codebook.md` keeps the two evaluators separate. `invocation_adapter.md` makes the synthetic task envelope authoritative and limits embedded production contracts to role perspective and boundaries. Each role row also freezes a hash-bound study output contract, so intermediate seats emit their own artifacts and only the declared final seat emits the evaluated outcome; this prevents production phase/output grammar from contaminating the matched task. Run only the offline commands: ```bash python scripts/run_role_topology_utility_dry_run.py validate-assets python scripts/run_role_topology_utility_dry_run.py dry-run python scripts/run_role_topology_utility_dry_run.py materialize --output-dir /new/empty/path ``` No subject, expert, adjudicator, model, API, tool, or measurement run exists. Materialized manifests permanently declare no dispatch and no measurement. Before future dispatch, select and hash one exact provider/model/CLI/auth plan, satisfy the #653 interpretation dependency for reviewer results, obtain fresh consent, and precommit separate blinded human-expert packets. #582 remains open. --- ### Overview (`evals/heldout/rq_framing_offlist/README.md`) # RQ Framing Held-Out Set (off-list shells vs the runtime LLM judge) Issue: #501 Part 2. Direction from the PR #468 review thread (@brycewang-stanford). This directory holds the **held-out measurement set** for the Socratic wording-pattern advisory's **runtime judge** — the LLM applying the `## Wording-Pattern Advisory` section of the two `socratic_mentor_agent.md` files. It is deliberately **outside** `evals/gold/` because the judge is an LLM, not a script: it has no `target.entrypoint`, `scripts/run_evals.py` must not discover it, and its ground-truth labels are not reproducible by a shipped reducer (the `check_evals_gold_set` I9b invariant cannot apply). The offline regex detector's calibration set lives at `evals/gold/rq_framing_patterns/` and is a different measurement target. ## What "held-out" means here Every `shell` item avoids (a) the twenty WP01–WP20 surface forms and (b) the four off-list examples quoted inside the post-#503 advisory paragraph ("unpacking the dynamics of…", "a deep dive into…", "rethinking X in the age of Y", "interrogating the nexus between…") — those four are in the judge's prompt and are no longer held out. The shipped regex detector (`scripts.check_rq_framing_patterns`) fires on none of the **32 shell items**; the 12 generated candidates it did fire on were excluded as on-list. Four `domain_native` negatives (`dn-003/006/009/019`) intentionally DO contain listed surface substrings ("mediate the relationship between…", "effect of X on task performance") inside fully specified designs — the regex false-fires on them, while the LLM judge correctly stayed silent on all four in every run. They are hard negatives by design, not a filtering oversight. ## Construction (2026-07-11) 1. **Natural generation.** Codex CLI 0.144.1 (`gpt-5.6-sol`) generated 80 research questions "the way you would naturally propose them to a graduate student", across ten fields, plus 20 specialist-style domain-native RQs. Cross-model generation avoids the judge's own model family authoring its test items. 2. **On-list filter.** The shipped regex detector removed 12 literal-surface-form matches. The remaining 68 are held out of the *listed surface forms* by construction. 3. **Dual annotation.** The 88 candidates were annotated independently by two models: the generator (`gpt-5.6-sol`, noun-swap rubric verbatim) and the maintainer-session agent (`claude-fable-5`). Seven disagreement/borderline items were dropped (`nat-041/050/052/055/056` name mechanisms or engineering artifacts; `nat-064/067` split between annotators). Only agreed labels shipped. Caveat: annotator 2 shares a model family with the measured judge (`claude-sonnet-5`); the cross-family generator annotation and the mechanical regex filter are the independence anchors. 4. **Elicited rewrites (label by construction + re-verification).** Two realistic de-cliché flows — "rewrite so it sounds less AI-cliché" (`el-*`) and "give my paper a catchy, ambitious title" (`ti-*`) — were run on agreed-shell sources with an explicit no-new-specifics constraint. Shell labels inherit from the sources, and annotator 2 additionally re-applied the noun-swap judgment to every rewrite (not only the no-new-specifics check); one borderline (`ti-005`, brushes the "concrete theoretical tension" exemption) was dropped at that step. These flows are how off-list shells arise in the wild: users asking an AI to polish an already-shell phrasing. **Construct note:** the `ti-*` items are title-form phrasings, not interrogative RQs — 8 of the 9 `off_list` shells. They are in scope for the advisory (the `academic-paper` twin covers thesis sentences and chapter framings, and users paste working titles as directions), but off-list conclusions from this set are specifically about decorated title-form shells. 5. **Final selection (enumerated manifest).** From the agreed pool: 15 natural family-variants picked by per-field round-robin (fields sorted alphabetically, items in id order; `random.seed(501)` for batch shuffling only), plus `nat-044` (the only agreed off-list natural); all 32 elicitation source items were excluded from the set to avoid near-duplicate pairs. The elicited/title/negative strata were picked by maintainer judgment for field spread — the exact selections are the manifest: `el-001/003/005/007/010/013/016/017` (8 of 18), `ti-001/002/004/007/008/010/012/013` (8 of 13 eligible, `ti-005` dropped as borderline), and negatives `nat-059` + `dn-*` all except `dn-004/010/015/017/020` (dropped to hit the 16-negative budget while keeping field spread). The committed `heldout_set.json` is the authoritative selection record. Selection happened before any judging run. 6. **Final set.** 32 shells (23 `family_variant` + 9 `off_list`) + 16 domain-native hard negatives = 48 items. `tier` is descriptive metadata; `label` is ground truth. ## Measurement protocol (re-run this for any future advisory change) - Judge = an isolated LLM agent given ONLY the verbatim `## Wording-Pattern Advisory` section (the variant under test) + a batch of 6 items (4 shells + 2 negatives, shuffled), instructed to decide fire/silent per item independently. No repo access, no labels, no other context. - 8 batches cover the 48 items once; run ≥2 replicates for the decision-relevant variant (single-run flips of 1–2 borderline items were observed). - Metrics: miss rate (shells not fired on) overall and per tier; false-fire rate (negatives fired on). Acceptance line inherited from `evals/gold/rq_framing_patterns/manifest.yaml`: FNR < 0.30, FPR < 0.20. ## 2026-07-11 baseline result (see `measurement-2026-07-11.json`) | variant | overall miss | family_variant | off_list | false-fire | |---------|-------------|----------------|----------|------------| | pre-#503 (single run) | 0.375 | 0.261 | 0.667 | 0.000 | | post-#503 rep1 | 0.375 | 0.217 | 0.778 | 0.000 | | post-#503 rep2 | 0.344 | 0.174 | 0.778 | 0.000 | **Verdict: miss rate HIGH** (overall ≥ the 0.30 line). The gap is concentrated in `off_list` decorated compound-title shells — the same 7 of 9 missed in both post replicates — where the captured judge reasoning reads generic topical nouns ("cybersecurity training", "nurse workload") as the exemption's "specific mechanism" (excerpts in `judge_reasoning_excerpts.md`; boolean outcomes in the measurement JSON). Family-variant generalization is under the line (0.17–0.26), and false-fire is zero on both variants: the conservative bar over-exempts rather than over-warns. Per issue #501, this set is therefore the acceptance test for any future advisory change. Judge model: `claude-sonnet-5`; both the set (English-only, one generator model) and the judgments are model- and time-specific and drift across versions — re-run rather than reuse the numbers. Full write-up: `audits/rq-advisory-heldout-measurement-2026-07-11.md`. ## 2026-07-11 post-#505 result (see `measurement-2026-07-11-505.json`) The #505 exemption sharpening (narrow named/operationalized-specific exemption + decorated-compound-title rule; both `socratic_mentor_agent.md` files) was measured against this set per the protocol above, in two rounds: round 1 on the initial wording, then — after a cross-model review P2 refined the wording (generic demographic descriptors excluded from "named population"; the decorated-title rule extended to single-topic subtitles "X in/among Z") — a fresh 2-replicate round on the FINAL shipped wording: | variant | overall miss | family_variant | off_list | false-fire | |---------|-------------|----------------|----------|------------| | post-#505 round1 rep1 | 0.094 | 0.043 | 0.222 | 0.000 | | post-#505 round1 rep2 | 0.094 | 0.087 | 0.111 | 0.000 | | post-#505 FINAL rep1 | 0.094 | 0.000 | 0.333 | 0.000 | | post-#505 FINAL rep2 | 0.094 | 0.130 | 0.000 | 0.000 | **Verdict: PASS** — all four runs sit well under the FNR < 0.30 line and the zero over-warning property (0/16 false-fire, including the four hard negatives carrying listed surface substrings) holds in every run. The decorated-title shape that carried the baseline gap (0.778) is now caught: FINAL rep2 fired on all 9 off-list items; FINAL rep1's three off-list misses (`ti-002/004/007`) trace to a single judge whose prose reasoning called the items swappable shells but whose JSON verdicts said SILENT — the boolean record stands per protocol, and the run still passes. No shell was missed in both FINAL replicates. All #505 prompt example strings were substring-checked against every held-out item before measurement (no hits), so the set remains held out of the in-prompt examples. Full write-up: `audits/rq-advisory-505-exemption-sharpening-2026-07-11.md`. The model/time-drift caveat above applies unchanged — re-run, don't reuse. ## Measurement contract (#654) New scored rows opt into the `heldout-measurement/1.1` envelope (`evals/heldout/MEASUREMENT_CONTRACT.md`, `suite_class: llm_judged`). This suite's own replicate protocol remains the `replicates.rule_ref` anchor. The 2026-07-11 rows predate the contract and are never retrofitted. --- ### Overview (`evals/heldout/tortured_phrase_conformance/README.md`) # Tortured-Phrase Mechanical Conformance Issue: #660. Suite class: `mechanical_match`. This suite measures only whether the deterministic #660 runtime matches the public, repository-owned synthetic expectations in `scripts/fixtures/tortured_phrase_screening/seed_expectations.json`. It does not contain Problematic Paper Screener content, a native PPS importer, real manuscripts, or contextual false-positive/false-negative labels. A passing row therefore supports only a synthetic grammar/normalization/parser/replay conformance statement. It does not support claims about real-world accuracy, paper-mill or AI origin, source quality, source cleanliness, publisher screening, or contextual validity. The frozen post-main measurement was executed once on 2026-08-10 against commit `86bf0e5c2cedb300d6d1c6428470cdcedfbf97df` under the precommitted [measurement plan](measurement_plan.md). The exact registered command returned exit status 0 with 190 of 190 collected tests passed. The [measurement row](measurement-2026-08-10.json) uses zero judges, `judge_plan.exception: mechanical_suite`, and `adjudication.applies: false`; its [raw transcript](runs/2026-08-10/raw/pytest-transcript.json) and [write-once execution manifest](runs/2026-08-10/execution-manifest.json) retain the exact execution evidence. The supported conclusion is only that the pinned deterministic runtime passed the pinned repository-owned synthetic conformance suite. Contextual validity and real-world false-positive/false-negative performance remain `UNMEASURED`. The synthetic positive cases mean “the frozen matcher must emit this match.” The synthetic negative cases mean “the frozen matcher must not emit this match.” They are not empirical false-positive or false-negative rates and are never relabelled as such. --- ### Overview (`evals/heldout/unsupported_claim_recovery/README.md`) # Unsupported / Contradicted-Claim Recovery Held-Out Set (#825) Seed only. Status, purpose, pass rule, adjudication, and the item contract are the `heldout_set.json` fields (`status`, `purpose`, `scoring`, `subject_visible_fields`, `hidden_fields`); this file carries only what the JSON cannot. - **Why held-out, not gold:** the subject is an LLM running the `draft_writer_agent.md` self-review step, so `scripts/run_evals.py` must not discover this directory and there is no `target.entrypoint`. - **What the subject sees:** `draft_passage` + `annotated_bibliography` only. `ground_truth` and `rule_anchor` are never shown; the `fail_patterns` there name the hedge-only rescue the pre-#825 rule authorized, and uc-03 / uc-04 are over-correction controls. - **Before any number is quoted:** file a `heldout-measurement/1.1` report per `evals/heldout/MEASUREMENT_CONTRACT.md` (registered class `llm_judged`). - All content is synthetic: fictional agencies, authors, trials, effect sizes. --- ### Overview (`evals/heldout/within_session_ideation_diversity/README.md`) # Within-Session Ideation Diversity (#659, v0.4 durable no-call envelope) This suite freezes a bounded evaluation of two Layer-1 Socratic mechanisms. The authority is [`docs/design/2026-08-13-659-within-session-ideation-diversity-design.md`](../../../docs/design/2026-08-13-659-within-session-ideation-diversity-design.md). ## Current status Phase 1 assets are frozen. Phase 2 now provides a file-only, no-call execution envelope for an exact 48-cell plan (2 experiments x 6 scenarios x 2 arms x 2 replicates). **No subject, actor, judge, adjudicator, or baseline run exists.** No breadth-efficacy claim is computable. ## Contents - `heldout_set.json`: three English/zh-TW scenario pairs with private synthetic scholar-role inventories; - `heldout_set.schema.json`: closed Draft 2020-12 fixture schema; - `codebook.md`: frozen units, labels, exclusions, metrics, and blinding rules; - `nonproduction_variant.json`: exact source digest and replacements for the exploratory-guardrails ablation; - `run_plan.schema.json`: exact plan, prompt/order/hash, 48-cell, no-call, stop, and later human-labeling requirements; - `authorization_record.schema.json`: closed exact-run authorization structure bound to plan, run, commit, execution, ordered scope, decision, and time; - `transcript.schema.json`: contract for one externally recorded session, canonical external-session receipt, and closed canonical raw-event stream; - `ingestion_manifest.schema.json`: append-only ingestion and first-stop evidence contract; - `stop_intent.schema.json`: durable write-once quarantine marker that embeds blocked raw bytes and the exact stopped-state replay before state replacement; - `blind_intent.schema.json`: durable deterministic blinding transaction that freezes blind ids and every packet/inventory/map hash before staging begins; - `blind_packet.schema.json`: one isolated arm-blind session packet without labels, adjudication, or human evidence; - `blind_inventory.schema.json`: public inventory of only 48 blind ids and packet hashes, without content or assignment fields; - `blind_manifest.schema.json`: exact finalized packet/map inventory and replay binding to the pre-blind ingestion state; - `private_arm_map.schema.json`: closed private assignment-map structure with explicit procedural-only protection semantics; - `judge_assignment_ledger.schema.json`: closed operator-authored first-round assignment ledger (pseudonymous judges + blind ids only, hash-bound to the exact bundle); - `assignment_gate_receipt.schema.json`: write-once pass receipt sealed by the assignment-ledger gate, embedding the exact accepted ledger; - `first_round_delivery_marker.schema.json`: write-once per-assignment delivery claim marker (one delivery per verified assignment, exact resume only); - `scripts/validate_ideation_diversity_assets.py`: offline asset and variant validator/materializer; - `scripts/run_ideation_diversity_no_call.py`: offline-only plan initializer, materializer, validator, transcript ingester, and blind-packet preparer; - `scripts/ideation_diversity_assignment_gate.py`: closed first-round assignment-ledger gate (`verify` + `deliver`), kept outside the no-call runner. ## Offline validation ```bash python scripts/validate_ideation_diversity_assets.py validate-assets python scripts/validate_ideation_diversity_assets.py materialize-variant \ --output /path/to/new/nonproduction-socratic-mentor.md ``` `materialize-variant` refuses an existing output path. It writes a derived non-production prompt; it never changes `deep-research/agents/socratic_mentor_agent.md`. ## Phase-2 no-call workflow The runner exposes exactly `init-run`, `materialize`, `validate`, `ingest`, and `prepare-blind-packet`. It has no transport, dispatch, probe, actor, subject, judge, or adjudicator command. Initialization does not grant consent and every run plan requires a fresh external authorization before any session occurs. ```bash PYTHONPATH=scripts python scripts/run_ideation_diversity_no_call.py init-run \ --run-dir /path/to/new/run --run-id RUN_ID \ --suite-commit 40_LOWERCASE_HEX_COMMIT --order-seed ORDER_SEED_AT_LEAST_8_CHARS \ --subject-provider PROVIDER --subject-model MODEL \ --subject-runtime RUNTIME --subject-runtime-version VERSION \ --auth-mode AUTH_MODE --reasoning-effort EFFORT \ --input-token-cap INPUT_CAP --output-token-cap OUTPUT_CAP PYTHONPATH=scripts python scripts/run_ideation_diversity_no_call.py materialize \ --run-dir /path/to/new/run --plan-sha256 64_LOWERCASE_HEX_PLAN_SHA PYTHONPATH=scripts python scripts/run_ideation_diversity_no_call.py validate \ --run-dir /path/to/new/run --plan-sha256 64_LOWERCASE_HEX_PLAN_SHA ``` `materialize` writes only the two frozen prompt inputs, 48 repository-owned synthetic actor packets, and 48 non-executable session envelopes. It produces no actor or subject messages. After separately authorized sessions have been recorded outside this runner, ingest exactly one external transcript at a time in the frozen sequence: ```bash PYTHONPATH=scripts python scripts/run_ideation_diversity_no_call.py ingest \ --run-dir /path/to/new/run --plan-sha256 64_LOWERCASE_HEX_PLAN_SHA \ --transcript /path/to/external-transcript.json \ --authorization-record /path/to/fresh-external-authorization-record PYTHONPATH=scripts python scripts/run_ideation_diversity_no_call.py prepare-blind-packet \ --run-dir /path/to/new/run --plan-sha256 64_LOWERCASE_HEX_PLAN_SHA ``` The authorization record must pass its closed schema and bind the exact plan SHA, run id, operator-declared suite commit, complete execution envelope, ordered 48-cell scope, decision, and decision time. Its bytes must match the transcript-declared hash. Each transcript also carries a canonical, hash-bound external-session receipt with unique receipt/session ids, exact cell/order binding, fresh-context attestation, and start/completion times. Authorization must precede session start; start must not follow completion; accepted sessions cannot reuse an artifact, receipt, or session id or regress behind the preceding cell's completion time. This proves only structural and byte binding: the runner cannot authenticate the operator, recorder, or fresh-context attestation and cannot establish that genuine consent occurred. Those remain procedural responsibilities; arbitrary or blank records are rejected. The no-call runner does not inspect Git objects. `suite_commit` is therefore an operator-declared, unverified provenance string, not proof that the commit exists; the plan SHA and per-file `asset_bindings` are the verifiable byte authority. Input/output token caps are likewise operator-declared and unverified here: this runner has no provider tokenizer, records no observed usage, and does not enforce or claim that either cap was honored. Every `raw_event_utf8` value is itself canonical closed JSON. The runner parses those bytes, derives event kind/turn index from them, rejects unknown fields or event kinds, classifies tool/network/partial/write-failure events before consulting outer normalized fields, and requires every transcript turn's text to equal its raw message-event text byte-for-byte. The stream contains exactly one first `session_started` and exactly one last `session_completed`; lifecycle events may not restart or complete midstream. A free-form runtime event cannot be mislabeled as benign and pass. The first binding/authorization mismatch, Unicode-normalized semantic arm/pair/ replicate leak, ineligible/partial session, unplanned tool or network event, evidence-write failure, actor-protocol deviation, out-of-order ingestion, or contract failure stops permanently. The runner first advances a materialization-time, per-cell attempt guard from `ready` to `active`, then publishes a closed write-once `stop-intent.json` (or its exact-byte fallback slot) that embeds and hash-binds the raw rejected bytes plus the exact stopped manifest. An input that cannot be acquired within the 4 MiB bound instead commits a bounded acquisition-failure record with bounded observed metadata and a 64 KiB prefix hash; that terminal record is also permanently retry-forbidden. If both marker slots fail, the already-active immutable guard remains the durable poison state, so later input cannot retry that cell. NFKC-normalized raw messages require a letter, number, or symbol, while execution and authorization identity text requires a letter or number. Spaces, invisible format characters, and isolated combining marks cannot stand in for provider identity, consent, or subject output. The runner then publishes content-addressed raw evidence and atomically replaces state. If that replacement fails, every command rejects retry and a later load may recover only the embedded exact stopped state. Validation replays marker, state, and blocked evidence. A compact canonical digest binds every unexpected file and directory without embedding unbounded paths in the marker. Registered marker hardlink aliases and manifest-replacement staging bytes are exact-replayed; a new, missing, or changed entry fails closed. If a write failure occurs after authorization/transcript/receipt or staging bytes were created before the success manifest commits, the surviving bytes remain evidence and are bound by the receipt or compact inventory rather than deleted. After all 48 transcripts pass, blinding rejects transcript free text containing any frozen cell/scenario/pair/experiment/arm/block identifier, explicit mapping marker, prior-label/adjudication marker, or claimed human-evidence marker. It first publishes a deterministic write-once `blind-intent.json` that freezes a one-time private nonce and binds the exact plan and complete ingestion state. The intent freezes all 48 blind ids and every packet, inventory, private-map, and manifest hash before writing the in-run `blind-staging/` transaction. The intent is a `0600` procedural-nondisclosure artifact that must never be delivered to judges. A partial staging failure may only resume the same exact transaction; a collision is quarantined, and a second set of ids or mappings cannot be generated. The staging directory is then atomically renamed into one bundle with 48 isolated packets, a content-free public inventory, an exact blind manifest, and a private map. The complete-byte staging path uses deterministic aliases derived from target refs and intent-bound hashes under a `0700` root. If alias cleanup fails, the hardlink remains registered evidence and exact-resumes against the same target without creating another mapping. The ingestion state becomes `blind_finalized`; validation reconstructs every packet from its source transcript and verifies the complete bundle inventory and hashes. A crash before bundle publication resumes only the bound staging transaction; a crash after publication but before the state update is recoverable only by exact replay of that bundle. A first-round judge may receive a packet only after the closed assignment-ledger gate verifies that the same judge has not and will not receive another arm or replicate that shares its scenario/pair/role card. The no-call runner still does not implement that ledger, and the bundle alone does not prove exposure blindness; the gate is the separate `scripts/ideation_diversity_assignment_gate.py` (see "First-round assignment-ledger gate" below). Never deliver the complete packet directory. These unlabeled packets cannot stand in for two independent human judges or the separate arm-blind human adjudicator. Packet flags state only that no structured label, adjudication, or human-evidence artifact is attached; they do not claim the codebook lacks label instructions. Once that gate passes, delivery is exactly one isolated packet per first-round assignment. ## First-round assignment-ledger gate `scripts/ideation_diversity_assignment_gate.py` implements the closed gate the paragraphs above require, deliberately outside the no-call runner so the runner's own boundary statement stays true. The operator authors a ledger (`judge_assignment_ledger.schema.json`): binding hashes for the exact run plan, inventory, blind manifest, and private arm map; a pseudonymous judge roster; an adjudicator handle (schema-excluded from first-round assignments); and the first-round assignments themselves as `judge_id`/`blind_session_id` rows only — the ledger never carries arm, pair, scenario, experiment, or replicate fields. ```bash PYTHONPATH=scripts python scripts/ideation_diversity_assignment_gate.py verify \ --run-dir /path/to/run --ledger /path/to/assignment-ledger.json PYTHONPATH=scripts python scripts/ideation_diversity_assignment_gate.py deliver \ --run-dir /path/to/run --judge judge-01 \ --blind-session-id blind-24_LOWERCASE_HEX --dest /path/to/judge-desk ``` `verify` replays the complete finalized bundle, checks the four hash bindings and private-map permissions, then enforces: every one of the 48 packets carries at least two distinct first-round judges; the roster equals exactly the set of assigned judges; and no judge receives two packets whose cells share a `pair_id` — the pair groups the en/zh-TW variants of one synthetic scholar context, so blocking at pair level subsumes scenario and role-card sharing, including the same scenario appearing in both experiments. Only when every check passes does it seal a write-once pass receipt (`assignment_gate_receipt.schema.json`) embedding the exact accepted ledger. A failed check writes nothing, and an exposure failure carries no judge or blind identifiers — even that pairing exists inside one judge's set is private-map information once combined with the ledger. Gate artifacts live in a sibling `-assignment-gate/` directory, never inside the run directory, so the runner's exact run-inventory validation stays green. `deliver` consumes only the sealed receipt (never a mutable ledger path), but treats it as evidence rather than authority: it replays the complete bundle validation and every semantic gate check — exactly what `verify` runs — against the embedded ledger, so a hand-fabricated receipt cannot authorize a delivery that `verify` would refuse, even over a consistently tampered bundle. It then requires the exact `judge_id`/`blind_session_id` assignment, verifies the packet's sealed inventory hash, and publishes exactly one isolated packet into a destination outside the run and gate directories that the gate itself creates — `mkdir` is the atomic desk claim, so a new delivery refuses any pre-existing destination and two racing deliveries cannot both own one desk. Each assignment is claimed by a write-once delivery marker (`first_round_delivery_marker.schema.json`) and closed by a write-once completion marker after publication: an interrupted identical delivery may exact-resume once, but a completed assignment is never re-issued, even by an identical command. One residual is accepted by design: a crash in the instant between packet publication and the completion marker leaves that one assignment resumable, which can re-materialize the identical bytes for the same judge — the same assignment, so no exposure or blinding property is affected. Desk ownership is acquired atomically at creation, and a post-publication isolation re-scan remains as defense in depth: a desk is certified only by a successful exit over exactly one packet. Two known bounded weaknesses are accepted for concurrent operator misuse (parallel gate commands aimed at one desk, itself a protocol violation), per the 2026-08-14 review disposition: an assignment that loses the atomic desk claim keeps its write-once claim marker bound to the lost destination and needs manual operator recovery before re-delivery, and two concurrent resumes of the same completed-identical delivery can both report success. Neither interleaving can expose a judge to a second scholar context, contaminate a certified desk, or re-issue a completed assignment. The gate verifies structural exposure constraints only: it cannot authenticate that two handles are two distinct people, and judge/adjudicator independence remains a procedural responsibility, as the ledger's `identity_boundary` block states. The assignment map is kept under a `0700` directory as a `0600` file and is declared `procedural_nondisclosure_only`, with `encrypted=false`. These local permissions reduce accidental disclosure but are not encryption or an enforced cannot-open seal. Operators must withhold the map until raw labels and adjudication are sealed. ## Claims and dispatch boundary The role cards are repository-owned synthetic material. They do not represent real scholars or measure real creativity. Count, dispersion, and facet follow-through remain separate; model-originated framings never earn scholar credit. The Phase-2 schema freezes and hashes the exact plan under the shared held-out measurement contract. A decision-relevant run still needs two complete independent replicates per scenario-arm cell, at least two independent blinded human judges, and a separate arm-blind human adjudicator. Obtain fresh consent for the exact subject/actor/judge plan. No earlier model-run consent applies, and the no-call envelope itself is never consent. The final report uses `heldout-measurement/1.1` with suite class `paired_controls`. #659 stays open until the per-mechanism baseline and its raw evidence are published. --- ### Overview (`evals/heldout/reviewer_seeded_defects/runs/raw/README.md`) # Raw panel outputs (audit evidence for the run records) One file per run, named `.review.md`: the complete output of that run's blinded full-mode panel under the ISOLATED dispatch design — the field-analyst report, then for each of the five seats the Phase 1 pre-commitment (produced by a physically separate, paper-blind call that received only the contract + title/field/word_count and was forbidden from reading any manuscript) and the Phase 2 paper-visible review, then the editorial synthesis. These files are the evidence base that makes the sibling `../*.json` records re-adjudicable: DETECTED/PARTIAL calls can be re-classified from the full text, severity can be recomputed from the seats' own tags, and the clean-control zero-false-findings claim can be verified only against the complete reports. **Fidelity note:** these files are verbatim subagent output (no redaction was needed this round; the repo-boundary deny-list scan came back clean). The `superseded/2026-07-24-in-context-dispatch/` directory holds the earlier single-context attempt these runs replace. **Voided artifacts:** `voided/` holds any synthesis output voided under the protocol §8.1 grammar rule, preserved verbatim; the operative retried synthesis lives in the parent run file's PART 3 with a provenance note. **Blocked artifacts:** `blocked/` holds partial panel evidence when fail-loud conformance stops a run before a complete panel exists, plus completed final panels whose retry provenance is incomplete because an earlier response was not preserved. These bundles are paired with machine-readable records under `../blocked/`, are never scored, and must never enter replicate means or gate calculations. Missing responses and downstream calls stay missing: do not reconstruct or impute them, redraw only the failed Phase 2, or launch a replacement panel to hide an abort. **Contamination reminder:** these are *outputs* of past measurement runs, not ground truth — but they live under `evals/`, which review sessions are already forbidden to read under the measurement protocol. Never paste them into a review session's context. --- ### Overview (`evals/heldout/reviewer_seeded_defects/runs/superseded/2026-07-24-in-context-dispatch/README.md`) # Superseded: 2026-07-24 in-context-dispatch baseline attempt These six run records (+ raw outputs under `raw/`) are the first baseline attempt, executed as ONE subagent per review simulating the whole panel in a single context. They are preserved for the audit trail but are NOT the operative baseline: the cross-model review of the attempt showed that single-context simulation cannot deliver the sprint contract's load-bearing mechanism ("physical separation of calls: Phase 1 never sees paper content", `sprint_contract_protocol.md` §1), and the leak was observable — several Phase-1 "paper-blind" scoring plans pre-registered manuscript-specific facts. The maintainer re-ran the full baseline the same day with physically separated per-seat Phase 1 / Phase 2 calls; the operative records live in the parent `runs/` directory. Comparative note (why these are still informative): the isolated re-run reproduced the in-context attempt's headline numbers almost exactly (MS01 strict recall 0.90 in both designs with the same sole GRIM miss; clean-control false findings 0 in both; severity agreement 0.625 operative vs 0.599 superseded) — evidence that the Phase-1 blindness leak was not inflating recall, and that the measurand is robust to this dispatch difference. The one substantive divergence: the in-context MS02-r1 panel only PARTIALLY caught the undescribed-interview-instrument defect, while both isolated MS02 panels named the absent protocol explicitly. --- ### Overview (`evals/heldout/reviewer_seeded_defects/runs/raw/blocked/2026-07-27-ms00_clean-post-r1/README.md`) # Provenance-invalid panel evidence: 2026-07-27 MS00 post r1 This panel reached synthesis, but it is invalid for scoring and gate calculations. The methodology seat's first Phase 1 response omitted the mandatory D3 `what_triggers_fatal:` line. The dispatcher used the protocol-permitted single paper-blind structural retry, but overwrote that first response. `dispatch.log` preserves the exact checker diagnostic and chronology; `final-panel.review.md` preserves the final conforming Phase 1, all five Phase 2 cards, and synthesis. Neither artifact can reconstruct the missing first response or prove its paper blindness independently. The final panel is retained as diagnostic evidence only. It is deliberately not named with the normal `.review.md` pattern, and its status record is under `../../../blocked/2026-07-27-ms00_clean-post-r1.json`. Do not adjudicate it as a valid run, include its observed finding count in a replicate mean, or use it for an acceptance gate. --- ### Overview (`evals/heldout/reviewer_seeded_defects/runs/raw/blocked/2026-07-27-ms00_clean-post-r2/README.md`) # Blocked panel evidence: 2026-07-27 MS00 post r2 This directory preserves the durable output that existed when the panel stopped. It is not a complete review and must not be adjudicated or included in gate means. The perspective model call emitted a `## Scoring Plan Dissent` heading without the required canonical `dimension_id:` line. `perspective.conformance.log` records the fail-loud diagnostic. The dispatch then exited before the Devil's Advocate seat and before synthesis, as shown by `dispatch.log` and `dispatch.stderr`. Preserved artifacts are: - the sprint contract, metadata, and field analysis; - the final conforming Phase 1 and completed Phase 2 outputs for EIC, methodology, and domain, with their layer-1 and conformance logs; - the final conforming Perspective Phase 1, the malformed Perspective Phase 2, and its conformance log; and - the dispatch chronology and terminal traceback. The methodology and perspective Phase 1 calls each used the protocol-permitted single structural retry. The runner overwrote each first malformed Phase 1 response before this bundle was assembled; their exact checker diagnostics remain in `dispatch.log` and the final conforming retries are preserved here. No Phase 2 retry or replacement panel was run. The machine-readable status record is `../../../blocked/2026-07-27-ms00_clean-post-r2.json`. --- ### ARCHITECTURE (`docs/ARCHITECTURE.md`) # ARS Pipeline Architecture (v3.23.0) Full pipeline view across stages × skills × artifacts × gates. Every completed stage requires a user-confirmation checkpoint (per `academic-pipeline/SKILL.md` and `pipeline_state_machine.md`); the diagrams below surface the **decision-heavy** checkpoints visually so they are easy to locate. The post-stage confirmation checkpoints at 2.5 and 4.5 are machine-verified first, then confirmed by the user — they are not skipped. ## How to read - **Flow diagram** (§2): macro view — which stage follows which, where loops exist, where gates block. Every rectangle ends in a post-stage user confirmation (elided for readability); 🧑 markers call out the decision-heavy moments where the user chooses a branch. - **Matrix** (§3): the only place where (stage × skill × mode × data_level × artifacts × agents × gate) all co-exist. Use this when asking "what happens at Stage X?" The Gate column lists both machine checks and the user-confirmation checkpoint that closes the stage. - **Data access flow** (§4) and **skill graph** (§6): orthogonal views answering "who sees what" and "who depends on what" respectively. - **Literature corpus flow** (§5): producer/consumer view of the optional Material Passport `literature_corpus[]` input port (v3.6.4) and Phase 1 consumer integration (v3.6.5). - **Quality gates** (§7): zoom on the blocking checks — both machine-enforced and human-enforced. §7.1 classifies the repo's CI workflows by enforcement strength (blocking / advisory / administrative / post-push detection). - **Timeline** (§8): why the architecture looks the way it does — each release added one honesty primitive or a new contract. - **Modes** (§9): reference when composing a pipeline invocation. The matrix alone is insufficient: it hides data-access hierarchy and skill dependency. The diagrams alone are insufficient: they hide artifact flow and per-stage agent detail. Together they are the full architecture. ## 1. Checkpoints (at-a-glance) The pipeline has **two classes of user checkpoint**. Both require the user to confirm before the pipeline advances; they differ in what the user is actually deciding. **Decision-heavy checkpoints** — the user chooses a branch or accepts a material decision: | # | Stage | What the user decides | |---|---|---| | 🧑 1 | 1. RESEARCH | RQ Brief + Methodology Blueprint | | 🧑 2 | 2. WRITE | Outline approval before drafting | | 🧑 3 | 3. REVIEW | Editorial decision (Accept / Minor / Major / Reject) | | 🧑 4 | 3 → 4 Revision Coaching | Revision strategy (up to 8 Socratic rounds; user can skip) | | 🧑 5 | 4. REVISE | Revision changes confirmed | | 🧑 6 | 3'. RE-REVIEW | Verification-review decision | | 🧑 7 | 3' → 4' Residual Coaching | Residual-issue trade-offs (up to 5 Socratic rounds; user can skip) | | 🧑 8 | 4'. RE-REVISE | Content frozen — no further review loop | | 🧑 9 | 5. FINALIZE | Output format selection (MD / DOCX / LaTeX / PDF) | | 🧑 10 | 6. PROCESS SUMMARY | Language confirmation + collaboration quality review | **Post-stage confirmation checkpoints** — machine verification runs first; the user then acknowledges the integrity report before proceeding. These are also user-gated (per `pipeline_state_machine.md` — every stage ends in `[checkpoint]`), but the decision is "acknowledge the automated report" rather than "choose a branch": | # | Stage | What runs | What the user acknowledges | |---|---|---|---| | ✓ 1 | 2.5 INTEGRITY | 7-mode failure checklist (see §3 for exact taxonomy) | Integrity Report PASS/FAIL + any SUSPECTED flags | | ✓ 2 | 4.5 FINAL INTEGRITY | Deep Mode 2 check, zero-tolerance | Final Integrity Report PASS + populated Material Passport | ## 2. Pipeline Flow ``` /* Detailed source-code truncated for AI context efficiency. */ ``` **Legend:** - **Solid red (🧑)** = decision-heavy human gate — the user chooses a branch or approves a material decision. - **Solid orange (✓)** = integrity gate — machine verification runs first, user then acknowledges the report. Not skipped. - **Green** = Socratic coaching sub-stage. User may engage or say "just fix it" to skip the dialogue. - **👁 observer** (v3.5.0) = `collaboration_depth_agent` dispatches at every FULL/SLIM checkpoint + pipeline completion. **Never blocks.** Advisory only. MANDATORY integrity gates (2.5 / 4.5) explicitly skip the observer so compliance checks are not diluted. - **Data level** (§3 column) = the data layer of that *stage's* output artifacts (a postcondition), not the owning skill's declared `data_access_level` (see §4 — e.g. `academic-pipeline` declares `raw`, while its gate stages consume unverified drafts and *produce* the verified artifacts their rows are labeled by). ## 3. Stage × Dimension Matrix | Stage | Skill / Mode | Data level | Artifact produced | Core agents | Gate / Checkpoint | |---|---|---|---|---|---| | **1. RESEARCH** | `deep-research` v2.12.1 (full / socratic / lit-review / three-way-scan / systematic-review / fact-check / review / quick) | RAW | RQ Brief; Methodology Blueprint; Annotated Bibliography (S2-verified); Synthesis Report; INSIGHT Collection. **Search Strategy report includes PRE-SCREENED block (v3.6.5)** when Material Passport carries `literature_corpus[]` | research_question_agent; research_architect_agent; **📚 bibliography_agent (v3.6.5+ corpus reader — corpus-first / search-fills-gap flow)**; source_verification_agent; synthesis_agent; meta_analysis_agent; editor_in_chief_agent; devils_advocate_agent; risk_of_bias_agent; ethics_review_agent; **🟦 socratic_mentor_agent (v3.5.1 reading-check probe layer, opt-in)**; report_compiler_agent; monitoring_agent (13 agents); **👁 collaboration_depth_agent (v3.5.0, advisory)** | 🧑 **Decision-heavy checkpoint:** user confirms RQ brief + methodology. Machine checks: S2 API Tier-0 verification (Levenshtein ≥ 0.70); evidence hierarchy graded; anti-sycophancy on DA (score 1-5, concede only ≥ 4); **corpus-first flow with 4 Iron Rules + F3/F4 provenance reporting (v3.6.5)** when corpus present. 👁 Observer runs post-checkpoint; never blocks | | **2. WRITE** | `academic-paper` v3.3.1 (full / plan / outline-only / lit-review / revision-coach / abstract-only / citation-check / disclosure / format-convert / revision) | REDACTED | Paper Configuration Record; Outline; Argument Map; Draft Text; Bilingual Abstract; Figures + Captions; Citation List. **Literature Search Report includes PRE-SCREENED block (v3.6.5)** when Material Passport carries `literature_corpus[]`; merged `final_included` set feeds the Literature Matrix and Research Gap Identification | 12-agent pipeline: intake_agent; **📚 literature_strategist_agent (v3.6.5+ corpus reader — corpus-first / search-fills-gap flow)**; structure_architect_agent; argument_builder_agent; draft_writer_agent; citation_compliance_agent; abstract_bilingual_agent; peer_reviewer_agent; formatter_agent; socratic_mentor_agent; visualization_agent; revision_coach_agent; **👁 collaboration_depth_agent (v3.5.0, advisory)** | 🧑 **Decision-heavy checkpoint:** outline approved before drafting. Machine checks: anti-leakage protocol (unsupported fill → `[MATERIAL GAP]`); VLM figure verification (10-pt APA checklist, max 2 refinements); style calibration vs user voice; Stage 2 parallelization (Phase 1 + visualization after outline); **corpus-first flow with 4 Iron Rules + F3/F4 provenance reporting (v3.6.5)** when corpus present. 👁 Observer runs post-checkpoint; never blocks | | **2.5 INTEGRITY** | `academic-pipeline` v3.23.0 (gate) | VERIFIED_ONLY | Material Passport (Schema 9, required) + `repro_lock` (v3.3.5, declared — populated or `null`); Claim Verification Report (pre-review sampling: #549 risk-stratified — 100% HIGH-IMPACT + 10% random sentinel, min(10, total) — per `claim_verification_protocol.md`); Data Provenance Audit | integrity_verification_agent; state_tracker_agent; pipeline_orchestrator_agent. **👁 collaboration_depth_agent: SKIPPED (MANDATORY gate — observer dilution explicitly prevented)** | ✓ **Integrity gate** + user ack. 7-mode AI failure checklist (Lu 2026, canonical order per `ai_research_failure_modes.md`): **M1** implementation bug passing AI self-review; **M2** hallucinated citation; **M3** hallucinated experimental result; **M4** shortcut reliance; **M5** implementation bug reframed as novel insight; **M6** methodology fabrication; **M7** frame-lock at early pipeline stage. Pre-review claim sampling mode. FAIL → fix + re-verify (max 3 rounds) | | **3. REVIEW** | `academic-paper-reviewer` v1.11.1 (full / guided / quick / methodology-focus / calibration) | VERIFIED_ONLY | **First-round review package** (per `academic-paper-reviewer/SKILL.md`): 5 review reports (Journal-Fit Reviewer + R1 methodology + R2 domain + R3 interdisciplinary + Devil's Advocate) + Editorial Decision (Accept / Minor / Major / Reject) + Revision Roadmap. **Schema 13.2 Sprint Contract (`shared/sprint_contract.schema.json`) is required** for `full` and `methodology-focus` modes (other modes reserved with pre-v3.6.2 behaviour). | field_analyst_agent (auto-detects domain, configures 3 field-adaptive reviewers); eic_agent; methodology_reviewer_agent; domain_reviewer_agent; perspective_reviewer_agent; devils_advocate_reviewer_agent; **🔒 editorial_synthesizer_agent (role-scoped three-step mechanical protocol + forbidden-ops list)** (7 agents); **👁 collaboration_depth_agent (v3.5.0, advisory)** | 🧑 **Decision-heavy checkpoint:** user reviews editorial decision. Machine checks: concession threshold protocol (DA rebuttal scored 1-5, no concede below 4); attack intensity preserved through revisions; cross-model DA critique (optional, `ARS_CROSS_MODEL` env); read-only constraint (no new claims). **Sprint Contract two-phase protocol**: each reviewer runs paper-content-blind Phase 1 (eligible-dimension trigger commitments) then paper-visible Phase 2 via ``; `check_phase_conformance.py` validates the boundary and the synthesizer applies two-stage eligible-seat arithmetic. 👁 Observer runs post-checkpoint; never blocks | | **3 → 4 Revision Coaching** | `academic-paper-reviewer` (Journal-Fit Reviewer Socratic sub-stage) | VERIFIED_ONLY | Revision strategy dialogue (not an artifact handed forward; feeds Stage 4 revision plan) | eic_agent | 🧑 **Decision-heavy checkpoint:** Socratic dialogue with the Journal-Fit Reviewer (max 8 rounds). User may say "just fix it for me" to skip. Source: `two_stage_review_protocol.md` | | **4. REVISE** | `academic-paper` v3.3.1 (revision / revision-coach) | REDACTED | Point-by-Point Response; Revised Draft; Delta Report (what changed + why) | revision_coach_agent (v3.3 Socratic mode); draft_writer_agent (re-entry); argument_builder_agent (if structural); **👁 collaboration_depth_agent (v3.5.0, advisory)** | 🧑 **Decision-heavy checkpoint:** user confirms changes. The orchestrator performs a narrative, evidence-anchored comparison per named criterion; unresolved decision-bearing regressions trigger review and changed criteria become `NOT_COMPARABLE`. No numerical delta or typed machine trajectory is currently emitted. 👁 Observer runs post-checkpoint; never blocks | | **3'. RE-REVIEW** | `academic-paper-reviewer` v1.11.1 (re-review — the default; a user-requested fresh full review at 3' dispatches full mode instead) | VERIFIED_ONLY | **Verification package** (re-review mode, per the spec in `academic-paper-reviewer/SKILL.md`): Revision response checklist + residual issues list + new Decision (Accept / Minor / Major) + **R&R Traceability Matrix (Schema 11)** with Author's Claim + Verified? columns. Fresh full review at 3' emits the normal full-review package instead (no R&R matrix) | **Contract-governed re-review dispatch**: orchestrating layer + three sequential fenced calls (Phase 1/2A use frozen-card routed personas; Phase 2B is one dedicated integration call; neither loads the first-round `eic_agent` / `editorial_synthesizer_agent` files), followed by the mandatory synthesis checker. Round-1 cards are reused and field_analyst_agent is NOT re-run except the protocol's visible regeneration fallback; a user-requested fresh full review at 3' runs full mode instead. **👁 collaboration_depth_agent (v3.5.0, advisory)** | 🧑 **Decision-heavy checkpoint:** user reviews verification decision. Hard cap: **max 1 RE-REVISE round; 2 revision loops total** across Stages 4 + 4'. Major outcome at 3' → Residual Coaching → Stage 4'. 👁 Observer runs post-checkpoint; never blocks | | **3' → 4' Residual Coaching** | `academic-paper-reviewer` (Journal-Fit Reviewer Socratic sub-stage) | VERIFIED_ONLY | Residual-issue dialogue | eic_agent | 🧑 **Decision-heavy checkpoint:** Socratic dialogue on trade-offs for residual issues (max 5 rounds). User may skip. Source: `two_stage_review_protocol.md` | | **4'. RE-REVISE** | `academic-paper` v3.3.1 (revision) | REDACTED | Final Revised Draft (terminal; advances to 4.5) | draft_writer_agent; revision_coach_agent; **👁 collaboration_depth_agent (v3.5.0, advisory)** | 🧑 **Decision-heavy checkpoint:** user confirms content frozen. No further review loop permitted. 👁 Observer runs post-checkpoint; never blocks | | **4.5 FINAL INTEGRITY** | `academic-pipeline` v3.23.0 (gate) | VERIFIED_ONLY | Updated Material Passport (`verification_status: VERIFIED`) + `repro_lock` declared — populated or explicit `null` (honest opt-out); Claim Verification Report (**final-check mode: 100% of E1 registered claims; semantic extraction completeness unknown** per `claim_verification_protocol.md`) | integrity_verification_agent (deeper re-run of 7 modes); state_tracker_agent. **👁 collaboration_depth_agent: SKIPPED (MANDATORY gate — observer dilution explicitly prevented)** | ✓ **Integrity gate** + user ack. Zero named gate defects within the registered/sampled populations; no skip permitted. Any mode SUSPECTED at 2.5 must be CLEAR or user-Overridden by 4.5. `repro_lock` is **not** read by the integrity gate at runtime (per `artifact_reproducibility_pattern.md`); if populated, `stochasticity_declaration` must be verbatim and is validated by the standalone `check_repro_lock.py` — this is post-hoc documentation, not a runtime block or global correctness certificate | | **4→5 CLAIM-AUDIT** (v3.8, opt-in via `ARS_CLAIM_AUDIT=1`) | `academic-pipeline` v3.23.0 (gate) | VERIFIED_ONLY | `claim_audit_results[]` + `claim_drifts[]` + `uncited_assertions[]` + `constraint_violations[]` + `audit_sampling_summaries[]` aggregates; reads `claim_intent_manifests[]` (writer-side pre-commitment baseline). Emits 5 HIGH-WARN annotation classes consumed by Stage 5 formatter REFUSE rules 6-10 | claim_ref_alignment_audit_agent (Stage 4→5 dispatch slot, after v3.7.1 cite finalizer, before formatter hard gate; in pipeline runs first at Stage 4.5, #929) | ✓ **Audit gate** (default OFF for v3.8.0). Per-citation LLM-as-judge against retrieved excerpt; 8-row finalizer matrix discriminates paywall (LOW-WARN) / fabricated (HIGH-WARN) / anchorless (HIGH-WARN) / audit_tool_failure (MED-WARN) via `ref_retrieval_method`. Calibration runner (`scripts/test_claim_audit_calibration.py`) checks the FNR<0.15 + FPR<0.10 gate on the shipped 25-tuple gold set with a stub judge that returns the gold labels, so it validates the tooling; no live-judge calibration result is recorded. Spec: `docs/design/2026-05-15-issue-103-claim-alignment-audit-spec.md` | | **5. FINALIZE** | `academic-paper` v3.3.1 (format-convert / disclosure) | VERIFIED_ONLY | Publication-ready MD; DOCX (Pandoc, if available); PDF (tectonic, from LaTeX) and the .tex source when the user asks for them; default venue AI applicability/status bundle (`REQUIRED` / `ACTION_ONLY` / `NOT_REQUIRED` / `UNKNOWN` plus typed halt) or policy-anchor-specific render | formatter_agent | 🧑 **Decision-heavy checkpoint:** user selects format before render. The disclosure output must match the selected venue or policy anchor (15-entry venue database: ICLR / NeurIPS / Nature / Science / ACL / EMNLP + medical-publishing targets incl. ICMJE, NEJM, The Lancet, JAMA, BMJ, PLOS, Frontiers, and two Chinese-language policy targets — one publisher-wide and one journal; see `venue_disclosure_policies.md`). **v3.8 terminal hard gate (formatter_agent REFUSE rules 6-10)** refuses output on any unresolved `[HIGH-WARN-CLAIM-NOT-SUPPORTED]` / `[HIGH-WARN-NEGATIVE-CONSTRAINT-VIOLATION]` / `[HIGH-WARN-FABRICATED-REFERENCE]` / `[HIGH-WARN-CLAIM-AUDIT-ANCHORLESS]` / `[HIGH-WARN-CONSTRAINT-VIOLATION-UNCITED]` annotation when `ARS_CLAIM_AUDIT=1` was set upstream. **v3.10 rule 11** refuses on any `severity=HIGH-BLOCK` terminal-policy token (generic; co-emitted by the finalizer under a strict `terminal_policies` mode). **v3.11 rule 12 (#182)** refuses on a `lookup_verified == false` citation-existence row ONLY under `terminal_policies.citation_existence == strict` — default advisory passes (`/ars-mark-read`-ack-able); the narrowed ID-keyed `false` never fires on a title-only-unmatched `unresolvable` citation | | **6. PROCESS SUMMARY** | `academic-pipeline` v3.23.0 | VERIFIED_ONLY | Paper Creation Process Record (MD, plus PDF when asked for); AI Self-Reflection Report (concession rate, sycophancy risk, health alerts, Failure Mode Audit Log); narrative criterion-regression notes when recorded; **Collaboration Depth Chapter (v3.5.0)** summarising the per-checkpoint observer reports from `collaboration_depth_history[]` | state_tracker_agent; pipeline_orchestrator_agent; **👁 collaboration_depth_agent (v3.5.0, pipeline-completion dispatch — final advisory report)** | 🧑 **Decision-heavy checkpoint:** language confirmed with user. Collaboration quality evaluated. No typed criterion-trajectory visualization is claimed until its producer/validator is implemented. Post-publication audit report (if peer-review published). 👁 Observer runs final pipeline-completion dispatch; never blocks | ## 4. Data Access Level Flow (v3.3.2+) ```mermaid flowchart LR User[User input
web / PDFs / queries / pasted drafts] Raw[deep-research
data_access_level: raw] Red[academic-paper
data_access_level: raw] Ver1[academic-paper-reviewer
data_access_level: raw] Orch[academic-pipeline
data_access_level: raw] User --> Raw User -- standalone modes: ungated drafts,
reviewer comments, search-fills-gap --> Red User -- standalone /ars-reviewer:
ungated pasted manuscript --> Ver1 User -- Stage 1 request / mid-entry paper --> Orch Raw -- source_verification elevates artifacts --> Red Red -- Gate 2.5: 7-mode integrity --> Ver1 Orch -. orchestrates .-> Raw Orch -. orchestrates .-> Red Orch -. orchestrates .-> Ver1 classDef raw fill:#fff1f0,stroke:#cf1322 class Raw,Red,Ver1,Orch raw ``` Rules (per `shared/ground_truth_isolation_pattern.md`): - `data_access_level` is a **declarative** annotation, not a runtime-enforced permission system. The CI lint `scripts/check_data_access_level.py` pins each skill's value and confirms the vocabulary; it does not inspect context windows at runtime. - `raw` skills consume layer-1 data (arbitrary, possibly adversarial). - `redacted` (sanitized material, no new raw ingestion) and `verified_only` (runs only after upstream integrity gates) remain in the legal vocabulary, but since #773 no top-level skill qualifies for either: the annotation takes the **dirtiest input across all modes and entry paths**, and every skill has at least one legitimate Layer-1 entry. Artifact-level elevation is tracked per stage *output* in the §3 Data-level column, not per skill. - `academic-pipeline` is `raw` (#756) because Stage 1 accepts raw user requests and mid-entry accepts raw existing papers — the integrity gates run *inside* the pipeline, downstream of its intake. - `academic-paper` is `raw` (#773) because its standalone modes ingest ungated user drafts and third-party reviewer comments, and `literature_strategist_agent`'s search-fills-gap flow ingests external-index search results inside the skill. The former `redacted` described the orchestrated pipeline path, where Stage 2 inputs arrive as Stage-1 sanitized artifacts (Gate 2.5 runs after Stage 2, not before it). - `academic-paper-reviewer` is `raw` (#773) because the standalone `/ars-reviewer` entry legitimately consumes an ungated pasted manuscript — the former `verified_only` was at best true for the pipeline's initial Stage 3 dispatch (post-Gate-2.5; Stage 3' re-review consumes a freshly revised manuscript before Stage 4.5). That Stage 3 sequencing is unchanged. - The reviewer side **may hold a rubric privately** — the key guarantee is that rubric / gold-label content must not be present in the candidate-generating agent's context. Calibration gold sets are runtime-supplied by the human researcher, not bundled into the repository. - Stage 2.5 and Stage 4.5 (plus the user's review at each gate) are the actual enforcement points. This pattern document explains the data-flow structure that makes those gates meaningful; it is not itself a runtime lock. ## 5. Material Passport `literature_corpus[]` Flow (v3.6.4 input port + v3.6.5 consumers) The Material Passport's `literature_corpus[]` is an **optional** Schema 9 input port for user-curated literature. Producers (out-of-band, before an ARS session) and consumers (Phase 1 literature agents at runtime) sit on opposite sides of the passport. ```mermaid flowchart LR subgraph Producer["Out-of-band (user-owned)"] Source[User corpus
Zotero / Obsidian /
folder of PDFs / etc.] Adapter["Adapter
(reference: scripts/adapters/
folder_scan.py / zotero.py /
obsidian.py — v3.6.4+)"] Source --> Adapter Screener["sr-screener report
*_literature_corpus.yaml
(screened records)"] end Passport["Material Passport
passport.yaml
+ rejection_log.yaml
(literature_corpus[]
optional Schema 9 field)"] Adapter --> Passport Screener --> Passport subgraph Consumer["Phase 1 ARS runtime (v3.6.5)"] BA["📚 deep-research
bibliography_agent
(Phase 1)"] LS["📚 academic-paper
literature_strategist_agent
(Phase 1)"] end Passport -- presence-based
auto-engage --> BA Passport -- presence-based
auto-engage --> LS BA --> SR1[Search Strategy report
+ PRE-SCREENED block
final_included = pre_screened ∪ external] LS --> SR2[Literature Search Report
+ PRE-SCREENED block
+ corpus-aware Matrix / Gap] classDef producer fill:#fffbe6,stroke:#d48806 classDef passport fill:#f0f5ff,stroke:#2f54eb,stroke-width:2px classDef consumer fill:#f6ffed,stroke:#52c41a classDef report fill:#f9f0ff,stroke:#9254de class Source,Adapter,Screener producer class Passport passport class BA,LS consumer class SR1,SR2 report ``` **Producer side (v3.6.4 input port).** Adapters run out-of-band — before an ARS session, not during. They read a user corpus source and emit `passport.yaml` with `literature_corpus[]` populated and a parallel `rejection_log.yaml` (always emitted; empty when no rejections). Three reference Python adapters ship at `scripts/adapters/{folder_scan,zotero,obsidian}.py`; the `sr-screener` skill's `report` mode writes the records a screening advanced as a ready-to-paste `literature_corpus[]` list (`adapter_name: sr-screener`); users are expected to write their own adapters for non-reference sources following [`academic-pipeline/references/adapters/overview.md`](../academic-pipeline/references/adapters/overview.md). Schema validated by `scripts/check_literature_corpus_schema.py`. **Consumer side (v3.6.5).** Two Phase 1 literature agents read `literature_corpus[]` via the **corpus-first, search-fills-gap** flow — `deep-research/agents/bibliography_agent.md` and `academic-paper/agents/literature_strategist_agent.md`. The flow is presence-based: it auto-engages when the passport carries a non-empty `literature_corpus[]` and parses cleanly. When the corpus is absent, empty, or fails the minimal shape check, each consumer runs its existing external-DB-only flow unchanged (Iron Rule 4 graceful fallback for the failure cases). **Five-step shared flow.** Step 0 minimal shape check → Step 1 pre-screen corpus against current RQ → Step 2 search-fills-gap (4-case dispatch on `uncovered_topics` × `user_corpus_only`) → Step 3 merge into `final_included` → Step 4 emit Search Strategy report with PRE-SCREENED block. `final_included` stays neutral — no provenance tags on bibliography entries, no provenance column in the Literature Matrix. **Four Iron Rules** govern every consumer: 1. **Same criteria.** Apply the same Inclusion / Exclusion criteria to corpus entries and external database results. No exceptions. 2. **No silent skip.** Any skipped corpus entry is recorded in the PRE-SCREENED block's skipped sub-section with a reason. Silently dropping an entry is a prompt-layer violation. 3. **No corpus mutation.** Consumer agents never modify, backfill, or derive new content into `literature_corpus[]`. Read only. 4. **Graceful fallback on parse failure.** Consumer agents do NOT re-validate schema, do NOT parse JSON Schema at runtime, and do NOT dereference `source_pointer` URIs. When the corpus cannot be parsed, emit `[CORPUS PARSE FAILURE: ]` and fall back to external-DB-only flow. **PRE-SCREENED reproducibility block.** Lives inside the Search Strategy section of each consumer's report, immediately before the existing `Databases` line. Enumerates included / excluded / skipped citation_keys with reasons; carries F3 zero-hit note when corpus is non-empty but 0 entries survived screening; carries F4a–F4f provenance reporting for `obtained_via` (adapter origin) and `obtained_at` (snapshot date) covering full / partial / undeclared / wide-spread sub-cases. CI lint `scripts/check_corpus_consumer_protocol.py` enforces nine invariants L1-L9 with manifest-driven consumer list (`scripts/corpus_consumer_manifest.json`) and tuple-matched closed-set state machine. **Out of v3.6.5 scope.** `citation_compliance_agent` corpus reading is deferred (target version TBD post-v3.8). `source_pointer` URI dereferencing and source verification remain a future `source_verification_agent` concern. Schema is unchanged from v3.6.4 — existing user adapters work without modification. Authoritative references: [`academic-pipeline/references/literature_corpus_consumers.md`](../academic-pipeline/references/literature_corpus_consumers.md) (consumer protocol) + [`academic-pipeline/references/adapters/overview.md`](../academic-pipeline/references/adapters/overview.md) (adapter contract) + [`docs/design/2026-04-26-ars-v3.6.5-consumer-integration-design.md`](design/2026-04-26-ars-v3.6.5-consumer-integration-design.md) (consumer design). ## 6. Skill Dependency Graph ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## 7. Quality Gates Two classes of gate: **🧑 decision-heavy** (user chooses a branch or approves material) and **✓ integrity** (machine verification + user ack). Pure machine-enforced 🤖 lint checks run in CI. | Gate | Class | Stage | What blocks advancement | Failure handling | |---|---|---|---|---| | RQ + methodology confirmation | 🧑 | 1 | User hasn't approved RQ Brief and Methodology Blueprint | Revise and re-present | | S2 API verification | 🤖 | 1 | Citation not in Semantic Scholar; title Levenshtein < 0.70 | Flag; user decides to drop or re-cite | | Outline approval | 🧑 | 2 | User hasn't approved outline | Revise and re-present | | Anti-leakage (v3.3) | 🤖 | 2 | Draft contains parametric fill not grounded in session materials | `[MATERIAL GAP]` tag; user provides material or accepts gap | | VLM figure verify (v3.3) | 🤖 | 2 | Rendered figure fails 10-pt APA 7.0 checklist | Max 2 refinement iterations | | Stage 2.5 integrity + ack | ✓ | 2.5 | Any mode SUSPECTED on 7-mode checklist, or Modes 1/3/5/6 INSUFFICIENT EVIDENCE (exceptions in `academic-pipeline/references/ai_research_failure_modes.md`), or user hasn't acknowledged report | Fix + re-verify (max 3 rounds); or user override with reasoning (logged) | | Editorial decision review | 🧑 | 3 | User hasn't reviewed decision letter | Present decision; await user | | Concession threshold | 🤖 | 3 | DA rebuttal scored < 4/5 by responder | No concession; frame-lock detector runs | | Revision Coaching | 🧑 | 3→4 | User hasn't engaged or explicitly skipped (max 8 rounds) | User may say "just fix it" to skip | | Revision confirmation | 🧑 | 4 | User hasn't confirmed changes | Revise; re-present | | Revision loop cap | 🤖 | 4 / 3' / 4' | 2 revision loops already consumed | Forced advance to Stage 4.5 | | Residual Coaching | 🧑 | 3'→4' | User hasn't engaged or explicitly skipped (max 5 rounds) | User may say "just fix it" to skip | | Content-frozen confirmation | 🧑 | 4' | User hasn't confirmed freeze | Await user; no further review loop permitted | | Stage 4.5 final integrity + ack | ✓ | 4.5 | ANY issue on deeper 7-mode re-run; mode still SUSPECTED since 2.5 and unresolved | ZERO-tolerance; no skip; fix + re-verify | | Format selection | 🧑 | 5 | User hasn't chosen output format | Await user format choice | | Disclosure check | 🤖 | 5 | Venue-specific AI disclosure absent or wrong form | Block render until fixed | | `repro_lock` (v3.3.5) | 🤖 (standalone) | — | `repro_lock` key required in Material Passport v3.3.5+; value must be either a populated block or explicit `null` (honest opt-out). Populated blocks validated by `check_repro_lock.py`. Per `artifact_reproducibility_pattern.md`: **not** wired into the CI lint suite by default and **not** read by the Stage 2.5 / 4.5 integrity gate at runtime — this is post-hoc documentation, not a pipeline block | Run `check_repro_lock.py ` on demand | | Language + collaboration review | 🧑 | 6 | User hasn't confirmed output language / reviewed self-reflection | Await user | | `benchmark_report` (v3.3.5, external) | 🤖 | — | Publishing a benchmark without honest disclosure | Users run `check_benchmark_report.py` before publishing | | Collaboration Depth Observer (v3.5.0) | 🤖 observer | Every FULL/SLIM checkpoint + pipeline completion | **Never blocks.** Advisory only. Scores user-AI collaboration pattern on 4 dimensions (Delegation Intensity / Cognitive Vigilance / Cognitive Reallocation / Zone Classification) per `shared/collaboration_depth_rubric.md`. Injects a named section into checkpoint presentation and a chapter in the Process Record. MANDATORY integrity gates (2.5 / 4.5) do NOT invoke the observer. | n/a — output is advisory; user `Ready to proceed?` prompt unchanged | | Reading-check probe (v3.5.1, opt-in) | 🤖 (Mentor sub-layer) | Stage 1 (Socratic mentor session) | Opt-in via `ARS_SOCRATIC_READING_PROBE=1`. Goal-oriented intent only; fires at most once per session when user has cited a specific paper. Decline logged without penalty. Outcome inline in Research Plan Summary; carried into Stage 6 AI Self-Reflection Report. | n/a — non-blocking; flag OFF preserves pre-v3.5.1 behaviour byte-for-byte | | Sprint Contract hard gate (Schema 13.2) | 🤖 + 🧑 | 3 (REVIEW) | Each reviewer receives the contract BEFORE the paper (Phase 1 metadata-only), commits triggers only for dimensions where its role is eligible, then emits role-scoped Phase 2 scores. `check_phase_conformance.py` enforces the boundary; `editorial_synthesizer_agent` applies per-dimension eligible-seat quantifiers and emits one four-value decision. Templates: `shared/contracts/reviewer/full.json` (panel 5, six dimensions) + `methodology_focus.json` (panel 2). | Synthesizer rejects post-hoc trigger/fatality edits; user sees the pre-commitment and any DA-CRITICAL-vs-Accept escalation | | Passport reset boundary (v3.6.3, opt-in) | 🤖 (orchestrator) | FULL checkpoints | Opt-in via `ARS_PASSPORT_RESET=1`. Promotes every FULL checkpoint to a context-reset boundary. `systematic-review` mode with the flag ON makes reset mandatory; other modes treat reset as the flag-gated default. New `resume_from_passport=` mode in `academic-pipeline` lets users resume in a fresh session from the Material Passport ledger alone. Schema 9 `reset_boundary[]` append-only ledger with `kind: boundary` + `kind: resume` entry types; hash via JSON Canonical Form + SHA-256 + canonical placeholder for self-reference safety. Concurrency contract: POSIX `fcntl.flock LOCK_EX` + bounded timeout ≤60s + non-POSIX fail-loudly. Authoritative protocol: [`academic-pipeline/references/passport_as_reset_boundary.md`](../academic-pipeline/references/passport_as_reset_boundary.md). Validated by `scripts/check_passport_reset_contract.py`. | Flag OFF preserves pre-v3.6.3 continuation behaviour byte-for-byte | | Corpus consumer protocol (v3.6.5) | 🤖 + Phase 1 agents | 1 / 2 (when `literature_corpus[]` present) | Presence-based auto-engage when Material Passport carries non-empty `literature_corpus[]` and parses cleanly. Four Iron Rules (Same criteria / No silent skip / No corpus mutation / Graceful fallback on parse failure). PRE-SCREENED reproducibility block in Search Strategy report (F3 zero-hit + F4a–F4f provenance). Validated by `scripts/check_corpus_consumer_protocol.py` (9 invariants L1-L9 with manifest-driven consumer list). | Parse failure → emit `[CORPUS PARSE FAILURE: ]` and fall back to external-DB-only flow (Iron Rule 4) | | Model tiering (v3.16.0, opt-in) | 🤖 (dispatch layer) | All stages (agent dispatch) | **Never blocks.** Opt-in via `ARS_MODEL_TIERING=economy\|quality-boost`. `economy` (frontier-tier session): the 14 execution-type agents dispatch one tier below the session model, floor Opus-class. `quality-boost` (below-frontier session): judgment-type agents at the Stage 2.5/4.5 integrity gates and final-review surfaces step up to the frontier tier; nothing is ever downgraded. Tiers are relative positions, never hard-pinned model ids. Frozen 43-agent classification (29 judgment / 14 execution; sr-screener's screening calls take per-role models from its config) in `scripts/model_tiering_manifest.json` + `shared/model_tiering.md`, pinned to each other and to the agent files by `scripts/check_model_tiering.py`. | Unset = byte-equivalent pre-#517 behaviour; unknown value warns once and behaves as unset (fail-open to the safe default) | | Stage 5/6 boundary semantics (v3.17.0) | 🤖 + 🧑 | 5 (entry gate), 6 (terminal checkpoint) | Stage 5's "before finalization: always MANDATORY" names exactly one checkpoint — the entry gate between Stage 4.5 PASS and Stage 5 dispatch. Stage 6 gains a defined Stage 5→6 transition, a non-mandatory decline path, a terminal checkpoint after the Process Record is delivered, and canonical terminal-acknowledgement vocabulary (`finish`/`end`/`done`/`confirm`) that sets pipeline global state to `completed`. All five pipeline surfaces (`academic-pipeline/SKILL.md`, `agents/pipeline_orchestrator_agent.md`, `agents/state_tracker_agent.md`, `references/pipeline_state_machine.md`, `references/process_summary_protocol.md`) carry whole-file sha256 content locks in `scripts/check_pipeline_boundary_semantics.py` (66 mutation tests). | Any byte change to a locked surface fails CI until the pinned hash is updated in the same commit | | Cross-model handoff envelope (v3.17.0) | 🤖 (dispatch layer) | Design freeze (1), Final editorial decision (3), DA critique (3) | Canonical `[CROSS-MODEL-HANDOFF v1]` envelope + normative Python grammar (`scripts/cross_model_handoff.py`) for the #523 owner→dispatcher→owner transport path. Malformed envelope/result → `[CROSS-MODEL-ERROR]` → outcome `unavailable`, never a fabricated judgment; agreement → mechanical fill with no owner re-invocation; divergence → re-invoke the owner with minimum context. `scripts/check_cross_model_handoff_contract.py` pins the contract across all five surfaces. | `ARS_CROSS_MODEL` unset stays byte-equivalent; malformed transport degrades to `unavailable`, never silently treated as a deliverable | ### 7.1 CI workflow enforcement classes (#755) The files under `.github/workflows/` are often described collectively as CI gates, but they enforce at four different strengths (origin: ISO/IEC 42001-spirit gap assessment, finding T-5). Classes: **Blocking** — a failure on the guarded event must be fixed (or explicitly bypassed) before proceeding · **Advisory** — the audited condition warns, never fails · **Administrative** — produces work items, audits nothing · **Post-push detection** — triggered by a tag push that has already happened; nothing in GitHub Actions can reject a push after the fact, so a failure is a remediation signal, not prevention (the three tag-only workflows are conventionally *called* release gates; their stop-power is the maintainer acting on the failure). One GitHub Actions subtlety the Trigger column accounts for: an unfiltered or paths-only `push:` trigger ALSO matches tag pushes — GitHub does not evaluate `paths` filters for tags — so `spec-consistency`, `command-invariants`, and `freshness-check` additionally run on every `v*` tag push, where their failures are post-push detection exactly like the three tag-only workflows. | Workflow | Trigger | What it checks | Class | Bypass | |---|---|---|---|---| | `spec-consistency.yml` | push (all branches **and tags**) + PR | the full lint/pytest battery: spec surfaces, contracts, content locks, the pytest manifest | Blocking | none | | `pytest.yml` | PR + push to main, both path-filtered (scripts/tests/contracts/config, adapter references, `bibliography_agent`) | adapter + script test suite | Blocking | none | | `command-invariants.yml` | push (path-filtered for branch pushes; **also every tag push**) + PR (all) | SessionStart announce list matches the command inventory; plugin-version ↔ CHANGELOG lockstep; command frontmatter `name` validation | Blocking | none | | `repository-hygiene.yml` | PR targeting main + push to main | gitleaks secret scan | Blocking | none | | `eval-harness.yml` | PR + push to main, both path-filtered (scoring/generation surfaces + gold sets) | eval gold-set thresholds (aggregate + per-class) | Blocking on `pull_request` events only; report-only on push | `[eval-regression-acknowledged]` in the PR body + ≥1 open tracking-issue URL in this repo | | `test-count-monotonic.yml` | PR targeting main | collected test count must not drop | Blocking | `[skip-test-count]` in the PR body (justification requested, not machine-validated) | | `pr-closes-issue.yml` | PR targeting main | PR body references an issue via an auto-close keyword | Blocking | `[skip-closes-check]` in the PR body (justification requested, not machine-validated) | | `changelog-covers-merges.yml` | PR targeting main; the job runs only when the head branch is `release/**` | every release-worthy merge since the last tag is documented in CHANGELOG | Blocking (ordinary merges rely on the manual CONTRIBUTING fallback) | none | | `platform-port-reminder.yml` | PR targeting main | new top-level directory → platform-ports policy reminder | Advisory (one `::warning::`, always exits 0; the merge decision stays with the maintainer) | none | | `freshness-check.yml` | weekly schedule + push (two-file path filter for branch pushes; **also every tag push**) + manual dispatch | PRISMA-trAIce snapshot staleness | Advisory for staleness (warns on stderr, exits 0); malformed protocol metadata is a hard failure | none | | `harness-retirement-monthly.yml` | monthly schedule + manual dispatch | opens the monthly prompt-debt audit issue | Administrative | none | | `defer-label-gate.yml` | tag push `v*` | open `defer:` issues must be closed or relabelled | Post-push detection | `[skip-defer-check]` in the tagged commit message | | `release-cooldown.yml` | tag push `v*` | paces consecutive release tags | Post-push detection | `[skip-cooldown]` in the commit/tag message | | `tag-version-match.yml` | tag push `v*` | re-runs the full version-consistency lint at the tag | Post-push detection | none | Count, honestly stated: **14 workflows — 8 blocking on at least one event class, 2 advisory, 1 administrative, 3 post-push detection.** Inventory sync, the count line, and the bypass tokens are pinned by `scripts/check_workflow_classification.py`; the class *semantics* — whether a row honestly describes its workflow's behavior — stay owned by code review, mirroring the degradation-registry posture. ## 8. ARS Evolution Timeline ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## 9. Skill Modes | Skill | Modes | |---|---| | `deep-research` v2.12.1 | full, quick, socratic, review, lit-review, three-way-scan, fact-check, systematic-review (8) | | `academic-paper` v3.3.1 | full, plan, outline-only, revision, revision-coach, abstract-only, lit-review, format-convert, citation-check, disclosure, rebuttal-audit (11) | | `academic-paper-reviewer` v1.11.1 | full, re-review, quick, methodology-focus, guided, calibration (6) | | `academic-pipeline` v3.23.0 | orchestrator (delegates to sub-skill modes) + `resume_from_passport=` (v3.6.3 — resume a prior pipeline run from a Material Passport reset boundary; no flag required to invoke. The producing session must have set `ARS_PASSPORT_RESET=1` to emit boundary entries.) + `ARS_CLAIM_AUDIT=1` (v3.8 — opt-in Stage 4→5 L3 claim-faithfulness audit gate; default OFF) + v3.9.4 temporal verification advisory layer (M1 timeline_extraction_agent + M2 5-pass verifier at Phase 4→5 + M3 IRON RULE + M6 first-party Crossref/pdftotext) | | `sr-screener` v1.0.0 | protocol, quick, pilot, ta-screen, ft-screen, adjudicate, audit, report (8) | --- ### CONTROL AVAILABILITY (`docs/CONTROL_AVAILABILITY.md`) # Control Availability by Install Channel **Purpose.** ARS documentation names several controls — the write-scope guard, the citation-verification gate, mandatory checkpoints, the tools allowlist. Which of those actually operate depends on *how you installed ARS*. This page maps mechanism × channel in one place, so a user evaluating an integrity claim can see whether it holds in their channel. The linked source documents remain authoritative for each fact. **Origin.** ISO/IEC 42001-spirit gap assessment ([`audits/iso42001-spirit-gap-assessment-2026-08-17.md`](../audits/iso42001-spirit-gap-assessment-2026-08-17.md), finding T-6, [#757](https://github.com/Imbad0202/academic-research-skills/issues/757)). Transparency here is one of this repo's distilled operating principles (with informative anchors to ISO/IEC 42001) — not an ISO-mandated artifact. ## Install channels | Channel | Documented in | Channel-wide limitation | |---|---|---| | **Plugin** — Claude Code plugin install | [SETUP Method 0](SETUP.md#method-0-claude-code-plugin-v370-recommended-for-claude-code-cli--ide-users) | None — the reference channel. | | **Skills copy** — skill folders copied into a project's `.claude/skills/` or the global `~/.claude/skills/` | [SETUP Method 1](SETUP.md#method-1-as-project-skills-recommended) | Nothing wired by the plugin manifest runs (note 3). | | **Repo clone** — Claude Code run inside a clone of this repository | [SETUP Method 2](SETUP.md#method-2-as-a-standalone-project) | Nothing wired by the plugin manifest runs (note 3). | | **Cowork** — skills uploaded to Claude Cowork (desktop) | [SETUP Method 3](SETUP.md#method-3-claude-cowork-desktop) | Each skill runs standalone: no Task-tool subagent dispatch, so the coordinated pipeline — and its staged checkpoints — does not run as designed. | | **claude.ai Project** — repo attached to a claude.ai Project as retrievable knowledge | [SETUP Method 4](SETUP.md#method-4-use-with-claudeai-web) | Read-only knowledge: Claude can read and cite the skill bodies, but nothing executes — no activation, routing, hooks, scripts, or orchestration. (The Method 4a upload path is documented but not recommended; see SETUP § Method 4a.) | | **Claude Science** — skills imported via "Import from GitHub" | [SETUP Method 5](SETUP.md#method-5-claude-science-import-v3140) | Methodology layer only; Claude Code-specific machinery does not transfer, and Claude Science substitutes its own agent system (details: SETUP Method 5). Imports are point-in-time snapshots. | | **Pi port** — community-maintained wrapper for the Pi coding agent | [`pi/README.md`](../pi/README.md) | Two documented boundaries: the wrapper itself supplies no agent isolation or orchestration — an installed Pi orchestration capability is used when available, otherwise roles run sequentially (degraded execution, disclosed, not independent multi-agent review) — and no Claude hooks (write-scope enforcement stays prompt-level). `/ars-pi-doctor` reports what the local environment supplies. | ## Availability matrix Legend: **Active** = operates as documented · **Conditional** = operates only under the noted conditions, with a defined degraded state otherwise · **Absent** = does not operate in this channel. Read down your channel's column: a claim about a mechanism holds only where its row says Active — or Conditional with the linked note's conditions met — after applying your channel's channel-wide limitation above. | Mechanism | Plugin | Skills copy | Repo clone | Cowork | claude.ai Project | Claude Science | Pi port | |---|---|---|---|---|---|---|---| | Methodology layer (the five skills' `SKILL.md` protocols) | Active | Active | Active | Active | Conditional | Active | Active ⁽¹⁰⁾ | | Skill auto-routing (trigger keywords → skill activation) | Active | Active | Active | Active | Absent | Conditional | Conditional | | Cross-skill routing discipline (routing core: route directly or ask which workflow, #133 / #892) | Conditional ⁽⁸⁾ ⁽⁹⁾ | Conditional ⁽⁹⁾ | Active ⁽⁹⁾ | Conditional ⁽⁹⁾ | Absent | Conditional ⁽⁹⁾ | Conditional ⁽⁹⁾ | | `/ars-*` slash commands | Active ⁽¹⁾ | Absent | Absent | Absent | Absent | Absent | Conditional | | SessionStart announce, update reminder, and compaction handoff reminder | Conditional ⁽⁸⁾ | Absent ⁽³⁾ | Absent ⁽³⁾ | Absent | Absent | Absent | Absent | | Write-scope guard (`PreToolUse` hook) | Conditional ⁽²⁾ | Absent ⁽³⁾ | Absent ⁽³⁾ | Absent | Absent | Absent | Absent | | Plugin agents with tools allowlist (#514) ⁽⁴⁾ | Active | Absent ⁽³⁾ | Absent ⁽³⁾ | Absent | Absent | Absent | Absent | | Subagent orchestration (Task-tool multi-agent dispatch) | Active | Active | Active | Absent | Absent | Absent | Conditional | | Python-backed opt-in features (repo `scripts/`) | Conditional ⁽⁵⁾ | Conditional ⁽⁵⁾ | Conditional ⁽⁵⁾ | Absent | Absent | Absent | Conditional ⁽⁵⁾ | | Cross-model verification (consent-gated second model) | Conditional ⁽⁶⁾ | Conditional ⁽⁶⁾ | Conditional ⁽⁶⁾ | Absent | Absent | Absent | Absent | | Prompt-level checkpoints and integrity gates | Active ⁽⁷⁾ | Active ⁽⁷⁾ | Active ⁽⁷⁾ | Conditional | Absent | Conditional | Conditional | | sr-screener dual blinded screening (two reviewer subagents + adjudicator) | Active ⁽¹⁰⁾ | Conditional ⁽¹⁰⁾ | Conditional ⁽¹⁰⁾ | Absent ⁽¹⁰⁾ | Absent | Absent ⁽¹⁰⁾ | Absent ⁽¹⁰⁾ | CI-side checks (mutation-tested lints, content locks, changelog gates) are deliberately not a matrix row: they run in this repository's GitHub Actions, protecting the published artifact all channels ship from, and never run on a user machine — identical for every channel. They do not all enforce at the same strength or fire on every change: the per-workflow classification (blocking / advisory / administrative / post-push detection, with triggers and bypass tokens) is [ARCHITECTURE.md §7.1](ARCHITECTURE.md#71-ci-workflow-enforcement-classes-755). The machine-readable index of the suite's runtime graceful-degradation mechanisms is [`shared/contracts/degradation_registry.json`](../shared/contracts/degradation_registry.json). ## Notes 1. Plugin installs namespace commands as `/academic-research-skills:ars-`; a bare `/ars-` alias also works on recent Claude Code versions — the exact minimum version and the older-version behavior are documented in SETUP Method 0 (#633). 2. The write-scope guard needs a **real Python interpreter** and a `bash` to run its launcher; missing either produces a *defined degraded state*, never a block — see the [environment degradations](#environment-degradations-within-a-channel) table. The guard is optional subagent hardening; core skills are unaffected when it is inactive (README Requirements). It fences a Bucket A role only when that role runs as a subagent dispatched under its own agent type, which today means the plugin agents `synthesis_agent` and `research_architect_agent`; roles that run inline in the main session, the default for most, are not fenced (#973). 3. Hooks and plugin agents are wired by the **plugin manifest** (`hooks/hooks.json`, `agents/`, resolved via `CLAUDE_PLUGIN_ROOT`). A skills-copy or repo-clone install does not wire them: the `PreToolUse` write-scope guard and the SessionStart announce do not run, and agent dispatch uses the in-skill prompt templates without the frontmatter tools allowlist. A user may wire the hook into their own Claude Code settings manually, at which point the note-2 conditions apply. 4. The allowlist's canonical contents live in the three plugin agents' frontmatter, pinned in CI by `scripts/check_tools_allowlist.py` — this page deliberately does not restate the list. 5. These features (e.g. the citation-verification gate CLI, the revision token-conservation checker, the submission-package verifier, the PDF read preflight, the cache commands — a non-exhaustive, growing set) shell out to Python scripts at the repository root (`scripts/`, `shared/`), not inside the four skill folders. They need (a) a real Python interpreter and (b) the repo checkout present: automatic for the plugin channel (the plugin root is the repo snapshot) and repo clones; for a skills-copy install, keep the original clone — the copied skill folders alone cannot run them. On Pi, they work if Python and the repo are present (`pi/README.md`). The pipeline's run ledger (`scripts/run_ledger.py`, #887) has the same conditions; where they are not met, the orchestrator says that no ledger is kept. 6. Note-5 conditions, plus a transport: provider API credentials and `curl` for the general transports, or — for the citation-only calls — a Codex CLI ChatGPT-subscription login (`ARS_CROSS_MODEL_TRANSPORT=codex`). All transports sit behind the same boundary defined in [`shared/cross_model_verification.md`](../shared/cross_model_verification.md): the user's **explicit consent per session** — the `ARS_CROSS_MODEL` environment variable is configuration, not consent. Unset, the feature is invisible and makes zero network calls. 7. The MANDATORY checkpoints, integrity gates, and IRON RULE constraints are **prompt-level, trust-based controls with audit trails**, executed by the session model following the skill instructions — not coercive runtime enforcement. Documented overrides require recorded reasoning, and final integrity responsibility stays with the human researcher (see the [gap assessment §3](../audits/iso42001-spirit-gap-assessment-2026-08-17.md)). This row says the *instructions* are present and active in the channel, nothing stronger. 8. The SessionStart hook is launched through `bash`, so on Windows it needs Git Bash (the same PowerShell limitation as the guard launcher); its update-reminder path additionally needs `curl`, stays silent on any failure, and is disabled entirely by `ARS_UPDATE_CHECK=0` (SETUP Method 0). After a compaction or a resume, the announce adds one sentence asking the session to run the pipeline's handoff check against the run ledger (#887); the sentence is a reminder and runs no check itself. 9. The routing core (Routing Discipline Steps 0-3 and the #133 anti-pattern; canonical file [`shared/references/routing_core.md`](../shared/references/routing_core.md)) decides whether a request goes straight to a skill or the user is first asked which workflow they want. Claude Code loads this repository's `.claude/CLAUDE.md` only for sessions started inside the checkout, so the core also rides in the SessionStart announce and in each skill's `SKILL.md` (#892). **Plugin:** before any skill loads, through the announce (note-8 conditions), and again once a skill loads. **Repo clone:** from `.claude/CLAUDE.md` at session start, and from `SKILL.md`. **Skills copy:** at session start from the project's `.claude/CLAUDE.md` when it carries the merged ARS text, as SETUP Method 1 instructs for a project install (a merged copy does not update with ARS). **Skills copy without that text (a global install, or the step skipped), Cowork, Claude Science, Pi port:** only after a skill has loaded, from its `SKILL.md`: a request that no skill picks up gets no routing guard, and a clarifying question can come only after a skill call. As with note 7, these are prompt-level instructions, not runtime enforcement. The #133 routing fixtures were run with the plugin loaded, in sessions started inside and outside the checkout. Outside it, Claude Fable 5.1 still honored a mid-message `[direct-mode]` token in some sessions after the routing core gained a sentence against it (#892), single cells of other fixtures failed (#960), and fixture 05 passed on both models only under two scoring readings ([`CALIBRATION_LOG.md`](../tests/fixtures/issue_133_routing/CALIBRATION_LOG.md)). The other channels are covered only by the copy check (`scripts/check_routing_core_sync.py`). 10. sr-screener's dual review needs subagents plus its own Python scripts, which live inside the skill folder (`sr-screener/scripts/`, standard library only), so a copied skill folder carries them. **Plugin:** the `screening_reviewer_agent` (tools: Read, Grep) is wired from `agents/`. **Skills copy / repo clone:** active once `sr-screener/agents/screening_reviewer_agent.md` is copied into `.claude/agents/` and named in `screening_config.json`; with an empty `agent_type` the default workflow subagent runs the reviewers, which works but carries more tools than Read and Grep. **Cowork, Claude Science:** no Task-tool subagent dispatch of this skill's reviewer, so only `quick` mode runs, disclosed as single-reviewer triage (one model reading twice is not two independent reviewers). **Pi port:** the community-maintained wrapper does not yet package this skill. ## Environment degradations within a channel Independent of install channel, the write-scope guard has documented degraded states. None of them *introduces* a block: launcher failure paths resolve to **pass-through**, and a missing or unusable `timeout` binary is a bounding-mechanism swap under which the guard keeps operating normally — its real decisions (including deny) still apply, with only an overrun resolving to pass-through. The guard is an optional hardening layer, and a broken guard must not lock a user out of their own files (maintainer decision recorded in `hooks/run_guard.sh`; user-facing summary in the README Requirements bullet). This table is a convenience summary, not a second authority: the indexed rows are the `write_scope_guard_*` mechanisms in [`shared/contracts/degradation_registry.json`](../shared/contracts/degradation_registry.json) (#769), whose anchors into `hooks/run_guard.sh` are lint-pinned, and which also cover two states this table omits — guard subprocess misbehaves or the launcher fails internally (→ pass-through), and the documented multi-megabyte payload edge (an accepted, untested case with no pinned outcome): | Condition | Behavior | |---|---| | No real Python found (Git Bash / POSIX shell present) | Guard silently no-ops (pass-through); core skills unaffected. On Windows, the 0-byte Microsoft Store `python3` stub is rejected, not mistaken for Python. | | Windows without Git Bash | Claude Code falls back to PowerShell, which cannot run the `.sh` launcher: guard inactive, and the `PreToolUse` hook logs an error per call (accepted degradation — noisy, never blocking). | | No usable `timeout` binary (none on PATH, or one that rejects the GNU `timeout s cmd` form, such as the Windows built-in `timeout.exe` under Git for Windows) | Portable background-watchdog fallback with the same wall-clock bound and the same pass-through-on-overrun posture. Before #970 a foreign `timeout` was trusted, and the guard passed every write through. | --- ### DATA FLOWS (`docs/DATA_FLOWS.md`) # Data Flows: What Leaves the Machine, What Is Stored, and for How Long **Purpose.** ARS touches the network in a small number of places and persists a small number of local stores. Each is documented at its own feature page; this file is the single user-facing map — one row per network touchpoint and one row per local store. **Origin.** ISO/IEC 42001-spirit gap assessment ([`audits/iso42001-spirit-gap-assessment-2026-08-17.md`](../audits/iso42001-spirit-gap-assessment-2026-08-17.md), finding T-7, [#758](https://github.com/Imbad0202/academic-research-skills/issues/758)). Transparency here is one of this repo's distilled operating principles (with informative anchors to ISO/IEC 42001) — not an ISO-mandated artifact. ## Scope This page covers the network calls and stores that **ARS's own scripts** perform. The boundaries around that scope: - **The Claude session itself is not on this map.** Everything you type, every file the session model reads, and any web search/fetch the model performs while executing the research skills travels over your Claude platform connection under your Anthropic account settings. That path exists with or without ARS and is governed by the platform, not by this repo. - **Maintainer/evaluation harnesses are not user paths.** A few repo scripts exist only for maintainers running measurements (e.g. `scripts/dispatch_e4_panel.py` and `scripts/dispatch_calibration_panel.py` through `claude -p` — the latter also sends the operator's own `ANTHROPIC_API_KEY` to the Anthropic API's `GET /v1/models` as a zero-cost credential preflight before the first billed call — `scripts/run_review_criteria_constructive_value.py` through the Codex CLI, `scripts/check_ranking_lift.py` through `gh api`). They send content through locally authenticated CLIs when a maintainer invokes them, are never triggered by any user-facing feature, and are deliberately excluded from the touchpoint tables below. - **Nothing here publishes.** No ARS component is designed to submit, post, or upload your work anywhere autonomously — every network row below is a *lookup* (sending queries or citation metadata to read public indexes), an *update check*, or an *explicitly consented* verification call. This is the same first-party scope boundary stated in [`POSITIONING.md`](../POSITIONING.md), and it inherits that page's qualifier: a scope boundary and review criterion, not a runtime guarantee. ## Network touchpoints ### The citation-verification gate (four bibliographic indexes) The deterministic citation-existence gate (#182) fires at the Stage 2.5 / 4.5 integrity gates or on standalone `verify_passport.py` / `verify_citation` calls, cache-through by default. Each resolver sends the same payload class — identifiers (DOIs/arXiv ids) and title query strings of your references; author/year metadata is used locally for matching, not transmitted — and all four work without any account or key; keeping the gate key-free is a deliberate reproducibility choice. Off switch: don't run script-backed verification; prompt-only modes make no calls. | Resolver | Endpoint | Credentials (optional) | Per-index note | |---|---|---|---| | `scripts/semantic_scholar_client.py` | `api.semanticscholar.org` | `S2_API_KEY` (raises rate limit) | | | `scripts/openalex_client.py` | `api.openalex.org` | `OPENALEX_API_KEY` | Polite-pool email sent if you configure one | | `scripts/crossref_client.py` | `api.crossref.org` | none (polite email optional) | Polite-pool email rides the `User-Agent` header | | `scripts/arxiv_client.py` | `export.arxiv.org` | none | ToU-aligned ≥3 s pacing | ### Everything else | Touchpoint | When it fires | What is sent | Recipient | Credentials | Off switch | |---|---|---|---|---|---| | Chinese-literature resolver (`scripts/chinese_literature_client.py`, #595) | **Callable client only (no CLI wrapper) — deliberately NOT wired into the four-index verification gate** (the same reproducibility choice) | DOI prefixes / DOIs of the works you resolve; the PubMed path additionally sends your NCBI contact email (required by E-utilities) and bibliographic search coordinates (journal/volume/page, author/year) | `doi.org` (RA lookup + resolution), `hdl.handle.net`, NCBI E-utilities (`eutils.ncbi.nlm.nih.gov`) | NCBI email required for the PubMed path; NCBI API key optional (does not relax the client's polite pacing) | Don't call the client | | Claim-standing discovery adapters (`scripts/claim_standing_discovery.py`, #655) | Only under an explicit consent-bound query plan; the evaluation substrate is offline by default | **Claim-derived search query strings** (they can derive from unpublished claims — hence the consent gate and the per-transmission ledger) + date filters | The same four indexes above | none (fixed User-Agent; the resolver clients' env keys are not consumed) | No consent → no calls; every transmission is ledgered | | Timeline bootstrap (`scripts/bootstrap_timeline_yaml.py`, v3.9.4 opt-in) | Standalone CLI you invoke to seed `timeline.yaml` from a literature corpus; `--dry-run` makes no calls | DOIs of your corpus entries | `api.crossref.org` | none (needs the optional `requests` package; absent, lookups are treated as an outage) | Don't run it, or pass `--dry-run` | | Cross-model verification transport | Only when `ARS_CROSS_MODEL` is configured **and** you give explicit per-session consent — the env var is configuration, not consent ([`shared/cross_model_verification.md`](../shared/cross_model_verification.md)) | Up to **manuscript content**: integrity-gate samples, the blind Devil's-Advocate critique input, the full paper for the consent-gated Reviewer-2 seat, checkpoint judgments | `api.openai.com`, `generativelanguage.googleapis.com`, or the OpenAI-compatible base URL you set (`ARS_OPENAI_COMPAT_BASE_URL`, e.g. DeepSeek) | `OPENAI_API_KEY` / `GOOGLE_AI_API_KEY` / `ARS_OPENAI_COMPAT_API_KEY` (required for this feature) | Leave `ARS_CROSS_MODEL` unset (zero calls), or decline consent per session | | Codex audit wrapper (`scripts/run_codex_audit.sh`) | Only when a human, CI step, or SubagentStop hook invokes it (same-session in-LLM invocation is forbidden by its header contract); `--dry-run` writes and sends nothing | Audit prompts containing the **deliverable and supporting files' contents** | OpenAI, through the local Codex CLI login | Codex CLI auth | Don't invoke it | | ChatGPT-subscription citation transport (`scripts/cross_model_codex_transport.py`, #630) | Only when `ARS_CROSS_MODEL_TRANSPORT=codex`; citation-integrity calls **only** (never DA / reviewer / judgment calls) | One reference's citation text plus its exact `citation_context` — the sentence where it is cited, which can contain unpublished manuscript text — sent through the local Codex CLI in a read-only sandbox with an auth-only ephemeral home | OpenAI, through your Codex CLI ChatGPT login | Codex CLI subscription login | Unset the transport selector; any other value fails visibly, no fallback | | SessionStart update check (`scripts/ars_update_check.sh`, #544; plugin installs only) | At most one *successful* check per 24 h (a failed attempt writes no state and may retry at the next session start); 3-second total ceiling, redirects followed, silent on failure | **No user data** — fetches a public `plugin.json` and compares versions | `raw.githubusercontent.com` by default; `ARS_UPDATE_CHECK_REMOTE_URL` overrides the endpoint | none | `ARS_UPDATE_CHECK=0` | | Manual smoke tests (`scripts/cross_model_smoke_test.sh`, `scripts/cross_model_smoke_test_codex.sh`) | Only when you run them by hand; CI never does | Public sample citation metadata | The provider under test | The provider's key / login | Don't run them | Notes: - **Agent-side lookups use the same indexes.** Outside the script clients, the research/verification agents (`bibliography_agent`, `source_verification_agent`, `integrity_verification_agent`) query the same four indexes at ingest and verification time, following the per-index API protocol docs under `deep-research/references/`. Same payload class (citation metadata), same endpoints; executed through the session's tooling. - Resolver clients never log or echo credentials; polite-pool emails and API keys are stripped from error messages (see each client's redaction comments). - CI runs against checked-in synthetic fixtures (`scripts/test_transport_fixture_citation_gate.py`); no CI job performs live resolver or provider calls. ## Local stores | Store | Path | Content | Lifetime | How to delete | |---|---|---|---|---| | Citation-verification cache | `~/.cache/ars/verification.db` (override: `ARS_VERIFICATION_CACHE_PATH`) | Per-citation resolver outcomes (SQLite) | 90-day TTL per entry — expiry means a cache miss, not deletion; expired rows persist on disk until overwritten by a later re-verification, invalidated, or the file is deleted. Staleness advisory after `ARS_CACHE_STALE_ADVISORY_DAYS` (default 30), live re-validation via `ARS_CACHE_REVALIDATE=1` | `/ars-cache-invalidate ` per key, or delete the file | | Update-check state | `~/.cache/ars/` (override: `ARS_UPDATE_CHECK_STATE_DIR`) | A state label (`UP_TO_DATE` / `UPDATE_AVAILABLE`) plus installed and remote version strings | Re-fetched when older than 24 h | Delete the directory; `ARS_UPDATE_CHECK=0` stops new writes | | Retraction-status cache (`scripts/retraction_status.py`) | A caller-supplied SQLite path (no default location) | DOI-keyed resolver observations with timestamps | No automatic expiry; observations older than the 30-day threshold are marked stale, not deleted | Delete the file | | Material Passport + project ledgers | The passport path **you** name per run (never a hidden global location) | Your research content: corpus entries, the run-wide constraints you confirmed, in your words (`standing_constraints[]`, #927), references an integrity gate judged `NOT_FOUND` (`excluded_sources[]`, #936), read-attestation ledger, reset boundaries, compliance history, claim-standing consent receipts / transmission ledgers, rejection logs | No TTL — user-owned project files | Delete with your project | | Run ledger (`scripts/run_ledger.py`, #887) | `_run_ledger.yaml` beside the passport you name, with an empty peer lock file named after it | Your initial instructions and checkpoint answers **in your exact words**, checkpoint questions, partly collected answers, receipts for tool steps, progress counters, and hashes of transient input files. The whole file is never put into a dispatch; a dispatch that relays one decision quotes that decision's words, and the session model reads the file over your Claude platform connection | No TTL — user-owned project file | Delete the file and its lock file; during a run, deleting it means earlier decisions are asked again. Do not name it as a supporting file for the Codex audit wrapper, which sends the contents of the files it is given | | Evidence source folder (#933) | The folder the pipeline names for each integrity gate: `_evidence_sources/stage-2.5/` or `stage-4.5/` beside the passport you name, or run-local storage outside the repository when the run has no passport file | One `.txt` file per cited source the integrity check matched a claim against: the source text it retrieved during verification, which can be a whole article. The checkpoint reads only the files its evidence rows name, to check the displayed excerpts against them; the files are not put into other dispatches | No TTL — user-owned project files; a correction round at the same gate rewrites the files for the sources it re-checks and rebuilds every row that names them | Delete the folder; the evidence rows of that gate's report then cannot be rendered again, and the gate's verdict is unchanged. Do not name the files as supporting files for the Codex audit wrapper, which sends the contents of the files it is given | | sr-screener work and output folders (`sr-screener/scripts/`) | The `--work` and `--out` folders **you** name per review | Parsed and de-duplicated records (titles, abstracts, keywords), batch files, generated run scripts, reviewer decisions with reasons, merged decisions, QC candidates, your overrides; outputs: screening log, RIS groups, PRISMA counts, methods draft, `literature_corpus[]` file (no abstracts). The scripts make no network calls; reviewer subagents read the batch files over your Claude platform connection | No TTL — user-owned project files | Delete the folders | | Codex transport working dir | A per-call `ars-codex-citation-*` temporary directory | Auth-only ephemeral home, empty working root | Removed automatically when the call returns | Automatic | The tunable numbers above (TTLs, thresholds) are documented where they are set — [`SETUP.md`](SETUP.md) § Citation verification cache and § Optional environment flags are the user-facing authority for changing them. The suite itself contains no telemetry and no analytics endpoint, and requires no account with the ARS project; the inventory above is exhaustive for this repository's own scripts as of this revision. A CI lint (`scripts/check_data_flows.py`) holds one direction of that mechanically — a script that gains a *direct* network import, or a shell script that gains a `curl`, fails CI until it has a row here. Paths that reach the network indirectly — through a spawned CLI (the codex transport rows), or through the session's own tooling following a protocol doc — are held by review, not by the lint. ## Related - [`SECURITY.md`](../SECURITY.md) — data exfiltration and credential leakage are explicitly in scope for vulnerability reports; anything beyond this map is report-worthy. - [`THIRD_PARTY.md`](../THIRD_PARTY.md) — community projects around ARS and their own policies (this map covers the core suite only). - [`docs/SETUP.md`](SETUP.md) — installation, environment flags, and the canonical home of the cache and cross-model configuration this map summarizes. - [`shared/cross_model_verification.md`](../shared/cross_model_verification.md) — the consent boundary and provider table for the cross-model rows. - `docs/CONTROL_AVAILABILITY.md` (lands with PR #768) — which of these code paths even exist in your install channel. --- ### PERFORMANCE (`docs/PERFORMANCE.md`) # ARS Performance Notes > **Recommended models: Claude Fable 5.1 or Claude Opus 5.5** (at the time of writing) with **Max plan** (or equivalent configuration). Claude Code starts on Opus 5.5 by default; select Fable 5.1 with `/model fable` (in Claude apps gateway sessions, where `fable` means Fable 5, use `/model claude-fable-5-1`). Neither model is better at everything: the Opus 5.5 system card puts Opus 5.5 ahead of Fable 5.1 on every row of its capability summary table, but behind it on DRACO deep research and OfficeQA document reasoning (pp. 174, 187, 208). Current Claude models use adaptive thinking; you no longer set a fixed thinking budget. > > **Set the effort level on Opus 5.5.** Claude Code starts Opus 5.5 at `medium` effort and Fable 5.1 at `high`. On the card's two research benchmarks, DRACO deep research and WANDR wide search, Fable 5.1 at its `high` default scores 86.5 and 66.7; Opus 5.5 scores 86.7 / 71.3 at `xhigh`, 83.9 / 62.8 at `medium`, and 72.5 / 31.2 at `low` (card pp. 187-188). Before a heavy run on an Opus 5.5 session (`/ars-full`, `/ars-reviewer`, `/ars-revision-coach`, or a `deep-research` run started in plain language), make sure effort is at least `high`: if it is lower, raise it with `/effort high` or `/effort xhigh` (a level typed after `/effort` also becomes your saved default for that model), and keep `xhigh` or `max` if you already use them. Avoid `low`. ARS deliberately does not pin effort in command frontmatter: a pin would also lower a user who chose `xhigh` or `max`. > > **Give third-party text to ARS as a file when you can.** Opus 5.5 is more likely than earlier models to act on instructions hidden in text you paste into your own message (card §6.5.1). In the card's test, the same text arriving through a tool, such as a file read, was never acted on (0 of 105). Claude Code marks large pastes for the model only in sessions that fetch feature flags; sessions with telemetry disabled, Claude apps gateway sessions, and most third-party-provider sessions do not fetch them ([pasted text](https://code.claude.com/docs/en/terminal-config#how-claude-treats-pasted-text)). > > The full academic pipeline (10 stages) consumes a **large amount of tokens** — a single end-to-end run can exceed 200K input + 100K output tokens depending on paper length and revision rounds. Budget accordingly. > > Individual skills (e.g., `deep-research` alone, or `academic-paper-reviewer` alone) consume significantly less. ## Estimated token usage by mode | Skill / Mode | Input Tokens | Output Tokens | Estimated Cost | |---|---|---|---| | `deep-research` socratic | ~30K | ~15K | ~$0.60 | | `deep-research` full | ~60K | ~30K | ~$1.20 | | `deep-research` systematic-review | ~100K | ~50K | ~$2.00 | | `academic-paper` plan | ~40K | ~20K | ~$0.80 | | `academic-paper` full | ~80K | ~50K | ~$1.80 | | `academic-paper-reviewer` full | ~50K | ~30K | ~$1.10 | | `academic-paper-reviewer` quick | ~15K | ~8K | ~$0.30 | | **Full pipeline (10 stages)** | **~200K+** | **~100K+** | **~$4-6** | | + Cross-model verification | +~10K (external) | +~5K (external) | +~$0.60-1.10 | *Estimates based on a ~15,000-word paper with ~60 references. Actual usage varies with paper length, revision rounds, and dialogue depth. Costs measured on Opus 4.x at Anthropic API pricing as of April 2026 — treat as order-of-magnitude anchors under newer models rather than exact quotes.* > **2026-09 list-price re-derivation.** At 2026-09 list prices, the full-pipeline token figures above (~200K in + ~100K out) come to roughly **~$7** per run on Claude Fable 5.1 (US$10 / US$50 per million input / output tokens) and **~$2.80** on Claude Opus 5.5 (US$4 / US$20, with cache reads at US$0.20 per million; Opus 5.5 system card p. 180), before any cache discount. This is arithmetic on the token columns, not a re-measurement: no pipeline run has been re-timed on either model. Both models always reason (thinking cannot be disabled), so dialogue-heavy modes may spend more output tokens than the Opus 4.x rows recorded, and a higher effort level spends more still. > **v3.11 citation verification (#182).** The deterministic citation-existence gate calls external bibliographic APIs (Semantic Scholar / OpenAlex / Crossref / arXiv), not the LLM, so it adds **no Claude token cost** to the figures above — only network latency on first lookup. The persistent SQLite cache (`~/.cache/ars/verification.db`, 90-day TTL) means each paper is verified once and reused across drafts; a re-run over an already-cached bibliography does no network work. See [SETUP](SETUP.md#citation-verification-cache-v3.11-182). ## Recommended Claude Code settings | Setting | What it does | How to enable | Docs | |---|---|---|---| | **Agent Team** (optional) | Enables `TeamCreate` / `SendMessage` tools for manual multi-agent coordination. **ARS's internal parallelization does not require this flag** — skills spawn subagents via the built-in `Agent` tool directly. Only useful if you want to manually orchestrate persistent team workflows across sessions. | Set `CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1` (research preview) | Experimental feature — no stable docs yet | | **Auto mode** (recommended) | Auto-accepts most tool actions so long pipeline runs keep moving, while a server-side classifier still blocks actions that escalate beyond what you asked for (e.g. production deploys, force-pushes or direct pushes to main, data exfiltration). Explicit ask rules and classifier blocks can still prompt. The middle ground between manual approval and zero checks. | Launch with `claude --permission-mode auto` (when available), or set `"permissions": { "defaultMode": "auto" }` in `~/.claude/settings.json`; verify the active mode after startup (research preview) | [Permission modes](https://code.claude.com/docs/en/permission-modes) | | **Skip Permissions** | Skips routine tool-use confirmations with no safety checks. Faster than auto mode but removes all guardrails. Intended for ephemeral isolated sandboxes without internet access, not real development machines. | Launch with `claude --dangerously-skip-permissions` (equivalent to `--permission-mode bypassPermissions`) | [Permission modes](https://code.claude.com/docs/en/permission-modes) | > **⚠️ Choosing a mode**: For most unattended pipeline runs, auto mode is the recommended setting. It keeps most long runs moving while a classifier gates dangerous escalations, though ask rules and classifier blocks can still prompt. Auto mode is a research preview: it does not guarantee safety and is not a replacement for human review on sensitive operations, and its behavior may change. Skip Permissions removes that safety net entirely and should only be used in an isolated sandbox without internet access, where you are comfortable with Claude executing file reads, writes, and shell commands with no checks. ### v3.7.0 Plugin agents and model routing When ARS is installed as a Claude Code plugin (`/plugin install academic-research-skills`), three downstream worker agents are exposed as plugin-shipped subagents: `synthesis_agent`, `research_architect_agent`, and `report_compiler_agent`. Each declares `model: inherit` in its frontmatter, which means they run under the **dispatching session's model** rather than a pinned floor: - An Opus session running the full pipeline gets Opus agents, preserving the integrative depth those agents were designed for. - A Sonnet session gets Sonnet agents, matching the cost/latency profile of the parent run. - The agents never silently fall back to Haiku — `inherit` resolves through the parent session's model, which is itself gated by the project policy of "no Haiku for ARS runs." Since #514 (shipped in #521), each of the three also carries a pinned tools allowlist in the same frontmatter — `tools: Read, Write, Edit, Grep, Glob`, no shell and no network fetch — so dispatch-time capability is least-privilege; the exact value is CI-locked by `scripts/check_tools_allowlist.py` (#524). This means **plugin-agent token costs track the per-mode estimates above unchanged** (with `ARS_MODEL_TIERING` unset); there is no separate plugin agent surcharge or discount, because dispatched agents inherit the same model the parent run already pays for. Under `ARS_MODEL_TIERING=economy`, plugin-exposed execution-type agents (e.g. `report_compiler_agent`) follow the tiering rule instead — one tier below the session model, floor Opus-class (see `shared/model_tiering.md`). If you change the main session model mid-pipeline (e.g., downshift to Sonnet for a long revision pass), the next agent dispatch picks up the new floor automatically. Other ARS agents (`bibliography_agent`, `literature_strategist_agent`, etc.) are not plugin-exposed in v3.7.0; they remain in-skill prompt templates that the main session executes inline, with no separate model routing layer **by default**. The opt-in `ARS_MODEL_TIERING` switch (#517) adds a dispatch-time routing rule on top: when a tiering direction applies to a role, the session dispatches it as a subagent pinned to the target tier (inline roles included — dispatch-as-subagent is the mechanism); with the flag unset, this paragraph describes behavior unchanged. See `shared/model_tiering.md`. Wider plugin-agent coverage is deferred to a future release. **Tiering with Fable 5.1 and Opus 5.5 (2026-09).** The tier ladder puts Fable 5.1 above Opus 5.5 because that is the vendor's lineup order, not because Fable 5.1 scores higher on every task (see the model note at the top of this page). On an Opus 5.5 session, `quality-boost` sends the checkpoint calls to Fable 5.1 at 2.5 times the per-token list price, for a benefit ARS has not measured; raising the session effort is the cheaper first step. On a Fable 5.1 session, `economy` sends execution-type agents to the Opus class, which Claude Code resolves to Opus 5.5 on the Anthropic API (some providers resolve the `opus` alias to an older Opus model); the quality cost of that trade is also unmeasured on ARS. ## Long-running session management The full academic pipeline is designed for human-in-the-loop execution, with mandatory user confirmation at every stage. In practice, a full run often spans hours to days — longer than Anthropic's prompt cache TTL (5 minutes). Two consequences: 1. **Cache misses between checkpoints are normal.** When a stage checkpoint pauses longer than 5 minutes, the next stage reads its context uncached. This is an unavoidable cost of human-paced pipelines. 2. **Cross-session resume relies on Material Passport.** ARS does not maintain its own orchestrator state between sessions. To resume in a new session, paste your Material Passport YAML back; the orchestrator reads `compliance_history[]` and stage completion markers to locate your breakpoint. ### v3.6.2 Sprint Contract reviewer cost (always-on for `full` / `methodology-focus`) The Schema 13 sprint contract gate splits each reviewer agent's run into Phase 1 (paper-content-blind, commits scoring plan) + Phase 2 (paper-visible review). For modes that ship templates (`full` panel 5 + `methodology-focus` panel 2), each reviewer therefore costs roughly two LLM turns instead of one. Reserved modes (`re-review` / `calibration` / `guided` / `quick`) keep pre-v3.6.2 behaviour. | Skill / Mode | Effect on tokens | Notes | |---|---|---| | `academic-paper-reviewer full` | ~+30-40% input + small output bump per reviewer × 5 reviewers | Each reviewer reads the contract template + paper metadata in Phase 1, then full paper in Phase 2 | | `academic-paper-reviewer methodology-focus` | Same shape, panel 2 | Two reviewers (Journal-Fit Reviewer + methodology) each run two phases | | Synthesizer (always one) | +~2-3K input | Reads contract + reviewer outputs to run three-step mechanical protocol | Empirical measurement pending real review runs at scale. The two-phase shape is non-optional for the gated modes; treat as fixed overhead, not a tunable. ### v3.4.0 compliance agent cost Adding the mode-aware `compliance_agent` to Stage 2.5 and Stage 4.5 increases full-pipeline SR tokens by approximately: | Skill / Mode | Input Tokens | Output Tokens | Estimated Cost | |---|---|---|---| | `deep-research systematic-review` (2.5 only) | +~5–8K | +~3–5K | +~$0.15 | | Full pipeline SR (2.5 + 4.5) | +~10–15K | +~5–8K | +~$0.30 | | `academic-paper full` (pre-finalize) | +~3–5K | +~2–3K | +~$0.08 | These are on top of the existing per-skill costs in the table above (same 15,000-word / 60-reference basis; see footnote on line 23). Cross-model verification costs (if enabled) are unchanged. ### v3.6.3 Passport reset boundary (opt-in) When `ARS_PASSPORT_RESET=1` is set, every FULL and MANDATORY checkpoint becomes a context-reset boundary. The intended workflow is: 1. Run a stage to FULL checkpoint in session A. 2. Copy the `[PASSPORT-RESET: hash=, stage=, next=]` tag from the checkpoint notification. 3. Start a fresh Claude Code session (session B) and paste `resume_from_passport=`. Optional overrides: `resume_from_passport= stage= mode=`. 4. Session B loads only the passport ledger; no replay of session A's turns. The orchestrator locates the matching `kind: boundary` entry, appends a `kind: resume` entry to consume it, and continues. The resumed stage is determined by: a `stage=` CLI override if supplied, else the matched option's `next_stage` when the boundary carries a `pending_decision` (the orchestrator re-prompts the user first), else the recorded `next` field. `next` MAY be `null` when all decision branches terminate. **When reset beats continuation:** - Long pipelines where session A has accumulated >100K input tokens of context that the next stage does not actually need. - `systematic-review` mode runs where stage independence is cleanly defined by the Material Passport. - Any case where you hit the 5-minute prompt-cache TTL mid-pipeline; a reset lets the next stage start fresh instead of paying a cache miss on a bloated context. **When continuation still wins:** - Short pipelines (< 30K input tokens end-to-end). - Stages with implicit in-session state that the passport does not capture (e.g., a Socratic dialogue branch the user wants to keep warm). - When the flag is OFF, continuation is the unchanged pre-v3.6.3 default. **Passport file location convention:** By default, the orchestrator looks for the passport file in `./passports//` or matching `./material_passport*.yaml` relative to the current working directory. Resolving the hash to a passport file on disk is the integrator's responsibility; the orchestrator loads whichever passport the enclosing tool provides. See §"Passport file location convention" above for the `./passports//` default. The resume command only defines the hash and optional stage/mode overrides: ``` resume_from_passport= [stage=] [mode=] ``` There is no path syntax on the resume command itself. Custom passport locations are configured in the project's `CLAUDE.md` or handled by the integrator's tooling before the orchestrator is invoked. **Empirical token savings:** measurement pending a real `systematic-review` run with instrumentation. This section will be updated with observed token deltas once available; until then, no numeric claim is made. See [`../academic-pipeline/references/passport_as_reset_boundary.md`](../academic-pipeline/references/passport_as_reset_boundary.md) for the full protocol. ## Literature corpus ingestion (v3.6.4+) The Material Passport `literature_corpus[]` field is populated by user-written adapters, not ARS itself. Three reference adapters ship with v3.6.4: `scripts/adapters/folder_scan.py`, `scripts/adapters/zotero.py`, `scripts/adapters/obsidian.py`. See [`scripts/adapters/README.md`](../scripts/adapters/README.md) for how to run them and how to write your own. ### Performance posture - Adapters run out-of-band (before an ARS session, not during). Their runtime is the user's problem, not ARS's. - Adapters must be deterministic: re-running on identical input produces byte-identical output modulo timestamps. - `literature_corpus[]` entries are sorted by `citation_key`; rejections are sorted by `source`. - Adapter output size grows linearly with corpus size. A 500-entry Zotero library typically produces a passport of ~300 KB YAML. ARS consumers should lazy-load when the corpus is large. ### Ingestion-layer boundaries - Does not ingest PDFs, extract text, or run OCR. - Does not call the Zotero Web API, Notion API, or any live service. - Does not fetch paywalled content or use user credentials to access institutional libraries. These boundaries are deliberate and reflect the ARS data-layer decision: ARS is a writing/review-layer framework; corpus integration stays in user-owned code. Users who want API-based live-sync adapters are expected to write them themselves, using the three reference adapters as starting points. ### Consumer-side integration As of v3.6.5, two Phase 1 literature agents read `literature_corpus[]` via the **corpus-first, search-fills-gap** flow: `deep-research/agents/bibliography_agent.md` and `academic-paper/agents/literature_strategist_agent.md`. Both consumers follow the same five-step shared flow and four Iron Rules (Same criteria / No silent skip / No corpus mutation / Graceful fallback on parse failure). Search Strategy reports gain a PRE-SCREENED reproducibility block that enumerates included / excluded / skipped corpus entries with F3 zero-hit and F4 provenance reporting. Consumer integration is presence-based — auto-engages when the passport carries a non-empty `literature_corpus[]` and parses cleanly; parse failures fall back to external-DB-only flow with a `[CORPUS PARSE FAILURE]` surface. See [`academic-pipeline/references/literature_corpus_consumers.md`](../academic-pipeline/references/literature_corpus_consumers.md) for the full consumer protocol. `citation_compliance_agent` corpus integration is deferred (target version TBD post-v3.8). ### v3.6.5 corpus consumer cost (presence-gated) When the Material Passport carries a non-empty `literature_corpus[]`, Phase 1 reads scale with corpus size. The PRE-SCREENED block emit itself is prompt-layer (effectively free); the LLM cost is Step 1 pre-screening — applying the current Inclusion / Exclusion criteria to each corpus entry's `title` (always present) and any populated optional fields (`abstract` / `tags`). | Corpus size | Step 1 pre-screening (per consumer) | Notes | |---|---|---| | Empty / absent | 0 | External-DB-only flow runs unchanged | | ~50 entries (typical Zotero subset) | +~3-5K input + ~1-2K output | Title + abstract scan | | ~200 entries | +~10-15K input + ~3-5K output | Title-only scan dominates; abstract scan only when populated | | ~500 entries (large library) | +~25-40K input + ~8-12K output | Consider trimming the corpus before passport emit | Step 2 search-fills-gap reduces external-DB cost when `uncovered_topics` is small (case A), which can offset Step 1 cost. Empirical net delta pending real systematic-review run instrumentation; until then, no aggregate numeric claim is made. Parse failures cost roughly one short turn (parse + emit `[CORPUS PARSE FAILURE]` + fall back). --- ### PERFORMANCE.Zh TW (`docs/PERFORMANCE.zh-TW.md`) # ARS 效能說明 > **建議模型：Claude Fable 5.1 或 Claude Opus 5.5**（以撰寫當下為準），搭配 **Max plan**（或同等配置）。Claude Code 預設使用 Opus 5.5；要用 Fable 5.1，請執行 `/model fable`（在 Claude apps gateway 裡，`fable` 指的是 Fable 5，請改用 `/model claude-fable-5-1`）。兩個模型沒有誰全面勝出：Opus 5.5 system card 的能力總表上，每一列 Opus 5.5 都高於 Fable 5.1；但在 DRACO 深度研究與 OfficeQA 文件推理兩項測驗，Opus 5.5 較低（card 第 174、187、208 頁）。現行 Claude 模型採用 adaptive thinking，不需要手動指定 thinking budget。 > > **用 Opus 5.5 時請調整推理強度（effort）。** Claude Code 讓 Opus 5.5 以 `medium` 起跑，Fable 5.1 則以 `high` 起跑。card 有兩項研究測驗：DRACO（深度研究）與 WANDR（廣域搜尋）。Fable 5.1 在預設的 `high` 分別得 86.5 與 66.7；Opus 5.5 在 `xhigh` 為 86.7／71.3，`medium` 為 83.9／62.8，`low` 為 72.5／31.2（card 第 187-188 頁）。在 Opus 5.5 上跑重度任務前（`/ars-full`、`/ars-reviewer`、`/ars-revision-coach`，或直接用文字啟動的 `deep-research`），請確認推理強度至少是 `high`：低於 `high` 時，用 `/effort high` 或 `/effort xhigh` 調高（在 `/effort` 後面直接打強度，也會存成這個模型之後的預設）；原本就用 `xhigh` 或 `max` 的話維持不變。避免用 `low`。ARS 刻意不在指令 frontmatter 固定推理強度：固定之後，自己選了 `xhigh` 或 `max` 的使用者也會被壓回來。 > > **第三方文字盡量存成檔案再交給 ARS。** 你貼進自己訊息裡的文字如果藏了指令，Opus 5.5 比先前的模型更容易照做（card §6.5.1）。在 card 的測試中，同一段文字若經由工具進來（例如讀檔），105 次中 0 次被照做。Claude Code 只在會抓取 feature flag 的 session 標記大段貼上文字；關閉遙測的 session、Claude apps gateway 的 session，以及多數第三方雲端平台的 session 都不會抓取（[貼上文字說明](https://code.claude.com/docs/en/terminal-config#how-claude-treats-pasted-text)）。 > > 完整學術 pipeline（10 階段）會消耗**大量 token** — 單次完整執行可能超過 200K 輸入 + 100K 輸出 token，視論文長度和修訂輪數而定。請依預算斟酌使用。 > > 單獨使用個別 skill（如只用 `deep-research` 或 `academic-paper-reviewer`）的消耗明顯較少。 ## 各模式 Token 消耗估算 | Skill / 模式 | 輸入 Token | 輸出 Token | 估算費用 | |---|---|---|---| | `deep-research` socratic | ~30K | ~15K | ~$0.60 | | `deep-research` full | ~60K | ~30K | ~$1.20 | | `deep-research` systematic-review | ~100K | ~50K | ~$2.00 | | `academic-paper` plan | ~40K | ~20K | ~$0.80 | | `academic-paper` full | ~80K | ~50K | ~$1.80 | | `academic-paper-reviewer` full | ~50K | ~30K | ~$1.10 | | `academic-paper-reviewer` quick | ~15K | ~8K | ~$0.30 | | **完整 pipeline（10 階段）** | **~200K+** | **~100K+** | **~$4-6** | | + 跨模型驗證 | +~10K（外部）| +~5K（外部）| +~$0.60-1.10 | *以 ~15,000 字論文、~60 篇引用為基準估算。實際消耗隨論文長度、修訂輪數、對話深度而異。費用以 Opus 4.x 實測、Anthropic API 2026 年 4 月定價計算；換用更新模型時請當成數量級參考，不是精確報價。* > **2026-09 牌價換算。**以 2026-09 的牌價換算，上表完整 pipeline 的 token 數（約 200K 輸入 + 100K 輸出）每次約為：Claude Fable 5.1 **~$7**（每百萬輸入／輸出 token 各 US$10／US$50）；Claude Opus 5.5 **~$2.80**（各 US$4／US$20，cache 讀取每百萬 US$0.20；Opus 5.5 system card 第 180 頁）。兩者都尚未計入 cache 折扣。這是用 token 欄位算出來的數字，不是重新實測：沒有任何 pipeline 在這兩個模型上重跑計時。兩個模型都一律會推理（thinking 無法關閉），所以對話密集的模式可能比 Opus 4.x 列多用一些輸出 token；推理強度越高，用得越多。 > **v3.11 引用查驗（#182）。** 確定性引用存在性 gate 呼叫的是外部書目 API（Semantic Scholar / OpenAlex / Crossref / arXiv），不是 LLM，因此**不增加上表的 Claude token 成本**——只在首次查詢時有網路延遲。持久化 SQLite cache（`~/.cache/ars/verification.db`，90 天 TTL）讓每篇論文只查驗一次、跨草稿重用；對已 cache 的書目重跑不做任何網路請求。見 [SETUP](SETUP.zh-TW.md#引用查驗-cachev3.11182)。 ## 建議 Claude Code 設定 | 設定 | 功能說明 | 啟用方式 | 官方文件 | |---|---|---|---| | **Agent Team**（選用） | 啟用 `TeamCreate` / `SendMessage` tools 做手動多 agent 協作。**ARS 內部平行化不需要這個 flag** — skills 透過內建 `Agent` tool 直接 spawn subagent。僅在你想手動跨 session 協作持久 team 時有用。 | 設定 `CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1`（研究預覽） | 實驗性功能 — 尚無穩定文件 | | **Auto 模式**（建議） | 自動接受大多數工具動作，讓長時間 pipeline 大幅減少中斷；同時由伺服器端 classifier 擋下超出你請求範圍的危險動作（例如部署到 production、force-push 或直接 push main、資料外洩）。明確的 ask 規則與 classifier 攔截仍可能跳出確認。是「手動逐項確認」與「完全不檢查」之間的折衷。 | 啟動時加上 `claude --permission-mode auto`（若可用），或在 `~/.claude/settings.json` 設定 `"permissions": { "defaultMode": "auto" }`；啟動後確認當前模式（研究預覽） | [Permission modes](https://code.claude.com/docs/en/permission-modes) | | **Skip Permissions** | 跳過例行的工具使用確認，且不做任何安全檢查。比 auto 模式更快，但移除所有護欄。設計用途是用完即拋、無網路連線的隔離沙箱，不適合真實開發機器。 | 啟動時加上 `claude --dangerously-skip-permissions`（等同 `--permission-mode bypassPermissions`） | [Permission modes](https://code.claude.com/docs/en/permission-modes) | > **⚠️ 模式選擇**：大多數無人值守的 pipeline，建議使用 auto 模式。它讓長時間執行大幅減少中斷，同時由 classifier 擋下超出你請求範圍的危險動作，但 ask 規則與 classifier 攔截仍可能跳出確認。auto 模式是研究預覽：它不保證安全，也不能取代敏感操作的人工審查，且行為可能變動。Skip Permissions 則完全移除這層安全網，僅應在無網路連線的隔離沙箱中使用，且你要確定可以接受 Claude 在無檢查的情況下執行檔案讀寫與 shell 指令。 ### v3.7.0 Plugin agent 與模型路由 當 ARS 以 Claude Code plugin 方式安裝（`/plugin install academic-research-skills`）時，會把三個下游 worker agent 暴露為 plugin-shipped subagent：`synthesis_agent`、`research_architect_agent`、`report_compiler_agent`。三個 agent frontmatter 都標 `model: inherit`，意思是它們**繼承派工 session 的模型**而非寫死特定 floor： - Opus session 跑完整 pipeline 時 agent 是 Opus，保留這三個 agent 設計的整合深度。 - Sonnet session 取得 Sonnet agent，跟主 session cost / latency 對齊。 - Agent 永遠不會默默掉到 Haiku — `inherit` 走的是主 session 模型，主 session 本身又被「ARS 全程不用 Haiku」政策守住。 自 #514（於 #521 出貨）起，這三個 agent 的 frontmatter 同時帶固定的 tools 白名單——`tools: Read, Write, Edit, Grep, Glob`，無 shell、無網路抓取——派工當下即是最小權限；白名單內容由 `scripts/check_tools_allowlist.py`（#524）在 CI 鎖定。 意涵：**plugin agent 的 token 成本完全跟著上表各模式估算走，沒有額外加減**（`ARS_MODEL_TIERING` 未設定時）。dispatched agent 跟主 session 同一個模型，主 session 已經付的成本沒有再多一層 plugin agent 收費。設定 `ARS_MODEL_TIERING=economy` 時，plugin 暴露的 execution 型 agent（如 `report_compiler_agent`）改走分層規則——比 session model 低一階、樓地板 Opus 級（見 `shared/model_tiering.md`）。如果 pipeline 中途換模型（例如 revision pass 改用 Sonnet 省成本），下一輪 agent 派工自動跟上。 其他 ARS agent（`bibliography_agent`、`literature_strategist_agent` 等）在 v3.7.0 不暴露為 plugin agent；它們仍是 in-skill prompt template，由主 session 內聯執行，**預設**沒有獨立的模型路由層。Opt-in 的 `ARS_MODEL_TIERING`（#517）在其上加了一層 dispatch 時的路由規則：當分層方向適用於某角色時，session 會把該角色以子代理形式派發、鎖定目標層級（內聯角色也一樣——「派發為子代理」正是其機制）；flag 未設定時，本段描述的行為完全不變。見 `shared/model_tiering.md`。更廣的 plugin agent 覆蓋留到後續版本。 **Fable 5.1 與 Opus 5.5 的分層（2026-09）。**分層階梯把 Fable 5.1 排在 Opus 5.5 之上，是因為原廠的產品排序如此，不是因為 Fable 5.1 每項任務都比較強（見本頁開頭的模型說明）。在 Opus 5.5 session 上，`quality-boost` 會把檢核點的呼叫送到 Fable 5.1，每個 token 的牌價是 2.5 倍，換來的效益 ARS 沒有量測過；先調高 session 的推理強度比較便宜。在 Fable 5.1 session 上，`economy` 會把執行型 agent 送到 Opus 級，Claude Code 在 Anthropic API 上會解析成 Opus 5.5（部分雲端平台會把 `opus` 別名解析成較舊的 Opus）；這筆取捨對品質的影響，ARS 同樣沒有量測過。 ## 長時間 session 管理 完整 pipeline 設計為 human-in-the-loop，每個階段都需使用者確認。實務上一次完整執行會跨越數小時到數天，遠長於 Anthropic 的 prompt cache TTL（5 分鐘）。兩項結果： 1. **階段間 cache miss 是常態。** 當 stage checkpoint 停留超過 5 分鐘，下一階段會以未快取狀態讀取 context。這是 human-paced pipeline 不可避免的成本。 2. **跨 session 續跑依賴 Material Passport。** ARS 本身不跨 session 保留 orchestrator 狀態。要在新 session 續跑，把 Material Passport YAML 貼回即可；orchestrator 讀取 `compliance_history[]` 與階段完成標記定位中斷點。 ### v3.6.2 Sprint Contract 審稿成本（`full` / `methodology-focus` 模式必跑） Schema 13 sprint contract 把每個 reviewer agent 切成 Phase 1（不見論文、先承諾評分準則）+ Phase 2（看論文做審稿）兩階段。已 ship template 的兩個模式（`full` panel 5 + `methodology-focus` panel 2）下，每位 reviewer 約等於跑兩個 LLM turn。保留模式（`re-review` / `calibration` / `guided` / `quick`）維持 pre-v3.6.2 行為。 | Skill / 模式 | Token 影響 | 備註 | |---|---|---| | `academic-paper-reviewer full` | 每位 reviewer 約 +30-40% input + 小幅 output × 5 位 | Phase 1 讀 contract template + 論文 metadata；Phase 2 讀完整論文 | | `academic-paper-reviewer methodology-focus` | 同上 shape，panel 2 | Journal-Fit Reviewer + methodology 兩位 reviewer 各跑兩階段 | | Synthesizer（固定一個）| +~2-3K input | 讀 contract + 各 reviewer 輸出，跑三步機械協議 | 實測待真實大規模審稿後校準。兩階段架構是 gated mode 的不可選 overhead，不是 tunable。 ### v3.4.0 compliance agent 成本 在 Stage 2.5 與 Stage 4.5 加上 mode-aware `compliance_agent` 會讓 SR 全 pipeline token 多出： | Skill / 模式 | 輸入 Token | 輸出 Token | 估算費用 | |---|---|---|---| | `deep-research systematic-review`（僅 2.5）| +~5–8K | +~3–5K | +~$0.15 | | 全 pipeline SR（2.5 + 4.5）| +~10–15K | +~5–8K | +~$0.30 | | `academic-paper full`（pre-finalize）| +~3–5K | +~2–3K | +~$0.08 | 以上為既有 per-skill 成本之上的額外增量（與上表共用 15,000 字 / 60 篇引用基準，見上表下方 footnote）。跨模型驗證成本（若啟用）維持不變。 ### v3.6.3 Passport 重置邊界（opt-in） 設定 `ARS_PASSPORT_RESET=1` 後，每個 FULL 與 MANDATORY checkpoint 變成 context 重置邊界。預期工作流程： 1. Session A 跑完一個 stage 到 FULL checkpoint。 2. 從 checkpoint 通知抄下 `[PASSPORT-RESET: hash=, stage=, next=]` tag。 3. 開新的 Claude Code session（session B），貼入 `resume_from_passport=`。支援可選覆蓋：`resume_from_passport= stage= mode=`。 4. Session B 只讀 passport ledger，不重播 session A 的對話。Orchestrator 找到相符的 `kind: boundary` entry，append 一個 `kind: resume` entry 完成消費，然後繼續。繼續的 stage 由以下順序決定：使用者在 resume 指令附上 `stage=` 時以其為準，否則當 boundary 帶 `pending_decision` 時由 orchestrator 先重新詢問使用者再用對應選項的 `next_stage`，否則才採用記錄的 `next` 欄位。所有選項都終止時，`next` 可以是 `null`。 **何時重置比延續划算：** - 長 pipeline，session A 累積 >100K input token，下個 stage 不需要這些上下文。 - `systematic-review` 模式，stage 獨立性由 Material Passport 精確界定。 - 撞到 5 分鐘 prompt cache TTL：重置讓下個 stage 重新起算，不用在臃腫 context 上付 cache miss。 **何時延續仍然比較好：** - 短 pipeline（end-to-end < 30K input token）。 - Stage 有 in-session 隱含狀態、passport 沒帶的情況（例如使用者想保溫的 Socratic 對話分支）。 - Flag OFF 時，延續是不變的 pre-v3.6.3 預設。 **Passport 檔案位置規約：** Orchestrator 預設在目前工作目錄下尋找 `./passports//` 或 `./material_passport*.yaml`。將 hash 解析到磁碟上的 passport 檔案是整合方的責任，orchestrator 載入呼叫端工具提供的 passport。預設位置見上方 `./passports//` 規約。 Resume 指令只定義 hash 與可選的 stage/mode 覆蓋： ``` resume_from_passport= [stage=] [mode=] ``` Resume 指令本身沒有路徑語法。客製 passport 位置在專案的 `CLAUDE.md` 設定，或由整合方的工具在呼叫 orchestrator 前處理。 **實測 token 節省：** 尚待真實 `systematic-review` 搭配儀器化測量。取得實測資料後會回填本節。目前不做任何數值宣稱。完整協議見 [`../academic-pipeline/references/passport_as_reset_boundary.md`](../academic-pipeline/references/passport_as_reset_boundary.md)。 ## 文獻語料庫導入（v3.6.4+） Material Passport 的 `literature_corpus[]` 欄位由**使用者自行撰寫的 adapter** 產出，不是 ARS 本身。v3.6.4 附三個 reference adapter：`scripts/adapters/folder_scan.py`、`scripts/adapters/zotero.py`、`scripts/adapters/obsidian.py`。執行方式與自行撰寫 adapter 的指引見 [`scripts/adapters/README.md`](../scripts/adapters/README.md)。 ### 效能定位 - Adapter 在 ARS session 之外執行（跑 ARS 前跑）；執行時間由使用者自行負責，不進 ARS 的時間預算。 - Adapter 必須具備 determinism：同一份 input 重跑產出 byte-identical 輸出（時間戳除外）。 - `literature_corpus[]` 依 `citation_key` 排序；`rejection_log.rejected[]` 依 `source` 排序。 - Adapter 輸出大小與語料庫大小線性成長。500 筆 Zotero 書目約產出 300 KB 的 passport YAML。大型語料庫建議 ARS 消費端採 lazy load。 ### 導入層邊界 - 不讀 PDF 內容、不做文字抽取、不跑 OCR。 - 不呼叫 Zotero Web API、Notion API 或任何遠端服務。 - 不抓付費牆後內容、不用使用者憑證連線機構圖書館。 這些邊界是刻意的，反映 ARS 的 data-layer 定位：ARS 是 writing / review layer 的框架，語料整合留在 user-owned code。如需 API-based live-sync adapter，由使用者以三個 reference adapter 為起點自行撰寫。 ### 消費端整合 v3.6.5 起，Phase 1 兩個文獻 agent 透過 **corpus-first、search-fills-gap** 流程讀取 `literature_corpus[]`：`deep-research/agents/bibliography_agent.md` 與 `academic-paper/agents/literature_strategist_agent.md`。兩者走相同的五步流程與四條 Iron Rule（Same criteria / No silent skip / No corpus mutation / Graceful fallback on parse failure）。Search Strategy 報告新增 PRE-SCREENED 可重現區塊，列出已納入／排除／略過的 corpus entry，並含 F3 zero-hit 與 F4 provenance 報告。消費端啟動採 presence-based — passport 帶非空 `literature_corpus[]` 且解析成功時自動進入；解析失敗時 fallback 到 external-DB-only flow，並 surface `[CORPUS PARSE FAILURE]`。 完整 consumer 協定見 [`academic-pipeline/references/literature_corpus_consumers.md`](../academic-pipeline/references/literature_corpus_consumers.md)。`citation_compliance_agent` 的 corpus 整合留到 v3.6.6+。 ### v3.6.5 corpus consumer 成本（presence 觸發） Material Passport 帶非空 `literature_corpus[]` 時，Phase 1 讀取量隨 corpus 大小線性增長。PRE-SCREENED block 的 emit 本身屬 prompt-layer（成本可忽略）；LLM 成本來自 Step 1 pre-screening — 對每筆 corpus entry 套用當前 Inclusion / Exclusion 條件，比對 `title`（一定有）與已填的選填欄位（`abstract` / `tags`）。 | Corpus 規模 | Step 1 pre-screening（每位 consumer）| 備註 | |---|---|---| | 空 / 不存在 | 0 | external-DB-only flow 維持原樣 | | ~50 筆（典型 Zotero 子集）| +~3-5K input + ~1-2K output | title + abstract 掃描 | | ~200 筆 | +~10-15K input + ~3-5K output | title-only 掃描為主，abstract 視填充情況 | | ~500 筆（大型文獻庫）| +~25-40K input + ~8-12K output | passport emit 前考慮先精簡 corpus | Step 2 search-fills-gap 在 `uncovered_topics` 小（case A）時會降低 external-DB 成本，可部分抵銷 Step 1。淨效應實測待真實 SR run instrumentation 後校準；目前不下總體數字結論。Parse 失敗約一個短 turn 成本（parse + emit `[CORPUS PARSE FAILURE]` + fallback）。 --- ### RISK REGISTER (`docs/RISK_REGISTER.md`) # Risk Register: Risk → Control → Evidence Status → Residual Gap **Purpose.** The suite's known risks are each handled somewhere — the citation gate, the claim-drift guards, the consent surfaces, the capability matrix — but those mechanisms live in different files. This register is the single artifact that links each standing risk to its existing controls, the evidence status behind them, and what is still open, so that when a new mechanism ships there is one place to answer: which risk does this retire, and what remains? **Origin.** ISO/IEC 42001-spirit gap assessment ([`audits/iso42001-spirit-gap-assessment-2026-08-17.md`](../audits/iso42001-spirit-gap-assessment-2026-08-17.md), finding F-1, [#759](https://github.com/Imbad0202/academic-research-skills/issues/759)). This is the deliberately lightweight substitute for a formal AIMS risk process, which was assessed and rejected as disproportionate for a solo-maintained repo (assessment §6). Feasibility here is one of this repo's distilled operating principles (with informative anchors to ISO/IEC 42001) — not an ISO-mandated artifact. **Posture.** The register indexes, it never re-authors. Each row cites its controls where they are defined and mirrors its evidence status from the stage capability matrix ([`shared/contracts/capability/stage_capability_matrix.json`](../shared/contracts/capability/stage_capability_matrix.json), rendered at [STAGE_CAPABILITY_MATRIX.md](STAGE_CAPABILITY_MATRIX.md)); it never redefines a control or upgrades a status. Statuses use the matrix vocabulary (`DESIGNED` / `NOT_RUN` / `MEASURED` / `MIXED`). A status citing a named matrix row is lint-pinned to that row's recorded `behavioral_evidence.status` by `scripts/check_risk_register.py`; a status marked as asserted-here is a maintainer assertion with no matrix backing, and may only carry a no-measurement status: the matrix stays the sole authority for `MEASURED`/`MIXED`. An open issue in a Residual gap cell is the honest answer, not a defect being hidden: `NOT_RUN` means exactly that. Two authoring constraints keep this file checkable: a backtick code span containing a slash is treated as a repo path that must exist, and the citation phrasings are reserved for well-formed citations. ## Risks ### R1 — Hallucinated citations A generated reference does not exist, or a real reference is cited for a claim it does not support. - **Existing controls**: deterministic four-index existence gate (`scripts/verify_passport.py`, `scripts/verification_gate/__init__.py`) with opt-in terminal policies (`shared/contracts/passport/terminal_policies.schema.json`); locator-bearing citation emission (`scripts/check_v3_7_3_three_layer_citation.py`); Phase E claim verification for the supports-the-claim half; the passport list that keeps a reference an integrity gate judged `NOT_FOUND` off later writer dispatches (`shared/contracts/passport/excluded_source_entry.schema.json`, #936). - **Evidence status**: `NOT_RUN` (capability matrix row `retrieval.citation_existence_gate`); `NOT_RUN` (capability matrix row `integrity_check.claim_verification`). - **Residual gap**: no measured hallucinated-citation catch rate — that needs an independently-authored ground-truth set, not one derived from the gate's own reducer; the Claim Registry's semantic completeness is unknown by contract. No open issue schedules either measurement yet. The vendor evaluations of both supported session models point the same way. The Claude Fable 5.1 system card (§2.2.4) reports fabricated references as rare and misrepresented findings or conclusions as the residual error class that needs domain familiarity to catch; the Claude Opus 5.5 system card reports reliance on abstracts over full papers and partial checks described as full reads (mapped in `audits/harness-retirement-2026-09-opus-5-5.md` DM-015). Both move the weight of this row onto the supports-the-claim half — exactly the unmeasured claim-verification row above. ### R2 — Silent claim-strength drift in revision A revision round strengthens or weakens a claim without an authorizing roadmap item. - **Existing controls**: claim-strength ladder (`shared/references/claim_strength_ladder.md`); deterministic token conservation (`scripts/check_revision_token_conservation.py`); advisory Phase E6 drift check. - **Evidence status**: `MEASURED` (capability matrix row `revision.claim_drift_guard`). - **Residual gap**: the measured row covers a condensed guard-block prompt, not the shipped pipeline path as-wired, and licenses no causal claim; re-measurement after any change is a manual maintainer action. ### R3 — Indirect prompt injection via retrieved content Third-party text, whether an agent retrieves it or the user pastes it, carries instructions that an agent follows as if they came from the user. - **Existing controls**: the retrieved-content instruction/data boundary as a standing principle (#367), inlined verbatim in the deep-research retrieval agents; since #883, in the revision coach that ingests pasted reviewer comments and decision letters; and since #890, in twelve agents on the dispatch and passport-import paths (the pipeline orchestrator, the integrity gates, literature intake, reviewer Phase 0 and the editorial synthesis, systematic-review risk of bias, temporal extraction, and deep-research `review` mode) and in two prompts a model receives without the agent file, the claim-audit judge prompt and the cross-model devil's advocate prompt; since #894, in five agents that read third-party text through their own tool calls (the formatter, citation compliance, and the three citation emitters), in the single-reference verification prompt the integrity gates send to a cross-model verifier on the API route, and in the four `SKILL.md` files, which every install path loads when a skill runs (`scripts/check_instruction_data_boundary.py` lists each); the reviewer-side untrusted-materials rule (`academic-paper-reviewer/SKILL.md`) and its manuscript fence in the five panel agents (pinned by `scripts/check_reviewer_data_fences.py`); the offline structural probe (`scripts/run_indirect_prompt_injection_probe.py`), which by design never dispatches a model. - **Evidence status**: `DESIGNED` (asserted here; no capability-matrix row) — the probe checks structure, so no behavioral injection-resistance number exists. - **Residual gap**: live injection behavior unmeasured ([#675](https://github.com/Imbad0202/academic-research-skills/issues/675)); structural instruction/data isolation at the task-envelope boundary is design work ([#676](https://github.com/Imbad0202/academic-research-skills/issues/676)). The Claude Opus 5.5 system card reports a regression on the pasted channel: the model acts on instructions planted in text the user pasted into their own message, but acted on them in 0 of 105 attempts when the same text arrived as a tool result (§6.5.1; rates and product-side caveats in `audits/harness-retirement-2026-09-opus-5-5.md` DG-1). The #675 scenarios include pasted reviewer and committee comments. The prompt-level boundary, including the interim #890 and #894 extensions, is trust-based; its effect on this regression is unmeasured, and #676 stays open with its structural requirements unmet. The inventories, ranking, and placement are in `docs/design/2026-09-23-890-instruction-data-boundary-extension.md` (#890) and `docs/design/2026-09-24-894-instruction-data-boundary-tool-calls.md` (#894). Surfaces they leave uncovered: agents whose dispatch carries third-party text only as quotations inside artifacts from covered agents (for example `meta_analysis_agent` and `argument_builder_agent`); agents whose input is the user's own text or dialogue; the advisory collaboration-depth observer and monitoring agent; the cross-model result at the design-freeze checkpoint; requests a fallback model serves; the ChatGPT-subscription transport's instructions to the verifier; and the routing decision a session makes before it loads a `SKILL.md`. The #675 seed runs one generic guided prompt rather than any agent's prompt, so as seeded it evaluates none of these paths; a measured claim for a path needs a #675 scenario that loads that agent's assembled prompt. ### R4 — Unpublished-content exposure via cross-model transport A consented second-model call carries unpublished manuscript content to an external provider. - **Existing controls**: consent-gated transport with explicit per-call surfaces (`shared/cross_model_verification.md`); the single user-facing network map (`docs/DATA_FLOWS.md`); the contained, bounded subscription transport protocol. - **Evidence status**: `NOT_RUN` (asserted here; no capability-matrix row) — the deterministic envelope pieces are CI-pinned; behavioral compliance with the consent contract is unmeasured. - **Residual gap**: consent surfaces are prompt-contract layers whose behavior varies with the session model; the no-autonomous-publication line is a scope boundary and review criterion, not a runtime guarantee (`POSITIONING.md`). Provider-side safety monitoring may escalate a flagged conversation to human review at the provider (`shared/cross_model_verification.md` § Provider-side monitoring and safety interventions), so consent has to assume provider staff may read what is sent. ### R5 — Upstream model-version behavior drift A new session-model version silently changes behavior that older measurements described. - **Existing controls**: commit-frozen measurement rows re-measured per change; the matrix staleness rule (`stale_after_days`); periodic harness-retirement audits (`audits/harness-retirement-2026-09-opus-5-5.md` is the most recent completed report, run when Claude Opus 5.5 joined Claude Fable 5.1 as a supported session model; `audits/harness-retirement-2026-09-model-update.md` covers the Fable 5 → Fable 5.1 and GPT-5.6 Sol → GPT-6 Astra change). - **Evidence status**: `NOT_RUN` (asserted here; no capability-matrix row) — no measurement of cross-version drift itself exists. - **Residual gap**: measured rows are model- and time-specific by contract; re-running them on a model change is a manual maintainer action, not automated. The session model can also change mid-run without a version change: a classifier-triggered fallback moves a Claude Code session onto the fallback model until the user runs `/model`, with a transcript notice the user sees and the suite never reads (`shared/model_tiering.md`, the declared-model note under Resolving a tier at dispatch time; vendor specifics in `audits/harness-retirement-2026-09-opus-5-5.md` DM-005), so every "session model" in a provenance record is the declared model, not a per-call attestation. ### R6 — Correlated same-family model error Generator and reviewer share a model family, so both miss the same failures. - **Existing controls**: optional cross-model verification and the cross-model reviewer track (`shared/cross_model_verification.md`); the Review Panel Provenance block's single-family correlated-error caveat; the same-family calibration epistemic note. - **Evidence status**: `NOT_RUN` (capability matrix row `review.calibration`). - **Residual gap**: first measured reviewer error profile pending ([#653](https://github.com/Imbad0202/academic-research-skills/issues/653)); a second model is an error-detection control, not organizational independence. A second model is also only as observable as its outputs: the current OpenAI verifier's vendor reports that covert underperformance by that model would likely evade the vendor's own monitors (GPT-6 Astra system card §9.2.2.3), and ARS's typed evidence anchors bound what a verifier can assert, not what it withholds. ### R7 — User over-reliance / rubber-stamping The researcher approves AI output without reading it, and attestations become rubber stamps. - **Existing controls**: read-scope attestation as declaration-not-proof (`/ars-mark-read` requires an explicit scope); the override ladder with recorded reasoning (`shared/compliance_checkpoint_protocol.md`); the advisory collaboration-depth observer (`shared/collaboration_depth_rubric.md`). - **Evidence status**: `NOT_RUN` (asserted here; no capability-matrix row). - **Residual gap**: these are trust-based controls with audit trails, not coercive controls — final integrity responsibility stays with the human researcher, and no evaluation measures over-reliance itself. ### R8 — Venue-policy staleness Disclosure guidance cites a venue policy the venue has since changed. - **Existing controls**: disclosure mode with per-venue policy anchors (`academic-paper/references/disclosure_mode_protocol.md`). - **Evidence status**: `NOT_RUN` (asserted here; no capability-matrix row). - **Residual gap**: no live policy re-validation exists; the user must confirm against the venue's current text before submission. ### R9 — Dependency / supply-chain compromise A compromised or drifting third-party dependency alters suite behavior. - **Existing controls**: stdlib-first runtime scripts; a small declared dev-dependency manifest (`requirements-dev.txt`, lower-bound pins only, no lockfile); the plugin-agent tools allowlist (`scripts/check_tools_allowlist.py`); CI lints running repo-side in GitHub Actions. - **Evidence status**: `NOT_RUN` (asserted here; no capability-matrix row). - **Residual gap**: no SBOM or supplier-review program — assessed and rejected as disproportionate (assessment §6); dev dependencies carry no exact version or hash locking; dependency changes ride ordinary code review. ### R10 — Install-channel enforcement loss A user installs through a channel where documented enforcement never runs. - **Existing controls**: the per-channel availability matrix (`docs/CONTROL_AVAILABILITY.md`); the write-scope guard's disclosed degrade posture (`hooks/run_guard.sh`), indexed as five `write_scope_guard_*` rows in the degradation registry (`shared/contracts/degradation_registry.json`, [#769](https://github.com/Imbad0202/academic-research-skills/issues/769)); per-channel notes in `docs/SETUP.md`. - **Evidence status**: `NOT_RUN` (asserted here; no capability-matrix row) — availability is documented; loss-of-enforcement behavior is not separately measured. - **Residual gap**: what each channel loses is per-mechanism (see the availability matrix): prompt-level protocols survive in most non-plugin channels but are absent in claude.ai Projects, and hook enforcement is absent outside the plugin channel unless the user wires the hook into their own settings manually (matrix note 3). ### R11 — Fabricated or distorted checkpoint authority The orchestrating model treats something other than the researcher's own turn as a checkpoint decision — an automated message, a subagent's report, a template default, or its own paraphrase — or restates the researcher's decision to a subagent as a broader authorization than was given. - **Existing controls**: MANDATORY checkpoint templates that wait for an explicit user turn (`academic-pipeline/references/pipeline_state_machine.md`); the orchestrator's checkpoint-authority fidelity rule (`academic-pipeline/agents/pipeline_orchestrator_agent.md`); deterministic authorization inputs that bind an author choice to exact patch bytes (`scripts/revision_roadmap.py`, the #670 integrity-correction authorization); read attestations that are declared, never inferred (`/ars-mark-read`). - **Evidence status**: `NOT_RUN` (asserted here; no capability-matrix row) — the deterministic authorization inputs are CI-pinned; the prompt-level rule is not measured on any session model. - **Residual gap**: the failure class is vendor-documented, not ARS-measured (evidence mapped in `audits/harness-retirement-2026-09-model-update.md` G-1 for Fable 5.1 and `audits/harness-retirement-2026-09-opus-5-5.md` DM-013 for Opus 5.5); the prompt rule is trust-based. ### R12 — Handoff loss across compaction and subagent returns Context compaction or a subagent return drops a pending checkpoint decision, the researcher's exact words, a partly collected answer, a step's outcome, or a transient input, and the run continues as if its record were complete. Loss and fabrication fail differently: fabricating a decision is R11's risk; this row covers losing one. - **Existing controls**: the run ledger beside the passport and its deterministic report (`scripts/run_ledger.py`, schema `shared/contracts/passport/run_ledger.schema.json`); the orchestrator's run-ledger and handoff-check rules (`academic-pipeline/agents/pipeline_orchestrator_agent.md`) and the state machine's run-ledger rule (`academic-pipeline/references/pipeline_state_machine.md`); the SessionStart reminder after a compaction or resume (`scripts/announce-ars-loaded.sh`); the opt-in passport reset for MANDATORY decisions (`academic-pipeline/references/passport_as_reset_boundary.md`); the user-confirmed `standing_constraints[]` passport list that later dispatches quote (`shared/contracts/passport/standing_constraint_entry.schema.json`, #927). - **Evidence status**: `NOT_RUN` (asserted here; no capability-matrix row) — how the report reads a ledger is CI-pinned by six synthetic scenarios, the `show` read that prints only entries before a break by synthetic tests, and the handoff-check block the report renders in English and Traditional Chinese by line-exact tests (#898, `scripts/test_run_ledger.py`); whether the orchestrator writes the entries, reads the ledger only through the script, and inserts the block unchanged is not measured on any session model. - **Residual gap**: anything lost before its entry is written cannot be recovered; the hashes catch accidental damage, not deliberate edits, a lost tail, or a restored older copy of the ledger (`docs/design/2026-09-23-887-handoff-integrity-design.md`, section 6); the collaboration observer still needs the original turns; outside the plugin channel the reminder runs only if the user wires the hook (`docs/CONTROL_AVAILABILITY.md`, note 3). --- ### ROADMAP V3.11 (`docs/ROADMAP-v3.11.md`) # v3.11 Roadmap (draft) **Status:** DRAFT — candidate scoping inferred from open issues, not a commitment. **Date:** 2026-06-01 (just after v3.10.0). **How this was built:** the repo has no milestone mechanism; this draft is synthesized from the open-issue set as of v3.10.0 ship. Every item links its source issue. Priorities and final selection are the owner's call — this is a menu, not a plan of record. > Theme candidate: **provenance + calibration hardening across artifact types** — almost > every mature open item EXTENDS existing machinery (claim decomposition, field-norm > calibration, citation verification, instruction/data boundary) rather than introducing a > greenfield system, which keeps v3.11 scope tractable. The alternative framing is a > narrower **citation-integrity release** anchored on the one P0 (#182). --- ## Tier 0 — ship-early, dependency-free (do regardless of theme) The Co-Scientist design-lesson docs are S-scope, pure documentation, no code, no dependencies — and **#223 (L4) updates the paper-derived issue template**, which every future paper-derived sub-issue depends on. Landing these first costs little and unblocks the boundary discipline for everything else. - **#220** Co-Scientist L1 — hidden-ranking red line (design-lesson doc) - **#221** Co-Scientist L2 — feedback-propagation user-approval gate (design-lesson doc) - **#222** Co-Scientist L3 — transferable vs non-transferable mechanisms (transfer matrix) - **#223** Co-Scientist L4 — control-plane ownership (who may write / rank / route) + issue-template update - Parent epic: **#219** --- ## Tier 1 — mature spec, no blocking deps, high leverage ### Citation integrity (the one self-assessed P0) - **#182** — Deterministic citation verification gate (Crossref/DOI/arXiv/PubMed lookup + source-text anchor; three status columns `lookup_verified` / `anchor_present` / `reviewer_approved`; a draft with any `lookup_verified: false` cannot reach "ready for human review"). **Scope: L** (new resolver module + caching + schema + tests). The #184 eval harness already left a hook ("verify_citation has not shipped — reconcile when it does"), so this closes a known forward-reference. **Highest leverage in the citation cluster.** - **Note (raised in review):** the gate FLAGS unverified citations (they stay visible — it is not hidden culling, so it does not cross the Co-Scientist L1 hidden-ranking red line). But "a draft cannot reach ready-for-review" is a *deterministic block*, and ARS philosophy is "the human decides at every gate." The #182 spec should state explicitly whether this block is hard (auto) or surfaced-for-human-override, so it reads as a flag-and-surface gate, not an AI culling decision. This is a #182 spec clarification, not a roadmap blocker. - **#250** (historical accepted limitation; no action item) — the static gold corpus has no stable, first-party-verifiable real work unmatched across all four resolvers, so its `false` class is exercised only by fabricated tuples. This is a representativeness limit, not a missing title-match regression: #431/#432 fixed and directly test resolver-client behavior. ### Kim et al. peer-review failure modes (epic #217) All four sub-issues have full specs naming exact files/lines. - **#213** — Decompose multi-part claims before citation judgment (adds Step 0 sub-claim split + `PARTIAL` verdict; addresses the 76% partial-evidence error class). **M** - **#214** — Sub-claim inventory before consensus aggregation in `editorial_synthesizer` (inventory primary key `Weakness` → `sub_claim`). Independently shippable from #213 (no hard blocking dependency), **but coordinate the `sub_claim` shape/vocabulary with #213** — both introduce sub-claim-level reasoning at different layers (citation vs synthesis) and will drift if specced separately. Land them in one pass or share the schema. **M** - **#215** — Field-norm severity calibration (domain_reviewer must cite external literature for field-norm claims; DA gains `field_norm_boundary`; calibration miscalibration histogram). **M** - **#216** — Reviewer-type asymmetry parity audit in Devil's Advocate (4th anti-sycophancy rule). DA text change is **S**; the authorship-metadata design question splits off as a separate, larger decision — not required to ship the text change. ### Kong et al. Tier A — ALL SHIPPED (no v3.11 work here; see stale-open cleanup below) Cross-checked against git history (not just issue state): the entire Kong Tier A is already in v3.10, even though several issues are still OPEN. - A1 #256 — shipped (Schema 11 ledger + follow-ups #266/#268/#269). - A2 #257 — shipped via `#270 feat: add idea-diversity advisories` (commit `9f60d11`, **before the v3.10.0 tag**). - A3 #258 — shipped via `#267 feat: add version-family citation reconciliation` (commit `48378bb`, before the v3.10.0 tag). - A4 #259 — shipped (CLOSED). → #256 / #257 / #258 are **open-but-shipped stale issues**, NOT v3.11 work. See the stale-open cleanup item under Tier 2. ### Opus 4.8 behavioral-signal cluster (recent, 2026-05-29) - **#273** — Calibration may be optimistic under same-model / rubric-aware judging; recommend cross-model evaluation as opt-in "for best results" (epistemic framing, not a gate). **S** - **#274** — Concise reviewer output + a behavioral check that hard boundaries / DA non-softening hold under 4.8 after user pushback (guidance + verification). **S** - **#272** — Treat retrieved external content as data, not instructions. **This is a separate, still-open SAFETY TRACK, not a low-cost guidance add.** The #134 design spec (§ "open items", line ~138) is explicit: "#272 remains a separate, still-open safety track … #134 must NOT close #272 or claim any mitigation … revisit #272's home only when a concrete envelope substrate exists (Slice 3+)." Do **NOT** co-locate #272 with #134 or treat it as a quick guidance patch. If taken on in v3.11, it needs its own design pass on the instruction/data trust boundary — scope accordingly, or defer until the envelope substrate exists. (Resolved against the #134 spec text, which is explicit on this point.) --- ## Tier 2 — deferred-from-v3.10 + hygiene (S each) - **#160** — Move `scripts/_test_helpers.py` → `tests/test_helpers.py` (drop the `sys.path` hack across ~24 import sites). `defer:v3.10`. - **#151** — Evaluate generic repo-hygiene CI (gitleaks / detect-secrets). `defer:v3.10`; scope explicitly constrained to avoid maintainer-specific rules in a public repo. - **#138** — Parallelize OpenAlex + Crossref calls per entry in the migration script (`ThreadPoolExecutor`, ~halves backfill wait). Verify the script is still the right target post-v3.10. - **#242** — `venue_disclosure_policies.md` ACL row: revisit canonical source (Admin Wiki URL returns HTTP 418 to curl → needs a CI link-checker exemption or alternate URL). - **Stale-open issue cleanup (not feature work, but do it this cycle).** Several Kong issues are OPEN but their implementation already shipped in v3.10 — git history confirms, even though the issues never got auto-closed: **#256** (A1, via #266/#268/#269), **#257** (A2, via #270 `9f60d11`), **#258** (A3, via #267 `48378bb`), and #259/#263 (already CLOSED). Verify each against the merged work and close. Leaving them open distorts any future roadmap read (this very draft's first pass mis-scoped #257/#258 as v3.11 work because they looked open). --- ## Tier 3 — needs an owner decision before scoping - **#89 Item 7 — diff/patch revision mode.** Ranked #1 by #89's own analysis as the highest- leverage *unfiled* work; **no design doc, no dedicated issue**. To take it on, write a spec first. Decide consciously: scope into v3.11 or defer. - **#244** (external — TriggerMinds "Structural Limitations" critique). Not a proposal; a discussion thread. Its two actionable children — **#246** (discipline-relative grading, also cross-linked from Kong A4 #259) and **#247** (dual-use scope clarification) — **already shipped and closed** (`#280 docs(#246,#247)`, `Closes #246` / `Closes #247`). So #244 itself is now "verify residual discussion → likely close or re-scope," NOT waiting on children. --- ## Explicitly NOT in v3.11 (per source epics) - **Kong Tier B (#260 / #261 / #262)** — the Kong epic #255 tags B-track as a **v3.12 candidate**, not v3.11. (B1 experiment provenance, B2 figure/table fidelity, B3 cross-paper contradiction — all have full specs and could be pulled forward, but the epic's own tiering puts them after v3.11.) - The anti-pattern non-features the source epics list as first-class rejections (autonomous idea generation, autonomous experiment execution, hidden ranking, open-ended self-play). These are boundary entries for `POSITIONING.md` / `NEGATIVE_SCOPE.md`, not features. --- ## Open questions for the owner 1. **Theme:** broad "provenance + calibration hardening" (Kim subs + #182 + Opus-4.8 guidance) vs a narrow "citation-integrity release" (#182-centered)? Note Kong Tier A is already shipped, so the v3.11 substance is really #182 + Kim (#213-216) + the two Opus-4.8 guidance items (#273/#274), with the Co-Scientist docs as the cheap Tier 0 opener. 2. **#272 (instruction/data boundary):** the #134 spec quarantines it as a separate safety track needing an envelope substrate. Take it on in v3.11 as its own design pass, or defer until the substrate exists? (Do NOT fold it into general guidance.) 3. **#89 Item 7 (diff/patch revision):** the highest-leverage unfiled item — scope a design doc for v3.11, or defer? 4. **Pull Kong Tier B (#260/261/262) forward?** Specs are ready; only the epic's tiering holds them at v3.12. --- ### ROADMAP V3.20.1 V3.22 (`docs/ROADMAP-v3.20.1-v3.22.md`) # ARS Post-v3.20 through v3.22 Development Roadmap **Status:** PLAN OF RECORD — implementation order is committed; structural defaults remain evidence-gated. **Date:** 2026-08-15. **Scope source:** reproducible post-release repository contract audit plus reconciliation with the public issue backlog. **Tracking epics:** #734 (immediate contract honesty) and #741 (adaptive inquiry-state architecture). ## Outcome The next releases will first make current claims match current mechanisms. Only then will ARS add richer inquiry state, and that state will remain bounded, optional, and research-type aware until comparative evidence shows that it helps without imposing unacceptable burden. The roadmap deliberately separates four things: 1. **correctness fixes** that can ship now; 2. **structural mechanisms** that can be engineered but may add user burden; 3. **effectiveness questions** that require controlled evaluation; 4. **intrinsic limits** that ARS must disclose rather than promise to solve. ## Delivery order | Phase | Target | Goal | Default behavior | |---|---|---|---| | 0 | next patch | Correct contract, status, and claims-language mismatches | Hardened immediately | | 1 | following patch design/eval | Freeze research-family profiles and usability thresholds | No workflow expansion | | 2 | v3.21 alpha | Add a bounded inquiry branch ledger | Opt-in; simple path unchanged | | 3 | v3.21 beta | Carry relevant alternatives across later stages | Opt-in and profile-specific | | 4 | v3.21-v3.22 | Bind criteria and publish stage-level evidence ceilings | Generated evidence view; no inflated claim | | 5 | v3.22+ | Execute outcome-level study | No effectiveness claim before results | Dates are intentionally not promised for human-participant evaluation. Release order and gates take precedence over calendar targets. Status at v3.22.0 (2026-09-16): Phase 4 is not delivered in v3.22.0; its window carries forward. Per-phase shipping status is recorded in `CHANGELOG.md`, not here. ## Phase 0 — next-patch contract-honesty hardening Parent: #734. 1. **#735 — Socratic research-question boundary.** The default non-convergence path may summarize scholar-originated directions and offer continued dialogue or literature exploration. It may generate candidates only after an explicit user request and a visible exit from the non-generative Socratic contract. 2. **#737 — coverage-bounded verification language.** Use `100% of registered claims`, name the denominator, keep semantic extraction coverage unknown, and distinguish replayable deterministic checks from generative reproducibility. 3. **#736 — author disposition for detected claim-strength drift.** Replace default-open routing with `restore`, `authorize_with_reason`, or `pause`; bind every choice at build and replay to explicitly named run-local raw event bytes while keeping paths and messages out of the durable sidecar. 4. **#738 — user-attested reading state.** Unknown/legacy scope remains `coverage_unknown` and can never become `ok` for an anchored claim. 5. **#739 — reviewer score honesty.** Generic 0-100 summaries are `NOT_CALIBRATED`; no field-general score-to-editorial-decision mapping or universal reference-count threshold. 6. **#740 — typed review provenance.** Report role, context, peer-output, model-family, provider, and human separation independently; do not infer independent error processes from personas. ### Next-patch ship gates - Every changed behavior has a regression test that fails on its v3.20 form. - Command, skill, agent, schema, renderer, and maintained documentation surfaces agree. - Unknown/unmeasured states fail visibly. - Release notes describe corrections, not improved research outcomes. - The existing full deterministic suite remains green. ## Phase 1 — research-family profiles and a complexity budget Issue: #742. This phase changes the design substrate, not the default workflow. ### Minimal universal kernel ARS may treat the following as broadly reusable governance primitives: - author versus AI provenance and decision authority; - evidence and evaluation status; - active, parked, rejected, reopened, and stale state; - explicit adoption/disposition receipts; - dependency and invalidation links. These primitives do not imply that the same research stages, alternative types, evidence hierarchy, or review standards apply everywhere. ### Research-family overlays A user-confirmed, versioned profile declares applicable stages, vocabulary, alternative categories, reporting/design overlays, institutional authority points, exclusions, and unresolved fit. Initial evaluation strata may include quantitative empirical, qualitative, theoretical/conceptual, interpretive/humanities, evidence synthesis, computational, and clinical/human-subjects work. They are test strata, not an exhaustive taxonomy or a coverage claim. Unsupported or hybrid work receives the minimal kernel and an explicit fallback state. ARS must not silently infer a profile from manuscript quality or disciplinary stereotypes. ### Complexity budget - The current simple path remains available and receives no new mandatory branch prompt. - Rich state appears only at consequential or hard-to-reverse decisions. - Default views use one compact summary; the underlying graph is progressively disclosed. - Each profile freezes a live-branch budget plus merge, park, and archive behavior. - Every added interaction offers `skip`, `off`, and reset-to-simple-path behavior without discarding scholar-owned work. - Novice and experienced users are evaluated separately. - Task completion, unnecessary prompts, time, abandonment, perceived control, wrong-profile recovery, and decision usefulness remain separate outcomes. - No default-on change is licensed by an average improvement that hides a material burden or authority regression in one research family. - Default-on authorization is specific to an exact research-family profile and release: positive evidence in one family or version cannot authorize another. - Before implementation, #742 preregisters the maximum added interaction/time budget and non-inferiority guardrails for completion, abandonment, and perceived control in every research-family × experience stratum; a failed stratum cannot be rescued by a pooled average. ## Phase 2 — bounded inquiry branch ledger Issue: #743; ideation-measurement dependency: #659. The versioned ledger records stable branch/parent ids, author versus AI-facet provenance, assumptions, evidence sought, status, disposition reason, reopen conditions, and downstream invalidation links. Only author-expressed or explicitly adopted framings become scholarly branches. The alpha remains opt-in. A branch summary appears at a consequential freeze or when evidence satisfies a reopen condition. Reopening marks dependent artifacts stale; it never silently mutates them. The ledger may show that alternatives were preserved and recoverable, but not that they were novel, correct, or valuable. Before the #743 alpha ships, it must register its exact mechanism version, `DESIGNED` / `NOT_RUN` evidence state, transport limits, and claim ceiling in a minimal #745-compatible evidence-matrix scaffold. Structural code may not precede the record that bounds what can be claimed about it. Promotion beyond opt-in requires paired evidence on recovery, breadth, burden, time, abandonment, and research-family fit. ## Phase 3 — cross-stage alternative register Issue: #744. Profile-relevant alternatives may be carried through theory, design, measurement, analysis, synthesis, interpretation, drafting, and review. Each profile decides which stages and categories apply. `not_applicable` is lawful and visible; more alternatives are not treated as inherently better. The register must track evidence, author disposition, unresolved state, reopen conditions, and dependent artifacts. Evaluation must report useful follow-through separately from irrelevant-alternative inflation and user burden. It stays opt-in until the Phase 1 gates pass. ## Phase 4 — criteria, capability evidence, and claim ceilings 1. **#575/#684 — target criteria.** Complete a bounded set of source-backed venue × track × contribution-type profiles with author confirmation, freshness governance, and constructive-value evaluation. No venue fit score becomes an acceptance prediction. 2. **#745 — stage capability/evidence matrix.** Publish one machine-readable source for mechanism status, conformance evidence, behavioral evidence, model/version/population, transport limits, claim ceiling, and next evaluation. 3. **#582 — role topology utility.** Compare panels with matched solo/minimal baselines; role count is never competence evidence. 4. **#659 — within-session ideation breadth.** Keep scholar-originated framings separate from AI-surfaced facets and report burden separately. 5. **#684 — criteria usefulness.** Execute the frozen evaluation rather than treating a registered profile as evidence of constructive review. The evidence matrix must preserve at least `DESIGNED`, `NOT_RUN`, `MEASURED`, `MIXED`, `OUT_OF_SCOPE`, and stale/transport-limited states. README and release-note claims cannot exceed the recorded ceiling. ## Phase 5 — outcome-level study Issue: #746; governing frozen design: #658. Engineering can build schemas, validators, packet manifests, assignment and masking records, judge assignment, and deterministic scoring replay. It cannot substitute simulated participants or ARS reviewer agents for recruitment and independent external judging. A pilot establishes feasibility only. The full study must follow the preregistered estimand, governance, intervention version, active-control parity, masking, stopping rules, and result-publication commitment. A positive result may support a bounded manuscript/process-quality claim; it cannot establish that underlying procedures were executed, raw data are true, results reproduce, or scientific conclusions are valid. ## What is and is not portable across research domains | Layer | Reasonably field-general | Must be profile-specific or externally governed | |---|---|---| | Authority | The author adopts, rejects, or reopens scholarly choices | Institutional authority, co-author roles, community-specific consent | | Provenance | Human/AI/source origin and version can be recorded | What counts as adequate evidence or expertise | | State | Alternatives can be active, parked, rejected, or stale | Which alternatives are meaningful and when they should appear | | Workflow | Dependencies and invalidation can be explicit | Stage order, optional/absent stages, iteration patterns | | Review | Findings can carry criteria, anchors, uncertainty, and provenance | Venue, genre, method, and interpretive standards | | Evaluation | Mechanism, behavior, and outcome evidence can remain separate | Gold standards, judges, utility measures, and acceptable burden | Therefore the proposed architecture is not universally applicable in one fixed form. The universal claim is limited to a small governance kernel; all substantive workflow content requires demonstrated profile fit or an explicit fallback. ## Intrinsic limits No roadmap item can make ARS independently establish: - whether reported procedures were actually performed; - whether raw data are authentic or complete; - whether results reproduce in the world; - whether a research question is important, original, or worth pursuing; - whether persona-separated reviewers have independent error processes; - tacit editorial judgment or future acceptance decisions; - suitability for every discipline, genre, jurisdiction, or institutional setting. ARS can make these boundaries, decisions, evidence, and uncertainty more inspectable. External experts, institutions, replications, and empirical studies remain necessary. ## Repository data boundary - Committed tests and examples use synthetic fixtures or materials with explicit public/redistribution permission. - Confidential correspondence, unpublished research artifacts, and maintainer-local screenshots or working documents do not belong in issues, commits, fixtures, examples, or pull-request descriptions. - Maintainer-local `deliverables/` is ignored at repository root. - A release candidate receives a **manual** changed-file privacy/provenance scan in addition to ordinary tests until an automated gate ships. ## Decision rule Correctness and honesty fixes ship when their contracts and regression tests pass. Structural features ship opt-in when their schemas and state transitions pass. A structural feature becomes a default only after its usability and research-family evaluations support that exact change without hiding subgroup harm. Effectiveness language changes only after the relevant preregistered study is complete. --- ### SETUP (`docs/SETUP.md`) # ARS Setup Prerequisites and optional setup for Academic Research Skills. If you only need Markdown output and the default Claude pipeline (the inherited session model), you can skip most of this — see "Minimum viable setup" below. --- ## Minimum viable setup 1. Install Claude Code (see below). 2. Export `ANTHROPIC_API_KEY`. 3. `claude` in this repo (or any project that has ARS in `.claude/skills/`). That is enough for Markdown output + DOCX conversion instructions. Everything else in this document is optional. ## Python (optional) The core skills (research / write / review) need no Python — they are prompt-driven. A **real Python interpreter** is needed only for: the `PreToolUse` write-scope guard (optional subagent hardening — if no real Python is found it cleanly no-ops and the guard is simply inactive; core skills are unaffected), plus a few opt-in features that shell out to Python (revision-patch mode, the submission-package verifier, and the `/ars-cache-invalidate` / `/ars-mark-read` / `/ars-unmark-read` commands). On Windows, note that `python3` is often a non-functional Microsoft Store placeholder rather than real Python; install Python from python.org (or via `winget`) so the launcher can find a real interpreter. The guard launcher is a POSIX shell script and `hooks.json` invokes it through `bash`, so on Windows it needs **Git Bash** (bundled with Git for Windows). With Git Bash present, a missing real Python degrades cleanly (the guard no-ops, silently). Without Git Bash, Claude Code falls back to PowerShell, which cannot run the `.sh` launcher at all: the guard is inactive and the `PreToolUse` hook will log an error per call rather than no-op quietly (accepted degradation — the guard is optional and never blocks your writes, but the hook noise is the trade-off until Git Bash is installed). --- --- ## Install Claude Code **Recommended: Native installer** (no Node.js required, auto-updates): ```bash # macOS / Linux curl -fsSL https://claude.ai/install.sh | bash # Windows (PowerShell) irm https://claude.ai/install.ps1 | iex ``` **Platform support.** macOS and Linux are the tested platforms; CI runs on Ubuntu only. Windows is best-effort: the scripts that lock files share one helper (`scripts/file_lock.py`) with an `msvcrt` backend, no Windows CI job exists, and Windows behaviour rests on contributor verification (#843, #845). On Windows, shared read locks degrade to exclusive locks with a short wait, indefinite lock waits are capped at 30 seconds, and the inquiry branch ledger alpha refuses to run.
Alternative: npm install (deprecated) Requires Node.js 18+. ```bash npm install @anthropic-ai/claude-code ```
## Set up API key Get an Anthropic API key at . ```bash # Claude Code will prompt for your API key on first run claude ``` Or set it as an environment variable: ```bash export ANTHROPIC_API_KEY=sk-ant-xxxxx ``` ## DOCX output (optional) Direct `.docx` generation uses [Pandoc](https://pandoc.org/). If Pandoc is unavailable, the formatter falls back to Markdown + DOCX conversion instructions. ```bash # macOS brew install pandoc # Linux (Debian/Ubuntu) sudo apt-get install pandoc # Windows — download from https://pandoc.org/installing.html ``` ## LaTeX / PDF output (optional) PDF output requires [tectonic](https://tectonic-typesetting.github.io/) and specific fonts. **This is optional** — Markdown output and DOCX conversion instructions work without any of this. ```bash # macOS brew install tectonic # Linux (Debian/Ubuntu) curl --proto '=https' --tlsv1.2 -fsSL https://drop-sh.fullyjustified.net | sh # Windows — download from https://tectonic-typesetting.github.io/en-US/install.html ``` **Required fonts** (for APA 7.0 CJK output): - **Times New Roman** — usually pre-installed on macOS/Windows; on Linux install `ttf-mscorefonts-installer` - **Source Han Serif TC VF** (思源宋體) — download from [Google Fonts](https://fonts.google.com/specimen/Noto+Serif+TC) or [Adobe GitHub](https://github.com/adobe-fonts/source-han-serif) - **Courier New** — usually pre-installed > If you only need Markdown output or DOCX conversion instructions, skip this entirely. Direct `.docx` generation requires Pandoc, and PDF generation requires `tectonic`. --- ## Material Passport `literature_corpus[]` adapters (v3.6.4+, optional) If you maintain a curated literature corpus (Zotero, Obsidian, a folder of PDFs, etc.), you can pre-load it into a Material Passport so Phase 1 ARS agents read your library *before* searching external databases. This is opt-in and presence-based — when no corpus is supplied, ARS runs the external-DB-only flow unchanged. Three reference Python adapters ship with v3.6.4 at `scripts/adapters/`: ```bash # 1. Install the dev dependencies (the adapter requirements are declared in requirements-dev.txt) pip install -r requirements-dev.txt # 2. Run a reference adapter (pick one that matches your corpus source). # Both --passport and --rejection-log are required. python scripts/adapters/folder_scan.py --input /path/to/pdfs --passport passport.yaml --rejection-log rejection_log.yaml python scripts/adapters/zotero.py --input my-zotero-export.json --passport passport.yaml --rejection-log rejection_log.yaml python scripts/adapters/obsidian.py --input ~/Obsidian/Lit\ Notes --passport passport.yaml --rejection-log rejection_log.yaml # 3. Pass the resulting passport.yaml into your ARS session # (concrete invocation depends on which skill you're running — see scripts/adapters/README.md) ``` Each adapter emits two files: `passport.yaml` (Schema 9 with `literature_corpus[]` populated) and `rejection_log.yaml` (always emitted, empty when no rejections — closed enum of categorical reasons). Users with non-reference corpus sources are expected to write their own adapters following [`academic-pipeline/references/adapters/overview.md`](../academic-pipeline/references/adapters/overview.md). v3.6.5 wires `bibliography_agent` (deep-research, Phase 1) and `literature_strategist_agent` (academic-paper, Phase 1) as the consumers — both run the corpus-first / search-fills-gap flow when a non-empty corpus is present and parses cleanly. See [`academic-pipeline/references/literature_corpus_consumers.md`](../academic-pipeline/references/literature_corpus_consumers.md) for the consumer protocol. ## Optional environment flags (v3.5.1+) ARS exposes a few opt-in flags. All default to OFF; setting them changes behaviour for the current session only. These flags are behavior toggles; for standing *content* preferences (citation style, search inclusion), see [§"Standing preferences via CLAUDE.md"](#standing-preferences-via-claudemd) below. | Flag | Since | What it does | Reference | |---|---|---|---| | `ARS_CROSS_MODEL` | v3.0 | Enable cross-model verification (see next section) | [§"Cross-model verification"](#cross-model-verification-optional) | | `ARS_CROSS_MODEL_TRANSPORT=codex` | #630 | Use ChatGPT subscription transport for citation-integrity calls only; all DA/reviewer/judgment paths remain API-key based | `shared/cross_model_verification.md` | | `ARS_SOCRATIC_READING_PROBE=1` | v3.5.1 | Activate the Socratic reading-check probe layer in `socratic_mentor_agent`. Goal-oriented intent only; fires at most once per session when user has cited a specific paper; decline logged without penalty. | `deep-research/agents/socratic_mentor_agent.md` | | `ARS_PASSPORT_RESET=1` | v3.6.3 | Promote every FULL and MANDATORY checkpoint to a context-reset boundary. Required to *emit* boundary entries; **not** required to invoke `resume_from_passport=` in a fresh session. With the flag ON in `systematic-review` mode, reset is mandatory at every FULL and MANDATORY checkpoint. | `academic-pipeline/references/passport_as_reset_boundary.md` | | `ARS_AUDIT_ARTIFACT_GATE=1` | #925 | Turn on the v3.6.7 Audit Artifact Gate, which has you run `scripts/run_codex_audit.sh` outside the session at each transition after a `synthesis_agent` / `research_architect_agent` (survey-designer) / `report_compiler_agent` (abstract-only) deliverable; the wrapper sends the deliverable to an external model, and the gate blocks on its findings. The flag is configuration, not consent: the gate runs only after you agree for the run. Off by default; the Stage 2.5 / 4.5 integrity gates run either way. | `academic-pipeline/agents/pipeline_orchestrator_agent.md` § 3.5 | | `ARS_CROSS_MODEL_SAMPLE_INTERVAL` | v3.5.0 | With cross-model on, run the second-model collaboration-depth observer every N checkpoints instead of every one (default `1`; the Stage 6 whole-pipeline pass always runs). It does not change integrity-gate sampling. | `academic-pipeline/agents/pipeline_orchestrator_agent.md` § Collaboration Depth Observer | | `ARS_CLAIM_AUDIT=1` | v3.8 | Turn on the claim-faithfulness audit: claims are judged against their cited sources, and unsupported ones block final output. In pipeline runs it first runs at Stage 4.5, so a finding can still be corrected. | `academic-pipeline/agents/pipeline_orchestrator_agent.md` § 3.6 | | `ARS_SOCRATIC_ADJACENT_PROBE=1` | v3.13.0 (#461) | Let the Socratic Mentor raise one adjacent framing at a time, as a question, at most twice per session. | `deep-research/agents/socratic_mentor_agent.md` | | `ARS_RE_REVIEW_LEGACY=1` | v3.20.0 (#576) | Run Stage 3' re-review with the archived pre-contract schemas and checker instead of the three-gate contract. | `academic-paper-reviewer/references/re_review_mode_protocol.md` § Legacy Mode | | `ARS_INQUIRY_LEDGER=1` | v3.21.1 (#743) | Alpha inquiry branch ledger. It needs a research-workflow profile binding, which no pipeline step creates; without one the run stays linear and says so. | `academic-pipeline/agents/pipeline_orchestrator_agent.md` § Inquiry Branch Ledger | | `ARS_VERIFICATION_CACHE_PATH` | v3.11 | Override the citation-verification cache location (see below). Not an on/off flag — the cache is on by default; this only relocates it. | `scripts/verification_cache.py` | | `ARS_CACHE_STALE_ADVISORY_DAYS` | v3.18.0 (#541) | Age threshold (days) for the cache-staleness advisory: cache-served verifications older than this surface as `ADV-CACHE` advisory rows at the integrity checkpoints (never gating). Default 30; `0` disables; malformed/negative values fall back to the default. | `scripts/verification_cache.py` | | `ARS_CACHE_REVALIDATE=1` | v3.18.0 (#541) | Opt-in live re-verification at the gate: cached rows past the staleness threshold are re-verified live (per-row bypass, then re-populated) instead of served. Cost scales with the stale-row count. Default off = advisory-only. | `scripts/verification_gate/__init__.py` + `integrity_verification_agent.md` § A0.5 | | `ARS_MODEL_TIERING` | v3.16.0 (#517) | Opt-in model tiering: `economy` (frontier session — execution-type agents step down one tier, floor Opus-class) or `quality-boost` (below-frontier session — judgment-type agents jump up to the frontier tier at the checkpoint surfaces: Stage 2.5/4.5 gates, the opt-in Stage 4→5 claim–ref audit, and final review). Unset = session model everywhere; unknown values warn once and behave as unset. | `shared/model_tiering.md` | --- ## Standing preferences via CLAUDE.md ARS deliberately ships no user-level configuration file. The supported way to get "set once, applies every run" behavior for content preferences is a preferences block in your project's `CLAUDE.md`: Claude Code loads it at session start, and every ARS agent dispatched in that session inherits it as context. This is a stated design position, not a missing feature — search-related preferences are per-project inclusion/exclusion criteria that belong in the Annotated Bibliography's `search_strategy` (Schema 2), and a machine-global config silently inherited across projects would be a methodological hazard for systematic-review work. Decision record: #634. Copy-pasteable template (adjust to taste): ```markdown ## ARS standing preferences - Citation style: APA 7th unless a venue template says otherwise. - Literature search: exclude preprints unless I explicitly ask; prefer peer-reviewed journal articles. - Journal tier: when ranking or shortlisting sources, prefer higher-tier journals in the field, and say so when unsure of a journal's tier. - Open access: prefer OA versions when citing, and link the OA copy. ``` Two honest limitations: - **Journal tiers are model judgment.** None of the four lookup indexes (Semantic Scholar, OpenAlex, Crossref, arXiv) serves quartile or tier data, so a tier preference is applied from the model's own knowledge and should be stated as such — treat tier claims as advisory, never as verified metadata. - **There is intentionally no output-directory setting.** The user-supplied Material Passport path is the single discovery anchor (v3.6.8 design round, decision R4-003); a standing output-location preference would create a second source of truth. Name the destination per run. If a preference changes what a systematic review may include (preprint exclusion, language limits, date windows), record it in that project's `search_strategy` rather than relying on the ambient block — the ambient block sets defaults, the Schema 2 `search_strategy` is the auditable record. Re-evaluation trigger (recorded in #634): an ARS-owned preferences surface gets revisited only if additional users independently request it, or a platform port lacks a `CLAUDE.md` equivalent. If ever built, the architecturally consistent shape is a Material Passport-level `user_preferences` input aggregate (like `literature_corpus[]`), not a machine-global config file. --- ## Citation verification cache (v3.11, #182) The deterministic citation-existence gate (#182) cross-checks each reference against Semantic Scholar, OpenAlex, Crossref, and arXiv. To avoid re-querying the same paper across drafts, results are cached in a local SQLite store. - **No setup required.** The cache is created automatically at `~/.cache/ars/verification.db` on first use; entries expire after 90 days. The arXiv resolver needs no API key. - **Relocate it** by exporting `ARS_VERIFICATION_CACHE_PATH=/your/path.db` (e.g. to share one cache across projects, or to keep it on a faster disk). - **Invalidate one citation** with `/ars-cache-invalidate ` — removes every cached row for that key (all four resolvers, all query forms); idempotent no-op if nothing is cached. The cache is single-process (SQLite WAL); concurrent multi-user access to one cache file is out of scope. For the complete picture of what leaves the machine (resolvers, optional cross-model calls, the update check) and every local store's lifetime and deletion path, see [DATA_FLOWS.md](DATA_FLOWS.md). --- ## Cross-model verification (optional) ARS works with the inherited Claude session model alone. For higher confidence, you can optionally enable a second AI model to independently verify integrity checks and challenge the devil's advocate. ### Quick setup ```bash # Step 1: Set your API key (choose one or both) export OPENAI_API_KEY="sk-your-key-here" # For GPT-6 Astra / GPT-5.6 Sol / GPT-5.5 export GOOGLE_AI_API_KEY="AIza-your-key-here" # For Gemini 3.1 Pro # Step 2: Choose your cross-verification model export ARS_CROSS_MODEL="gpt-6-astra" # Current OpenAI flagship — provisional pending ARS validation (run scripts/cross_model_smoke_test.sh) # or: export ARS_CROSS_MODEL="gemini-3.1-pro-preview" # Current Google flagship — validated, strong at factual verification # or: export ARS_CROSS_MODEL="gpt-5.6-sol" # Previous generation — validated on the ChatGPT-subscription citation transport, provisional on this API route # or: export ARS_CROSS_MODEL="gpt-5.5" # Previous generation — validated (designated API-route bakeoff baseline) # Optional: reasoning effort for OpenAI verifier calls (unset = provider default) # GPT-6 Astra API: low|medium|high|xhigh|max (the Codex citation transport rejects ultra) # export ARS_CROSS_MODEL_REASONING_EFFORT="medium" # Step 3: Run Claude Code as normal — cross-verification activates automatically claude ``` ### What changes when enabled | Feature | Without cross-model | With cross-model | |---|---|---| | Integrity verification | Single-model 100% check | + risk-stratified verification by 2nd model: 100% of high-impact references (final gate adds 100% of new/changed-claim references) + a sampled remainder | | Devil's Advocate | Single-model DA | + Cross-model generates a blind, separately executed critique; novel findings added | | Peer Review | 5 reviewers (same model) | Same 5 reviewers + cross-model DA critique/calibration support | | Irreversible checkpoints | Single-model decision | + Blind cross-model decision at design freeze + final editorial decision; divergence escalated to you, never averaged | ### Cost Full pipeline adds ~$0.60-1.10 in cross-model API costs (order-of-magnitude; measured at GPT-5.4 Pro pricing). See [`shared/cross_model_verification.md`](../shared/cross_model_verification.md) for the current model lineup and detailed breakdown. ### No API key? No problem Without `ARS_CROSS_MODEL` set, everything works exactly as before. The cross-model features are invisible and add zero overhead. ### ChatGPT subscription transport (citation integrity only) If Codex CLI 0.147.0+ is logged in through a ChatGPT subscription, citation-integrity calls can use that subscription without an OpenAI API key. This does not cover Devil's Advocate, Reviewer 2, calibration, re-review, or checkpoint judgments. ```bash # Citation-integrity calls only. General DA/reviewer/judgment calls remain on API transport. export ARS_CROSS_MODEL_TRANSPORT="codex" # gpt-6-astra: current OpenAI flagship — provisional on this transport (entry-gate # smoke PASS 2026-09-05 on codex-cli 0.153.4; no bakeoff run yet). export ARS_CROSS_MODEL="gpt-6-astra" # gpt-5.6-sol is validated for THIS transport (2026-08-19 codex-transport bakeoff, # superiority on recall + latency — audits/bakeoff-gpt-5-6-sol-codex-2026-08-19.md): # export ARS_CROSS_MODEL="gpt-5.6-sol" python3 scripts/cross_model_codex_transport.py detect # The producer sends one closed codex_citation_request/1.0 object on stdin: printf '%s' "$CITATION_REQUEST_JSON" | scripts/cross_model_codex_verify.sh ``` The detector honors custom `CODEX_HOME` and requires the exact subscription attestation `Logged in using ChatGPT`; it never prints credentials. The adapter uses an auth-only ephemeral home, empty working root, read-only sandbox, no local tools, and binds accepted source URLs to structured search results. The optional live smoke `scripts/cross_model_smoke_test_codex.sh` spends subscription/model/network capacity and is never run by CI. --- ## Installation methods Claude discovers skills at `//SKILL.md`. This repo contains five separate skills, each with its own `SKILL.md`: - `deep-research` - `academic-paper` - `academic-paper-reviewer` - `academic-pipeline` - `sr-screener` Do not install the whole repository as one nested skill folder under `.claude/skills/academic-research-skills/`; that buries the five `SKILL.md` files one level too deep for discovery. See Anthropic's [Claude Code Skills documentation](https://code.claude.com/docs/en/skills). The methods below differ in more than convenience: hooks, slash commands, the tools allowlist, subagent orchestration, and script-backed checks are each available in some channels and degraded or absent in others. Before relying on any of those mechanisms, check the per-channel map: [CONTROL_AVAILABILITY.md](CONTROL_AVAILABILITY.md). ### Method 0: Claude Code Plugin (v3.7.0+, recommended for Claude Code CLI / IDE users) If you use Claude Code CLI, VS Code extension, or JetBrains extension, install ARS as a plugin: ```text /plugin marketplace add Imbad0202/academic-research-skills /plugin install academic-research-skills ``` The five skills (`deep-research`, `academic-paper`, `academic-paper-reviewer`, `academic-pipeline`, `sr-screener`) are auto-discovered from the plugin's `skills/` directory. **Slash command forms (#633).** Plugin installs namespace commands as `/academic-research-skills:ars-` — that form is canonical. Every command also declares an explicit frontmatter `name`, so on Claude Code v2.1.216 or later the bare `/ars-` alias works too whenever no other command claims that name; the bare form is what the session-start announce lists. On Claude Code older than v2.1.216, the frontmatter `name` replaces the whole command name, so commands appear only in their bare `/ars-` form (still invocable; the namespaced form loses autocomplete). **Strongly recommended: open auto-update.** Open the `/plugin` UI, find `academic-research-skills`, and toggle auto-update on. ARS releases roughly every 1–2 weeks; auto-update keeps you in sync without manual refreshes. To refresh manually: `/plugin update academic-research-skills`. (`/plugin marketplace update academic-research-skills` only refreshes the marketplace source list, not the installed plugin itself.) **Built-in update reminder.** The plugin also nudges you on its own: at session start it compares your installed version against `main` (at most one network request per day, 3-second ceiling, silent on any failure) and, when you are behind, prepends one line to the session-start announce pointing at `/plugin update academic-research-skills`. Set `ARS_UPDATE_CHECK=0` to disable the check entirely. Privacy: the check performs a single HTTPS GET of this repository's public `.claude-plugin/plugin.json` and transmits no user data. **Plugin platform scope:** - ✅ Claude Code CLI / VS Code extension / JetBrains extension — full support - ❌ claude.ai web / Claude for Work / Anthropic API direct calls — plugins not supported; use Method 1 / 2 / 3 below - ➡️ Codex CLI — install the sibling distribution [`Imbad0202/academic-research-skills-codex`](https://github.com/Imbad0202/academic-research-skills-codex) (same workflow content, Codex-native packaging) ### Method 1: As project skills (recommended) Use this when you want ARS available inside an existing Claude Code project. Clone the repo to a stable local path, then copy each skill folder into your project's `.claude/skills/` directory: ```bash git clone https://github.com/Imbad0202/academic-research-skills.git ~/academic-research-skills cd /path/to/your/project mkdir -p .claude/skills cp -R ~/academic-research-skills/deep-research .claude/skills/deep-research cp -R ~/academic-research-skills/academic-paper .claude/skills/academic-paper cp -R ~/academic-research-skills/academic-paper-reviewer .claude/skills/academic-paper-reviewer cp -R ~/academic-research-skills/academic-pipeline .claude/skills/academic-pipeline cp -R ~/academic-research-skills/sr-screener .claude/skills/sr-screener ``` Expected path shape: ```text /path/to/your/project/.claude/skills/deep-research/SKILL.md /path/to/your/project/.claude/skills/academic-paper/SKILL.md /path/to/your/project/.claude/skills/academic-paper-reviewer/SKILL.md /path/to/your/project/.claude/skills/academic-pipeline/SKILL.md /path/to/your/project/.claude/skills/sr-screener/SKILL.md ``` `sr-screener` runs its screening reviewers as a subagent. For a project-skills install, also copy `sr-screener/agents/screening_reviewer_agent.md` into `.claude/agents/` (details: `sr-screener/references/orchestration.md` § 1). Then copy the `.claude/CLAUDE.md` content into your project's `.claude/CLAUDE.md` (merge with existing if you have one). > **Global Claude Code installation:** To make these skills available across your Claude Code projects, install the five folders to `~/.claude/skills/` instead: > > ```bash > git clone https://github.com/Imbad0202/academic-research-skills.git ~/academic-research-skills > > mkdir -p ~/.claude/skills > cp -R ~/academic-research-skills/deep-research ~/.claude/skills/deep-research > cp -R ~/academic-research-skills/academic-paper ~/.claude/skills/academic-paper > cp -R ~/academic-research-skills/academic-paper-reviewer ~/.claude/skills/academic-paper-reviewer > cp -R ~/academic-research-skills/academic-pipeline ~/.claude/skills/academic-pipeline > cp -R ~/academic-research-skills/sr-screener ~/.claude/skills/sr-screener > ``` ### Method 2: As a standalone project Use this when you want to work directly inside the ARS repository. ```bash git clone https://github.com/Imbad0202/academic-research-skills.git cd academic-research-skills claude ```
No Git? Download as ZIP instead 1. Go to 2. Click the green **Code** button → **Download ZIP** 3. Extract the ZIP to your desired location 4. For Method 1: copy the five extracted skill folders (`deep-research`, `academic-paper`, `academic-paper-reviewer`, `academic-pipeline`, `sr-screener`) into `.claude/skills/` inside your project 5. For standalone use: open a terminal in the extracted folder and run `claude`
### Method 3: Claude Cowork (desktop) Use this when you want the five ARS skills available in [Claude Cowork](https://support.claude.com/en/articles/13345190-get-started-with-claude-cowork), Claude Desktop's agentic workspace. > **Cowork does not read `~/.claude/skills/`.** That directory belongs to Claude Code (the CLI / IDE), and Cowork does not scan it. Cowork loads skills you upload through **Settings → Capabilities → Skills**, each as its own zip. Symlinking or copying the skill folders into `~/.claude/skills/` will not make them appear in Cowork, no matter how many times you restart. #### Prerequisites - Claude Desktop latest version on macOS or Windows. Download from Anthropic's [Claude Desktop page](https://claude.ai/download). - Active internet connection; Cowork tasks call the Anthropic API. - Keep Claude Desktop open while Cowork tasks run. Cowork runs inside the Desktop process. - A paid plan with Cowork access. See Anthropic's [Cowork requirements](https://support.claude.com/en/articles/13345190-get-started-with-claude-cowork) for current plan availability. - **Code execution / file creation must be enabled** in **Settings → Capabilities**, or the Skills section will not appear. See Anthropic's [Use Skills in Claude](https://support.claude.com/en/articles/12512180-use-skills-in-claude). - On Team or Enterprise plans, your organization admin may have disabled Skills. If the Skills section is missing after enabling code execution, ask your admin to check org-level controls. #### Step 1: Build one zip per skill Clone the repo, then zip each of the five skill folders individually so that each zip has its own `SKILL.md` at the top level (not nested under an extra folder). The `-x "*.DS_Store"` flag keeps macOS metadata out of the archive. ```bash git clone https://github.com/Imbad0202/academic-research-skills.git cd academic-research-skills for s in deep-research academic-paper academic-paper-reviewer academic-pipeline sr-screener; do (cd "$s" && zip -r "../$s.zip" . -x "*.DS_Store") done ``` This produces five zips in the repo root: `deep-research.zip`, `academic-paper.zip`, `academic-paper-reviewer.zip`, `academic-pipeline.zip`, `sr-screener.zip`. Each zip's top level looks like (`sr-screener` also carries `scripts/`): ```text SKILL.md agents/ examples/ references/ templates/ ``` #### Step 2: Upload each zip 1. In Claude Desktop (or claude.ai — uploaded skills sync to the same account), go to **Settings → Capabilities → Skills**. 2. Use the **+** in the Skills panel to upload a skill, and select one of the five zips. Repeat for all five, one at a time. 3. Each skill then appears under **Personal skills**, already enabled, with **Trigger: Slash command + auto**. Re-uploading a skill with the same name replaces the existing one (useful when updating to a new ARS release). Verified on Claude Desktop (June 2026): `deep-research.zip` built this way installs cleanly, the full skill description is preserved (no 200-character truncation), and `/deep-research` appears in the Cowork command palette. #### Step 3: Use the skills in a Cowork Task Type `/` in a Cowork Task to open the command palette and select a skill, or describe your intent in plain language (e.g. "do a deep literature review on X") and Cowork routes by the skill's `description`. #### One trade-off versus Claude Code Uploaded this way, each skill runs on its own as a standalone instruction set. This is a different experience from Claude Code. In Claude Code the skills work as a coordinated team: `academic-pipeline` chains them (research → write → review → revise) and each skill drives its own group of sub-agents. Cowork's uploaded-skill runtime does not provide that sub-agent orchestration, so the individual skills respond, but the full end-to-end pipeline does not run the way it does in Claude Code. For the full orchestrated experience, install ARS in Claude Code via Method 0 (plugin) or Method 1 (project skills) above. ### Method 4: Use with claude.ai (web) ARS is a Claude Code-native suite. The skills are multi-agent teams (4 to 13 agents each) that depend on multi-agent orchestration, executable scripts under `scripts/`, and Material Passport file handoffs. claude.ai's web interface delivers a different runtime than Claude Code, and the two access paths it offers reach this repository in different ways: - **Method 4b — Project + GitHub integration** (recommended for claude.ai users): brings the repository into a claude.ai Project as retrievable knowledge. Claude can read the skill bodies, references, schemas, and example outputs, and answer questions or draft against them. Not a Skill install — auto-loading and skill routing do not happen, but the content is fully available for reading and citation. - **Method 4a — Custom Skill upload**: claude.ai's standard Skill install path (Settings → Capabilities → Skills, one zip per skill). Not recommended for this suite — see the rationale below before using it. #### Prerequisites - A claude.ai account. Plan availability differs by sub-method (see below). - **For Method 4b**: claude.ai Projects are available across plan tiers per Anthropic's [What are Projects?](https://support.claude.com/en/articles/9517075-what-are-projects); paid plans (Pro, Max, Team, Enterprise) get larger knowledge capacity and stronger retrieval. GitHub authentication is required through the Anthropic connector — see [Using the GitHub integration](https://support.claude.com/en/articles/10167454-using-the-github-integration) and [Set up Claude integrations](https://support.claude.com/en/articles/10168395-set-up-claude-integrations). Private repositories require the Anthropic GitHub App to be authorized on the repo or organization. Team and Enterprise plans require owner-level connector enablement before users can add GitHub-sourced files. - **For Method 4a**: Custom Skills are available on Free, Pro, Max, Team, and Enterprise per Anthropic's [Use Skills in Claude](https://support.claude.com/en/articles/12512180-use-skills-in-claude). The same article notes that Skills require **code execution to be enabled** in Settings → Capabilities. No GitHub authentication is needed for Method 4a — you zip each skill folder locally and upload one zip per skill through Settings → Capabilities → Skills. Zip structure errors and the 200-character `description` cap surface as upload-time errors; see Anthropic's [Custom Skills packaging documentation](https://claude.com/docs/skills/how-to) and [How to create custom Skills](https://support.claude.com/en/articles/12512198-how-to-create-custom-skills). #### Method 4b: Project + GitHub integration (recommended for claude.ai) claude.ai Projects deliver content as static knowledge for Claude to retrieve and cite — see Anthropic's [What are Projects?](https://support.claude.com/en/articles/9517075-what-are-projects). This is NOT a Skill install. Skill auto-loading does not happen. Trigger phrases do not route. Claude can read the repo content for reading and citation, and answer questions about it, but does not execute the skills as agentic workflows. Use this when you want claude.ai to have access to the repo content — including the agent definitions, references, and example outputs — for reading and citation, without needing agentic skill execution. For agentic execution, use Method 3 (Cowork) on the desktop, or Methods 1-2 in Claude Code. 1. Sign in to [claude.ai](https://claude.ai). 2. Create a new Project: **Projects** → **Create Project**. 3. Import from GitHub: in the Project, click **Files** → **+** → **GitHub** → select `Imbad0202/academic-research-skills`. 4. Select the folders/files below. | Select | Directory / file | Why | |---|---|---| | ✅ | `deep-research/` | Core skill content for reading | | ✅ | `academic-paper/` | Core skill content for reading | | ✅ | `academic-paper-reviewer/` | Core skill content for reading | | ✅ | `academic-pipeline/` | Core skill content for reading | | ✅ | `sr-screener/` | Core skill content for reading | | ✅ | `shared/` | Cross-model verification, handoff schemas, shared protocols | | ✅ | `scripts/` | `literature_corpus[]` adapters (`folder_scan`, `zotero`, `obsidian`) + schema validators; required for Material Passport corpus mode and CI-style validation | | ✅ | `MODE_REGISTRY.md` | Mode definitions | | Optional | `.claude/` | Project-level routing rules. Skip if you set Project Instructions in step 5 below (recommended path); include only if you prefer to keep routing rules visible as Project files. | | Optional | `examples/` | Useful for reference examples; skip if you want a smaller Project knowledge set | | Optional | `.github/`, READMEs, LICENSE, etc. | Repository metadata; not needed for core reading context | 5. (Recommended) Set **Instructions** in the Project to the content of `.claude/CLAUDE.md` for better routing. 6. Start chatting: "Guide my research on X" or "Help me write a paper about Y". Anthropic's current [Project file limits](https://support.claude.com/en/articles/8241126-upload-files-to-claude) state that Project file count is not artificially capped at 200; files have a 30 MB per-file limit and total usable content is still subject to context-window limits at runtime. Keep the Project focused so Claude retrieves the relevant files reliably. #### Method 4a: Custom Skill upload (not recommended for this suite) Method 4a is claude.ai's standard Custom Skill install path: zip each skill folder, upload through Settings → Capabilities → Skills, and Claude treats it as an installed Skill with auto-loading and routing. claude.ai's Custom Skills do support multi-file skill packages including `scripts/` (see Anthropic's [How to create custom Skills](https://support.claude.com/en/articles/12512198-how-to-create-custom-skills) on supporting files and code execution), so Method 4a is mechanically capable of hosting skills with executable assets. The reasons not to recommend it for this specific suite are different and compound: 1. **ARS depends on Claude Code-only orchestration features**. Each ARS skill drives 12-13 specialised agents through Claude Code's Task / subagent tooling and Material Passport file handoffs that resume across sessions. The Anthropic-documented scope of claude.ai's Custom Skill runtime — a containerised code-execution environment per session, with the Skills user guide ([Use Skills in Claude](https://support.claude.com/en/articles/12512180-use-skills-in-claude)) describing skill activation but not multi-agent dispatch — does not include Claude Code's Task / subagent control surface. Method 4a is therefore expected to surface ARS as the SKILL.md body's instructions, without the multi-agent dispatch that produces the suite's actual outputs. We have not run a live upload to characterise this in detail; the recommendation is forward-looking based on the Claude Code-specific assumptions baked into the agent orchestration, not on a measured failure. 2. **Cost to Claude Code and Cowork routing**. claude.ai limits each skill's `description` field to 200 characters per the [Custom Skills documentation](https://claude.com/docs/skills/how-to), while the [Agent Skills specification](https://agentskills.io/specification) and [Claude Code Skills documentation](https://code.claude.com/docs/en/skills) allow up to 1,024 characters. The five ARS descriptions each exceed the 200-character claude.ai cap while staying under the 1,024-character Claude Code allowance, front-loading routing keywords that Claude Code and Cowork use to discriminate between research, writing, review, and orchestration. Trimming them to fit Method 4a would weaken routing on Claude Code and Cowork — the platforms ARS was built for — in exchange for an unverified partial fit on claude.ai. **Recommended paths instead:** - For skill execution on the desktop, use [Method 3 (Cowork)](#method-3-claude-cowork-desktop). The five skills upload as standalone Cowork skills; the multi-agent pipeline orchestration is only available in Claude Code (Methods 0–2). - For claude.ai web access to the repo content, use [Method 4b (Project + GitHub integration)](#method-4b-project--github-integration-recommended-for-claudeai). Claude reads the skill bodies, references, and examples, and you can ask questions or draft against them in a normal claude.ai chat. - For Claude Code projects, use [Method 1 (project skills)](#method-1-as-project-skills-recommended) or [Method 2 (standalone)](#method-2-as-a-standalone-project). If you still want to try Method 4a despite the limitations above, zip each skill folder so the archive's top-level entry is `/SKILL.md` (not `//SKILL.md` — that nesting buries the discovery file one level too deep). The `zip -r` commands below produce that shape correctly: ```bash git clone https://github.com/Imbad0202/academic-research-skills.git cd academic-research-skills zip -r deep-research.zip deep-research zip -r academic-paper.zip academic-paper zip -r academic-paper-reviewer.zip academic-paper-reviewer zip -r academic-pipeline.zip academic-pipeline zip -r sr-screener.zip sr-screener ``` Then in claude.ai: 1. Sign in to [claude.ai](https://claude.ai). 2. Open **Settings**. 3. Open **Capabilities**. 4. Open **Skills**. 5. Upload `deep-research.zip`. 6. Upload `academic-paper.zip`. 7. Upload `academic-paper-reviewer.zip`. 8. Upload `academic-pipeline.zip`. 9. Upload `sr-screener.zip`. The upload UI will reject each zip with a description-too-long error because every ARS description exceeds claude.ai's 200-character cap. The descriptions are intentionally not trimmed; see the rationale above. **claude.ai vs Claude Code:** - Method 4b is for content reading, not active Skill execution. For agentic skill execution, prefer Methods 1-3. - claude.ai does not support local shell commands; results may be less comprehensive than Claude Code workflows that rely on local scripts. - Cross-model verification (`ARS_CROSS_MODEL`) requires Claude Code with API keys. - Direct `.docx` generation requires Pandoc, and LaTeX/PDF output requires Claude Code with `tectonic`; claude.ai can still produce Markdown and DOCX conversion instructions. ### Method 5: Claude Science import (v3.14.0+) Claude Science imports the five ARS skills straight from GitHub: 1. Open **Customize → Capabilities → Skills → Import from GitHub**. 2. Paste `https://github.com/Imbad0202/academic-research-skills` and click **Preview**. 3. All five skills (`academic-paper`, `academic-paper-reviewer`, `academic-pipeline`, `deep-research`, `sr-screener`) appear — click **Import** (the v3.14.0 check, before `sr-screener` was added, showed **Import 4 skills**). **Notes:** - Requires repo state v3.14.0+ — the importer reads the explicit skill paths declared in `.claude-plugin/marketplace.json`. Earlier tags exposed skills only through the symlinked `skills/` directory, which GitHub-API importers cannot traverse (they report "no skills/ dirs with SKILL.md"). - Imports are **point-in-time snapshots**: Claude Science does not track the repo. Re-import after an ARS release to pick up changes. - **What transfers:** the methodology layer — each skill's `SKILL.md` and its protocols (research / writing / review), which Claude Science's agent reads when relevant. - **What does not transfer:** Claude Code-specific machinery — the `/ars-*` slash commands, hooks (including the write-scope guard), cross-model verification scripts, and Task-tool subagent orchestration. Claude Science runs its own specialist-agent system and a built-in citation-checking reviewer; treat a Claude Science run as "ARS methodology + Claude Science's own machinery", not a 1:1 pipeline port. --- --- METRICS --- - Files Extracted: 45 - Estimated Token Budget: ~129565 tokens - Canonical Reference: https://codewiki.google/github.com/Imbad0202/academic-research-skills