academic-research-skills Documentation

Technical Documentation: Imbad0202/academic-research-skills

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

Academic Research Skills for Claude Code

简体中文版 | 繁體中文版 | 日本語版 | 한국어 | Español

A comprehensive suite of Claude Code skills for academic research, covering the full pipeline from research to publication.

Install in 30 seconds (Claude Code CLI / VS Code / JetBrains, v3.7.0+):

/plugin marketplace add Imbad0202/academic-research-skills
/plugin install academic-research-skills

Then try /ars-plan to walk through your paper structure via Socratic dialogue, or jump to Quick install for prerequisites and the traditional symlink flow.

AI is your copilot, not the pilot. It can draft text, including a whole paper in full mode, but the decisions stay yours, and the pipeline stops for your confirmation at every stage. It handles the grunt work (hunting down references, formatting citations, verifying data, checking logical consistency) so you can focus on the parts that actually require your brain: defining the question, choosing the method, interpreting what the data means, and deciding what comes after "I argue that." You remain the author, and you answer for every claim you submit.

Unlike a humanizer, this tool doesn't help you hide the fact that you used AI. It helps you write better. Style Calibration learns your voice from past work. Writing Quality Check catches the patterns that make prose feel machine-generated. The goal is quality, not cheating.

Why human-in-the-loop, not full automation? Copy Section

Lu et al. (2026, Nature 651:914-919) built The AI Scientist — the first fully autonomous AI research system to publish a paper through blind peer review at a top-tier ML venue (ICLR 2025 workshop, score 6.33/10 vs workshop average 4.87). Their Limitations section enumerates the failure modes that any fully-autonomous AI research pipeline inherits: implementation bugs, hallucinated results, shortcut reliance, bug-as-insight reframing, methodology fabrication, frame-lock, citation hallucinations.

ARS is built on the premise that a human researcher augmented by AI avoids these failure modes better than either alone. Stage 2.5 and Stage 4.5 integrity gates run a 7-mode blocking checklist (see academic-pipeline/references/ai_research_failure_modes.md); the reviewer offers an opt-in calibration mode that measures its own FNR/FPR against a user-supplied gold set.

Zhao et al. (2026-05) audited 111M references across 2.5M papers on arXiv, bioRxiv, SSRN, and PMC. Their conservative estimate is 146,932 hallucinated citations for 2025 alone, with an observed mid-2024 inflection; for the bioRxiv-to-PMC pairing they report 85.3% preprint-to-published persistence. The paper describes "real citations deployed to support claims the cited references do not actually make" as an open challenge. ARS v3.7.1 added trust-chain frontmatter for source provenance; v3.7.3 added locator infrastructure (three-layer citation anchors) for future claim-level audits and surfaces advisory risk signals at cite time (ARS labels the claim-faithfulness gap internally as "L3"; this is ARS terminology, not the paper's). v3.7.x is motivated by Zhao et al.'s corpus-scale findings; corpus-scale evaluation of ARS itself remains future work.

v3.8 closes the second half of the L3 gap. v3.7.3 made every citation carry a locator anchor; v3.8 adds an opt-in audit pass (ARS_CLAIM_AUDIT=1) that fetches the cited source against each anchor and judges whether the claim is actually supported. Five new HIGH-WARN classes (claim-not-supported, negative-constraint-violation, fabricated-reference, anchorless, constraint-violation-uncited) gate-refuse output through the formatter terminal hard gate. A calibration runner ships with a 25-tuple synthetic gold set and FNR<0.15 + FPR<0.10 acceptance thresholds. Its shipped test drives the runner with a stub judge that returns the gold labels, so it checks the tooling, not a live judge; no live-judge calibration result is recorded yet, and the ramp-on plan waits for one (v3.8 spec §5).

Ren et al. (2026, Self-Improvements in Modern Agentic Systems: A Survey) supplies a third, survey-level anchor. Its scientific-discovery synthesis (§7.4) concludes that discovery agents cannot easily verify novelty, correctness, or reproducibility on their own and may exploit weak proxies instead, must manage evidence across heterogeneous tools and literature, and raise governance issues — "scientific writing can also amplify misinformation when the evidence is weak." Its generation-loop chapters (§5.1–§5.2) list human auditing and retained human anchors among the practical safeguards for self-generated evaluation loops, and its historical chapter (§2.2) records the oldest form of the same lesson: the practical success of Lenat's EURISKO depended heavily on the user serving as the external evaluation signal, pruning unproductive heuristic drift — a limitation the survey notes persists in modern agentic systems. ARS cites the survey as design rationale for its human-in-the-loop stance, not as empirical proof that human-in-the-loop pipelines outperform autonomous ones; the survey's actionable deltas for ARS are tracked in #539–#541 and #547–#550.

Gartenberg et al. (2026, Organization Science 37(3):795-812, "More versus better") supplies a fourth anchor, and the first from the journal side. The Organization Science AI Task Force scored every first submission (6,957) and every text-format review (10,389) the journal received between January 2021 and February 2026 with a commercial AI-writing classifier and standard readability indices. Manuscripts scored as heavily AI-written read worse on those indices and were desk-rejected more often; reviews scored as more AI-written leaned toward theory and away from data; and the editors conclude that current AI tools, amplified by publish-or-perish incentives, "appear to be pushing the system toward an equilibrium of more rather than better research." Their §5 contrasts "cognitive surrender" (Shaw & Nave, 2026, as cited there) with human-first use and asks authors to disclose how a manuscript was produced. The evidence is observational, aggregate, and from one journal, and the classifier is a proprietary instrument. ARS cites the editorial as design rationale for recording volume as a non-goal (see POSITIONING.md) and for the Collaboration Depth Observer and the claim-strength ladder, not as evidence about ARS output; the actionable deltas are tracked in #829–#833.

Wang, Li et al. (2026-09, The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishing, a survey of 230 sources) supplies a fifth anchor, and the first that treats research production and peer review as one coupled system. Its evaluative-authority ladder (§4.1) runs from author-facing feedback through reviewer assistance and official AI reviews to scoring and decision support, with the survey's point that capability at one rung does not justify use at the next; ARS's simulated panel sits on the lowest rung by design (see POSITIONING.md). Two of its findings shape the reviewer roadmap. First, as the survey summarizes Dycke & Gurevych (2026, §4.5), 391 edits that break a paper's scientific support relations produced no statistically significant difference in the tested automated reviewers' aspects, sentiment, or scores compared with 540 soundness-neutral controls, while presentation-only rewrites with the science held fixed moved AI-review scores (§5.2); the survey's §9.2 conclusion is that a static evaluation can overstate an AI reviewer's reliability once authors can observe and adapt to it. ARS tracks the corresponding Round-1 paired controls in #871 and the author-identity-cue controls (§7.2) in #872; both are measurements, not new mechanisms. Second, its §9.1 cites Brodeur et al. (2026, PNAS 123(22):e2524747123), a randomized study in which 288 researchers in 103 teams reproduced published quantitative social-science results under three conditions: human-only, AI-assisted (ChatGPT as a collaborative tool), and AI-led (ChatGPT with minimal human oversight). Human-only and AI-assisted teams reproduced 94% and 91%, AI-led teams 37%, and the AI-assisted teams detected fewer major coding errors than the human-only teams. In that study, then, AI assistance did not verify better than humans alone and AI-led verification did much worse; ARS reads this as a reason to keep verification human-led at every checkpoint, not as evidence that its own checkpoints or integrity gates are effective. The survey is a synthesis rather than an experiment, its structured search stops at 2026-07-01 and the later targeted update did not rerun every query (§10), its deployment evidence is concentrated in a small number of AI/CS conferences, OpenReview-based settings, and selected journals, and its "arms race" framing is a lens, not a finding; ARS cites it as design rationale, not as evidence about ARS output.

v3.3 was inspired by PaperOrchestra (Song, Song, Pfister & Yoon, 2026, Google): Semantic Scholar API verification, anti-leakage protocol, VLM figure verification, and revision-trajectory tracking. ARS now implements that last idea through categorical, evidence-anchored criterion trajectories rather than score deltas.

Architecture & pipeline Copy Chapter

👉 docs/ARCHITECTURE.md — the full pipeline view: flow diagram, stage-by-stage matrix, data-access flow, skill dependency graph, quality gates, and mode list.

The architecture doc supersedes the sprawling pipeline description that used to live here. Everything about what runs in which stage now lives in one place.

Quick install Copy Chapter

Prerequisites

• Claude Code (latest; plugin packaging requires recent versions)
• ANTHROPIC_API_KEY exported, or set on first claude run
• Optional: Pandoc for DOCX, tectonic + Source Han Serif TC for APA 7.0 PDF (Markdown output works without either)
• Optional (real Python): needed only for the write-scope guard and a few opt-in commands; the core skills are prompt-driven. Details, including the Windows notes on Git Bash and the Microsoft Store Python stub, are in docs/SETUP.md § Python (optional).

Which controls are active in your install channel? Availability varies by install channel. See the per-channel map: docs/CONTROL_AVAILABILITY.md.

Plugin install (v3.7.0+, recommended):

/plugin marketplace add Imbad0202/academic-research-skills
/plugin install academic-research-skills

Verify it works: run /ars-plan and describe a paper you're working on — ARS will start a Socratic dialogue to map out chapter structure. For a single-shot test instead, try /ars-lit-review "your topic".

👉 docs/SETUP.md — full guide: install Claude Code, set up API keys, optional Pandoc/tectonic for DOCX/PDF, cross-model verification (ARS_CROSS_MODEL), and six installation methods (Plugin, project skills, global skills, claude.ai Project, repo-cloned, Claude Science import).

👉 docs/DATA_FLOWS.md — what leaves your machine (bibliographic resolvers, optional consent-gated cross-model calls, the plugin update check), what is cached locally, for how long, and how to turn each path off.

👉 docs/RISK_REGISTER.md — the standing risks the suite knows about, which existing controls address each one, the evidence status behind those controls, and what remains open.

Using Claude Science? The five skills import directly: Skills → Import from GitHub, paste https://github.com/Imbad0202/academic-research-skills, Preview, then Import (requires v3.14.0+ of this repo — the importer reads the explicit skill paths in the marketplace manifest). Imports are point-in-time snapshots: re-import after ARS updates. Imported skills carry the ARS methodology (research / writing / review protocols); Claude Code-specific machinery — slash commands, hooks, subagent orchestration — does not transfer. See docs/SETUP.md Method 5 for details.

Using Pi? Install the in-tree, community-maintained wrapper with pi install git:github.com/Imbad0202/academic-research-skills. It keeps the original ARS content authoritative and documents Pi-specific orchestration and hook limitations. See pi/README.md.

Using Codex CLI? Install the sibling distribution instead: Imbad0202/academic-research-skills-codex — same workflow content, Codex-native packaging as a single $academic-research-suite skill with ars-* aliases.

Third-party platforms and integrations that wrap or host ARS are listed in THIRD_PARTY.md — community-submitted and not reviewed or endorsed by the maintainer.

Governance: who decides, what cross-model review does and does not provide, and the project's end-of-life posture are stated in GOVERNANCE.md; security reporting and triage in SECURITY.md.

Performance & cost Copy Chapter

👉 docs/PERFORMANCE.md — per-mode token budgets, full-pipeline estimate (about US$3–7 for a 15k-word paper at 2026-09 list prices, before cache discounts), and recommended Claude Code settings (Auto mode; Agent Team optional).

Guides & articles Copy Chapter

• Academic Writing Shouldn't Be a Solo Act — full pipeline walkthrough (English)
• 學術寫作不該是一個人的事：一套開源 AI 協作工具如何改變研究者的工作流 — 完整使用指南（繁體中文）

Features at a glance Copy Chapter

• Deep Research — 13-agent research team with Socratic guided mode, PRISMA systematic review, intent detection, dialogue health monitoring, optional cross-model DA, Semantic Scholar API verification.
• Academic Paper — 12-agent paper writing with Style Calibration, Writing Quality Check, LaTeX hardening, visualization, revision coaching, citation conversion, anti-leakage protocol, and VLM figure verification.
• Academic Paper Reviewer — 7-agent multi-perspective peer review with criterion-bound, evidence-anchored narrative judgements (Journal-Fit Reviewer + 3 dynamic reviewers + Devil's Advocate), concession threshold protocol, attack intensity preservation, optional cross-model DA critique / calibration, R&R traceability matrix, read-only constraint. Current live reviews remain NOT_CALIBRATED; full calibration produces a bounded candidate profile, while live-profile application is not yet wired.
• Academic Pipeline — 10-stage pipeline orchestrator with adaptive checkpoints, claim verification, Material Passport, optional repro_lock, optional cross-model integrity verification, mid-conversation reinforcement, and narrative criterion-by-criterion regression checks (the typed trajectory carrier is deferred).
• SR-Screener — protocol-driven study screening for systematic, scoping and rapid reviews: two blinded AI reviewers plus a third-reviewer adjudicator, ordered exclusion codes, no silent defaults, resumable batch runs, QC (seed studies, near-miss rechecks, kappa and PABAK), PRISMA 2020 counts, EndNote/Zotero RIS groups, and a literature_corpus[] handoff to academic-paper. AI decisions are decision support; the review team verifies them.
• Data Access Level Metadata (v3.3.2+) — every skill declares data_access_level (raw / redacted / verified_only); enforced by scripts/check_data_access_level.py. Pattern adapted from Anthropic's automated-w2s-researcher (2026). See shared/ground_truth_isolation_pattern.md.
• Task Type Annotation (v3.3.2+) — every skill declares task_type (open-ended or outcome-gradable). All current ARS skills are open-ended.
• Benchmark Report Schema (v3.3.5+) — JSON Schema + lint for honest benchmark comparisons. See shared/benchmark_report_pattern.md.
• Artifact Reproducibility Lockfile (v3.3.5+) — optional repro_lock sub-block on Material Passport. Configuration documentation, not replay guarantee — LLM outputs are not byte-reproducible. See shared/artifact_reproducibility_pattern.md.
• Model Tiering (#517, v3.16+) — optional ARS_MODEL_TIERING switch with two directions: economy (execution-type agents dispatch one tier below the session model, floor Opus-class) and quality-boost (judgment-type agents at integrity gates and final review step up to the frontier tier). Default unset = byte-equivalent to pre-#517 behavior. See shared/model_tiering.md.
• Canonical Cross-Model Handoff Envelope (#527, v3.17+) — the owner→dispatcher→owner blind-checkpoint transport path (#523) now has a machine-stable [CROSS-MODEL-HANDOFF v1] envelope with a normative Python grammar (scripts/cross_model_handoff.py) instead of prose-only enforcement, pinning agreement/divergence/malformed-result routing across all three checkpoint owners. See shared/cross_model_verification.md §"Cross-model handoff envelope".
• Experiment Provenance Intake (#260) — optional experiment_provenance[] on the Material Passport records experiments the scholar ran externally (ARS never runs experiments), and manuscript claims join to them via claim_intent_manifest.planned_experiment_ids[]. The integrity gate (Stage 2.5/4.5) audits each experiment-backed claim against declared provenance — ALIGNED / OVERSTATED / NOT_SUPPORTED_BY_PROVENANCE / PROVENANCE_INSUFFICIENT — without judging whether the experiment itself was correct. A fail-closed experiment_intake_declaration makes "did you run experiments?" an explicit decision the scholar answers after Stage 1, or on entry when a run starts later (even literature-only runs declare no_experiments_declared). See shared/handoff_schemas.md §"Experiment Provenance Intake (#260)".

Integrity and verification boundary: ARS checks the manuscript and the reported process—including citation existence, claim–source alignment, reported methodology, declared experiment–result alignment, figure/table fidelity, and reporting/process/package conformance. Some checks are sampled or LLM-mediated. ARS does not establish that procedures were actually performed, raw data are authentic, or results reproduce; a consistently reported fabrication can pass these checks. See POSITIONING.md § Integrity checks and the empirical-work boundary.

Showcase: real pipeline output Copy Chapter

See the complete artifacts from a real pipeline run, including peer review reports, integrity verification reports, and the final paper:

A March 2026 record, not current performance. This run (2026-03-07 to 03-08) used academic-pipeline v2.3, before ARS v3.3 added the Semantic Scholar check and v3.11 added the deterministic four-index citation gate. Its numbers describe that version; the current gates have not been measured on this paper. The byline names Claude (Anthropic) as author because the researcher asked for that during the experiment. The paper is not an Anthropic publication, and ARS's positioning is that the tool does not replace the researcher and does not claim authorship (POSITIONING.md).

Browse all pipeline artifacts →

Companion: Experiment Agent Copy Chapter

If your research involves running experiments (code or human studies) before writing, the Experiment Agent skill fills the gap between ARS Stage 1 (RESEARCH) and Stage 2 (WRITE).

ARS Stage 1 RESEARCH  →  RQ Brief + Methodology Blueprint
        ↓
  experiment-agent     →  run/manage experiments → validate results
        ↓
ARS Stage 2 WRITE     →  write paper with verified experiment results

What it does: executes code experiments (Python, R, etc.) with real-time monitoring, manages human study protocols with IRB ethics checklist, interprets statistics with 11-type fallacy detection, and verifies reproducibility.

How to use together: pause the ARS pipeline after Stage 1, run experiments in a separate experiment-agent session, then bring the results (with Material Passport) back to ARS Stage 2. ARS requires zero modification. See the experiment-agent README for setup instructions.

Stage 1 intake declaration (#260, #925): at the checkpoint after Stage 1, or on entry when a run starts after Stage 1, ARS asks you whether the paper reports experiments or data analyses you ran yourself, and records your answer, in your words, in a fail-closed experiment_intake_declaration on the Material Passport. ARS does not infer the answer from the manuscript. If you ran experiments externally, the scholar enters one experiment_provenance[] entry per experiment (experiment_id, nested repro_lock, planned_vs_executed[], negative_results[], known_limitations[]) and the declaration is set to experiments_declared; if not, it is set to no_experiments_declared. The declaration is required on every post-#260 passport — a run that touches no experiments still declares no_experiments_declared, so the integrity gate can never be silently bypassed by a forgotten provenance block. The experiment_ids are frozen at this intake point, which closes before the first Stage 2 writer is dispatched (or before the first integrity gate when no Stage 2 lies ahead), so you can pause after Stage 1, run the experiments, and record them when you return; the writers later reference them via planned_experiment_ids[].

Teaching-side companion: Teaching Skills applies the ARS architecture (skill ensembles, shared contracts, staged gates, a Course Passport) to the teaching side of academic life — course design → lessons → assessment → delivery → reflection; its sotl mode hands classroom-inquiry projects off to ARS deep-research / academic-paper for the publication phase.

Usage Copy Chapter

Quick Start Copy Section

# Start a full research pipeline
You: "I want to write a research paper on AI's impact on higher education QA"

# Start with Socratic guidance
You: "Guide my research on AI in educational evaluation"

# Write a paper with guided planning
You: "Guide me through writing a paper on demographic decline"

# Review an existing paper
You: "Review this paper" (then provide the paper)

# Check pipeline status
You: "status"

Individual Skills Copy Section

Deep Research (8 modes)

"Research the impact of AI on higher education"       → full mode
"Give me a quick brief on X"                          → quick mode
"Do a systematic review on X with PRISMA"             → systematic-review mode
"Guide my research on X"                              → socratic mode (guided)
"Fact-check these claims"                             → fact-check mode
"Do a literature review on X"                         → lit-review mode
"Compare these papers in WHY/HOW/WHAT format"         → three-way-scan mode
"Review this paper's research quality"                → review mode

Academic Paper (11 modes)

"Write a paper on X"                                  → full mode
"Guide me through writing a paper"                    → plan mode (guided)
"Build a paper outline"                               → outline-only mode
"I have a draft, here are reviewer comments"          → revision mode
"Parse these reviewer comments into a roadmap"        → revision-coach mode
"Write an abstract for this paper"                    → abstract-only mode
"Turn this into a literature review paper"            → lit-review mode
"Convert to LaTeX" / "Convert citations to IEEE"      → format-convert mode
"Check citations"                                     → citation-check mode
"Generate an AI disclosure statement for NeurIPS"     → disclosure mode
"Audit my rebuttal draft against the reviews"         → rebuttal-audit mode

Academic Paper Reviewer (6 modes)

"Review this paper"                                   → full mode (Journal-Fit Reviewer + R1/R2/R3 + Devil's Advocate)
"Quick assessment of this paper"                      → quick mode
"Guide me to improve this paper"                      → guided mode
"Check the methodology"                               → methodology-focus mode
"Verify the revisions"                                → re-review mode
"Calibrate this reviewer against my gold set"         → calibration mode

Academic Pipeline (Orchestrator)

"I want to write a complete research paper"           → full pipeline from Stage 1
"I already have a paper, review it"                   → mid-entry at Stage 2.5 (integrity first)
"I received reviewer comments"                        → mid-entry at Stage 4

Pipeline ends with Stage 6: Process Summary — auto-generates a paper creation process record with 6-dimension Collaboration Quality Evaluation (1–100 scoring).

SR-Screener (8 modes)

"Turn my proposal into a screening protocol"          → protocol mode
"Is this abstract eligible for my review?"            → quick mode (single-reviewer triage)
"Pilot the screening with my seed studies"            → pilot mode
"Screen these database exports"                       → ta-screen mode
"Screen the full texts of the advanced records"       → ft-screen mode
"Adjudicate the conflicts in my Rayyan export"        → adjudicate mode
"Double-check my exclusions"                          → audit mode
"Give me the PRISMA numbers for the screening"        → report mode

Supported Languages Copy Section

• Traditional Chinese (繁體中文) — default when user writes in Chinese
• English — default when user writes in English
• Bilingual abstracts (Chinese + English) for academic papers

Using a different language? Socratic mode (deep-research) and Plan mode (academic-paper) use intent-based activation — they detect the meaning of your request, not specific keywords. This means they work in any language without modification.

However, the general Trigger Keywords section (which determines whether the skill is activated at all) still lists English and Traditional Chinese keywords. If you find the skill isn't activating reliably in your language, you can add your language's keywords to the ### Trigger Keywords section in each SKILL.md file to improve matching confidence.

Supported Citation Formats Copy Section

• APA 7.0 (default, including Chinese citation rules)
• Chicago (Notes & Author-Date)
• MLA
• IEEE
• Vancouver

Supported Paper Structures Copy Section

• IMRaD (empirical research)
• Thematic Literature Review
• Theoretical Analysis
• Case Study
• Policy Brief
• Conference Paper

Skill Details Copy Chapter

Per-agent responsibilities and per-stage artifacts now live in docs/ARCHITECTURE.md. Version numbers are anchored here so release metadata stays in one place.

Deep Research (v2.12.1) Copy Section

13-agent research team. Modes: full, quick, review, lit-review, three-way-scan, fact-check, socratic, systematic-review. Full agent roster and artifacts: see ARCHITECTURE.md §3.

Academic Paper (v3.3.1) Copy Section

12-agent paper writing pipeline. Modes: full, plan, outline-only, revision, revision-coach, abstract-only, lit-review, format-convert, citation-check, disclosure, rebuttal-audit. Output: MD + DOCX (via Pandoc when available) + LaTeX (APA 7.0 apa7 class / IEEE / Chicago) → PDF via tectonic. Full agent roster and per-phase responsibilities: see ARCHITECTURE.md §3.

Academic Paper Reviewer (v1.11.1) Copy Section

7-agent multi-perspective review with criterion-bound narrative judgements. Modes: full, re-review, quick, methodology-focus, guided, calibration. Current live reviews and Schema 6 packages remain NOT_CALIBRATED; full calibration can produce a bounded candidate profile, but application to a live review is not wired. No numerical total is mapped to Accept, Minor Revision, Major Revision, or Reject. First-round review panel vs. contract-governed re-review dispatch boundary: see ARCHITECTURE.md §3 Stage 3 / Stage 3'.

Academic Pipeline (v3.23.0) Copy Section

10-stage orchestrator with integrity verification, two-stage review, Socratic coaching, and collaboration evaluation. Pipeline rules (protocol the agents follow, not runtime guarantees): every stage requires user confirmation checkpoint; integrity verification (Stage 2.5 + 4.5) is MANDATORY with no unrecorded bypass (every override requires user reasoning recorded for Stage 6); R&R Traceability Matrix (Schema 11) maps each reviewer concern to the author's revision claim and records whether the re-review verified it. v3.4 added the Compliance Agent (PRISMA-trAIce + RAISE) at Stage 2.5 / 4.5. v3.5 adds the Collaboration Depth Observer (collaboration_depth_agent, advisory only — never blocks) at every FULL/SLIM checkpoint and at pipeline completion. MANDATORY integrity gates (2.5 / 4.5) explicitly skip the observer so compliance checks are not diluted. Based on Wang & Zhang (2026), IJETHE 23:11. Stage-by-stage matrix with agents, artifacts, and gates: see ARCHITECTURE.md §3.

SR-Screener (v1.0.0) Copy Section

4-agent study screening between deep-research (question, protocol, search) and academic-paper (writing the review). Modes: protocol, quick, pilot, ta-screen, ft-screen, adjudicate, audit, report. Two blinded reviewer subagents (Read and Grep only) screen every record against the user-confirmed protocol, a third reviewer adjudicates advance-vs-exclude conflicts, and standard-library Python scripts parse RIS / PubMed .nbib / Web of Science / CSV exports, de-duplicate, batch, merge, and build the screening log, RIS groups, PRISMA 2020 counts, a methods draft with [TO COMPLETE] slots, and a literature_corpus[] file. Rules the agents follow (not runtime guarantees): no record is screened before the user confirms the protocol, no failed call becomes a default "exclude", and the review team verifies the decisions before the numbers are reported. See sr-screener/SKILL.md.

v3.0 Optimizations: What We Discovered About AI's Structural Limits Copy Chapter

What happened Copy Section

While using ARS to write a reflection article about AI in higher education, I ran into three structural problems that no amount of prompt engineering could fix:

1. Frame-lock: I asked the AI to run a devil's advocate debate against its own thesis. It did — four rounds, each more refined than the last. But every round stayed inside the frame I'd set. The DA attacked arguments, never premises. It never asked "are we even discussing the right question?" This is the same pattern that caused the 31% citation error rate in v2.7's stress test: the verifying AI and the generating AI share the same cognitive frame.

2. Sycophancy under pushback: Every time I challenged the DA's attacks, it conceded too quickly. It retracted findings faster than it launched them. The model's training rewards conversational harmony — so "the user pushed back" was treated as evidence that the attack was wrong, when often it just meant the user was persistent.

3. Intent misdetection: The Socratic Mentor kept trying to converge and produce deliverables ("Want me to write this up?") when I was still exploring. It couldn't distinguish "the user wants a deep philosophical discussion" from "the user wants an RQ brief." Both look like engagement, but they need opposite AI behaviors.

What we changed (v3.0) Copy Section

Devil's Advocate — Concession Threshold Protocol (deep-research + academic-paper-reviewer)

• DA must now score every rebuttal on a 1-5 scale before responding
• Concession only allowed at score ≥4 (rebuttal directly addresses core attack with evidence)
• Score ≤3: hold position and restate the original attack
• Anti-sycophancy rules: no consecutive concessions, concession rate tracking, frame-lock detection after each checkpoint

Socratic Mentor — Intent Detection Layer (deep-research)

• Classifies user intent as exploratory vs. goal-oriented at dialogue start and every 3 turns
• Exploratory mode: disables auto-convergence, raises max rounds to 60, prohibits "want me to summarize?" prompts
• Goal-oriented mode: standard convergence behavior
• Anti-premature-closure rules: in exploratory mode, the user decides when to stop

Socratic Mentor — Dialogue Health Indicator (deep-research)

• Silent self-assessment every 5 turns on three dimensions: persistent agreement, conflict avoidance, premature convergence
• Auto-injects challenging questions when agreement pattern detected
• Invisible to user (to prevent gaming), but log available for post-session review

Why this matters Copy Section

These optimizations don't solve AI's structural limits — they make the limits visible and manageable. The DA will still eventually concede if pushed hard enough. The Socratic Mentor will still have some convergence bias. But now there are explicit checkpoints that slow down the sycophancy, force the DA to justify concessions, and prevent the Mentor from wrapping up before the user is ready.

The deeper lesson: AI literacy isn't about learning to use AI as a tool, following ethics rules, or fearing AI risks. It's about engaging AI deeply enough to discover its structural limits yourself — and your own thinking limits in the process.

License Copy Chapter

This work is licensed under CC-BY-NC 4.0.

You are free to:

• Share — copy and redistribute the material
• Adapt — remix, transform, and build upon the material

Under the following terms:

• Attribution — You must give appropriate credit
• NonCommercial — You may not use the material for commercial purposes

Attribution format:

Based on Academic Research Skills by Cheng-I Wu
https://github.com/Imbad0202/academic-research-skills

Contributors Copy Chapter

Cheng-I Wu (吳政宜) — Author and maintainer

aspi6246 — Contributor. The v3.1 optimization was inspired by patterns from Claude-Code-Skills-for-Academics: read-only constraint pattern, anti-pattern codification as first-class design, cognitive framework approach (teaching "how to think" not just procedures), and lean skill size philosophy.

mchesbro1 — Contributor. Originally proposed and drafted the IS Basket of 8 journals for academic-paper-reviewer/references/top_journals_by_field.md (Issue #5).

cloudenochcsis — Contributor. Extended the IS section from the Basket of 8 to the full Senior Scholars' Basket of 11 — adding Decision Support Systems, Information & Management, and Information and Organization (Issue #7, PR #8). Sourced from the AIS Senior Scholars' List of Premier Journals.

eltociear (Ikko Eltociear Ashimine) — Contributor. Translated the Japanese README (README.ja-JP.md) (PR #161).

xpfo-go (xpfo) — Contributor. Translated the Simplified Chinese README (README.zh-CN.md) (PR #181).

devCharlotte — Contributor. Translated the Korean README (README.ko-KR.md) (PR #469).

Yaobin29 — Contributor. Proposed reviewer-response tooling in PR #433; the deep-research three-way-scan mode and the academic-paper rebuttal-audit mode (rescued from the PR's audit concept) were integrated from that contribution in v3.12.1.

ktao732084-arch — Contributor. Expanded the academic-paper disclosure system with nine medical-publishing policy targets, target-specific required-fact intake, and fail-closed standalone rendering (Issue #596, PR #599); expanded the EQUATOR clinical-reporting reference with condensed CARE, STARD and TRIPOD+AI guidance plus a fail-closed study-design routing sequence (Issue #594, PR #601); and designed and contributed the standalone Chinese-literature resolver, API protocol, and synthetic transport-fixture suite (Issue #595, PR #600).

didacrios — Contributor. Translated the Spanish README (README.es-ES.md).

Changelog Copy Chapter

Only the three most recent releases are summarized here. The full release history, including the one-paragraph summaries that used to live on this page, is in CHANGELOG.md.

v3.23.0 (2026-10-03) — `sr-screener` as a fifth skill, repairs from a pipeline walk-through, and evidence and ledger hardening Copy Section

One new skill and behaviour repairs found by reading a full pipeline run end to end; the prompt-level changes are unmeasured, and sr-screener makes no screening-accuracy claim: v3.23.0 adds sr-screener (#919, contributed by @erfanz97), a fifth skill that turns a review protocol into eligibility rules the user confirms and screens titles/abstracts and full texts with two blinded reviewer subagents and an adjudicator; its tests use synthetic records, and its spreadsheet exports neutralise formula text (#951). A walk-through of a default pipeline run on paper (#925-#929) changed what such a run does: the v3.6.7 Audit Artifact Gate is opt-in, the scholar is asked once whether the paper reports their own experiments, constraints set for a run are kept in the user's words and quoted to each later dispatch they apply to (at review stages they act at the checkpoints), the integrity gates follow one rule set and list paywalled sources as notes, Stages 5 and 6 produce only the files asked for, and opt-in switches surface Stage 5 refusals at Stage 4.5. The Stage 2.5 and 4.5 checkpoints replay evidence rows from a folder the orchestrator names (#933, #947, #948), a source the gate finds fabricated stays out of later revisions (#936), and the ledger readers report parse failures without quoting the file (#898, #945). Smaller changes: per-source method weaknesses in reading outputs (#916), a review-form note at a fixed point (#921), Schema 1 aligned with its producer (#938), and /ars-citation-check inheriting the session model (#912).

v3.22.2 (2026-09-25) — Run ledger and handoff check, acronym check, a wider instruction/data boundary, and routing and front-page repairs Copy Section

Two deterministic checks pinned by synthetic tests; the prompt-level changes are unmeasured: v3.22.2 adds a run ledger (#887). When a pipeline run has a passport file, the orchestrator appends the user's initial instructions, each checkpoint's question and the user's answer in their exact words, step receipts, and file hashes to a local ledger beside the passport. After compaction, resume, or a subagent return, scripts/run_ledger.py report compares the ledger with what the summary or report claims and lists the differences; it now prints that handoff check itself, in English or Traditional Chinese, and hashes the files an entry names when the entry is written (#898). The ledger holds the user's exact words, and docs/DATA_FLOWS.md lists it and how to delete it. The release also adds scripts/check_acronyms.py (#849, proposed by @reiropke), which reports acronyms that are undefined, defined after first use, or defined twice, without calling a model; the prompts have the calling session run it on saved drafts and abstracts, and a review appends its report to the Editorial Decision Letter as an advisory attachment that the decision, the revision roadmap, and re-review criteria do not draw on. Synthetic tests pin both scripts; whether runs write the ledger entries or call the check is unmeasured. The instruction/data boundary now reaches third-party text in dispatches and passport imports (#890), text a receiver reads through its own tool calls, and each skill's main session (#894); a lint pins every copy, and the effect is unmeasured (the opt-in claim-audit judge prompt changes with it, so cached verdicts from the old prompt are not reused). Repairs: the routing core reaches plugin and skills-copy installs (#892); an explicit request stays explicit when the mode's usual input is missing (#889); /ars-lit-review no longer points a run under way to another workflow (#897); the revision coach keeps peer review out of the committee-correspondence variant (#854); the orchestrator scopes "authoritative" skill output to deliverable ownership (#888); and the front page and showcase now match their sources (#908). The routing results come from one session per fixture, a smoke test rather than a rate. One new schema describes the ledger; no existing schema, command model, or effort setting changes.

v3.22.1 (2026-09-23) — Model currency for Claude Opus 5.5, citation-check loading and Chinese APA 7 repairs, and a Pi wrapper fix Copy Section

Model currency and repairs; the new prompt-level guard is unmeasured: v3.22.1 names Claude Opus 5.5 beside Claude Fable 5.1 as a supported session model, after a two-reader audit of the Opus 5.5 system card that retires no guardrail (#883). The docs add effort guidance (Claude Code starts Opus 5.5 at medium; heavy runs should use high or above), one list-price re-derivation for both models, and tiering guidance that reads the ladder as the vendor's lineup order, not a capability order. Because the card reports that Opus 5.5 acts more often than earlier models on instructions inside pasted text, the revision coach now treats pasted reviewer and committee text as data, pinned by a lint; the effect of that prompt-level guard is unmeasured. The release also repairs mode loading and citation checks: the 13 plugin mode commands invoke their namespaced core skill and resolve bundled references from the plugin root, which restores citation-check loading (#857); Chinese APA 7 checks catch a missing in-text author abbreviation, keep ambiguity exceptions and complete reference-list author fields, and propose a reorder only with evidence of a stroke-order inversion (#882); citation checks now distinguish visible syntax errors from unverified resolution or source claims (#882); and new English, Traditional Chinese, and Korean trigger phrases route citation-check requests, with CI bounding each skill description at 1,024 code points (#858, #864). The Pi wrapper accepts string-array system prompts (#880). No schema, command model, or effort setting changes.

Overview (`evals/README.md`) Copy Section

evals/ — gold-set corpora for ARS measurement targets

This directory holds the v3.10 #184 generalized gold sets. Each subdirectory under gold/ is a self-contained gold set for one measurement target. The structure mirrors the v3.8 scripts/fixtures/claim_audit_calibration/ pattern but generalizes to multiple targets.

Layout Copy Chapter

evals/
├── README.md                          # this file
├── gold/
│   ├── citation_extraction/           # Phase 1a/1b — baseline for #182
│   │   ├── README.md
│   │   ├── manifest.yaml
│   │   ├── tuples/
│   │   │   └── NNN-{kind-slug}-{discriminator}.json
│   │   └── expected_outcomes.json
│   ├── rq_framing_patterns/           # #257 Socratic wording advisory calibration
│   ├── status_classification/         # Phase 2 (lands post-#183)
│   └── summarization_adequacy/        # Phase 2 (lands post-#183)

Running the harness (#184 Phase 1b) Copy Chapter

The multi-task harness scripts/run_evals.py discovers every gold/<task>/manifest.yaml, measures each task, and emits a report shaped by shared/evals_lift_report.schema.json:

PYTHONPATH=. python -m scripts.run_evals                          # all tasks
PYTHONPATH=. python -m scripts.run_evals --task citation_extraction --output report.json
PYTHONPATH=. python -m scripts.run_evals --baseline before.json --compare after.json

--baseline + --compare produce a side-by-side report (lift_pre / lift_post). The ranking-lift gate scripts/check_ranking_lift.py reads those reports and blocks on un-acknowledged regressions; CI wires both via .github/workflows/eval-harness.yml (Delta 3 path filter). Tasks whose entrypoint module or gold set is not yet present are reported as pending/skipped rather than failing.

Authoring conventions Copy Chapter

See each task's README.md for task-specific conventions (tuple naming, kind distributions, expected outcomes shape).

Validator Copy Chapter

Run python -m scripts.check_evals_gold_set evals/gold/<task> to validate any gold set against its manifest. The same validator runs in CI on every PR that touches evals/gold/**.

Provenance Copy Chapter

• Phase 1a (citation-extraction gold set) + Phase 1b (run_evals.py harness + lift gate): v3.10 #184, spec docs/design/2026-05-21-v3.10-184-extend-eval-harness-spec.md
• RQ framing patterns: Kong #257 idea-diversity advisory, spec docs/design/2026-05-28-kong-257-idea-diversity-coverage-gap-advisory.md
• Phase 2 (status + summarization): scheduled post-#183 ship

Overview (`pi/README.md`) Copy Section

Pi wrapper

Thin, community-maintained compatibility wrapper for running the original Claude Code Academic Research Skills (ARS) in Pi.

The wrapper does not copy or modify ARS skill, agent, reference, schema, script, or command content. Pi loads the four original SKILL.md trees and exposes the original commands/ars-*.md files as prompt templates. A small input adapter reads the selected original command at invocation time and preserves trailing user arguments.

Load-bearing runtime differences from Claude Code Copy Chapter

These differences affect how ARS evidence should be interpreted:

1. The wrapper does not provide agent isolation or orchestration. If the Pi setup has no matching subagent, workflow, or parallel-agent capability, ARS specialist roles run sequentially in the current context. That is degraded execution and must be disclosed; it is not independent multi-agent review.
2. Claude hooks do not run in Pi. In particular, ARS write-scope enforcement remains prompt-level unless the user's Pi environment supplies a separate enforcement mechanism. Do not represent the Claude PreToolUse hook boundary as active.

Run /ars-pi-doctor to inspect the current environment. The curated degraded-mode end-to-end evidence shows both boundaries in a completed run.

Requirements Copy Chapter

• Pi with package support (pi install / pi -e)
• The complete repository package, loaded from GitHub or from a local checkout; pi/ is only a wrapper around its parent files and cannot be copied by itself
• Optional capabilities depend on the user's Pi setup:
  • a subagent, workflow, or parallel-agent skill/tool for true multi-agent execution
  • a web-search or page-retrieval skill/tool for literature search and verification
  • Python, Pandoc, and tectonic for the optional ARS features that already require them

There are no required Pi orchestration or web-search dependencies. When a capability is unavailable, the wrapper tells Pi to use an installed equivalent or disclose degraded execution instead of pretending the operation ran.

Try without installing Copy Chapter

Load the package temporarily from GitHub for the current Pi run:

pi -e git:github.com/Imbad0202/academic-research-skills

To try a local checkout instead, clone the repository first:

git clone https://github.com/Imbad0202/academic-research-skills.git
cd academic-research-skills
pi -e ./pi

Then inspect optional capabilities and try a mode:

/ars-pi-doctor
/ars-plan
/ars-lit-review AI-assisted systematic reviews
/ars-reviewer
/ars-full

The original skills are also directly available:

/skill:deep-research
/skill:academic-paper
/skill:academic-paper-reviewer
/skill:academic-pipeline

For automatic ARS skill selection from subsequent natural-language prompts, explicitly enable ARS mode first:

/ars-pi-start

Install from GitHub Copy Chapter

Pi can install the canonical repository directly because the root package.json points to this wrapper and the original ARS resources:

pi install git:github.com/Imbad0202/academic-research-skills

Pin a branch, tag, or commit that contains the Pi wrapper when needed:

pi install git:github.com/Imbad0202/academic-research-skills@REF

Update installed Git packages with pi update --extensions. Remove this package with:

pi remove git:github.com/Imbad0202/academic-research-skills

Install from this checkout Copy Chapter

Keep the checkout at a stable path, then run from the repository root:

pi install .

The nested manifest also supports local installation from the wrapper directory:

pi install ./pi

Pi stores a local path rather than copying the repository. Update ARS with normal Git operations; the wrapper continues to load the original files.

If an optional sandbox restricts reads to the current working directory, allow read access to the repository checkout or run Pi from the repository root. Supporting files are intentionally not copied into the wrapper.

Remove using the same source form used during installation:

pi remove .
# or: pi remove ./pi

Capability doctor Copy Chapter

/ars-pi-doctor runs without an LLM call. It reports discovered orchestration and web-retrieval capabilities plus Python, PyYAML, Pandoc, tectonic, sandbox, and Claude-hook status. Missing optional dependencies remain the user's choice; the wrapper does not install them.

System-prompt scope Copy Chapter

For prompts submitted while Pi is idle, installing the package does not add ARS text to ordinary Pi prompts. While ARS mode is inactive, the wrapper removes only this package's four ARS skill entries from the newly started run's system prompt, preventing automatic model invocation without modifying the original SKILL.md files.

When Pi is idle, an /ars-* command or one of the four direct /skill:* entries above activates ARS mode before the new agent run starts, so that request receives both the original skill and the compatibility note. /ars-pi-start explicitly enables ARS mode and automatic skill selection for subsequent natural-language prompts. The state survives resuming the same session, follows the selected branch during /tree navigation, and resets in a new session. /ars-pi-doctor does not activate it.

Pi 0.83.0 does not rebuild the system prompt when a prompt is queued into an agent run that is already streaming. A mid-stream /ars-* or direct /skill:* prompt can therefore execute under that run's existing system prompt without the wrapper-injected compatibility note; similarly, /ars-pi-stop changes the persisted mode immediately but cannot remove ARS text from the in-flight run. The RPC steer and follow_up methods also bypass Pi's input event, so a direct ARS /skill:* sent through either method does not activate the wrapper. To receive the scoped system-prompt guarantee, wait for Pi to become idle and submit the request through the normal interactive or RPC prompt path.

To hide the ARS skills again and continue unrelated work without the compatibility note, run:

/ars-pi-stop

Wrapper regression test Copy Chapter

The wrapper accepts both upstream Pi's single-string system prompt and a string-array prompt (as observed in OMP 18.2.6). Array inputs remain arrays: the inactive XML filter runs per block, and activation appends the compatibility note as a separate block without mutating the host's array.

This is prompt-container compatibility, not a full OMP port. Skill hiding still recognizes Pi's XML <skill> entries; it does not hide OMP's different <skills> text listing. Package discovery and complete research workflows in other hosts are outside this change.

node --test pi/wrapper.test.mjs

The test covers idle-prompt skill hiding through canonical and symlink-spelled load paths, XML-escaped locations, preservation of missing and adjacent unrelated skills across alternate block formatting, same-request /ars-* and direct /skill:* activation while idle, manual start/stop toggling, /tree state restoration, argument-safe script-path rewriting, and single-pass argument-placeholder substitution.

The array regressions additionally cover selective XML filtering, preservation of block boundaries and frozen input arrays, activation, repeated preparation, and deactivation.

What the wrapper translates Copy Chapter

The wrapper reads /ars-* invocations from the original Claude command files, strips their frontmatter, appends trailing arguments when the command has no argument placeholder, converts executable python scripts/... paths to checkout-absolute paths, and expands the original target SKILL.md through Pi's native /skill:* mechanism. While ARS is active in the session, it also adds a short compatibility note to Pi's system prompt:

• repository-root ARS paths resolve against this checkout
• Claude tool names mean “use the equivalent available Pi capability”
• multi-agent work searches available tools and configured Pi skill locations for an installed orchestration capability, otherwise it uses sequential execution with a disclosure
• WebSearch, WebFetch, and /websearch search available tools and configured Pi skill locations for an installed web capability; no capability means no verification claim
• Claude-specific command model hints are ignored, so the active Pi model is inherited

The pi/package.json manifest performs the remaining mapping directly:

Scope Copy Chapter

This is intentionally a basic wrapper, not a reimplementation of Claude Code or ARS orchestration. The original ARS content remains authoritative and unmodified.

The project license remains CC BY-NC 4.0. Attribution and noncommercial restrictions apply.

Overview (`plugin-evals-citation-check/README.md`) Copy Section

plugin-evals-citation-check — `academic-paper` citation-check flow

Eval suite for claude plugin eval. One flow per suite: citation-check
(manuscript excerpt + reference list + a user-supplied "source pack" → citation
error report). Sibling of plugin-evals/ (revision-coach); the two suites do not
share cases.

Quality spec (author-defined, 2026-09-13): the four citation failures that
matter are 無中生有 (a reference the user has no source for), 張冠李戴 (right
paper, wrong authors), 小題大作 (a hedged or minor finding cited as an established
result), and 以訛傳訛 (citing a retracted or concern-flagged paper as live
evidence). Every fire case supplies a complete source pack so these are
detectable offline; the sandbox has no network, so "無中生有" is graded as
"flags it as unverifiable and does not claim it exists", never as a real lookup.
Secondary axes: mechanical errors (orphans, numbering, et al., & vs and),
no false positives on clean entries, no claim of online verification, no
rewriting of the manuscript.

All inputs are synthetic (fictional papers, journals, authors; DOIs use the
reserved 10.5555 example prefix and graders explicitly allow the model to
say so).

Run Copy Chapter

claude plugin eval . --eval-dir plugin-evals-citation-check --ablation with-without --judge-model opus

Add --no-publish to keep the HTML report local. Headline number is Δ
(with-plugin score − without-plugin score). runs: 3 per case. Cases request
model: sonnet, an alias that moves to each new Sonnet release, so compare runs
only when the resolved agent and judge versions match (the /ars-citation-check
command itself inherits the session model). The judge must be a different,
larger model — the runner's default judge is haiku,
which is both too small and never to be used here.

Cases Copy Chapter

Shared graders on 01–06: content-caught (w1.5), format-caught (w0.5, spec-
level mechanics), no-false-positive (w1), honest-unverified (w1),
no-overreach (w0.5), plus skill-fired (tool_used: Skill, matching
academic-research-skills:academic-paper, display-only under ablation, never
moves Δ). The exact core-skill match excludes command-stub-only invocation.
With --ablation none, this grader contributes to the score; report loading
coverage separately from output-quality grades. Every llm rubric is written as "work through the checks
one at a time and quote the evidence"; keep that style when adding graders.

Routing verification (2026-09-21) Copy Chapter

Claude Code 2.1.278, --ablation none --model sonnet --judge-model opus --runs 1 --no-publish, from a working directory outside the plugin checkout:

• All six positive cases called academic-research-skills:academic-paper.
  Cases 07 (format conversion) and 08 (unused Python imports) passed their
  negative-case rubrics; 08 made no Skill call.
• The full eight-case run passed six cases. Cases 01 and 02 hit the 600-second
  timeout after calling the core skill; their output-quality failures
  remain failures, not successful end-to-end runs.
• Isolated reruns: case 01 passed all rubrics; case 02 completed with the core
  skill invoked but failed format-caught and no-false-positive (score 0.75).
  It omitted the three-author 「等」 correction and incorrectly called the
  correctly stroke-ordered Chinese references misordered. These output-quality
  failures were the motivation for #882 below; the routing acceptance was not a
  suite-wide pass.
• A separate manual /academic-research-skills:ars-citation-check smoke,
  using case 03's synthetic material, called the core skill and read
  citation_compliance_agent.md. This is runtime evidence for citation-check,
  not a runtime coverage claim for all 13 mode commands.

These are loading/routing checks, not a with/without ablation or a measured
quality uplift. Earlier candidates are not pooled with this run: simply adding
nested Skill instructions still allowed command-only execution; hiding mode
commands without updating the SessionStart announcement led to attempted
calls to user-only commands. The announcement now names the core Skill-tool
targets for natural-language requests, including resume/compact events.

Chinese format repair verification (#882, 2026-09-21) Copy Chapter

The same case-02 prompt and all original quality rubrics remain unchanged.
Additional agent-loaded / locale-guide-loaded indicators record Read calls;
trace inspection checks that those reads succeeded. They are not quality scores.

The first repair candidate caught the missing three-author abbreviation and
accepted the correct stroke order in all three outputs. One output still
failed no-false-positive by promoting a DOI-prefix heuristic into a required
reference correction. The final candidate adds a general evidence boundary:
visible DOI syntax can be checked, but resolution/source claims require actual
resolver/source evidence. No fixture prefix or fixture author names are added
to the operative rules.

Final candidate, Claude Code 2.1.278, observed agent claude-sonnet-5, requested
judge opus, --ablation none --no-publish:

Across those final-candidate case-02 attempts: 3 passed, 2 timed out.
Timeouts remain failures and are not omitted from denominators. No completed
final-candidate output reproduced either reported defect or the DOI-prefix
false positive. This is bounded synthetic evidence, not a claim of universal
accuracy, a clean full-suite run, or measured with/without uplift.

The separate Chinese boundary suite
checks that the repair still detects real ordering errors, preserves two-author
and disambiguation cases, and honors an explicit venue romanization override. All three boundary cases
passed on the final candidate (one run each).

Side channels and ceilings (pilot 3, 2026-09-13, 1 run × 2 arms, sonnet agents) Copy Chapter

Pilot 1 (before calibration, opus agents) cost $6.89 and peaked at 155 s / 7
turns / $0.83 per run; pilot 2 (sonnet agents) cost $3.99. Pilot 3 followed a
cross-model review of the suite (13 findings, 12 applied: disputable plantings
replaced, presence regexes tied to an "unused" statement, metadata preservation
required on the conversion negative, real journal names replaced).

Known caveats Copy Chapter

• Historical loading failure (2026-09-13 pilot). The with-plugin arm
  invoked a command stub but could not find the mode prompt outside its cwd;
  citation_compliance_agent.md never loaded. Its near-zero Δ measured the
  stub plus base model, not the complete mode. #857 adds an explicit core-skill
  invocation and plugin-root references, and makes mode command stubs
  user-invocable only so automatic routing selects the core skill directly.
• Historical trigger rate was 3 of 6 fire cases. The old loose
  input_match: academic counted a command-stub invocation as firing. That
  number is not comparable to the stronger core-skill indicator now used.
  The Spanish trigger verificar citas subsequently arrived with #856;
  #858 adds English, Traditional Chinese, and Korean citation-check phrases.
• 03 was a /ars-citation-check slash-command case in pilot 1. The
  without-plugin arm answered "Unknown command" and Δ was +0.70 for the wrong
  reason (command existence). Switched to a prose trigger.
• Clean entries must be clean at the claim level too. Pilot 1 failed
  no-false-positive in 5 of 6 fire cases because the planted "clean" sentences
  overstated their own abstracts and the models (correctly) said so. Inputs were
  tightened and the grader now defines a false positive narrowly (entry-level
  error, listed under corrections, or proposed change); claim-wording comments,
  DOI-prefix remarks, and conditional house-style notes are allowed.
• 02 order of scripts is not graded. The reference list puts English
  entries before Chinese ones; the author accepts either order for Taiwan
  journals, so format-caught only checks the 「等」 rule.
• 02 bold vs italic on Chinese journal names is house style, not an error;
  entries are plain and the grader treats either remark as neutral.
• no-overreach allows per-sentence replacement wording for a misrepresenting
  claim and an offer to redraft; a rewritten whole excerpt or design-level
  critique fails it.
• Regex presence checks are secondary. orphan-named (w0.5) only proves the orphan key was mentioned; the paired llm format-caught decides whether it was flagged. In 08 the regexes require the import name within 120 characters of an "unused" statement (headings such as 「沒用到的引用：」 on the line above count).
• 04 does not require the style to be named. Both arms fixed the planted year mismatch without ever writing "Vancouver"; the grader only fails a response that applies author-date rules to the numbered list.
• 07 and 08 show Δ 0 — the base model already handles them. They stay as
  regression guards (07 must not become an audit report; 08 must not fire).

Overview (`plugin-evals-citation-locale/README.md`) Copy Section

Chinese citation-check boundary cases (#882)

Companion to plugin-evals-citation-check. All papers and people in these
inputs are fictional. This suite checks the scope of the abbreviation and
ordering repair, rather than changing the original case-02 acceptance rules.

claude plugin eval . --eval-dir plugin-evals-citation-locale --ablation none --model sonnet --judge-model opus --runs 1 --no-publish

The original mixed-overclaim case remains in the main citation suite and is
repeated three times for the two reported defects. Its existing quality
rubrics are unchanged. Additional tool_used graders record attempted reads
of the agent and Chinese guide; successful loading is checked in the traces.
Loading metrics are distinct from the output-quality judgements. These small
synthetic tests do not establish universal accuracy or a with/without uplift.

The supplied stroke counts in case 09 isolate applying the ordering rule from
looking up character data. They do not test a general-purpose stroke-count
lookup implementation. No such implementation is added by this prompt fix.

Verification (2026-09-21) Copy Chapter

On the final #882 candidate, Claude Code 2.1.278 with --ablation none --model sonnet --judge-model opus --runs 1 --no-publish passed all three
cases and all their graders (3/3). The observed agent was claude-sonnet-5.
Case 09 uses different personal names from the operative guide examples.
The result covers these three supplied-input boundaries; it does not certify
all Chinese ordering/author-name cases.

Overview (`plugin-evals/README.md`) Copy Section

plugin-evals — `academic-paper` revision-coach flow

Eval suite for claude plugin eval. One flow per suite: revision-coach (reviewer
comments → Revision Roadmap + Response Letter skeleton; committee-correspondence
variant when the user names a real committee).

Quality spec (author-defined, 2026-09-12): the primary axis is no unauthorised
rewriting — the response must not draft manuscript prose, must not change things
no reviewer asked for, and must not assert results or changes that have not
happened. Secondary axes: no comment dropped, push-back allowed on wrong
comments, committee letters get a tracker with no peer-review grading.

All inputs are synthetic (fictional studies, no real names or institutions).

Run Copy Chapter

claude plugin eval . --eval-dir plugin-evals --ablation with-without --judge-model sonnet

Add --no-publish to keep the HTML report local. Headline number is Δ
(with-plugin score − without-plugin score). runs: 3 per case.

Cases Copy Chapter

skill-fired (tool_used: Skill) on 01–05 is display-only under ablation and
never moves Δ. So is no-committee-branch on 03 (arm: with-only, #854).

Routing retest (2026-09-21, #854) Copy Chapter

After the #857 command-loading and SessionStart routing changes, case 03 was
run three times with Claude Code 2.1.278, --ablation none --model opus --judge-model opus --runs 3 --case '03-*' --no-publish from outside the plugin
checkout. All three called academic-research-skills:academic-paper, read
revision_coach_agent.md, and passed the existing no-fabrication and pushback
rubrics. Trace/output inspection found 0 of 3 applying the committee
variant. That branch assessment is manual, not an assertion made by the
existing graders; the same-family judge is not independent validation.

The observed model ID was claude-opus-5. This small synthetic retest does
not rule out the intermittent failure or close #854. Earlier intermediate
candidates included timeouts and are not counted as completed passing runs
here. No committee guard or agent definition changed.

Committee-branch guard (2026-09-24, #854) Copy Chapter

The counts and costs in this section are maintainer-reported: they come from
local runs whose outputs are not committed (plugin-evals/results/ is ignored
by git).

Case 03 gained no-committee-branch with the #854 guard (see CHANGELOG.md): a
with-only regex that fails when the answer emits a line of the committee
variant's output, that is, a concern-tracker or preserved-source heading or bold
label, a concern-ID heading or comment, the status line, or the blockquoted
human-subjects footer that committee_correspondence_protocol.md requires. A
heading or bold-label line with "no", "not", "none", "without", or "n/a" in it
does not count, and an answer that declines the variant in prose passes even
when it names the variant's parts.
scripts/test_check_committee_correspondence.py runs examples of both through a
JavaScript RegExp with the grader's flags, because the eval runner is a
JavaScript program. Against the maintainer's 2026-09-12 results it flags the two
runs named under Known caveats and none of the other nine, the without-plugin
arms included. Neither flagged run used the protocol's required strings; the
heading pattern caught both. The grader reads only the answer text, so it misses
a run that only announces the branch, or only writes the bundle's files, without
printing their lines. It would flag a peer-review answer that titles a section
"concern tracker" with no negating word, and a refusal that quotes one of those
lines on a line of its own.

Retest with Claude Code 2.1.281, --ablation none --model claude-opus-5-5 --judge-model sonnet --runs 7 --case '03-*' --no-publish, from outside the
plugin checkout: with the guard, 0 of 7 runs took the branch; on unchanged
main with only the new grader added, also 0 of 7 (5 of those 7 read
revision_coach_agent.md). The misroute was seen on claude-opus-5, and
Opus 5.5 did not reproduce it without the guard, so this retest shows no
effect of the guard; the guard is prompt-level and its effect is unmeasured.
None of the fourteen runs built the Revision Roadmap or the response skeleton;
each answered the push-back question directly. Cost: US$2.46 with the guard,
US$2.70 on main. The retest ran on an earlier text of the guard: the sentence
in committee_correspondence_protocol.md came later, and the mode table then
gave the variant its own row, which the final change dropped. It was also graded
with an earlier text of no-committee-branch; re-graded with the final text,
the counts are the same.

Case 04 (the ethics-committee letter), 7 runs on the final text with the same
settings: the skill fired in 2 runs, and both took the committee variant
(concern IDs and a segmentation for the user to confirm); tracker-covers-5
passed in 1 of 7. In the other 5 runs Opus 5.5 answered without invoking the
skill. The guard sits in files that load only after the skill fires, and the
skill descriptions and the SessionStart hook are unchanged from main, so those
5 runs do not test it. The stored 2026-09-12 runs on claude-opus-5 fired the
skill in 2 of 2. Cost: US$1.58.

Side channels and ceilings (pilot 2026-09-12, 1 run × 2 arms) Copy Chapter

Known caveats Copy Chapter

• 03 fires since #851. Before the #851 description fix the with-plugin arm
  answered the ICLR "should we push back" prompt without invoking the skill
  (0 of 2 pilots). After the fix: 7 of 7 verification runs invoked it. One of
  those seven said it was applying the #668 committee-correspondence branch to
  the ICLR peer reviews (the skill forbids inferring committee authority), and a
  second produced that branch's shape without naming it; four said the
  reviewers are peer review and one did not say. Tracked as #854 (see
  Committee-branch guard above).
• Judge style matters. With the runner's judge, rubrics phrased as a bare
  list of claims produced repeated 3-vote FAILs on outputs that a reasoning
  judge (same model, asked to quote evidence first) passed. Every llm rubric in
  this suite that showed that pattern (02 all-covered, 03 no-fabrication,
  03 pushback-per-reviewer) is written as "work through the checks one at a
  time and quote the evidence"; keep that style when adding graders.
• no-rewrite is judged, not regex-checked. Manuscript prose vs. quoted
  reviewer text cannot be told apart lexically. A single suggested sentence of
  manuscript text is borderline and judges have passed it.
• 05 and 07 show Δ 0 — the base model already handles them. They stay as
  regression guards.

Overview (`examples/pi/README.md`) Copy Section

Pi degraded-mode end-to-end evidence

This directory is a privacy-minimized evidence bundle from one completed /ars-full run through the thin Pi wrapper. It demonstrates checkpoint enforcement, source-read provenance gating, Stage 5 finalization, Stage 6 process recording, and persisted terminal completion when no Pi orchestration capability was installed.

This is evidence from one synthetic test scenario, not a general conformance certificate or a claim that degraded execution is equivalent to independent multi-agent review.

Run identity Copy Chapter

The run crossed a disclosed transport recovery and automatic context compaction. Continuity was established from persisted artifacts, state, and hashes; it was not represented as a new academic run.

After the run, the wrapper's system-prompt injection was scoped to ARS-active sessions and inactive prompts were hardened to hide this package's ARS skill entries. That guarantee and its probes cover prompts submitted while Pi is idle. Pi 0.83.0 does not rebuild the system prompt for prompts queued into an in-flight run, and its direct RPC steer and follow_up methods bypass the wrapper's input handler; those residuals are documented rather than represented as covered. The compatibility text used after /ars-full is unchanged; the final wrapper received fresh idle-path activation, automatic-invocation, and package-load regression tests, but these small post-run changes did not receive another full pipeline execution. See model-portability.md.

Load-bearing Pi boundaries Copy Chapter

1. No wrapper-provided agent isolation. runtime/ars-pi-doctor.txt reported Orchestration: none; sequential/degraded mode. Specialist roles therefore ran sequentially in one recovered Pi/model context. No independent reviewer, writer, verifier, observer, formatter, or cross-model judgment is claimed. The external RPC driver delivered synthetic test-operator checkpoint responses only; it was not available to the inner Pi session as an orchestration tool or skill.
2. No Claude hook enforcement. The same doctor record reported that Claude hooks are unavailable in Pi. ARS write-scope enforcement was prompt-level in this run; no deterministic Claude PreToolUse hook boundary is claimed.

The installed /skill:websearch capability was visible to Pi. Source verification remained bounded as disclosed in the provenance and process records. runtime/package-load-validation.json records fresh offline root/nested package loads that found all three wrapper commands, all 16 /ars-* prompt templates, and all four ARS skills, plus nine passing unit tests (including filesystem-backed symlink/XML-location, block-boundary, and single-pass argument-substitution probes) and five real Pi idle-prompt-scope probes.

Evidence chain Copy Chapter

Full stage chain Copy Section

run-log.md provides the chronological checkpoint record. Representative byte-for-byte outputs make each executed stage inspectable:

• Stage 1 Research: stage-01/revised-research-report.md
• Stage 2 Write: stage-02/course-paper-draft.md
• Stage 2.5 Integrity: stage-02.5/focused-reverification-report.md
• Stage 3 Review: stage-03/editorial-decision-and-revision-roadmap.md
• Stage 4 Revise: stage-04/response-to-reviewers.md
• Stage 3′ Re-review: stage-03p/verification-review-report.md
• Stage 4′ Re-revise: stage-04p/validation-report.md
• Stage 4.5 Final integrity: stage-04.5/final-integrity-verification-report.md
• Stage 5 Finalize and Stage 6 Process summary: final package, process records, checkpoints, and terminal state below

Source-read hard gate Copy Section

1. stage-05/cite-time-gate-refusal-report.md and stage-05/cite-time-gate-refusal.json show that Stage 5 refused converted output while all 79 citation occurrences remained unacknowledged LOW-WARN.
2. After the required human attestation was consumed, stage-05/cite-time-gate-pass.json recorded 21 attested sources, 79/79 markers resolved to OK, and a hard-gate PASS before clean outputs were emitted.
3. Exact attestation wording, Pi session identifiers, event identifiers, and human-read ledger were intentionally omitted from the public bundle. The aggregate gate state is sufficient for port evidence and is not independent proof that reading occurred.

Stage 5 Copy Section

• Retained public evidence: final paper.md, format validation, cite-time refusal/PASS records, and FULL checkpoint.
• The run also generated a DOCX and advisory submission-verifier artifacts. Those redundant binary/snapshot files are intentionally omitted; their hashes and outcomes remain in the checkpoint and persisted state.
• No paper PDF exists because the test operator declined paper LaTeX. HTML-to-PDF was not used.
• The submission-package policy was advisory. B1–B5 and C1–C2 remained NOT-CHECKED; this is not a clean universal submission-verifier pass.

Stage 6 and terminal state Copy Section

• The concise public process evidence is the English Markdown record. Redundant PDF, LaTeX, translated copy, and build-validation files are omitted.
• Terminal checkpoint was correctly nonterminal when presented.
• state/pipeline-state.json is the controlling proof: pipeline_state: completed, Stage 6 completed, checkpoint confirmed, terminal acknowledgement received, and completion timestamp persisted. Exact acknowledgement wording and transport identifiers are omitted.

Model portability Copy Chapter

model-portability.md compares observed openai-codex/gpt-5.6-sol behavior with the repository's illustrative Claude-oriented example. It separates runtime-established divergences from output differences that cannot be causally attributed to the model.

Privacy boundary Copy Chapter

• No maintainer name, email, account, home-directory path, credential, API key, authorization token, or Git remote is included.
• “Scholar” decisions in generated artifacts are synthetic test-operator inputs for the canonical example. Funding, conflict, affiliation, authorship, and placeholder statements are not biographical declarations about the Pi port maintainer.
• The public bundle records only aggregate human-read gate state; exact attestation wording and internal session/event identifiers are excluded.
• Public-source UUIDs appearing inside government document URLs are source locators, not Pi or user identifiers.

Preserved limitations Copy Chapter

• Sequential same-context role simulation is weaker than independent agents or context windows.
• Source verification and the scholar's source-read attestation have distinct evidentiary meanings.
• The paper is a non-systematic evidence synthesis and retains the nonblocking PRISMA-trAIce/RAISE reporting warning.
• The submission verifier was advisory and incomplete for checks requiring absent venue-profile or corpus inputs.
• Four title-page placeholders remain.
• The run experienced model-stream stalls, recovery, and compaction.

See inventory.json for source-to-curated path and hash provenance. SHA256SUMS covers every curated file except itself. Raw RPC streams, full session logs, internal identifiers, downloaded source files, redundant binaries/translations, and intermediate caches are intentionally excluded.

Overview (`examples/showcase/README.md`) Copy Section

Showcase: Full Academic Pipeline Output

Complete artifacts from a real academic pipeline run, demonstrating the end-to-end quality assurance process.

Paper: From Snapshots to Trajectories: How Agentic AI Will Redefine Student Learning Outcomes and Transform Student Success Measurement — Implications for Taiwan's Next Cycle of Institutional Accreditation

Version note. This run (2026-03-07 to 03-08) used academic-pipeline v2.3, before ARS v3.3 added the Semantic Scholar check and v3.11 added the deterministic four-index citation gate. The numbers below describe that version; the current gates have not been measured on this paper. The byline names Claude (Anthropic) as author because the researcher asked for that during the experiment. The paper is not an Anthropic publication, and ARS's positioning is that the tool does not replace the researcher and does not claim authorship (POSITIONING.md).

Paper Outputs Copy Chapter

Integrity Verification Reports Copy Chapter

Peer Review Reports Copy Chapter

Process Documentation Copy Chapter

These PDFs predate the public role rename. Any EIC text inside them is the legacy serialized source ID now displayed as Journal-Fit Reviewer; the archived binaries are intentionally unchanged.

Pipeline Stages Covered Copy Chapter

Stage 1: Deep Research (13-agent research pipeline)
Stage 2: Paper Writing (12-agent writing pipeline)
Stage 2.5: Integrity Verification ← MANDATORY, caught 22 issues
Stage 3: Peer Review Round 1 (Journal-Fit Reviewer + R1/R2/R3 + Devil's Advocate)
Stage 3': Author Revision + Re-review
Stage 3-R2: Peer Review Round 2
Stage 4: Final Revision
Stage 4.5: Final Integrity Verification ← MANDATORY, confirmed 0 regressions
Stage 5: Finalization (LaTeX → PDF, bilingual)
Stage 6: Process Summary
Post-Publication: Claude Code + WebSearch audit of all 68 refs → 21 issues found, 19 removed or corrected

Post-Publication Audit (2026-03-09) Copy Chapter

After the pipeline completed, Claude Code with WebSearch, run separately from paper generation, audited all 68 references and found problems in 21 of them (31% of the reference list) that had survived three rounds of automated integrity checking: 4 not found (likely fabricated), 6 with wrong authors, 7 with wrong metadata (year, title, pages, journal, or DOI), and 4 minor format or title issues. Afterward, the 4 were removed from the paper and 15 of the other 17 were corrected. The audit PDF calls the 31% a false-negative rate; it is the share of final references with a problem, not a miss rate over all problem references. At least seven of the 21 (cited as Banihashem, El-Banna, Gandara, Kestin, Stanford SCALE, Tao, and Temper) had been flagged at Stage 2.5 and marked FIXED at re-verification, yet the audit still found errors in them, such as wrong authors or an altered title. This stress test led to the integrity verification agent v2.0 overhaul (Anti-Hallucination Mandate, gray-zone elimination, known hallucination pattern library). See post_publication_audit_2026-03-09.pdf for the full report.

Overview (`scripts/adapters/README.md`) Copy Section

ARS Literature Corpus Adapters

Reference adapters that produce literature_corpus[] entries for the ARS Material Passport. Defined by the adapter contract at academic-pipeline/references/adapters/overview.md.

What these are Copy Chapter

Three starting-point adapters:

• folder_scan.py — scans a directory of files, parses citation metadata from filenames.
• zotero.py — reads a Better BibTeX JSON export.
• obsidian.py — reads an Obsidian vault (frontmatter Convention A or Karpathy-style Convention B).

Each writes two files: a passport.yaml with literature_corpus[] entries, and a rejection_log.yaml listing whatever the adapter could not map onto a valid entry.

These are references, not products. If your corpus is in Zotero, Obsidian, or a plain folder, you can run these directly; if it is in Notion, Readwise, Airtable, a custom SQLite, etc., copy a reference adapter and adapt it to your source.

Running a reference adapter Copy Chapter

folder_scan Copy Section

python scripts/adapters/folder_scan.py \
    --input /path/to/your/pdf/folder \
    --passport /tmp/passport.yaml \
    --rejection-log /tmp/rejection_log.yaml

Filenames are parsed with two conventions plus a fallback: {Family}_{Year}_{title}.{ext}, {Family}{Year}{title}.{ext}, or ... {Year} ... {Family} .... Files whose filename cannot yield both a family and a year are rejected. Non-ASCII filenames are rejected (Unicode support is a deferred extension). Symlinks are followed but the rejection log records the un-resolved path so collisions across subdirectories remain distinguishable.

zotero Copy Section

Export your Zotero library as Better BibTeX JSON (Zotero → File → Export Library → Format: Better CSL JSON, with the Better BibTeX extension installed). Then:

python scripts/adapters/zotero.py \
    --input ~/zotero_export.json \
    --passport /tmp/passport.yaml \
    --rejection-log /tmp/rejection_log.yaml

This adapter does NOT call the Zotero Web API. If you want live sync, write your own adapter using this file as a starting point. Items missing required fields (title, authors, parseable year, source_pointer) are rejected with a categorical reason; seasonal dates like Spring 2024 are rejected as year_unparseable.

obsidian Copy Section

python scripts/adapters/obsidian.py \
    --input ~/ObsidianVault \
    --passport /tmp/passport.yaml \
    --rejection-log /tmp/rejection_log.yaml

Files under _templates/ and .obsidian/ are skipped. Wikilinks ([[other note]]) are NOT resolved; they appear as literal text in user_notes. Notes with valid YAML frontmatter use Convention A (BibTeX-style: citekey:, authors:, year:, title:); notes without frontmatter fall through to Convention B (filename stem as citation_key, body as user_notes). Notes with malformed YAML frontmatter are rejected as invalid_field_format rather than silently re-classified as Convention B.

Validating adapter output Copy Chapter

Before using the passport in any ARS workflow:

python scripts/check_literature_corpus_schema.py \
    --passport /tmp/passport.yaml \
    --rejection-log /tmp/rejection_log.yaml

How to write your own adapter Copy Chapter

1. Read the adapter contract at academic-pipeline/references/adapters/overview.md.
2. Read the JSON schemas:
   • shared/contracts/passport/literature_corpus_entry.schema.json
   • shared/contracts/passport/rejection_log.schema.json
3. Copy one of folder_scan.py / zotero.py / obsidian.py as a starting point.
4. Change the input-reading and field-mapping code for your source.
5. Keep the output shape exactly (sorted literature_corpus[] by citation_key, always emit rejection_log.yaml, schema-clean entries only).
6. Set obtained_via: "other" and adapter_name: "<your-adapter-name>" on every entry you produce.
7. Reuse helpers in _common.py where possible — make_citation_key, ensure_unique_citekey, parse_csl_name, parse_semicolon_names, path_to_file_uri, write_passport, write_rejection_log, now_iso. They encode the contract details so you do not have to re-derive them.
8. Validate: python scripts/check_literature_corpus_schema.py --passport <your passport>.
9. Write tests modeled on scripts/adapters/tests/. The conftest fixtures clean_timestamps and load_yaml make golden-output testing easy.

Privacy reminder Copy Chapter

abstract and user_notes can contain publisher-copyrighted material. Before sharing a passport that contains these fields publicly (e.g., in a public repo or over the web), make sure you have the right to publish that text. ARS does not enforce this — it only warns you in the schema description.

Tests Copy Chapter

cd <repo_root>
pytest scripts/adapters/tests/ -v

CI runs these on every push / PR that touches scripts/adapters/**, shared/contracts/passport/**, the adapter overview, or either of the two lint scripts.

Overview (`shared/contracts/README.md`) Copy Section

ARS Shared Contracts

Schema files for cross-skill contracts: reviewer sprint contracts, Material Passport
ports, and (v3.6.7+) cross-model audit artifact pipelines.

Stage capability / evidence matrix (#745) Copy Chapter

• capability/stage_capability_matrix.json (stage-capability-matrix/1.0) is the
  single machine-readable source for per-stage mechanism status, deterministic
  conformance, behavioral-evidence provenance, transport limits, and the maximum
  claim that evidence licenses. Behavioral statuses (DESIGNED / NOT_RUN /
  MEASURED / MIXED / OUT_OF_SCOPE) cannot collapse: an unrun eval can never
  carry a result, a measured row must carry full provenance (in-repo eval_ref,
  model, population, ISO date, result summary), and stale measurements require a
  visible staleness note. A measured row whose suite publishes
  measurement-*.json reports must bind the current one (date-equal, sibling
  supersession detected), CI_GATED/TESTED conformance must name existing
  lints/tests (conformance_pinned_by, D4-style), and claim_anchors bind
  top-level capability sentences verbatim so rewording a claim without
  touching the matrix fails CI.
• docs/STAGE_CAPABILITY_MATRIX.md is GENERATED from the matrix
  (scripts/check_stage_capability_matrix.py --render) and byte-pinned by the
  same lint. The matrix indexes evidence, it does not create it: a row licenses
  at most its recorded max_licensed_claim, never more.

Sealed Promotion Bakeoff preregistration (#789) Copy Chapter

• cross_model/promotion_bakeoff_sealed_commitment.schema.json is the closed
  public pre-fleet carrier: campaign id, LF-normalized probe-set SHA-256, and
  the fixed 30-row aggregate composition only. It has no row, label, path, or
  free-text field in which the answer key can be exposed.
• cross_model/promotion_bakeoff_sealed_reveal.schema.json binds that immutable
  commitment to the post-fleet probe_set.json reveal. The standard-library
  checker scripts/check_promotion_bakeoff_preregistration.py validates the
  closed probe shape, strict Git introduction order, immutable history,
  canonical paths, worktree/index state, and detectable reuse across all local
  refs and every historical probe version. It rejects shallow history and
  unreadable historical objects; the sole 2026-08-19 unsealed grandfather is
  pinned to its canonical path, normalized digest, blob history, and file mode.
  Verification receipts enumerate every bound squash/cherry-pick reveal copy
  whose remote publication time must be witnessed after the fleet.

The checker proves repository evidence only. Remote publication time and fleet
start/end time cannot be derived from local Git, so its receipt explicitly
requires the public permalink, passing CI witness, and fleet-time bounds in the
run report before a result is gate-eligible. The full operator lifecycle and
claims boundary live in ../cross_model_verification.md § Promotion Bakeoff.

Research-family workflow profiles (#742) Copy Chapter

• research_workflow/research_workflow_profile.schema.json
  (research-workflow-profile/1.0) is the closed, manuscript-blind declaration
  of stage applicability and vocabulary. Omitted task families resolve to
  unresolved_fit, never to applicable. The profile embeds a SHA-256 over its
  JSON Canonical Form with content_sha256 replaced by 64 zeroes; persisted
  profile files must themselves be exact canonical bytes.
• research_workflow/research_workflow_profile_selection_receipt.schema.json
  (research-workflow-profile-selection-receipt/1.0) records an append-only
  user selection/confirmation/fallback chain. A correction appends a new
  binding and one visible profile_context_changed stale mark per supplied
  prior-profile stage output. It never rewrites or removes an artifact. Newly
  introduced authority requirements remain attached to each mark with an
  explicit authority_sensitive_reuse_gate: unmet, so a later consumer can
  block that reuse until the applicable check passes.
• ../research_workflow_profiles/field_general.json is the mandatory shipped
  fallback: every task family is unresolved except the field-general integrity
  gates, alternatives are unresolved, the live-branch budget is 3, and an empty
  authority list means “unknown; ask the user,” never “not required.” It ships
  with explicit English and zh-TW display names.

The standard-library runtime is scripts/research_workflow_profile.py. Its
select command treats an omitted --profile as a visibly recorded automatic
fallback; an explicit profile requires --selected-by. correct consumes an
explicit prior-profile binding and stage-output inventory, then emits the new
receipt to stdout only. No command reads manuscript content, infers a family,
changes the default workflow, or writes scholar-owned artifacts. The human
usability protocol and every outcome claim remain NOT_RUN.

Opt-in inquiry branch ledger (#743) Copy Chapter

• research_workflow/inquiry_branch_ledger.schema.json
  (inquiry-branch-ledger/1.0) is the closed event-source contract for
  author-originated branches, parked AI-surfaced facets, explicit adoption,
  disposition, reopen signals, profile rebound, and first-degree artifact
  staleness. Dense event ids plus a canonical previous-event hash chain detect
  interior rewrite/reorder; the separate passport digest is the required
  trusted head for truncation detection.
• passport/inquiry_ledger_ref.schema.json closes the optional Material
  Passport pointer to {ledger_path, ledger_version, content_sha256}. It never
  embeds a branch or profile and never widens a filesystem path outside the
  caller-supplied workspace root.

The deterministic offline runtime is scripts/inquiry_branch_ledger.py. Replay
requires every exact profile bound by the ledger so the live-branch budget is
never inferred from a current default. Reopen appends contiguous deterministic
stale-cause events; clearing one cause cannot clear another. Ledger/passport
publication uses a stable sidecar lock and durable recovery journal. The
feature is OFF unless ARS_INQUIRY_LEDGER=1, does not materialize before a
second branch, and shows compact summaries only at the frozen Stage 1, 2.5,
4.5, or still-actionable recorded-signal moments. Behavioral and usability evidence is
NOT_RUN.

PDF read-integrity and optional content advisory (#512 follow-up) Copy Chapter

• pdf/pdf_read_preflight.schema.json accepts the unchanged legacy structural sidecar
  or the all-or-nothing opt-in content extension. In that extension, verdict is
  explicitly verdict_scope: STRUCTURE_ONLY; OCR_RECOMMENDED never rewrites that
  structural value into a content claim. The schema binds the legacy shape to tool
  version 1.0.0 and the extension shape to 1.1.0.
• pdf/pdf_content_classifier_worker.schema.json closes the stdout of the fixed
  isolated worker to two classifications, three unavailable reasons, finite bounded
  confidence, and bounded non-negative page indexes. Runtime additionally binds every
  page to the structural page count.
• pdf/pdf_content_classifier_diagnostic.schema.json is the separate POSIX-only,
  local mode-0600 operator artifact. Platforms without fchmod reject its CLI
  option before path creation. Its explicitly untrusted detail is capped and never
  copied into or referenced by the sidecar. File-writing CLI invocations use
  conservative NFC/casefold keys, path resolution, and existing-inode checks to reject
  aliases before worker launch; the stdout-only legacy path adds no such precondition.
  POSIX sidecar output pre-binds the parent dirfd/inode, then uses a private 0700
  fixed-name staging directory and anchored dirfd-relative publication. Open-inode
  checks reject staging entry swaps; atomic final-entry replacement does not follow
  post-check links. Non-POSIX output fails closed; stdout classification remains
  available. The diagnostic parent is likewise dirfd-bound before the worker. These are
  instantaneous inode postconditions, not a general same-UID sandbox; callers control
  the output parent. A failed diagnostic unlinks only the no-follow leaf still matching
  its created fd inode, preserving the primary error and any attacker replacement while
  keeping its exclusive path retryable.

Runtime: scripts/pdf_read_preflight.py and
scripts/pdf_content_classifier_worker.py. Frozen opt-in scope and residual risk:
docs/design/2026-08-13-512-pdf-content-classification-sandbox-spec.md.

Claim-standing candidate ledger (#655 Track A) Copy Chapter

• claim_standing/query_plan.schema.json (claim-standing-query-plan/1.0)
  binds one exact high-impact checkpoint claim, at most three accepted queries,
  at most four discovery-index identities, filters, authorized content class,
  frozen caps, and retrieval-only consent through a closed consentable-plan
  projection (version 1.1 below adds the stance-authorizable consent).
• claim_standing/retrieval_input.schema.json
  (claim-standing-retrieval-input/1.0) carries already-retained,
  adapter-neutral attempts, closed retry-authorization receipts, raw hits,
  explicit version relations, and caller-supplied relevance success/failure
  evidence bound to exact claim/candidate inputs and canonical prompt bytes.
• claim_standing/candidate_ledger.schema.json
  (claim-standing-candidate-ledger/1.0) preserves every attempt and raw-hit
  terminal state while recording deterministic work-family selection.
• claim_standing/query_plan_v1_1.schema.json
  (claim-standing-query-plan/1.1) is identical to 1.0 except that the consent
  decision may be retrieval_plus_stance, in which case a top-level
  stance_plan names the exact stance provider/model surface, the consent
  binds its hash, and the authorized content classes extend by exactly
  claim_and_selected_evidence_to_stance_provider. A 1.0 plan stays valid
  under its own schema; the runtime validator accepts both versions and
  enforces the stance bindings only on 1.1.
• claim_standing/transmission_ledger.schema.json
  (claim-standing-transmission-ledger/1.0) accounts every event that left the
  session during one probe: retrieval-query events derived one-to-one from the
  retained attempts and stance-classification events copied verbatim from the
  stance runner's transmission records, each carrying recipient, purpose, exact
  content classes, byte count, local hash, time, consent receipt, retention
  disclosure, and result state. scripts/check_claim_standing_transmissions.py
  is the normative builder/validator: an event outside the plan's authorized
  content classes or consented recipient roster fails closed (design §9
  gate 14), and validation is exact replay.
• claim_standing/stance_record.schema.json
  (claim-standing-stance-record/1.0) is the stance-classification output for
  one probe run: full §7 probe-identity hashes, one row per selected work
  family under the closed §5.1 vocabulary and cross-field rules (a performed
  row requires at least one EVR-CS- evidence-row reference; metadata-only
  coverage can never be performed), the all-selected distribution whose seven
  buckets must sum to selected_total, a mandatory
  STANCE CLASSIFICATION UNMEASURED banner, and no scalar
  credibility/confidence/trust field anywhere (test-pinned).

Provider retention disclosure is also closed: known requires a non-empty
reference and unknown requires null.

The schema-level \\S checks are portable first screens. The runtime applies a
single NFKC visible-semantic-text predicate to the claim/query, disclosure,
identity, available-abstract, successful-assessment, and failure-detail
surfaces; it rejects surrogates and text made only of Unicode control/format,
separator, combining, whitespace, or punctuation characters. Failed malformed
assessment raw output remains exact evidence and may itself be whitespace- or
format-only. DOI text is stable identity only when strict NFKC/prefix-trimmed
10.<4-9 digits>/<suffix> validation succeeds.

The pure local finalizer is
scripts/build_claim_standing_candidate_ledger.py; the authoritative boundary
is shared/references/claim_standing_candidate_ledger_protocol.md. This slice
has no discovery adapter, network/model call, stance classification, rendering,
evidence-row extension, pipeline hook, or held-out dispatch. It is an offline
substrate only, remains unmeasured, and does not close #655.
Its CLI never creates an output for a session_only plan. build --output
requires the existing, hash-bound explicit_local_export consent state; there
is no command-line override. Authorized output must exactly match the
hash-bound absolute authorized_output_path, is exclusive/no-follow where the host
supports it, mode 0600 from creation, file- and directory-fsynced, and
truthfully carries the consented persistence/export/path state at the ledger root
and in each work family's sharing scope. Its separate rights basis remains
not_assessed; local persistence consent is not a rights claim.

Codex subscription citation transport (#630) Copy Chapter

• cross_model/codex_citation_request.schema.json — closed, bounded data-only
  request for one citation; no path or caller-authored prompt is representable.
• cross_model/codex_citation_receipt.schema.json — closed verdict plus exact
  app-server search-result bindings and a fixed containment receipt.

Runtime: scripts/cross_model_codex_transport.py. Protocol and limitations:
shared/cross_model_verification.md. Frozen design:
docs/design/2026-08-11-630-codex-subscription-citation-transport-spec.md.
The adapter is scoped to Stage 2.5 / 4.5 citation-integrity calls and is not a
general DA, reviewer, judgment, or handoff transport.

Sprint contracts (v3.6.2+) Copy Chapter

Sprint contract templates for reviewer hard-gate orchestration.

Schema: shared/sprint_contract.schema.json (Schema 13).
Validator: scripts/check_sprint_contract.py.
Spec: docs/design/2026-04-23-ars-v3.6.2-sprint-contract-design.md.
Protocol: academic-paper-reviewer/references/sprint_contract_protocol.md.

Shipped templates Copy Section

v3.6.2 (reviewer family):

• reviewer/full.json — panel 5, 5 dimensions, 4 failure conditions
• reviewer/methodology_focus.json — panel 2, 2 dimensions, 3 failure conditions

v3.6.6 / suite v3.6.8 (generator-evaluator family):

• writer/full.json — single-agent writer, 7 dimensions (D1 section_completeness / D2 citation_density / D3 argument_blueprint_fidelity / D4 total_word_count / D5 per_section_word_count / D6 acknowledged_limitations / D7 register_consistency), 5 failure conditions (F1 / F4 / F2 / F3 / F0). No scoring_plan field.
• evaluator/full.json — single-agent evaluator, 5 dimensions (D1 originality / D2 methodological_rigor / D3 evidence_sufficiency / D4 argument_coherence / D5 writing_quality), 7 failure conditions (F1 / F2 / F3 / F6 / F4 / F5 / F0). Carries full scoring_plan + disagreement_handling.

Both writer + evaluator templates ship under Schema 13.1 (allOf branches 11/12 require pre_commitment_artifacts for writer_full and disagreement_handling for evaluator_full; branches 5/6 pin failure_conditions[].action to mode-specific enums; branches 8/9 pin F0 contains to the mode's accept variant). Orchestration block lives in academic-paper/SKILL.md § "v3.6.6 Generator-Evaluator Contract Protocol" + the writer/evaluator agent files.

Reserved reviewer modes without shipped templates Copy Section

reviewer_calibration and reviewer_guided are in the schema enum but ship without
templates. Those modes continue to operate in their existing form (no contract, no
hard-gate) until a follow-up patch release adds their templates. reviewer_re_review
left the Schema 13 enum with #576 Spec B: re-review is governed by the dedicated
contract family under re_review/ (four schemas + scripts/check_re_review_synthesis.py),
not by a Schema 13 template — a contract claiming mode: reviewer_re_review no longer
validates.

How to add a new template Copy Section

1. Add the file under shared/contracts/<domain>/<mode>.json.
2. Run python scripts/check_sprint_contract.py --ars-version vX.Y.Z; expect
   zero errors and zero soft warnings.
3. If expression strings use new phrasing, update sprint_contract_protocol.md
   and the synthesizer prompt's recognised-pattern list in the same PR.

Passport contracts (v3.6.4+) Copy Chapter

Schemas for Material Passport input ports.

• passport/literature_corpus_entry.schema.json (v3.6.4) — Schema 9 literature_corpus[]
  entries produced by user-written adapters.
• passport/bibliographic_integrity_signal.schema.json (#678/#651/#660) — v1.0
  additive signal carrier, v1.1 authoritative retraction-status rows, and the
  v1.2 title/abstract tortured-phrase advisory profile,
  including resolver disagreement/reinstatement, judgment context, freshness,
  and opt-in finalizer policy eligibility. The separate
  retraction_status_cache_v1 namespace and pure resolver live in
  scripts/retraction_status.py.
• passport/rejection_log.schema.json (v3.6.4) — adapter output companion logging
  entries that could not be included in the corpus.
• passport/reset_ledger_entry.schema.json (v3.6.3) — reset_boundary[] ledger entries
  for the opt-in passport reset boundary protocol.
• passport/inquiry_ledger_ref.schema.json (#743) — optional digest-bound
  pointer to the separate canonical inquiry branch ledger; missing or
  mismatched targets fail visibly and unpointed candidates are ignored.
• passport/standing_constraint_entry.schema.json (#927) — standing_constraints[]
  entries: a run-wide constraint the user stated, in their words. Schema tests:
  scripts/test_standing_constraint_entry_schema.py.
• passport/excluded_source_entry.schema.json (#936) — excluded_sources[]
  entries: a reference an integrity gate judged NOT_FOUND, kept off later writer
  dispatches. Schema tests: scripts/test_excluded_source_entry_schema.py.
• passport/audit_artifact_entry.schema.json (v3.6.7 Step 6) — audit_artifact[] entries
  recording one cross-model audit run per downstream-agent deliverable. Two lifecycle
  states (proposal / persisted) share the schema via oneOf. Cross-artifact invariants
  are enforced by scripts/check_audit_artifact_consistency.py. Spec:
  docs/design/2026-04-30-ars-v3.6.7-step-6-orchestrator-hooks-spec.md §3.1-§3.2.
• passport/version_records.schema.json (Kong #258) — optional
  phase2_investigation/version_records.yaml sidecar for academic citation version
  families (preprint -> proceedings -> journal extension). This is deliberately a
  sidecar: literature_corpus_entry.schema.json stays adapter-owned and unmodified.
• passport/human_read_log.schema.json (#513/#738) — the user-owned
  USER_ATTESTED_READ ledger
  (_human_read_log.yaml, written by scripts/ars_mark_read.py),
  including required scope on new marks and legacy-compatible scope absence.
  Declaration-only: it is not verified reading or comprehension. Missing/unknown
  scope remains coverage_unknown and never promotes to ok.
• passport/user_attested_read_resolution.schema.json (#738) — the closed output
  of scripts/human_read_attestation_resolver.py; only state: covered is
  ok_eligible, while absence, rescind, partial/unknown coverage, unresolved
  anchors, and invalid ledgers remain explicitly non-promoting. This output is a
  transient_routing_decision, not a persisted audit receipt: it carries no
  input digest and must be recomputed from the current ledger and exact anchor
  on each finalizer pass. Its closed finalizer_disposition keeps absent or
  rescinded marks as unacknowledged LOW-WARN, partial/unknown active marks as
  acknowledged-partial, anchor failures on the locator-precedence route, and
  invalid ledgers on a blocking route.

Shared evidence rows (#656) Copy Chapter

shared/contracts/evidence/claim_registry.schema.json (#737) is the closed E1
population contract: exact raw-draft hash plus exact UTF-8 byte span and equal
text for every registered claim. claim_registry_coverage_report.schema.json
is the deterministic, replay-bound gap report emitted by
scripts/claim_registry_coverage.py; it binds exact raw draft and serialized
registry hashes and joins only those exact spans. It covers citation-bearing
and quantitative candidate sentences only. Its finite grammar includes common
Markdown/numeric/author-year/Pandoc/inline-reference citations plus
unit-bearing numbers, p-values, N=..., and common effect-size/ratio notation;
unrecognized scholarly syntax remains possible. The report always records
semantic_extraction_coverage: not_machine_detectable; it cannot certify that
every substantive claim was extracted into E1. Consumers replay the report
against both inputs; absence, stale bindings, or validation failure are
unresolved execution states, never a zero-gap result.

Review-panel provenance (#740) Copy Chapter

reviewer/review_panel_provenance_input.schema.json records actual seat-level
observations; reviewer/review_panel_provenance.schema.json is the closed,
replay-derived artifact built by scripts/review_panel_provenance.py. Both are
closed to reviewer_full, bind the exact raw-byte digest of
reviewer/full.json, and require the ordered EIC, R1, R2, R3, DA
roster. Actual execution observations remain nullable and are never filled from
those seat labels. The artifact keeps role separation, within-panel invocation
context separation, peer-output blinding, model-family diversity, provider
diversity, and accountable-human diversity as six separate true / false /
unknown axes. fresh_context_scope: within_panel_attempt_only makes explicit
that no cross-attempt history is checked. Missing facts remain unknown,
same-family or family-unknown execution carries a fixed correlated-error
disclosure, and no persona-derived binary independence field is admitted.

reviewer/review_panel_provenance_carrier.schema.json is the exact closed
Schema 6 valid/invalid union. The valid branch is accepted only after the
runtime hashes the referenced artifact's exact raw bytes, validates and replays
it, and compares the normalized-manifest digest, execution-topology digest,
fresh-context scope, and six axes. The invalid branch admits only
absent|unreachable|digest_mismatch|schema_invalid|replay_invalid, forces all
axes to unknown, and carries no path or digest. reviewer_full must emit one
branch; the other closed modes must omit the field. Use
scripts/review_panel_provenance.py validate-schema6 for mode-scoped presence
and replay validation.

shared/contracts/evidence/evidence_row.schema.json defines the closed
evidence-row/1.0 carrier for evidence shown at human-adjudication checkpoints.
V1 has one consumer discriminator, surface: phase_e_claim_verification: the
Stage 2.5/4.5 Phase E Claim Verification Report persists
one row per (claim_id, ref_slug, anchor) at Integrity Report
phases.E_claims.evidence_rows[]. The array is additive: a positively identified
pre-#656 report may omit it and, only with --allow-legacy-absence, renders
exactly LEGACY — EVIDENCE ROWS UNAVAILABLE; missing shape alone is not legacy
proof and never means that evidence was checked. The opt-in Stage 4→5
claim_audit_results[] contract is a separate lifecycle and is unchanged.

The one evidence-state vocabulary is:

• verified_exact_match — a once-decoded writer quote anchor matched the
  exact session-held source string;
• agent_extracted — a page/section passage selected by the auditing agent,
  also exact-substring checked, but labeled non-authoritative because passage
  selection is agent judgment;
• unconfirmed_anchor — session source was present but the writer's decoded
  quote did not match it;
• not_checked, source_missing, access_failed, retrieval_failed, and
  anchorless — explicit empty states. V1 maps paragraph to not_checked and
  none/a missing marker to anchorless.

These states describe excerpt provenance/availability only. They never compute
or change the independent Phase E verdict, issue severity/count, or PASS/FAIL
decision. Viewing a row is not reading the source: the runtime never reads or
writes human_read_log, never invokes /ars-mark-read, and never affects
LOW-WARN/read-scope promotion.

scripts/evidence_rows.py is the standard-library-only normative runtime:

strict_percent_decode(value)
build(row, session_source_or_none, *, extracted_text=None,
      failure_state=None, cached_row=None)
validate(row, session_source_or_none=None)
paginate(rows, page=1, page_size=25)
render_markdown(rows, page=1, page_size=25, *, session_sources=None)
render_html(rows, page=1, page_size=25, *, session_sources=None)

The decoder is strict UTF-8 and runs exactly once (+ is not a space;
%2520 becomes the literal %20). Positive excerpts must be an exact
contiguous substring of the explicitly supplied session source. Quote anchors
and excerpts are each capped at 25 words by whitespace split and 1000 Unicode
code points. source_content_sha256 hashes the exact session source encoded as
UTF-8 with no Unicode, whitespace, or newline normalization;
source_artifact_sha256 hashes optional raw artifact bytes and never substitutes
for the content hash. UTF-8 byte offsets bind an excerpt to the content hash.
row_sha256 covers canonical JSON for the complete row except the digest field
itself. It is an unkeyed integrity checksum, not a signature or independent
proof of provenance. Source-bound trust is established by replaying the row
against explicitly supplied session source text. Direct Python
validate(row) without its optional source argument checks only closed shape,
cross-fields, and self-contained integrity; it does not prove source provenance.

Cache keys bind schema/surface, claim and citation identity, source-content
hash, decoded anchor, resulting state, and (for page/section) the candidate
excerpt hash. A key match is replay-validated against the current session
source before reuse. Hits preserve the original excerpt and captured_at, then
set cache.status: hit and recompute row_sha256. Claim/source/content/candidate
drift is a miss. A corrupt candidate is ignored and rebuilt from current explicit
inputs; it is never returned as a hit. Current explicit failure states win without
reading cached evidence; failure/empty states are never cached.

Renderers are pure functions over persisted rows plus the explicit in-memory
session_sources mapping supplied by their caller. Every source-bound row must
have a matching ref_slug → exact source text entry and is replay-validated
before anything is displayed; missing or drifting text fails closed. Empty-state
rows need no source map. For V1 rows, the CLI's --source-dir builds the mapping from the
files named by source_file_name(ref_slug) that source-bound rows name in one
folder, read as exact UTF-8 bytes, and opens no other file (#933). The renderer
does not accept or follow a source pointer,
URL, DOI, retrieval client, model, cache, or read ledger. Integrity validation
checks the stored encoded/decoded-anchor relationship; display does not decode or
alter the stored anchor again. Markdown and HTML external strings are rendered
inert, including percent-decoded markup, table delimiters, bare URLs/email,
newlines, all Unicode Cc/Cf controls, and U+2028/U+2029. A render page contains
at most 25 rows, preserves document order, reports total/page bounds and
deterministic previous/next/explicit-page navigation, and never silently
truncates. There is intentionally no --all; any number of persisted rows
remains accessible by page.

For any Integrity Report wrapper with evidence_rows present, the adapter
requires integer E_claims.checked and E_claims.verified; it checks that unique
evidence claim_id count equals checked, unique VERIFIED claim count equals
verified, and every row sharing a claim ID carries the same claim object and
claim-level verdict. Thus checked > 0 with an empty present array fails. Raw
single-row and row-array inputs remain valid standalone runtime inputs. Only an
absent evidence_rows field is legacy-compatible. Exact omission
of a selected (claim_id, ref_slug, anchor) tuple cannot be proved from this
wrapper alone because the E1 selected-tuple registry is not machine-carried here;
the current producer must enforce that completeness before handoff.

Rows containing an excerpt or decoded quote anchor default to
session_only/not_assessed. Only an explicit user declaration may set the
paired user_confirmed_shareable/user_declared_authorized values. The length
limit is data minimization, not a licence, copyright exception, or publication
right; exported reports must retain the row handling label/caveat or strip the
external text.

Validate persisted rows or render exactly one page. On both CLI commands, any
source-bound row requires replay from an explicit ref_slug → source text JSON
map, or from a folder of per-source text files (#933; V1 rows only):

python scripts/evidence_rows.py validate evidence-rows.json \
  --source-map session-sources.json

python scripts/evidence_rows.py render evidence-rows.json \
  --format markdown --page 1 --page-size 25 \
  --source-dir paper_evidence_sources/stage-2.5

Exactly one of --source-map and --source-dir is required whenever either CLI
command receives a document with a source-bound state. The command opens the
JSON map, or only the folder's files that source-bound rows name, and no other
file. Missing
evidence_rows is a render failure by default. A positively identified pre-#656
Integrity Report renders the exact fixed legacy label with exit 0 only when the
caller adds --allow-legacy-absence; validate always rejects the absence.
Current producers may never use the compatibility flag.

shared/contracts/evidence/evidence_row_v1_1.schema.json is the separately
versioned extension for surface: authority_profile_content_coverage (#681).
It replaces Phase E claim/verdict fields with exact authority-requirement,
structured-expectation, packet-artifact, and document-locator bindings. Its
states are agent_extracted, checked_no_match, not_checked,
source_missing, access_failed, and retrieval_failed; they describe only
the provenance or absence of one bounded advisory passage. The shared 25-word,
1,000-code-point, strict once-decode, exact UTF-8 replay, inert rendering,
rights, and human-read-ledger boundaries still apply. V1.1 performs no cache
lookup. scripts/evidence_rows.py exposes build_advisory(...) for this
surface, while the existing evidence-row/1.0 builder and rendered bytes remain
unchanged. Its CLI refuses V1.1 rows with exit 2 and names
scripts/build_content_coverage_advisory.py instead (#947). The versioned surfaces cannot be mixed in one page.

shared/contracts/evidence/evidence_row_v1_2.schema.json is the separate closed
version for surface: cross_document_consistency (#672). It binds one complete
ordered bilateral or trilateral advisory observation to the exact accepted
manuscript and optional completed-preregistration bytes. It is finalized only by
scripts/build_cross_document_consistency_advisory.py; the existing 1.0/1.1
schemas and scripts/evidence_rows.py identities and behavior remain unchanged.

shared/contracts/evidence/evidence_row_v1_3.schema.json is the separate closed evidence-row/1.3 carrier for the claim_standing_advisory surface (#655 design §5.2): one row per (probe claim, selected work-family candidate) binding the bounded inspected-evidence excerpt to its exact source hash/span, reusing the family's excerpt/cache/content-handling blocks verbatim (test-pinned). The row is provenance-only — it never carries a stance or verdict, an exact excerpt match never determines stance, and abstract-level coverage is declared as such and never rendered as verified full text.

Non-ranking revision authority (#670) Copy Chapter

The current reviewer-to-author revision family lives under revision/:

• revision_roadmap.schema.json — immutable reviewer-owned
  revision-roadmap/1.0, exact draft/manifest bindings, source-trace order,
  independent severity/obligation/cost/consequence, and proposed target scopes;
• claim_surface_manifest.schema.json — exact registered Claim Intent surfaces,
  raw UTF-8 spans, hashes, blocks, current rungs, and byte-identical equality
  between every protected surface and its referenced ClaimIntent claim_text;
• author_adjudication_input.schema.json and
  author_adjudication.schema.json — explicit session-author choices and the
  deterministic raw-hash-bound sidecar; and
• integrity_correction_list.schema.json plus the integrity authorization
  input/output schemas — proposal-only issues and explicit author approval of
  the complete exact patch SHA-256; and
• revision_evidence_bundle.schema.json plus the integrity receipt schema
  — a continuous local chain from exact integrity PASS through current
  review-write/no-op/integrity rounds to the final draft; and
• claim_strength_drift_findings.schema.json,
  claim_strength_drift_disposition_input.schema.json, and
  claim_strength_drift_disposition.schema.json — the E6 semantic finding set,
  transient exact raw-event artifact paths plus choices, and deterministic
  hash-bound disposition sidecar. Build and replay validation safely reopen
  every explicitly named regular non-symlink event file and recompute its raw
  SHA-256; a digest assertion alone is insufficient. The sidecar retains no
  event path or raw message. This byte binding does not authenticate source,
  content meaning, or actor identity.

scripts/revision_roadmap.py builds, validates, renders, and bundle-replays this
family without a model, network, API, directory scan, or ambient clock. It opens
only explicitly named local artifacts through containment, symlink, and
read-once guards. Bundle validation reruns the pure current patch engine for
every write round and requires byte-exact replay output to equal the named post
draft; rewritten artifact/report hashes cannot substitute for authorized bytes.

The current patch schema accepts only format 1.1 and separates
review_roadmap from integrity_correction authority. Registered claim
replacements and declined-overlap collateral authority are exact and
single-use. An integrity issue list grants no write by itself: apply requires a
separate author sidecar whose explicit input already carries the exact proposed
patch digest. Apply report 1.3 records the replayed witness and explicitly leaves
unregistered semantic drift to E6 review. Once E6 reports a drift row, it has no
ordinary advisory default: scripts/claim_strength_drift_disposition.py requires
one explicit restore, authorize_with_reason, or pause choice per finding;
only an all-authorized sidecar derives authorized_to_continue. The sidecar
proves finding coverage and artifact binding, not semantic-detection completeness
or scientific warrant. Patch 1.0 lives only under
patch/legacy/v1_0/ with its archived loader.

The current #576 re_review/ family is version 1.1, uses
obligation_class, hard-requires original/revised drafts, roadmap, author
sidecar, and bundle, and copies
author fields exactly into Schema 11. Version 1.0 is archived under
re_review/legacy/v1_0/; mixed chains are invalid.

Spec: docs/design/2026-08-10-670-non-ranking-revision-roadmap-spec.md.

Human-subjects correspondence contract (#668) Copy Chapter

human_subjects/committee_correspondence.schema.json defines the standalone
academic-paper revision-coach committee-correspondence variant. It binds every
confirmed source comment to one concern record while preserving the entire UTF-8
letter byte-for-byte, including non-comment material. The contract carries
multi-label actions, explicit authority/provenance, optional profile enrichment,
fixed unresolved placeholders, the #665 administrative boundary, and no model
priority or severity.

Validate a bundle with:

python scripts/check_committee_correspondence.py \
  committee_correspondence/<source-sha12>/concern_tracker.json

The checker recomputes file/segment hashes, contiguous byte coverage, exact
comment-to-concern accounting, source order, full-permutation working views, and
response-skeleton coverage. Spec:
docs/design/2026-08-08-668-committee-correspondence-spec.md.

Human-subjects authority context (#666) Copy Chapter

The human_subjects/ authority family keeps selection explicit and separates three
closed artifacts:

• irb_context_record.schema.json — the author-confirmed facts plus exact,
  axis-qualified profile and overlay pins;
• authority_profile_registry.schema.json — curator-owned, versioned, bounded
  profiles and row-local source anchors;
• resolved_authority_context.schema.json — a pointer-only, deterministic
  three-valued applicability trace and downstream gate.

V1 has exactly two axes: review_ethics and data_protection. Institutional and
funder rules are additive overlays, never a third axis; display precedence cannot
remove, merge, or satisfy a requirement. The shipped registry demonstrates the
same contract with bounded US 45 CFR 46, Taiwan Human Subjects Research Act, and
GDPR research subsets. It is not a completeness, compliance, pathway, exemption,
or authorization claim.

Resolve an explicit context offline, lint the registry alone, or replay-check a
serialized result before consuming it:

python scripts/resolve_human_subjects_authority.py \
  --context context.json \
  --registry shared/human_subjects_authority_registry.json \
  --output resolved-authority-context.json

python scripts/resolve_human_subjects_authority.py \
  --registry shared/human_subjects_authority_registry.json \
  --check-registry

python scripts/resolve_human_subjects_authority.py \
  --context context.json \
  --registry shared/human_subjects_authority_registry.json \
  --check-resolved resolved-authority-context.json

The resolver is standard-library-only, opens only named files, evaluates a closed
Strong-Kleene predicate AST, never infers a jurisdiction, and rejects duplicate
JSON keys and non-finite numbers. Protocol:
shared/references/human_subjects_authority_protocol.md. Spec:
docs/design/2026-08-09-666-human-subjects-authority-contract-spec.md.

Human-subjects submission-packet manifest (#667) Copy Chapter

Two closed contracts define the deterministic packet layer:

• shared/contracts/human_subjects/submission_packet_inventory.schema.json — an author-declared
  list of exact packet-relative files, byte digests, evidence bindings, responsible
  holder roles, declared structure metadata, waiver/exception claims, and
  caller-supplied authorization status; and
• shared/contracts/human_subjects/submission_packet_manifest.schema.json — the pointer-only,
  replay-bound result with five structural status tokens, exact requirement and
  evidence pointers, independent #665 readiness/authorization fields, and the
  fixed non-authorization boundary.

The checker consumes a #666 authority result only after exact replay against its
named context and registry. It filters to submission_packet consumer rows and
uses only evidence_expected ids, artifact types, and holders. It never evaluates
or copies structured_expectations, descriptions, or attachment prose; exact
whole-row bytes are canonical-hashed only for replay integrity. It never infers a
jurisdiction, grants a waiver, verifies institutional acceptance, or updates the
caller-supplied authorization value. An evidence row is packet-owned only when
both its exact obligated actor and exact expected holder occur in the declared
packet-responsibility roles. Declared version, date, signature, and certificate
metadata is syntax/internal-consistency only unless a separately versioned,
source-backed mechanical expectation exists.
Once the authority and capability gates permit packet observation, every
inventoried path retains a DOCUMENTED or CONFLICTING row for
declared-vs-attached visibility, including extra files not consumed by the
selected profiles. A closed gate leaves observations empty and does not open the
packet root. Observation status alone does not change readiness; only an
applicable packet-owned evidence entry can create a listed packet gap.
The runtime rejects more than 512 copied consumer scopes, more than 4,096 entry
or exclusion rows, and any final canonical manifest larger than 8 MiB, ensuring
that every successful build can be replayed by the same CLI.

Build, replay-validate, or render a manifest offline with:

python scripts/build_submission_packet_manifest.py build \
  --inventory inventory.json \
  --packet-root packet \
  --context context.json \
  --registry shared/human_subjects_authority_registry.json \
  --resolved resolved-authority-context.json \
  --output submission-packet-manifest.json

python scripts/build_submission_packet_manifest.py validate \
  --manifest submission-packet-manifest.json \
  --inventory inventory.json \
  --packet-root packet \
  --context context.json \
  --registry shared/human_subjects_authority_registry.json \
  --resolved resolved-authority-context.json

python scripts/build_submission_packet_manifest.py render \
  --manifest submission-packet-manifest.json \
  --inventory inventory.json \
  --packet-root packet \
  --context context.json \
  --registry shared/human_subjects_authority_registry.json \
  --resolved resolved-authority-context.json

The authority triplet is all-or-none. An intentionally absent triplet produces an
unresolved manifest without a default profile; a partial or mismatched triplet is
a contract error. render requires the same inputs and exact replay, so a
self-consistent but forged manifest digest is insufficient.

Protocol: shared/references/submission_packet_manifest_protocol.md. Spec:
docs/design/2026-08-09-667-submission-packet-manifest-spec.md.

Review-pathway rule trace (#669) Copy Chapter

Two closed Draft 2020-12 contracts define the determination-adjacent trace:

• shared/contracts/human_subjects/review_pathway_trace_request.schema.json
  binds caller-owned candidate question labels to an exact, complete partition
  of every selected-profile pathway_trace requirement; and
• shared/contracts/human_subjects/review_pathway_rule_trace.schema.json
  carries matched, unmatched, and unresolved predicate work, profile-local
  alternatives, exact fact occurrences, responsible authority roles, exact
  requirement/anchor pointers, the fixed institutional result, and the #665
  footer.

The standard-library-only builder first replays the exact #666 context,
registry, and resolved artifact. It never invents a candidate name, profile,
predicate, authority anchor, determination, probability, rank, or timeline. All
selected profiles on both axes are accounted for, but candidates and
alternatives stay profile-local. An unknown requirement fact remains unresolved;
a missing profile halts without candidate rows at JURISDICTION_UNRESOLVED.

Build, replay, render, and lint only explicitly named artifacts:

python scripts/build_review_pathway_rule_trace.py build \
  --request pathway-trace-request.json \
  --context context.json \
  --registry shared/human_subjects_authority_registry.json \
  --resolved resolved-authority-context.json \
  --output pathway-rule-trace.json

python scripts/build_review_pathway_rule_trace.py validate \
  --request pathway-trace-request.json \
  --context context.json \
  --registry shared/human_subjects_authority_registry.json \
  --resolved resolved-authority-context.json \
  --trace pathway-rule-trace.json

python scripts/build_review_pathway_rule_trace.py render \
  --request pathway-trace-request.json \
  --context context.json \
  --registry shared/human_subjects_authority_registry.json \
  --resolved resolved-authority-context.json \
  --trace pathway-rule-trace.json \
  --output pathway-rule-trace.md

python scripts/check_review_pathway_output.py \
  --trace-json pathway-rule-trace.json \
  --rendered pathway-rule-trace.md

The banned-output lint is surface-scoped to those named generated files and
permits a pathway term only in the exact candidate grammar. Successful replay or
lint never changes a readiness, authorization, acceptance, verdict, checkpoint,
or workflow gate. Protocol:
shared/references/review_pathway_rule_trace_protocol.md. Spec:
docs/design/2026-08-11-669-review-pathway-rule-trace-spec.md.

Authority-profile content-coverage advisory (#681) Copy Chapter

shared/contracts/human_subjects/content_coverage_advisory.schema.json defines
the closed final content-coverage-advisory/1.0 carrier. It consumes a #667
manifest only after exact replay against the named inventory, packet root,
#666 context, authority registry, and resolved context. It then binds explicit
evaluator judgments to exact structured_expectations[] pointers and exact
session-held artifact strings through evidence-row/1.1 rows. The standard
library finalizer is scripts/build_content_coverage_advisory.py.

The output layer is always LLM-ADVISORY, and its independent field is
advisory_coverage_status. The finalizer copies deterministic packet status,
readiness, caller-supplied authorization, institutional-acceptance boundary,
authority/evidence pointers, and digests without changing them. A structural
gap, external dependency, or waiver/exception boundary cannot be converted into
a semantic missing-element finding. Applicability-false requirements remain
excluded, and every profiled structured expectation is either explicitly
checked or explicitly not_checked; missing session content never becomes an
implicit negative result.
An open authority/capability gate with an explicitly unprovided overlay
selection preserves each selected base requirement as
APPLICABILITY_UNRESOLVED without inspecting content; a fully closed gate has
no exact requirement to report and is rejected.

The final carrier is deliberately marked evaluation_status: UNMEASURED.
UNMEASURED is not a scored measurement row, and this feature makes no accuracy,
coverage-improvement, or efficacy claim. The finalizer and renderer open only
named inputs, perform no directory scan or retrieval, and invoke no model/API;
the draft judgments and positive-row capture timestamps are caller-supplied
advisory observations. Rendering first replays the deterministic manifest and
every source-bound evidence row, so a self-consistent digest alone is
insufficient. The carrier retains all rows, while one render call exposes only
one explicit page of at most 25 evidence rows with deterministic navigation;
there is no render-all mode.

Protocol:
shared/references/authority_content_coverage_advisory_protocol.md. Spec:
docs/design/2026-08-09-681-authority-content-coverage-advisory-spec.md.

Tortured-phrase screening contracts (#660) Copy Chapter

The #660 family is local, hash-bound, and advisory-only:

• audit/tortured_phrase_snapshot.schema.json defines the closed canonical
  literal / all / any / near AST and rule-level exclude_if grammar;
• audit/tortured_phrase_snapshot_manifest.schema.json binds the exact raw
  snapshot bytes, source/version/as-of metadata, preprocessing disclosure,
  zero unsupported rules, and rights; and
• audit/tortured_phrase_advisory.schema.json defines the replay-bound
  own-draft HEURISTIC-ADVISORY / UNMEASURED transcript.

scripts/tortured_phrase_screening.py accepts only explicitly named local
snapshot/manifest and input paths. A snapshot is either user supplied or a
repository-authored synthetic fixture. ARS includes no native PPS importer,
fetch option, or redistributed PPS list content, and this path invokes no
model, external API, human/model judge, ambient clock, file time, or network
time. Snapshot SHA-256 covers the exact UTF-8 file bytes; required timestamps
are explicit arguments.

The same runtime can return a new passport copy carrying
bibliographic-integrity-signal/1.2 rows for each cited title and abstract
surface. A missing abstract is an explicit not_checked / unresolved
ABSTRACT_MISSING row; a present whitespace-only abstract uses
ABSTRACT_EMPTY. Consumers remain read-only and the enricher refuses
in-place output. A detected row is only a phrase-list match requiring review;
a zero match is not a clean certificate. No result establishes origin,
paper-mill production, contextual validity, or accuracy, and no result creates
a marker, terminal gate, replacement text, or automatic rewrite. All corpus
rows render in the one canonical Bibliographic Integrity Advisories section.

Spec: docs/design/2026-08-10-660-tortured-phrase-screening-spec.md.

Cross-document consistency advisory (#672) Copy Chapter

The #672 family is a standalone replay-bound advisory:

• passport/preregistration_artifact.schema.json defines the persistent
  preregistration-artifact/1.0 handoff sidecar;
• audit/cross_document_source_manifest.schema.json binds exactly the accepted
  manuscript plus the exact sidecar projection;
• audit/cross_document_consistency_advisory_draft.schema.json accepts the
  closed caller-supplied semantic observations;
• evidence/evidence_row_v1_2.schema.json binds their exact bilateral or
  trilateral evidence; and
• audit/cross_document_consistency_advisory.schema.json defines the canonical
  final LLM-ADVISORY / UNMEASURED carrier.

Only scripts/build_cross_document_consistency_advisory.py may build or update
the preregistration sidecar. The non-shell research architect supplies only the
explicit caller status and companion handle. Academic-paper intake and every
pipeline handoff validate and carry the same sidecar and, when provided, the
same companion bytes unchanged. The repository template is guidance, not
evidence.

Finalization replays the sidecar, exact two-artifact manifest, accepted draft,
optional preregistration companion, and every quote or checked scope before
consuming observations. Methods absence requires an exact named counterpart
scope. An undisclosed preregistration deviation requires a third exact manuscript
disclosure-scope witness. Missing or unavailable inputs cannot become a
no-listed result.

At the one mandatory Stage-5 entry checkpoint, #660 runs first and #672 second
against the same accepted-draft artifact ID/SHA-256. Their carriers and failure
semantics stay separate. #672 failure writes no output and produces only bounded
ADVISORY_UNAVAILABLE:<CODE>; neither advisory changes Stage 4.5 or Stage-5
routing. Any manuscript revision stales both and requires both to rerun.

The final carrier has no score, pass/fail, gate, readiness, authorization,
rewrite, ClaimIntent, consent/protocol duplicate, or clean-document meaning.
Protocol: shared/references/cross_document_consistency_advisory_protocol.md.
Spec: docs/design/2026-08-10-672-cross-document-consistency-advisory-spec.md.

Audit artifact contracts (v3.6.7 Step 6) Copy Chapter

The audit/ directory carries the three wrapper-emitted artifact schemas that pair
with the passport-side audit_artifact_entry.schema.json above. Together they form
the four-schema contract that scripts/run_codex_audit.sh (Phase 6.1) writes and the
orchestrator agent reads at every per-agent audit gate.

• audit/audit_jsonl.schema.json — Layer 2 evidence: per-row schema for the codex CLI
  0.125+ --json event stream (thread.started / turn.started / item.completed /
  turn.completed / error). One JSONL line per event row.
• audit/audit_sidecar.schema.json — Layer 3 evidence: runner / timing / process /
  stream / prompt metadata. Cross-file rules linking sidecar fields to JSONL events,
  on-disk files, and passport entries (B1-B7 in spec §3.7 family B) are enforced by
  scripts/check_audit_artifact_consistency.py (Phase 6.3), not by this schema alone.
• audit/audit_verdict.schema.json — verdict file shape (PASS / MINOR / MATERIAL /
  AUDIT_FAILED). The artifact orchestrator parses for ship/block decisions; cross-field
  consistency with finding_counts and failure_reason is lint-enforced per
  spec §3.7 A1 / A2 / A5 / A6.

Spec: docs/design/2026-04-30-ars-v3.6.7-step-6-orchestrator-hooks-spec.md §3.

Review target contracts (#683) Copy Chapter

The review_target/ family keeps target selection author-owned and separates it
from deterministic criterion resolution:

• review_target_declaration.schema.json — the closed author-confirmed discipline,
  exact venue/track/contribution-type (or explicit no-venue fallback), overlay,
  selection, precedence, and as-of input;
• criteria_registry.schema.json — the versioned four-part authority registry with
  criterion provenance, applicability/exclusions, freshness, and blocking policy;
• review_criteria_source_receipt.schema.json — the closed discriminated
  receipts for mutable-web semantic snapshots and immutable Git repository-head
  verification;
• review_target_context.schema.json — the pointer-only resolved profile, three
  independent outcome dimensions, parallel conflicts, fallback state, and stable
  digests.

shared/review_criteria_registry.json ships a bounded source-backed proving set
for MSR 2027 Technical Papers Full Papers and ACM SIGSOFT's General and
Repository Mining standards, alongside the field-general fallbacks. This is not
a coverage claim: targets without an exact official profile remain unresolved,
and the SIGSOFT rows remain advisory unless venue adoption is separately
sourced. The executable exact-target declaration and hermetic source/digest
checks live under scripts/fixtures/review_target_context/ and
scripts/test_resolve_review_target_context.py; the dated source audit is
audits/575-source-backed-proving-set-2026-08-24.md. Mutable-page and
immutable-repository verification receipts live under
shared/review_criteria_sources/. The fixture models a hypothetical
author-confirmed declaration; it is not a real-manuscript attestation. The
registry release preserves its id, increments its version, and visibly rotates
V1 resolved digests so consumers must explicitly rebind.

Resolve a declaration and optionally emit the Phase 0/1 Target Criteria Brief:

python scripts/resolve_review_target_context.py \
  --context declaration.json \
  --output review-target-context.json \
  --brief target-criteria-brief.md

The resolver is standard-library-only and opens only the named context and registry
inputs. It never reads manuscript content, infers a venue, averages interdisciplinary
criteria, or applies adaptive numeric weights. Spec:
docs/design/2026-08-08-683-review-target-context-spec.md.

Review criteria consumer binding (#684) Copy Chapter

The #684 layer binds the #683 context across the three actual consumer classes
without copying criterion prose or inventing a second resolver:

• review_target/review_criteria_binding_manifest.schema.json — the closed,
  pointer-only manifest with exact formative, internal-evaluator, and
  five-seat external-panel receipt roles;
• review_target/constructive_review_findings.schema.json — the closed
  Critical/Major finding sidecar with exact criterion pointers, typed
  manuscript anchors, separate scholarly/target relevance, honest remedies,
  costs/trade-offs, and author-choice boundaries.

Build, record, render markers, and validate with
scripts/review_criteria_binding.py. The tool reads only explicit paths,
hashes artifacts itself, and has no model/API/network/clock/ambient-scan path.
The manifest is workflow-conformance authority only: it never supplies a
severity, verdict, checkpoint, or author triage. Operational protocol:
shared/references/review_criteria_consumer_protocol.md. Frozen spec:
docs/design/2026-08-11-684-review-criteria-consumer-binding-spec.md.

Overview (`tools/release-discipline/README.md`) Copy Section

vendored from [email protected] on 2026-07-03

This directory contains a snapshot of release-discipline-toolkit scripts and
GHA workflows. Do not edit; regenerate by running upstream sync-toolkit.sh.

Overview (`evals/gold/citation_extraction/README.md`) Copy Section

Citation-Extraction Gold Subset (Phase 1a)

51-tuple gold subset measuring the per-citation lookup_verified 3-class enum classification accuracy (50 original + one v3.11 by-design false-negative fixture, spec OQ-5 / C-V6(a)). Phase 1a shipped the data; Phase 1b (#263) shipped the harness (scripts/run_evals.py) — it now measures this gold set directly.

Run it with:

PYTHONPATH=. python -m scripts.run_evals --task citation_extraction

The harness computes the predicted lookup_verified from each tuple's resolver_outcomes via the #182 Delta 4 reducer citation_verification_summary.reduce_lookup_verified (the single source of truth). verification_gate.verify_citation (#182 Delta 5) runs the resolvers live and feeds the SAME reducer, so the harness and the shipped API agree by construction. The reducer uses the v3.11 narrowed-false definition (C-V6(a)): false requires an ID-keyed unmatched (queried_by: id); a title-only unmatched is a coverage gap and reduces to unresolvable, never false. The metric is symmetric 3-class accuracy — unresolvable is never collapsed into false. Because expected_outcomes.json was authored by the same rule, a fresh run scores ~1.0.

Spec reference Copy Chapter

docs/design/2026-05-21-v3.10-184-extend-eval-harness-spec.md §3.1.1

Tuple distribution Copy Chapter

Threshold contract (v3.10.0 binding defaults) Copy Chapter

• Aggregate: accuracy >= 0.90 across all 51 tuples
• Per-class: accuracy >= 0.85 for each of true / false / unresolvable

Changing these before v3.10.0 ship requires a spec amendment per #184 §3.1.1 / E-V2.

Tuple file naming Copy Chapter

NNN-{kind-slug}-{discriminator}.json (zero-padded NNN, lowercase-hyphenated). Filename stem must match the tuple_id field inside.

Fabricated-reference safety Copy Chapter

Tuples 031–040 and 046–050 use intentionally-bogus DOIs (10.99999/ars184.fake.<domain>.<seq>) and bogus titles. Each carries fabrication_intent: true per public-repo discipline. Do not source fabricated tuples from real-but-misattributed citations.

Amendment: `valid_unresolvable` class removed Copy Chapter

The original spec defined a valid_unresolvable class (tuples 031–040) — real citations
that are unmatched across all four resolvers. No stable, first-party-verifiable source
satisfying that constraint was found under current index coverage, so the class was
removed and 031–040 are now fabricated. The false class is carried by fabricated
tuples. The missing real-but-unindexed tuple is an accepted corpus-representativeness
limitation, recorded historically in #250; there is no current action item to add one.
It is not missing resolver regression coverage: this static harness reduces authored
resolver_outcomes and cannot exercise resolver-client title matching. The client-layer
false-positive behavior was isolated in #431, fixed by #432, and is covered by direct
client HTTP-path tests.

Human expert verdicts Copy Chapter

11 tuples (the original 10 at 20% per Delta 5, plus the v3.11 by-design FN fixture) carry an optional human_expert_verdict field. The Phase 1b harness emits an expert_concordance row per class from these labeled tuples (agreement of the expert verdict vs expected_outcomes.json). The verdicts are advisory only — synthetic ground truth in expected_outcomes.json is the source of truth for CI gates per E-V3, and concordance never gates.

Validator Copy Chapter

Run this command from the repo root to validate the corpus against its manifest:

python -m scripts.check_evals_gold_set evals/gold/citation_extraction

Lift gate Copy Chapter

When a PR changes ranking / scoring logic, run the harness on the base and the change, then compare:

PYTHONPATH=. python -m scripts.run_evals --output before.json   # on base
PYTHONPATH=. python -m scripts.run_evals --output after.json    # on change
python -m scripts.check_ranking_lift --baseline before.json --compare after.json --pr-body @pr.txt

The gate blocks on any polarity-corrected signed_lift < -0.05 (or a zero-baseline metric change) unless the PR body carries [ranking-regression-acknowledged] + an OPEN follow-up issue and declares the Affected metric: <task>.<class>.<metric>. CI wires this via .github/workflows/eval-harness.yml.

Overview (`evals/gold/field_norm_severity/README.md`) Copy Section

Field-Norm Severity Gold Set

Issue: #215

A first-party regression fixture of severity-miscalibration cases drawn verbatim from Kim et al. 2026 (arXiv:2605.20668v1, "On the limits and opportunities of AI reviewers"). It pins the two documented failure shapes the #215 changes target, so prompt/protocol edits can be checked against real, traceable cases.

What this is (and is not) Copy Chapter

This is a regression fixture, not a detector calibration set. There is no deterministic predictor for "field-norm severity miscalibration" — judging it requires the field prior the paper shows AI reviewers lack. So the validator (scripts/check_field_norm_severity.py) checks data integrity + first-party provenance, not FNR/FPR. With n=10 it makes no distributional-calibration claim.

Subtypes Copy Chapter

• field_norm_boundary — paper W1 (n=54, the largest weakness class): the AI critique is content-correct against a discipline-neutral standard but mis-rated because it lacks the subfield's accepted-practice prior. Example: a CERN/LHCb reproducibility request that ignores the collaboration's internal-artifact norm.
• significance_boundary — paper §F.3.4 (56 errors): the AI meta-reviewer's habitual "would addressing this change the core result?" formula under-weights methodological rigour / scope / translational relevance and over-weights presentation issues with technical terminology.

Both are severity_miscalibration: true; the subtype distinguishes the driver. They are not mutually exclusive in nature (a case can be both field-norm-like and significance-like, e.g. the Coulombic-efficiency case), so the subtype records the dominant driver rather than partitioning the space.

Provenance Copy Chapter

Every case carries provenance with a stable, extraction-independent anchor: the paper section/example ID (e.g. §F.3.4 Pattern A Example 1), the paper's own citation token (e.g. P9 · GPT-5.2 · item 5 · primary), the reviewer model, and a verbatim quote snippet. Anchors are section/example IDs, not session-scoped line numbers (those came from a transient pdftotext extract and would rot).

Exception fixtures Copy Chapter

sgb-005-exception (SAR safety at 11.7T MRI) carries exception: true + exception_reason. The paper notes experts agreed with the AI's direction in that specific case while still citing it under the broader under-rating pattern. It is retained — not as a clean positive case — to document that the "does not change the core result" formula is not always wrong, so a severity check must not blanket-flag it.

Run Copy Chapter

python -m scripts.check_field_norm_severity

Overview (`evals/gold/rq_framing_patterns/README.md`) Copy Section

RQ Framing Pattern Gold Set

Issue: #257

This gold set calibrates the Socratic wording-pattern advisory. It measures whether a conservative lexical detector catches common AI-typical research-question wording shells without over-warning domain-native research questions.

Labels Copy Chapter

• wording_cliche: surface wording matches one of the WP01-WP20 shell patterns.
• domain_native: wording is specific enough that the advisory should not trigger, even if the topic itself is broad.

Scope Copy Chapter

The detector and gold set evaluate wording only. They do not judge idea quality, novelty, feasibility, or contribution.

Acceptance Thresholds Copy Chapter

• FNR < 0.30
• FPR < 0.20
• balanced accuracy >= 0.75

Run:

python -m scripts.check_rq_framing_patterns

Overview (`evals/gold/surface_form_parity/README.md`) Copy Section

Surface-Form Parity Gold Set

Issue: #216

A mixed-provenance regression fixture for the §F.3.6 reviewer-type asymmetry documented in Kim et al. 2026 (arXiv:2605.20668v1, "On the limits and opportunities of AI reviewers"). It is anchored to first-party verbatim examples from the paper, with a small number of maintainer-authored variants that are labelled honestly as such.

What the paper found Copy Chapter

The AI meta-reviewer applies two different standards depending on the prose style of the review item (§F.3.6, "Asymmetry mechanism"):

• For human-written items (typically informal, vague, loosely scoped) it demands literal precision and penalises informal language — so it over-rejects correct human concerns. Of 41 correctness false negatives, 29 involved human reviewers.
• For AI-written items (typically naming specific technical concepts, code elements, math frameworks) it credits technical specificity and accepts claims without checking them against the paper — so it over-accepts incorrect AI concerns. Of 13 false positives, 10 involved AI reviewers.

The root cause the paper names is a learned prior that specificity correlates with correctness, which misfires in both directions.

Why "surface-form", not "authorship" Copy Chapter

The hook is the prose style (framing_style: informal_vague vs technical_precise), not the author label. Blinding the author does not remove the bias, because the prior keys off how the item is written, not a literal "human"/"AI" tag. So the mitigation (#216) is a Surface-Form Parity self-check at verdict time — judge the checkable claim against the paper, not against polish — and authorship is deliberately kept out of the runtime reviewer-item schema (see the design note).

What this is (and is not) Copy Chapter

This is a regression fixture, not a detector calibration set. There is no deterministic predictor for this bias, n is tiny, and the paper's 29-FN-human / 10-FP-AI numbers are directional (§H), not thresholds. The validator (scripts/check_surface_form_parity.py) checks data integrity + provenance honesty + pair invariants — it does not compute FNR/FPR or assert distributional asymmetry. A lint that claimed to measure detection accuracy on a prompt-based judge would be fluent wrongness: green while the real bias persists.

Provenance honesty (mixed-provenance) Copy Chapter

Because the set mixes real and maintainer-authored material, every item carries an explicit provenance_type:

• paper_verbatim (n=4) — a real §F.3.6 example with a verbatim_anchor quote that appears in the paper. Items: sfp-001, sfp-002, sfp-003, sfp-004.
• counterfactual_rewrite (n=2) — a maintainer-authored minimal-difference variant of a paper_verbatim case: same canonical_claim, same expected_correctness, rewritten in the opposite framing_style, truth conditions preserved. Carries derived_from + semantic_equivalence_rationale. NOT first-party — never cite it as paper-verbatim. Items: sfp-001-cf, sfp-003-cf.
• maintainer_boundary (n=1) — a maintainer-authored boundary case whose review_item_text is synthetic (the paper states the mechanism but gives no verbatim item for this boundary). Anchored to the paper's mechanism sentence via mechanism_anchor, not a verbatim review quote. Item: sfp-005-ambiguous.

Anchors are paper section/example IDs + verbatim quote snippets, not session-scoped pdftotext line numbers (those rot).

Pairs Copy Chapter

Two pair_id groups (pair-01, pair-03) hold a paper_verbatim + its counterfactual_rewrite. Pair invariant: members share canonical_claim and expected_correctness and differ only in framing_style. The discipline being fixtured: a correctness verdict that flips between paired members is keying off style, not substance — the §F.3.6 failure. pair_id is internal; it is in judge_blind_fields and must never reach the judge.

Judge-blind fields Copy Chapter

metadata.judge_blind_fields lists everything that must be stripped before an item is shown to a reviewing judge: pair_id, framing_style, provenance_type, expected_correctness, expert_verdict, meta_reviewer_verdict, asymmetry_direction. The judge sees only an opaque handle (derived internally from a position index) + review_item_text — not the fixture id, because the IDs encode the answer (-cf = counterfactual, -ambiguous = expected boundary verdict). A serializer-strip test enforces this (the #216 enforcement point for the "authorship/style invisible at judgment time" decision).

Exception / boundary fixture Copy Chapter

sfp-005-ambiguous (exception: true) documents the UNLESS clause of the parity rule: "do not down-rate informal/vague wording unless ambiguity changes truth conditions or makes the item unevaluable." It is a genuinely unevaluable vague item where a Not-Correct / ambiguous outcome is correct. Retained so the parity rule cannot be misread as "never penalise vagueness" — without it, a judge could over-correct into accepting empty concerns.

Relationship to #273 Copy Chapter

#273 (same-family / rubric-aware calibration) is a different mechanism and is not folded in here. Cross-reference only — no shared prompt, gold, or runtime wiring. See the design note.

Run Copy Chapter

python -m scripts.check_surface_form_parity

Overview (`evals/heldout/authority_content_coverage/README.md`) Copy Section

Authority-profile content coverage — held-out evaluation status

Current result Copy Chapter

UNMEASURED — no scored held-out measurement row exists.

The #681 implementation provides a closed advisory carrier, exact source-bound
evidence rows, replay validation, and hermetic contract tests. Those tests show
that the machinery preserves its inputs and boundaries. They do not measure
whether an LLM correctly identifies semantic coverage.

This directory deliberately contains no measurement-*.json, no synthetic
"unmeasured" row, no placeholder score, and no inferred baseline. The
evaluation_status=UNMEASURED marker in a final advisory is an honest product
boundary, not a measurement artifact.

Claims prohibited while unmeasured Copy Chapter

Until a real scored row exists, ARS must not claim or imply:

• content-coverage accuracy, precision, recall, sensitivity, or specificity;
• improved IRB/committee acceptance or submission readiness;
• fewer missing consent elements in practice;
• equivalence or superiority to a human reviewer; or
• validated performance for a jurisdiction, language, document type, profile,
  model, or prompt.

Contract acceptance phrases such as "exact replay passed" or "mutation test
passed" refer only to carrier integrity and noninterference, never efficacy.

Required future held-out design Copy Chapter

Any future efficacy claim requires a pre-registered held-out suite and a real
scored row under evals/heldout/MEASUREMENT_CONTRACT.md. The suite must register
as llm_judged before publishing that row and must at minimum freeze:

1. exact selected-profile, effective-date, context, registry, resolved-manifest,
   inventory, and content fixtures;
2. bilingual/multilingual passages where supported, including Unicode and
   document-locator variation;
3. positive coverage, checked absence, genuine conflict, missing access,
   conditional-false, external-actor, and waiver/exception cases;
4. gold expectation-level labels and bounded quoted anchors prepared without
   access to subject output;
5. a precommitted adjudication rubric that distinguishes "appears covered" from
   adequacy, compliance, acceptance, and authorization;
6. at least two disclosed judge configurations from different model families for
   a decision-relevant run, plus the measurement contract's replication,
   blinding, raw-output, and adjudication requirements; and
7. per-expectation false-positive and false-negative accounting, especially the
   prohibited false-missing cases.

Conditional-false requirements and profiled waiver/external-authority boundaries
must be scored as boundary-preservation cases, not ordinary negative-coverage
examples. Missing content must be scored as not_checked, never as a pass or a
located omission.

Separation from hermetic fixtures Copy Chapter

Fixtures under scripts/fixtures/content_coverage_advisory/ are implementation
tests. They may exercise deterministic mappings with caller-supplied draft
observations, but they are visible to maintainers and are not a held-out corpus.
Passing them cannot create a score or change this README's UNMEASURED status.

Overview (`evals/heldout/claim_standing_probe/README.md`) Copy Section

Claim-Standing Probe — Stance Seed Set (#655, v0.1)

Authority: docs/design/2026-08-13-655-search-bounded-claim-standing-probe-design.md
(§5.1 closed labels, §8 held-out stance set and baseline gate).

STANCE CLASSIFICATION UNMEASURED

Current status Copy Chapter

This directory holds the seed required by design §8: repository-owned
synthetic claim/candidate pairs, the closed label vocabulary as schemas, the
expert label guide, and a mechanical scorer. No probe implementation, no
expert labels, no adjudication, no subject run, and no baseline measurement row
exist. Accordingly:

• claim_standing_probe is deliberately absent from
  evals/heldout/suite_registry.json — registration happens in the
  implementation PR that carries a valid baseline row under the #654/#664
  contract (the validator fails if the suite is registered while the set says
  seed_unmeasured);
• nothing here supports any accuracy, usefulness, or coverage claim.

Contents Copy Chapter

• heldout_stance_set.json + .schema.json: 32 items (16 en + 16 zh-TW), one
  discipline each, covering per language: support ×2, contradict ×2, mixed ×2,
  not_addressed ×2, INSUFFICIENT_EVIDENCE ×2, AMBIGUOUS ×2, missing-abstract,
  metadata-only, irrelevant-candidate, and session-held-full-text slots.
  Claims and abstracts were cross-model authored (Codex gpt-5.6-sol,
  reasoning xhigh, 2026-08-14) so the future subject's model family did not
  write its own test items; all content is synthetic (fictional actors,
  instruments, venues; DOIs use the reserved 10.99999/csp-* form).
• label_guide.md: the expert labeling procedure operationalizing design
  §5.1's closed enum and cross-field rules (relevance first; missing evidence
  is never not_addressed; explicit both-directions → mixed, otherwise
  AMBIGUOUS; no scalar confidence anywhere).
• expert_label.schema.json: one blinded expert's raw label file (≥2
  independent domain-qualified experts required).
• adjudicated_labels.schema.json: the sealed ground truth — expert files are
  hash-bound, every disagreement resolved by a separate adjudicator blind to
  subject output, raw labels retained.
• subject_output.schema.json: the strict enum surface the future subject
  emits; deliberately closed with no confidence/probability/score field.
• stance_score_report.schema.json: the deterministic comparison artifact.
• scripts/validate_claim_standing_stance_assets.py: schema + invariant
  validator (slot coverage, id/doi binding, zh-TW simplified-character screen
  as a heuristic defect detector, registration guard).
• scripts/claim_standing_stance_scorer.py: the mechanical scorer required by
  design §8 — compares the frozen subject enum to the adjudicated label with
  no model in the loop; reports stance/relevance/check-state confusion counts
  by language, evidence-scope agreement counts, and failure-class counts
  (per-scope and per-failure-class accuracy strata belong to the future
  baseline-row compiler); macro recall and micro accuracy stay separate; an
  abstention on a gold-performed row lands in an explicit
  NOT_CHECKED confusion column so abstaining can never inflate recall;
  full-row accuracy includes evidence scope; blocked/partial rows keep their
  failure classes visible and are never imputed; the two-replicate
  decision-relevance flag is computed, not assumed; the adjudication file must
  bind at least two distinct expert label files.

design_target is not ground truth Copy Chapter

Each item carries a design_target (the relation its text was commissioned to
realize). It exists for construction coverage accounting only. It is never
ground truth, never enters the scorer as gold, and must be stripped from any
expert or subject packet. Ground truth comes exclusively from at least two
independent blinded expert label files plus adjudication
(adjudicated_labels.schema.json).

What must still happen before a baseline row (design §8) Copy Chapter

Frozen here: item set, label vocabulary/guide, record schemas, mechanical
scorer. Still future, owned by the implementation PR: the exact eligible
population and sampling rule binding the probe's trigger; the subject prompt,
model/runtime/token settings, replicate and stopping rules, and execution
manifest (hash-frozen before dispatch); expert recruitment, labeling, and
adjudication; the baseline stance-classification measurement row under the
#654/#664 contract; and suite registration. Any model or human dispatch
requires separate consent to the exact frozen plan.

Accepted seed boundaries Copy Chapter

• Static text can only realize the abstract_missing failure family; the
  runtime failure classes (source_missing, access_failed,
  retrieval_failed, judge_timeout, judge_error, parse_error) are
  subject/harness vocabulary, to be exercised by fault injection in the
  implementation PR's run plan, not by fixtures.
• The scorer verifies that the adjudication file names two distinct experts
  with distinct file hashes and covers every item, and
  validate-expert-file checks one expert file's complete distinct item
  coverage — but nothing here verifies the referenced expert files' bytes
  against the recorded hashes; sealing and hash-verifying expert packets is
  the implementation PR's labeling-workflow tooling.
• Design §5.1's "at least one evidence-row reference" applies to the
  implemented probe's #656 evidence rows; the eval subject output carries a
  bounded rationale and evidence scope instead, because no evidence-row
  surface exists in the seed.

Offline validation Copy Chapter

python3 scripts/validate_claim_standing_stance_assets.py validate-assets

Overview (`evals/heldout/cross_document_consistency/README.md`) Copy Section

Cross-document consistency advisory — UNMEASURED

This directory documents the measurement boundary for #672. The repository's
invented fixtures exercise only closed-contract validation, source/UTF-8 replay,
failure states, deterministic identifiers/digests, consumer wiring, paging, and
legacy isolation.

They do not measure semantic accuracy, inconsistency coverage, agreement,
false-positive rate, false-negative rate, usefulness, or efficacy. A
NO_LISTED_INCONSISTENCY_LOCATED fixture is not proof of agreement,
completeness, or a clean document.

There are no measurement-*.json files, raw model outputs, judge records, or
accuracy claims here. No live model, external API, network service, human/model
judge, existing Codex multi-file audit, or expensive evaluation is required or
authorized. The final carrier therefore remains
evaluation_status=UNMEASURED.

Overview (`evals/heldout/experiment_alignment_overclaim/README.md`) Copy Section

Experiment-Alignment Overclaim Held-Out Set (#915)

Seed only. Status, purpose, pass rule, and the item contract are the
heldout_set.json fields (status, purpose, scoring,
subject_visible_fields, hidden_fields); this file carries only what the
JSON cannot.

• Source: the DynaSpec-RAG showcase manuscript in ScientistTwo (Nam et al.,
  2026, arXiv:2609.19644 v1, Appendix D). The PDF is not vendored. Passports
  hold short quotes, numbers, and arXiv v1 page references only.
• Why held-out, not gold: the subject is an LLM running
  integrity_verification_agent.md section C4, so scripts/run_evals.py must
  not discover this directory and there is no target.entrypoint.
• What the subject sees: the passport file and manuscript_locators.
  ground_truth and rule_anchor are never shown.
• Why this is not cross_document_consistency: that suite compares two
  documents. Here one manuscript's claims are checked against its own tables.
• Controls: each passport carries one supported claim (st2-01 C-003,
  st2-02 C-002). A subject that marks every claim OVERSTATED fails them.
• Before any result is quoted: file a heldout-measurement/1.1 report per
  evals/heldout/MEASUREMENT_CONTRACT.md (registered class mechanical_match).
  Two cases from one paper are a smoke check, not a detection rate.

Overview (`evals/heldout/indirect_prompt_injection_behavior/README.md`) Copy Section

Indirect Prompt-Injection Behavior Paired Controls (#675, v0.1)

This held-out suite measures a frozen model's behavior when synthetic retrieved
or pasted content contains an inert embedded instruction. The authority is
docs/design/2026-08-13-675-indirect-prompt-injection-behavior-eval-spec.md.

Current status Copy Chapter

seed_version: 0.1.0 — design, fixtures, rubric, validator, diagnostic scorer,
and a Phase-2 no-call execution/ingestion/blinding envelope. No subject
baseline exists. No safety, robustness, or ARS-guidance efficacy claim is
computable from these assets.

The suite is behavioral. It does not enforce an instruction/data boundary.
Structural isolation remains #676.

Contents Copy Chapter

• heldout_set.json: eight synthetic scenarios in a closed 2 x 2 design;
• heldout_set.schema.json: strict fixture schema;
• prompt_ars_guided.txt: production-guidance condition;
• prompt_platform_only.txt: non-production platform-only control;
• adjudication_rubric.md: pre-dispatch semantic label and blinding rules;
• scripts/run_indirect_prompt_injection_probe.py: strict validator,
  one-replicate materializer, and mechanical response scorer;
• run_plan.schema.json, authorization_record.schema.json,
  transcript.schema.json, ingestion_manifest.schema.json, and
  stop_intent.schema.json: closed
  Draft 2020-12 contracts for an exact 64-cell externally executed run;
• journal_token.schema.json: closed contract for materialization-time
  pre-armed ingestion, pre-load-terminal, and blind-bundle transaction tokens;
• blind_session_packet.schema.json, blind_inventory.schema.json,
  blind_private_map.schema.json, and blind_manifest.schema.json: isolated
  judge-packet, public inventory, private unblinding, and atomic final-bundle
  contracts;
• judge_assignment_ledger.schema.json: frozen future pre-delivery ledger
  shape; this no-call runner does not create or validate a real ledger;
• scripts/run_indirect_prompt_injection_no_call.py: offline-only
  init-run, materialize, validate, ingest, and
  prepare-blind-packet commands. It contains no detect, dispatch, probe,
  model, provider, network, or subprocess path.

The eight scenarios cover web/source verification, PDF or manuscript ingestion,
bibliography intake, and pasted reviewer/committee comments in English and
zh-TW. Each scenario materializes four cells, so one replicate has 32 calls.

Validate and materialize without a model call Copy Chapter

python scripts/run_indirect_prompt_injection_probe.py validate-assets
python scripts/run_indirect_prompt_injection_probe.py materialize \
  --output-dir /path/to/new/neutral-run-directory

materialize refuses a non-empty output directory. It writes neutral prompt
filenames and a hash manifest. It does not dispatch a model, enable tools, or
contact a provider.

To exercise the mechanical parser on one retained response:

python scripts/run_indirect_prompt_injection_probe.py score-output \
  --scenario-id ipi-web-en \
  --condition injected \
  --response /path/to/raw-response.json

Mechanical output is diagnostic only. The final four outcomes require the
frozen blinded rubric and the held-out measurement contract.

Freeze a Phase-2 envelope without making a call Copy Chapter

init-run expands the exact 8 scenarios x 2 content conditions x 2 guidance conditions x 2 replicates = 64 cells. Its SHA-256 order is deterministic from
the supplied seed; replicate 2 reverses each scenario's four-factor order.
All provider/model/runtime values are declarations to bind future external
evidence, not capabilities of this runner.

python scripts/run_indirect_prompt_injection_no_call.py init-run \
  --run-dir /path/to/new-run \
  --run-id RUN_ID \
  --suite-commit 40_LOWERCASE_HEX \
  --order-seed PRECOMMITTED_SEED \
  --subject-provider PROVIDER \
  --subject-model MODEL \
  --subject-runtime RUNTIME \
  --subject-runtime-version VERSION \
  --auth-mode AUTH_MODE \
  --reasoning-effort EFFORT \
  --input-token-cap INPUT_CAP \
  --output-token-cap OUTPUT_CAP

python scripts/run_indirect_prompt_injection_no_call.py materialize \
  --run-dir /path/to/new-run --plan-sha256 EXACT_PLAN_SHA256
python scripts/run_indirect_prompt_injection_no_call.py validate \
  --run-dir /path/to/new-run --plan-sha256 EXACT_PLAN_SHA256

Materialization writes 64 neutral prompt files and 64 non-dispatch call
envelopes using exclusive creation. It also pre-arms one immutable journal
token for each ingestion, one pre-load-terminal token, and one for blind-bundle
construction. Claiming a token creates a second hard link to the same validated
inode before any external transcript bytes are acquired or any blind ids are
generated. A successful cell moves its claimed name to a same-inode completed
name only after the ingestion manifest replacement and directory fsync succeed.
Every
envelope fixes tools=[],
web_enabled=false, runner_transport=none, dispatch_available=false, USD 0
API spend, no fallback, and a fresh-external-authorization requirement.

ingest accepts exactly the next externally recorded transcript plus an
external authorization record bound to the exact plan bytes. The runner proves
schema, byte, hash, scope, and time ordering only. It explicitly does not verify
operator identity or turn a record into consent. Raw response and event bytes
are bounded and hashed. The runner's pinned canonical-event decoder derives the
action class from a closed native event type; every raw event also binds the
exact external session, cell, sequence, event index, and event id. An outer
class cannot hide a native tool or network event. A closed, hash-bound
external-session receipt must use a unique receipt id and fresh-context session
id, and session times must remain monotonic across all 64 cells. Observed input
and output token usage is mandatory and capped; a conservative response-byte
cap also ensures that every accepted transcript can fit a final packet.

The first blocked, partial, drifted, semantic mapping/prior-label leak,
forbidden-action, or evidence-write failure stops the whole run and permanently
forbids retry. The pre-armed journal claim is the first irreversible boundary,
including when transcript acquisition fails or the primary stop-intent write
itself cannot complete. Available raw bytes are preserved content-addressably;
a closed write-once stop intent containing exact base64 replay bytes is then
committed before stopped-state replacement. Surviving authorization,
transcript, and receipt artifacts are registered individually, while one
compact canonical tree digest binds every otherwise-unregistered file and
directory, including empty directories and replacement staging. A failed
manifest replacement is replayed from the intent, never retried as a call.
If replacement publishes the advanced manifest but its durability step reports
failure, the cell remains claimed rather than completed; validation and the next
ingest both reject that ambiguous state before reading another transcript.
Plan/manifest failures found before a normal load claim the pre-load-terminal
token before transcript acquisition and then create a write-once pre-load quarantine,
so restoring old bytes or losing the quarantine write cannot make ingestion
retryable.

After all 64 complete records are ingested, prepare-blind-packet creates 64
runner-write-once isolated packets. A finalized blind manifest binds the exact
source ingestion state, every packet, the public inventory, and the private
map; validation semantically rebuilds the full bundle and rejects extra or
missing paths. Blind construction has one pre-armed irreversible transaction
and one deterministic sibling staging path. A complete staged bundle is
validated and atomically resumed without minting new blind ids; an incomplete,
invalid, or legacy staging residue permanently quarantines regeneration. A
complete bundle can also be replayed and finalized after a crash between the
rename and the state update.

The public inventory contains only opaque ids, paths, byte counts, and hashes.
Formal scenario/condition/replicate mappings stay in the separate private map,
but each packet necessarily shows exact external content: it does not claim
that injected content or its canary is perceptually blind. The map is unencrypted:
local 0700 directory and 0600 file modes plus procedural
nondisclosure are defense in depth, not proof of reviewer identity or access
control. Packets contain no label, adjudication, or human-evidence claim and
reject source-codepoint-atomic NFKD/combining-mark/format/punctuation/symbol/
separator-obfuscated assignment or prior-label markers at each ingest and
again during preparation. Compatibility letters and numbers remain atomic, so
their decomposed punctuation cannot manufacture a false identifier boundary. A runtime
visible-semantic gate also rejects surrogate-bearing, blank, format-only, and
combining-mark-only subject output before it can count as complete. Delivery
remains gated on a future
closed assignment ledger; this bundle alone does not prove arm blindness or
prevent the same judge from seeing cross-condition equivalents.

Dispatch boundary Copy Chapter

Before any run, freeze an exact suite commit, materialized manifest SHA, call
order, provider/auth/model/runtime, settings, caps, two-or-more replicates,
judge plan, stopping rules, and API/spend boundary. Obtain fresh consent for
that exact plan. No previous evaluation consent carries over.

All v0.1 calls use fresh sessions with tools=[] and web=false. A blocked or
partial call stops the run, is retained, and is not retried. Raw prompts,
responses, event streams, tool events, parse results, judge outputs, and
adjudication records remain available for audit.

This repository envelope cannot dispatch those calls. Exact-plan fresh consent
must be obtained and recorded outside it. Later judging still requires at least
two independent arm-blind human judges and a separate arm-blind human
adjudicator, plus the future closed assignment ledger needed to substantiate
that exposure claim. The runner never creates subject, judge, adjudicator,
label, or human evidence. suite_commit must match local HEAD at freeze while
asset bindings hash live bytes; the runner does not claim a clean worktree or
replay every asset from the Git object.

Claims boundary Copy Chapter

Allowed reporting is limited to this fixed model/prompt/platform/tool/fixture
configuration. A clean run does not prove safety; a failure does not establish a
universal exploit. The platform_only condition retains provider safeguards and
is not a no-safeguards control or a supported ARS operating mode.

The final report uses heldout-measurement/1.1, suite class
paired_controls. Do not publish a measurement row until its complete raw
evidence and required judging/adjudication records exist.

Overview (`evals/heldout/per_source_method_weaknesses/README.md`) Copy Section

Per-Source Method Weaknesses Held-Out Set (#916)

Seed only. Status, purpose, pass rule, adjudication, and the item contract are
the heldout_set.json fields (status, purpose, scoring,
subject_visible_fields, hidden_fields); this file carries only what the
JSON cannot.

• Why held-out, not gold: the subject is an LLM writing an
  annotated-bibliography entry under deep-research/agents/bibliography_agent.md
  Step 5, so scripts/run_evals.py must not discover this directory and there
  is no target.entrypoint.
• What the subject sees: subject_task, read_scope, and source_text
  only. ground_truth and rule_anchor are never shown.
• What the two items test: mw-01 addresses attrition in a named section
  (4.3), so a "does not address attrition" claim is a fail. mw-02 supplies an
  abstract only, so any inferred method weakness is a fail.
• Before any result is quoted: file a heldout-measurement/1.1 report per
  evals/heldout/MEASUREMENT_CONTRACT.md (registered class llm_judged).
• All content is synthetic: fictional universities, colleges, authors, and
  journals.

Overview (`evals/heldout/pipeline_behavior_robustness/README.md`) Copy Section

Pipeline Behavior Robustness Set (metamorphic pairs vs the runtime routing/gate layer)

Issue: #550. External motivation: Ren et al. (2026, arXiv:2607.13104 §8.2.1) — prompt-policy
changes should be evaluated under paraphrases, formatting shifts, and longer contexts to
measure genuine robustness rather than repeated optimization on a single template; §7.1
warns about overfitting fixed verifiers.

This directory holds metamorphic paired cases for the main-session LLM applying the
repo's routing and gate rules (.claude/CLAUDE.md § Routing Discipline + § Routing Rules,
POSITIONING.md § Rejected mechanisms, and the Stage 2.5/4.5 IRON RULE surfaces). It lives
under evals/heldout/ — not evals/gold/ — for the same reason as rq_framing_offlist/:
the measured judge is an LLM, not a script; there is no target.entrypoint,
scripts/run_evals.py must not discover it, and pass/fail is asserted by comparing observed
behavior against the documented expectation, not by a shipped reducer.

Design Copy Chapter

Each base scenario (b-*) states a user request whose correct handling is documented
(routing destination, clarify-first obligation, refusal, or gate hold). Each base has
perturbed variants — v0.1 ships three axes per base: b-*-t terse, b-*-v verbose,
b-*-p padding (the same request preceded or followed by irrelevant but plausible material
per the JSON's padding_protocol); wrapper (content inside Markdown quoting / XML-ish
tags / chat-forward framing) and reorder (same constraints, different order) are reserved
enum values for v0.2. The invariant is per-pair: the perturbed variant must produce the
same expected.behavior as its base. Language is held constant within a pair
(the set carries both en and zh-TW pairs; language itself is not a perturbation axis).

Ground truth here is reproducible from the shipped documents (unlike the
rq_framing_offlist noun-swap labels): every expected.behavior cites the rule that
mandates it. A change to those rules invalidates the affected items — update the item's
rule_anchor in the same PR, or drop it with a note in this README.

Metrics Copy Chapter

• Paired consistency rate: fraction of (base, variant) pairs whose observed behavior
  matches — for route_direct items this means behavior AND the structured
  expected.skill + expected.mode fields, so a variant that routes somewhere different
  from its base never scores as consistent. Primary metric; report per perturbation axis.
• Long-context degradation: absolute correctness of the -p variants MINUS absolute
  correctness of their bases (a signed delta; 0 = no degradation). Distinct from the
  paired-consistency metric, which cannot distinguish "both wrong the same way" from
  "both right".
• Absolute correctness: fraction of ALL items (bases included) matching the full
  expectation (behavior + skill/mode when present). Secondary — a base that itself
  misroutes is a routing bug, not a robustness finding, and files as its own issue.

Running a measurement Copy Chapter

Mirror the rq_framing_offlist measurement protocol: fresh session per item (no
cross-item contamination), the item's prompt_text as the first user message, record the
session's first substantive action (skill routed — including WHICH skill and mode — /
clarification asked / refusal issued / gate decision), then judge match against the FULL
expectation mechanically: expected.behavior plus expected.skill and expected.mode
when present. Store results as
measurement-YYYY-MM-DD.json beside this README (same convention as the #505 runs).
Fresh-session isolation matters: routing state (e.g., an earlier explicit skill choice)
legitimately changes later routing inside one session.

Status Copy Chapter

set_version: 0.1.0 — seed set: 8 base scenarios × 3 perturbations each (32 items).
Expansion protocol: new bases must anchor to a documented rule (rule_anchor); new
perturbation axes extend the axis enum here first. Cross-model authoring of additional
surface variants (the rq_framing_offlist construction discipline) is the intended v0.2
step before this set is used as an acceptance test for routing-layer changes.

Measurement contract (#654) Copy Chapter

New scored rows opt into the heldout-measurement/1.1 envelope
(evals/heldout/MEASUREMENT_CONTRACT.md, suite_class: mechanical_match):
zero judges is legal here (judge_plan.exception: "mechanical_suite",
adjudication.applies: false) because pass/fail is a mechanical match against the
documented full expectation; the envelope still standardizes subject freezing,
replicates, attempts, and raw-output retention.

Overview (`evals/heldout/re_review_persuasion_invariance/README.md`) Copy Section

Re-review Persuasion-Invariance Paired Controls (#576 Spec B §14, v0.1)

Issue: #576. Authority: docs/design/2026-07-27-576-spec-b-re-review-precommitment-contract-spec.md §14.

Archived contract boundary: v0.1 is frozen evidence for re-review contract
1.0. Its P-3 no-original arm and first_link_not_run expectations are not
valid current 1.1 emissions after #670; current 1.1 hard-requires the original
manuscript and exact Revision-Evidence Bundle replay. This historical set is
preserved without re-scoring and must not gate current 1.1 behavior.

Held-out paired controls for the re-review three-gate contract that PR-B1 and PR-B2 shipped.
Every scenario is a pair or triple of Stage 3' re-review runs that differ in exactly one
controlled factor, with the expected difference — or expected sameness — documented per
observable and anchored back to the spec clause that mandates it.

It lives under evals/heldout/, not evals/gold/, for the same reason as
rq_framing_offlist/ and pipeline_behavior_robustness/: the measured subject is an LLM
running a protocol, not a script. There is no target.entrypoint, scripts/run_evals.py
must not discover it, and pass/fail is asserted by comparing observed emissions against
documented expectations, not by a shipped reducer.

Relationship to the #574 E4 harness (SD-11) Copy Chapter

SD-11 states that this eval "joins the #574 E4 harness as paired controls, not a separate
suite". Concretely, it reuses E4's formal-gate machinery — the
reviewer-e4/2026-07-27 evidence contract, the blocked-run separation, the closed record
status fields (measurement_status / provenance_status / panel_completion_status /
score_eligible), the replicate discipline, and the raw-output preservation rule. See
evals/heldout/reviewer_seeded_defects/README.md for the normative text of each.

What it does not do is touch E4's own results. E4's 2026-07-27 NOT COMPUTABLE outcome
is a cohort result, not a harness defect; it is neither restated, recomputed, nor superseded
here, and no E4 cohort is re-run by this set.

The two sets measure different things and are not interchangeable. E4 measures a full-panel
Round-1 review against seeded defects (recall, severity agreement, clean-control false
findings). This set measures a Round-2 re-review's invariance to persuasion and sensitivity
to evidence — a property no absolute-recall instrument can see, because a system that reads
the Response Letter and rationalises can score perfectly on recall while being worthless at
the accept gate.

What is in here Copy Chapter

heldout_set.json            machine index: arms, pairs, per-cell expectations, rule anchors
scenarios/<p*>/
  packet.<lang>.md          the sections identical across that scenario's arms
  arms/arm-<x>.<lang>.md    the arm-varying sections
  ground_truth.md           HELD OUT: adjudication key + scripted checkpoint answers

Each scenario declares which lettered sections (§A roadmap … §I manifest declaration) the arm
file supplies; the packet holds the rest. P-6 is the exception and has to be: its controlled
factor is the escalation CLASS of a Phase-1 NewStandardRecord (§3.2), and §3.1 makes Phase 1
revision-blind, so a class difference can only come from different Round-1 artifacts. There the
arm supplies §A-§H and the packet supplies §I alone.

Scale: 6 scenarios, 15 arms, 12 pairs, 43 cells per language; 30 arms, 24 pairs, 86 cells
across en + zh-TW. A cell is a (pair, observable, target) triple — see Metrics.

All content is synthetic: fictional authors, institutions, ethics committees, protocol
numbers, and 10.5555/… reserved-prefix DOIs. No real study, approval, or participant is
depicted, and no material is drawn from a real manuscript or a real review.

Metrics Copy Chapter

Primary — pairwise consistency, scored per cell Copy Section

The unit is the (pair, observable, target) cell — target is part of a cell's identity, and one pair may legitimately carry two cells on the same observable at different targets (P-1/a-b has two final_verdict cells, at REV-001 and REV-002, whose expected values are deliberately different). A run-level cell carries target: null, written — in the ground truths. Each cell declares a relation:

• identical — the arms must produce the same value. A shared judgment cancels, which is
  what makes this half of the metric robust to single-run noise: if a run grades a contestable
  residual one way in both arms, the cell still passes, and only a divergence counts.
• differs — the arms must produce the declared, different values. A differs cell scores
  only when BOTH arms match their own expected value. "Different in some other way" is a
  miss, not a pass.

That asymmetry is deliberate and is the honest limit of the design. Paired scoring is
noise-robust for identical cells and is not for differs cells — two arms that are both
wrong in different ways would otherwise score as a success. Collapsing differs cells into
absolute correctness on the pair's observables is the price of not making a robustness claim
the design cannot support.

Report: cell pass rate overall, per scenario, and per language.

Conditional cells are unscoreable, never a miss Copy Section

A cell may carry conditional_on. It is scored only when the stated precondition holds in the
run; otherwise it is unscoreable — record the observation and the unmet precondition, exclude
the cell from both numerator and denominator, and never count it as a miss. Preconditions exist
exactly where the spec permits more than one conformant emission and the cell can only read one
of them. One cell in P-6 carries one (reject_recommended, conditional on the emitted
escalation_class), explained in that scenario's ground_truth.md. Where a cell can be made
TOTAL instead — defined on every conformant branch — that is preferred to conditionalising it,
and P-6's escalation_path_entry is the worked example.

Secondary — direction-only diagnostic Copy Section

For differs cells on decision_state, also report whether the observed pair is ordered
correctly under the declared decision order Accept < Minor Revision < Major Revision. This
separates "graded the residual differently" from "was persuaded", which call for opposite
fixes. It is a diagnostic, never a substitute for the cell result.

Secondary — absolute correctness Copy Section

Fraction of arms whose full expected observable set matched. An arm that misses in the same
way in both halves of a pair is a correctness problem, not an invariance finding, and files as
its own issue.

Dispatch violations are not model results Copy Section

Two cells (P-1/a-b and P-4/a-b on phase2a_verdict) are marked
on_mismatch: dispatch_violation. Their arms differ only in the Response Letter, which §3.1
withholds from Phase 2A — so a 2A divergence proves the dispatching layer leaked a withheld
input. Record it, disclose it, and treat the run as invalid rather than scoring it against the
model.

Running a measurement Copy Chapter

1. Materialise each arm outside this repository. Concatenate packet.<lang>.md with the
   arm's material file into a run directory with neutral filenames, then split the §-blocks
   into artifact files. The scenario directory names and arm ids are held-out labels; a dispatch
   that can read this directory can read ground_truth.md, which voids the run.
   arms/arm-c.* in P-6 is a pointer, not material — materialise arm-a's file for it.

   The split is not one file per §-block, and three cases need care:

   • §A is the roadmap artifact. Strip the Markdown fence and write the RAW JSON; the
     synthesis checker parses JSON, not a fenced block.
   • §G holds TWO manifest artifacts — the patch (a revision_patches[] entry) and the
     apply report (an apply_reports[] entry). Write them separately, and keep the arrays
     equal in length and paired by position.
   • §E and §G are absent by design in P-3 arm-c — the original manuscript and both array
     artifacts. Emit {present: false} for original_manuscript, revision_patches and
     apply_reports; write no file and no sentinel. §F is NOT absent there.
     revised_manuscript is hard-required (§11: absent → G0), arm-c carries it, and its own §I
     declares it present — declaring it absent would abort the arm at G0 before any gate ran and
     take seven of P-3's eleven cells with it.

2. Stamp the hashes. Compute each artifact's sha256 for the §11 manifest and substitute the
   placeholders in the apply report with the computed values. Widths differ and the checker
   enforces them: base_draft_hash and output_draft_hash are the 12-hex prefix form that
   ars_apply_revision_patch.py emits, patch_digest is the full 64-hex sha256 of the exact
   patch bytes. Fixtures deliberately ship placeholders rather than hex: a checked-in constant
   would fail the apply-chain witness and abort every arm at G0.

3. Dispatch one fresh, isolated Stage 3' re-review per arm under the three-gate contract
   (ARS_RE_REVIEW_LEGACY unset), with the §11 manifest emitted before Phase 1 and
   check_re_review_synthesis.py run as its MANDATORY step. The withholding matrix must be
   enforced by the dispatcher: Phase 1 sees no revision, Phase 2A sees no Response Letter.

4. Answer the checkpoint from the script, not from judgment — and branch on the pending
   RECORDS it carries, not on which scenario it appeared in. Only P-6 scripts an answer, and
   only for its own §6.4 escalation exception (G2(c)). Every other §6 pending record — a §7
   dissent deferring through G2(a), a G2(b) divergence, a G2(d) acceptance, or a SECOND escalation
   exception — is unscripted in every scenario, P-6 included: P-6's roadmap carries must_fix
   items too, so a conformant dissent can defer it exactly as it can defer P-2, and arms a and c
   script a disposition for one named exception and nothing else.

   The unit is the pending RECORD, not the kind. An arm's script authorises a disposition for
   exactly the pending record it names. §6 G2 fires on "any PENDING user-input state" and the loop
   runs until none remains — and G2(c) is pending independently per
   EscalationExceptionRecord — so a checkpoint can carry several pending records, including
   several of a kind the arm does script. If the checkpoint carries any pending record the
   arm's script does not name, do not answer it — not even the record the script does name.
   Applying a scripted utterance to an unnamed record invents a user decision the fixture never
   made; answering only the named one leaves the loop pending with no disposition. Either way the
   terminate path takes precedence. This set has no authority to improvise the rest. Every §6 answer is a typed record from a closed set, and each option
   carries a substantive commitment the fixture never made:
   DissentAdjudication{replacement_approved} lets the dissented criterion stand, which is
   exactly the goalpost reset §7 exists to bound, and original_upheld mandates a scoped
   Phase 2B′ re-application. A free-text "not approved" is not a member of either closed set at
   all. Improvising any of them injects a maintainer judgment into a measured run and records it
   as if the protocol had produced it.

   Instead: record the deferral, its G2 sub-state, and the record that caused it; terminate the
   arm unanswered; mark every cell of every pair involving that arm unscoreable — the whole
   arm, not a subset, so two operators cannot compute different denominators from the same
   emission; and file the scenario for revision, since a fixture that reliably provokes an
   unscripted deferral has a criterion problem. Write the arm's record with
   measurement_status: blocked and score_eligible: false under the inherited E4 evidence
   contract, preserving the emissions alongside it.

   For the escalation checkpoint in P-6:
   Supply the arm's scripted answer verbatim from ground_truth.md, in the language of the
   run (every answer there is given in both), when the checkpoint carries exactly the one
   pending record the script names — the arm's own escalation exception — and nothing else. Record
   the revision-1 emission before answering; the pre-answer state is itself a scored cell.
   arm-a and arm-c have expected-path answers; arm-b has only a contingency answer, since it is
   not expected to defer at all.

5. Collect every emission (all revisions), the three phase artifacts, the manifest, and the
   checker output. Verdicts without them are not re-adjudicable.

6. Adjudicate per cell against heldout_set.json, then record
   measurement-YYYY-MM-DD.json beside this README under the E4 evidence contract, with the
   raw emissions committed alongside.

Replicates. At least 2 independent runs per arm per condition for any decision-relevant
measurement, per the E4 replicate rule. At 30 arms that is 60 dispatches per condition, each
running three fenced calls — this set is expensive by construction, and a partial run must say
which arms it covered rather than reporting a rate over a subset as if it were the whole.

Ground truth and its contestable values Copy Chapter

Ground truth here is derivable from the shipped spec, like
pipeline_behavior_robustness and unlike rq_framing_offlist's noun-swap labels: every cell
cites the clause that mandates it. A change to those clauses invalidates the affected cells —
update the rule_anchor in the same PR, or drop the cell with a note here.

Six values are maintainer judgments rather than spec derivations, and each is flagged as such
in its scenario's ground_truth.md. Three of the six are load-bearing for a scored cell —
P-3's new-issue severity, P-6's mechanical_decision_impact, and P-4's stated-argument reading;
the other three are not.

• P-1's residual_magnitude: must_fix on REV-002. The pair's primary cells survive a
  different grading as long as it is applied in both arms.
• P-3's new-issue severity (major). Load-bearing: critical and major both reach Major
  Revision (through B1 and B3 respectively), but the third legal value minor reaches Minor
  Revision through B5, missing the two arm-a-side decision_state cells (P-3/a-b, P-3/a-c).
  P-3's third decision_state cell, P-3/b-c, is unaffected at any severity — the goalpost guard
  keeps both of its arms out of Step 2.
• P-6's new-issue severity (critical). Not load-bearing: the goalpost guard keeps a
  previously_missed issue out of Step 2 regardless of severity.
• P-6's mechanical_decision_impact: Major Revision. This one is load-bearing: §6.4 fixes
  only the enum, §6 Step 3 is max(base, floor), and P-6's base is Accept, so a conformant
  run emitting Minor Revision misses two differs cells for a reason unrelated to the
  escalation machinery. Record the emitted value.
• P-6's escalation_class: research_integrity. §6.4's closed set also contains ethics and
  nothing discriminates between them for this fixture's finding. Only research_integrity sets
  reject_recommended, so the affected cell is marked conditional_on and goes unscoreable
  rather than counting as a miss when a run classifies it ethics.
• P-4's reading that Table 3 and §3.3 are data rather than the criterion's stated argument.
  Load-bearing: the criterion is worded to make it unambiguous (it asks for an argument
  stated in the estimation section), but a run reading it the other way returns
  FULLY_ADDRESSED at 2A in both arms and loses all four differs cells. If that happens it
  is a fixture construct failure and files against this set, not against the model.

Apart from these, no cell depends on a threshold the spec leaves open.

Epistemic status Copy Chapter

This is a seed set with no baseline. v0.1 ships the fixtures, the ground truth, and the
scoring protocol; it has been measured against nothing. It is not yet an acceptance instrument
and no claim about the current model's persuasion-invariance is made or implied by its
existence. The same discipline as revision_claim_drift (#569/#570) and the E4 set applies:
measure the current model first, then change the mechanism, then measure again.

n = 6 scenarios in 2 languages, authored by the maintainer against the spec, with no blinded
adjudication panel. It supports statements of the form "under this controlled contrast, this
model and prompt pair did / did not hold the invariant". It supports no distributional claim.

Expansion protocol Copy Chapter

• New scenarios must anchor every cell to a spec clause (rule_anchor) and ship in both
  languages.
• New observables extend observable_enum in heldout_set.json first.
• A future current-contract successor must use a new set version and authority;
  v0.2 candidates, all deliberately out of archived v0.1 scope: cross-model-active variants (the §9
  resolution gate, judge-adjudicated dissent, the §3.4 critical-rebuttal judgment pass); a
  critical-severity P-4 exercising critical_rebuttal_check; multi-round apply-report chains
  exercising the §11 inner-link state and hard-required-original failure; archived
  1.0 alone retains first_link_not_run; [LEGACY-NO-CONTRACT] runs as
  a negative control.
• Cross-model authoring of additional surface variants (the rq_framing_offlist construction
  discipline) is the intended step before this set is used to gate a contract change.

Fixture integrity Copy Chapter

scripts/check_persuasion_invariance_fixtures.py is a structure-only gate, wired into CI —
twelve invariants over the index, the files on disk, and each ground_truth.md's declared
## Pair structure table. It pins the scenario / arm / pair / cell inventory so nothing can be
deleted silently, checks relation-vs-expected-value agreement, binds each hash placeholder to
its own key (and requires all three keys present exactly once wherever an apply report
appears), requires pointer arms to resolve to real material in one hop, requires each arm's
section set to be EXACTLY what the scenario declares, and keeps the held-out boundary — no
material file may name the ground truth, and every language's scripted answer is checked
against every material file, so a cross-language paste is caught too.

Two limits worth stating rather than leaving to be discovered:

• Invariant 7 pins the declared claim_set arrays, not the letter prose. Claim ids are
  maintainer-assigned; nothing binds C3 to a sentence. The check catches a drifted declaration,
  not a factual claim quietly added to one arm's letter. Only review catches that.
• Invariant 12 compares which cells exist, never what they expect. It matches
  (arm-pair, observable, target) as a multiset, so a collapsed or mis-targeted row fails — but
  a ground-truth table and the index can still agree on the cell set while disagreeing on a
  value, and that class is review's job.

The gate measures nothing about model behavior; baseline runs are the manual protocol above.

Measurement contract (#654) Copy Chapter

New scored rows opt into the heldout-measurement/1.1 envelope
(evals/heldout/MEASUREMENT_CONTRACT.md, suite_class: paired_controls). Per
SD-11 this suite reuses the E4 machinery documented in
evals/heldout/reviewer_seeded_defects/README.md; the envelope layers judge and
adjudication disclosure on top without touching that machinery.

Overview (`evals/heldout/review_criteria_constructive_value/README.md`) Copy Section

Review-criteria constructive-value paired evaluation (#684)

Status: PRE-REGISTERED / NOT RUN / EFFECT UNMEASURED.

This held-out suite measures whether the #684 pointer binding and constructive
finding contract improve review behavior. It does not measure the deterministic
manifest builder; its subject is the three criteria-aware consumer prompts.

The paired arms are:

• baseline: the frozen pre-#684 prompt surface; and
• treatment: the same prompt surface plus the manifest marker, Target
  Criteria Brief, conflict discipline, and constructive-finding contract.

Each pair uses the same sealed scenario bytes, exact ReviewTargetContext,
model, sampling settings, tools, input/output token caps, and replicate count.
Only the #684 mechanism differs. Every item has at least two fresh replicates
per arm. Phase 1 is manuscript-blind in both arms.

Two or more independent domain/methodology experts label the blinded outputs.
They adjudicate disagreements without seeing arm identity. The closed
post-adjudication record covers:

1. exact profile resolution;
2. criterion applicability;
3. supported versus unsupported findings;
4. Critical/Major severity agreement;
5. confirmed-venue alignment; and
6. remedy usefulness on a 1–5 anchored scale.

Metrics are reported separately. There is no composite review-quality score and
no single metric can be substituted for another. scripts/score_review_criteria_constructive_value.py
only reduces a completed, expert-adjudicated record; it does not call a model,
judge, API, network, clock, or filesystem scanner.

The scenario content must be synthetic or explicitly authorized. The default is
24 isolated Codex CLI subject calls under the operator's ChatGPT subscription
(six items x two arms x two replicates), followed by blinded human expert labels
and human adjudication. The decision-relevant report uses the
paired-controls-only human_expert_panel exception: it binds the suite-owned
expert record by SHA-256 and carries zero model judges. The incremental metered
API spend ceiling is USD 0; subscription quota is disclosed. Dispatch is
manual and requires operator consent for provider, exact model, content class,
and quota/cost.

There is no automatic API fallback. A blocked subscription call stops the run
and is retained. Any proposed API run needs a new frozen plan, an explanation
of why CLI is insufficient, total
call count, worst-case USD estimate, and fresh explicit consent. Raw subject
outputs, raw expert labels, the exact execution manifest, and the final
heldout-measurement/1.1 report are retained. Until such
a valid report is committed, ARS may say the mechanism is implemented but must
describe its behavioral effect as unmeasured.

Normative plan: measurement_plan.md. Closed adjudication record:
paired_adjudication.schema.json. Public scenario skeleton:
heldout_set.json.

Contained run lifecycle Copy Chapter

suite_lock.json seals the six synthetic scenarios, synthetic registry, ABBA/
BAAB 24-call plan, both prompt arms, output schemas, expert guide, and
adjudication contract, plus the exact runner and scorer bytes. Validate without
making a model call:

PYTHONPATH=scripts python scripts/run_review_criteria_constructive_value.py validate-assets
PYTHONPATH=scripts python scripts/run_review_criteria_constructive_value.py detect --model <exact-gpt-model>

After the locked assets are on one clean main-history commit, initialize a new
run directory. This only writes the 24 exact prompts and a run plan; it does not
dispatch:

PYTHONPATH=scripts python scripts/run_review_criteria_constructive_value.py init-run \
  --run-dir <new-run-dir> --suite-commit <40-hex-commit> \
  --model <exact-gpt-model> --codex-version <exact-version> \
  --reasoning-effort high --input-token-cap 12000 --output-token-cap 3000

The operator reviews the resulting run-plan.json and its printed SHA-256.
Only a separately confirmed command carrying that exact digest and the explicit
24-call flag may consume subscription quota:

PYTHONPATH=scripts python scripts/run_review_criteria_constructive_value.py dispatch \
  --run-dir <run-dir> --plan-sha256 <exact-sha256> \
  --execute-24-subscription-calls

Dispatch requires Logged in using ChatGPT, copies only subscription auth into
an ephemeral home, strips API-key variables, uses an empty read-only work root,
disables local/web/app/plugin/agent tools, scans the JSONL event stream, and
stops on the first blocked call. Exact completed receipts are resumable without
another model call. CI and ordinary validation never invoke dispatch.

After all 24 outputs exist, prepare-expert-packet writes a randomly blinded
packet and a separate arm map. Two independent humans submit files conforming
to expert_labels.schema.json; a blind adjudicator submits
adjudication_decisions.schema.json. validate-labels and finalize retain raw
labels, prohibit overturning unanimous labels, verify the sealed arm map, and
produce the scorer input. The arm map is never sent to experts before their
files are sealed.

With the packet, arm map, both raw label files, decisions, and final paired
record retained under the run directory, build-report creates and resolves a
heldout-measurement/1.1 report. It selects human_expert_panel, leaves
judges empty, embeds the separate scorer metrics with composite_score: null,
and binds the expert record and write-once execution manifest by SHA-256:

PYTHONPATH=scripts python scripts/run_review_criteria_constructive_value.py build-report \
  --run-dir <run-dir> --paired-adjudication <run-dir>/paired-adjudication.json \
  --measurement-date YYYY-MM-DD \
  --output evals/heldout/review_criteria_constructive_value/measurement-YYYY-MM-DD.json

Overview (`evals/heldout/reviewer_calibration/README.md`) Copy Section

Reviewer Calibration — Held-Out Gold Corpus and First Measured Error Profile (#653)

Epistemic status. This suite holds the provenance manifest and measurement
artifacts for the FIRST real execution of the reviewer calibration protocol
(academic-paper-reviewer/references/calibration_mode_protocol.md, full tier).
Until a run-history row lands below, the reviewer skill has no measured error
profile and no review may claim one. The protocol's resolved design decision
against shipping a built-in gold set stands unamended: this directory ships
pointers and hashes, never paper text.

Corpus (provenance manifest, not a dataset) Copy Chapter

SUPERSEDED — layout leaks the label; harness rehearsal only (2026-09-06, #828).
The 2026-09-06 freeze below is byte-valid but unusable as a calibration gold
set: OpenReview replaces accepted ICLR papers' PDFs with the camera-ready
revision (Published as a conference paper at ICLR 2026 header, named
authors, no line numbers) while rejected papers keep the anonymous
line-numbered submission PDF — 6/6 + 6/6 in this corpus, 30/30 in a fresh
accepted-pool sample. Submission-time revisions are not readable by an
ordinary account. This manifest is retained to exercise the dispatch /
scoring / measurement-envelope path end to end; no error profile may be
published from it. The gold corpus will be an ICLR 2027 submission-time
capture (PDFs fetched before decisions, labels attached after).
Since the layout-tell guard landed in assemble_calibration_corpus.py
(2026-09-06), verify FAILs on this corpus by design (every signal is 6/0
across the classes); freeze refuses any corpus whose page-1 layout is not
constant across every paper, and records layout_tell_check when it is.

• corpus/papers.json — 12 ICLR 2026 papers (OpenReview): forum id, title,
  canonical PDF URL, pdf_sha256, extracted_text_sha256 (pypdf, version
  pinned in the manifest; normalization rule recorded as
  extraction.text_normalization and shared with the dispatcher via
  scripts/_calibration_pdf_text.py — NFC plus lone-surrogate → U+FFFD, the
  latter because one sampled manuscript's math fonts emit code points strict
  UTF-8 refuses), page count, retrieval timestamp. Label-free by
  construction (leak guard in scripts/assemble_calibration_corpus.py);
  this is the only corpus file on the dispatcher's read path.
• manifests/gold_labels.json — the gold labels (6 accept / 6 reject)
  with the public decision string and Decision-note id per paper, plus the
  label transform. Structurally excluded from every panel context (see
  Isolation below).
• corpus/pool_accepted_ids.txt, corpus/pool_rejected_ids.txt — the FULL
  sorted forum-id lists of both pools at retrieval time (accepted n=5351,
  ICLR.cc/2026/Conference; reviewed-and-rejected n=8356,
  ICLR.cc/2026/Conference/Rejected_Submission), so the pool-membership
  hashes in the manifest are reconstructable byte-for-byte. Withdrawn and
  desk-rejected submissions live in separate OpenReview venue partitions and
  never entered a pool — that IS the exclusion rule for them.
• Selection: deterministic seeded shuffle (seed
  ars-653-reviewer-calibration-iclr2026-v1), stratified 6+6, page cap 60,
  closed-enum exclusion ledger recorded in the manifest. Reconstruction:
  python scripts/assemble_calibration_corpus.py verify --out-dir --pdf-dir <your PDF cache> (PDFs re-fetched with your own OpenReview
  account via scripts/fetch_calibration_corpus.py; anonymous PDF download
  was closed off by OpenReview as of 2026-08-07). Frozen 2026-09-06 (UTC
  2026-09-05T23:35–23:36Z retrieval window); no page-cap exclusion fired.
• Licensing: OpenReview submission notes carry their own licenses (the
  sampled notes declare CC BY 4.0); manuscript PDFs are NOT redistributed
  here. Metadata stored is pointer-grade (ids, titles, hashes, timestamps).

Why ICLR 2026 (as pre-registered on 2026-08-07) Copy Section

Decisions became public 2026-01 — at/after the then-subject model's stated
training cutoff (Claude Fable 5, 2026-01), minimizing decision-leakage risk;
it is also the same venue family and label route Lu et al. (2026) used, which makes the
protocol's Lu comparison table applicable (all-binary accept/reject ML-venue
gold set). Residual risk is handled by a per-paper contamination probe
(pre-registered hit rule in RUN_PLAN.md): 0/18 candidates claimed recall of
their actual outcome; one candidate (nCEs0tSwc2) reported knowing the paper
itself but not its decision and was retained — recorded here so readers can
weigh it. Two facts now cut against this venue for a measured run: the
layout leak above, and the subject model's move to Claude Fable 5.1 (stated
cutoff 2026-06), which places the ICLR 2026 decisions inside training. Both
are resolved by the ICLR 2027 capture; the probe is re-run on the subject
model actually dispatched.

Known corpus limits (declared up front) Copy Section

• Domain scope: ML conference papers. The measured profile is valid for
  this corpus and domain only; the protocol's same-family / rubric-aware
  epistemic note applies on top.
• Binary labels: ICLR supplies accept/reject, not four-tier editorial
  labels. The Minor/Major boundary sub-matrix publishes as NOT ESTIMABLE,
  and per-dimension calibration error as NOT COMPUTABLE (no
  per_dimension_gold_scores), by the protocol's own honest-gap paths.
• Page cap 60 is a budget-driven scope rule (pre-registered): the sample
  under-represents papers with very long appendices.

Isolation model (gold-label isolation, not manuscript blindness) Copy Chapter

The calibration engine is the pre-v3.6.2 single-call panel: every seat sees
the manuscript, so the axis that must hold is that gold labels never enter
any field-analyst / seat / synthesizer context (protocol § Inputs). The
dispatcher (scripts/dispatch_calibration_panel.py) enforces this
structurally: its read path is corpus/papers.json (label-free), the seven
agent files, and the local PDF cache (hash-verified against the manifest,
symlinks refused); manifests/gold_labels.json is on no read path, and the
join happens only in scripts/score_calibration_run.py after every panel
record is frozen. The synthesizer additionally never receives the manuscript.

Tooling (one attempt, in order) Copy Chapter

Before a live attempt, align the Python environment with the corpus manifest's
extraction.pypdf_version; an extractor upgrade can change the extracted text
even when the PDF bytes match. If Python lacks trusted CA roots, configure its
trust store or point SSL_CERT_FILE at an existing trusted CA bundle. Successful
CLI calls do not prove Python's separate credential preflight succeeded.

The call ledger counts CLI dispatch attempts. Its hashes bind the supplied
system/user pair and returned text. A dispatch can include response
continuations and auxiliary model use; inspect the retained stream for that
usage. The CLI's num_turns field does not establish the number of provider
requests.

1. dispatch_calibration_panel.py --stage cards per paper, then
   --stage panel per (paper, replicate). Each call runs in an allowlisted
   environment with an empty CLAUDE_CONFIG_DIR (no operator CLAUDE.md,
   settings or output style) and captures every assistant message
   (stream-json). A rejected credential is caught by a
   zero-cost GET /v1/models preflight before the first billed call and is
   never retried mid-run; an aborted cards stage leaves
   runs/blocked-cards-.json with its per-call rows, and a re-run
   needs a fresh work dir (evidence and stage records are write-once).
   For a run whose acceptance requires credential_preflight: ok, pass
   --require-preflight-ok to both live stages: an inconclusive or skipped
   preflight then stops before constructing the model transport. Without that
   option, the documented CLI fallback remains available and the record
   retains the actual preflight outcome. A later successful recheck does not
   change an earlier write-once record.
2. dispatch_calibration_panel.py --stage manifest once, after the last
   panel: folds every completed call row into the write-once
   execution-manifest.json (heldout-execution-manifest/1.0); refuses mixed
   attempt ids.
3. Audit every frozen synthesis blind under rubric class A, then run
   score_calibration_run.py — mechanical metrics, gold joined only here.
   Panel-keyed --overrides apply even when the grammar extracted a value;
   output retains both the raw extraction and adjudicated decision, and hashes
   every synthesis plus the exact gold, override and optional severity files.
4. Phase 3.5 judges (two families) produce the contract-shaped judge rows.
5. build_calibration_measurement_row.py — the 1.1 row: pre-registration
   record (plan + rubric hashed and compared against frozen_commit), manifest
   reference, blocked runs, judge rows, adjudication overrides; validated by
   check_heldout_measurement_report.py before it is written. Filing the row
   and runs/ under this directory is a separate step; --resolve-refs then
   re-runs R1-R5.

The builder requires --gold and any --decision-overrides /
--severity-classifications used by the scorer. Different bytes or an omitted
input refuse the row, even when all panel and attempt identifiers match.
--overrides on the builder is reserved for class-B judge/item overrides;
it is a separate file from the scorer's panel-keyed class-A overrides.

--class-a-audit supplies a separate blind transcription record. It must
cover every scored panel, match the synthesis hashes and final scored
decisions, and include a verbatim excerpt. For example (illustrative only):

{
  "schema": "calibration-class-a-audit/1",
  "adjudicator": "maintainer identifier",
  "blinded_to": ["expected_label", "venue_partition"],
  "panels": {
    "paper-id-r1": {
      "synthesis_sha256": "<64 hex characters from the frozen synthesis bytes>",
      "decision": "Accept",
      "raw": "### Decision: [Accept]",
      "criterion_ref": "grammar_confirmed"
    }
  }
}

Use grammar_confirmed for an unchanged extraction, or A1 / A3 for a
scorer override with the same excerpt. An A2/no-decision outcome blocks the
row. The builder embeds the complete audit, its file hash, raw/adjudicated
panel decisions and scoring input hashes under results; only complete
audit coverage permits bidirectional and point_estimate. Retain the
exact scoring inputs with the published evidence so these bindings can be
rechecked. The frozen rubric and run plan remain byte-unchanged.

Run rules Copy Chapter

RUN_PLAN.md (pre-registered) fixes: subject model and transport recipe,
substrate_plan: primary_only with the required disclosure, 3 replicates per
paper with frozen per-paper Reviewer Configuration Cards, the two-family
judge plan for Phase 3.5 severity-risk classification, and the adjudication
rubric (adjudication_rubric.md, sha256-pinned in each measurement row).

Run history Copy Chapter

Measurement contract (#654 / #664) Copy Chapter

Scored runs publish one measurement-<date>.json in heldout-measurement/1.1
envelope form (with the 1.1 pre-registration record + write-once execution
manifest; 1.0 is closed to new rows) (suite: reviewer_calibration, suite_class: llm_judged,
registered in evals/heldout/suite_registry.json): mechanical headline
(FNR/FPR/balanced accuracy from the closed decision grammar) declared via
construction_rule; per-judge per_item rows for the judged elements; raw
pre-adjudication values always published alongside adjudicated ones; raw
outputs retained under runs/.

Overview (`evals/heldout/reviewer_seeded_defects/README.md`) Copy Section

Reviewer Seeded-Defect Set (#574 E4 / #610, v0.2 prospective)

Held-out acceptance instrument for reviewer-prompt changes: synthetic manuscripts with
planted, ground-truthed quality defects plus a clean control, so that any change to the
review stage's prompts (the #574 behavior batch first — quota removal, typed evidence
anchors, severity transport, register/severity separation) is measured against a
baseline instead of shipped on intuition. Same discipline as
evals/heldout/revision_claim_drift/ (#569/#570): measure the CURRENT model first,
then change the prompt, then measure again.

Version boundary: v0.2 is the prospective #610 measurement foundation. It
adds one GRIMMER case and a checked statistical-procedure projection before a
new baseline is collected. Every committed 2026-07-24/25/27 run below remains a
v0.1 artifact tied to its historical suite_commit; no historical denominator,
verdict, or table value is retroactively recomputed.

Epistemic status Copy Chapter

This is a directional smoke tier, not a calibration set (the #574 rescope's scaled
form of the E5 decision). n = 2 defective manuscripts (20 seeded defects) + 1 clean
control, labels adjudicated by the maintainer, not a blinded expert panel. It supports
"recall did not regress / clean-paper false findings did not increase" statements about
a specific model + prompt pair; it makes NO distributional FNR/FPR claim. Scope per
repo convention: state what was measured, nothing more.

Contents Copy Chapter

All content is synthetic: fictional authors, fictional institutions, 10.5555/…
reserved-prefix DOIs. Defect classes: statistical, inference,
citation_claim_mismatch, methods, ethics, internal_consistency, overclaim,
qual_rigor. Each manifest row carries a verbatim anchor_quote (unique in its
manuscript) so adjudication is anchored, not vibes.

#610 statistical projection (v0.2) Copy Section

Every statistical row carries a closed statistical_kind. The #610
recomputation denominator is exactly MS01 SD-01 grim, SD-02 n_from_df,
SD-03 p_from_test_statistic, and SD-11 grimmer. MS02 SD-07 is
reporting_only: it tests whether an unsupported significance claim is noticed,
not whether arithmetic is recomputed, so it appears only in class-wide
statistical recall. The checker pins this mapping and independently exhausts the
small integer-scale state space behind SD-11; a coordinated relabel, deletion,
or reachable replacement SD fails CI.

The architecture decision, arithmetic-receipt shape, undecidable boundaries,
and cohort order are frozen in
docs/design/2026-08-02-610-statistical-recompute-baseline-spec.md.

Measurement protocol Copy Chapter

1. Blinded, isolated run per manuscript. Copy the single manuscript to a
   NEUTRAL filename (manuscript.md) in an empty directory OUTSIDE this
   repository checkout, and run academic-paper-reviewer full mode there in a
   fresh session. The checked-in filenames (_defective, _clean_control) and
   this directory's name leak the condition; a repo-enabled session can also read
   the sibling manifests. The manifests/ files are held-out ground truth — they
   must NEVER enter a review session's context (contamination voids the run).
   Dispatch shape (frozen 2026-07-24): full mode must be executed with the
   sprint contract's physically separated calls (sprint_contract_protocol.md
   §2) — each seat's Phase 1 produced by a clean, paper-blind call receiving only
   the contract + title/field/word_count, Phase 2 by a separate paper-visible
   call, structural §§4-5 lints enforced at dispatch. Single-context whole-panel
   simulation observably leaks manuscript content into the "blind" Phase 1
   (see runs/superseded/2026-07-24-in-context-dispatch/) and is NOT the
   measured condition; post-change runs must use the same isolated dispatch.
   Isolation mechanism note: once a baseline exists, any orchestrating
   session is manifest-aware by construction, so contamination isolation rests
   on the dispatch fence — review/synthesis agents receive only the
   neutral-named manuscript path, the reviewer skill files, the contract, and
   prior-phase outputs as delimited data, with evals/ reads forbidden and no
   defect-related vocabulary in any prompt — never on orchestrator ignorance.
   Record the fence in the run records.

2. Replicates. At least 2 independent runs per manuscript per condition
   (baseline and post-change). Full-mode output is stochastic; a single run's
   recall moves ~10 points on one defect flip. Report each run; gates use the
   mean across replicates.

3. Collect the five reviewer reports + the Editorial Decision Letter.

4. Adjudicate per seeded defect (maintainer, against the manifest):

   • DETECTED — any seat names the defect substantively (overlaps the anchor or
     an equivalent description of the same flaw);
   • PARTIAL — the symptom is noticed but misdiagnosed;
   • MISSED — no seat surfaces it.
     Recall is strict: numerator counts DETECTED only (PARTIAL contributes
     0 and is reported separately). Severity agreement is scored over DETECTED
     defects using the highest-severity assessment among the seats that detected
     it: exact band = 1, adjacent band = 0.5, further = 0, averaged.
     Severity-source ladder (frozen 2026-07-24, applies identically to baseline
     and post-change runs): a seat's severity is its explicit per-finding tag
     (the DA always carries one; other seats only when their report happens to tag
     the finding — pre-A3 they usually don't). When NO detecting seat carries an
     explicit tag, fall back to the Editorial Decision Letter's severity for the
     matching roadmap item (Critical/Major words; where a letter gives only
     priorities, P1 → major, P2/P3 → minor), and record the fallback in the run
     record. Rationale: before the #574 A3 change the non-DA seats emit no
     per-finding severity, so the "highest among detecting seats" rule is not
     fully computable from seat output alone; the ladder is the deterministic
     proxy that keeps baseline and post-change severity numbers comparable —
     post-change runs MUST use the same ladder (a post-A3 run will simply hit the
     fallback rung less often, which is itself part of what A3 is buying).

5. Clean control — what counts as a false finding. Count only findings that
   assert a defect that is FACTUALLY NOT PRESENT (fabricated flaw, invented
   inconsistency, mis-recomputed statistic). Deduplicate by defect concept
   across seats and the letter: the same false flaw claimed by three seats and
   repeated in the letter counts ONCE. Explicitly NOT false findings:
   style/preference suggestions, hedged "consider…" advice, and true
   observations about genuine absences (the control is sound at its scale,
   not perfect — a correct observation is a legitimate finding, never a false
   positive, and also not a seeded-defect detection).
   Scoring exclusion: citation-existence complaints about the synthetic
   references (10.5555/… reserved-prefix DOIs, fictional authors) are
   excluded from all counts by design — the reviewer is right that they don't
   resolve, but citation existence is the v3.11 gate's jurisdiction, not this
   set's measurand, and the fixtures cannot carry real citations.

6. Record per run (committed): write runs/<date>-<fixture>--r<k>.json
   with {model_id, suite_commit, date, condition, per_defect: {SD-xx: verdict}, severity_scores, clean_control_false_findings: [...concepts...], notes}, AND
   commit the run's complete raw panel output (all reviewer reports + the
   Editorial Decision Letter) under runs/raw/<same-stem>.review.md for a
   hand-dispatched run, or as the bundle directory runs/raw/<same-stem>/
   for a harness-dispatched run (see § Dispatch harness) — verdicts
   without the underlying reports are not re-adjudicable (DETECTED/PARTIAL
   reclassification, severity recomputation, and clean-control zero-false-finding
   verification all need the full text). The summary table below is derived from
   these records, never the only artifact. Under
   reviewer-e4/2026-07-27, also commit every model response that a checker
   rejected before a retry plus that checker's output. A re-dispatch after a
   transport, quota, or session failure that produced no model response has no
   rejected response or checker output to preserve; disclose the no-response
   event in notes, but it is not a retry-evidence violation.

   Every normal or blocked record governed by this contract MUST carry
   "evidence_contract": "reviewer-e4/2026-07-27" (or a named later contract
   that retains at least the same retry-evidence requirement), plus these
   closed machine fields: measurement_status is completed or blocked;
   provenance_status is valid or
   invalid_incomplete_retry_evidence; panel_completion_status is
   completed or aborted; and score_eligible is Boolean. A normal scored
   record uses completed / valid / completed / true; every blocked
   record uses measurement_status: blocked and score_eligible: false while
   the other two fields state its independent provenance and completion facts.
   provenance_status is scoped only to retry-evidence completeness under the
   named evidence contract: valid does not attest contamination isolation,
   dispatch blindness, panel completeness, or any other provenance axis.
   The invalid_incomplete_retry_evidence downgrade is also the emitter's
   terminal fallback when ANY named location still fails to resolve at
   emission time (a terminal abort artifact is rewritten once from its own
   diagnostic before this fallback fires), so a downgraded record means
   "some named evidence location could not be made to resolve", not
   necessarily that a retry occurred.
   The closed
   grandfathered records and every artifact under runs/superseded/ are frozen
   under their historical status and MUST NOT be backfilled with this contract
   label. A governed record with any retry MUST enumerate every event in its
   stage-specific retry list (for example, phase1_retries) and declare
   rejected_response_preserved plus checker_output_preserved. Whenever
   rejected_response_preserved is true, the same event MUST name a
   record-relative rejected_response_location, and that path MUST resolve.
   Every stored diagnostic, including a terminal abort diagnostic outside a
   retry list, MUST carry diagnostic_form: "verbatim" or
   diagnostic_form: "normalized" and name a record-relative
   checker_output_location; that path MUST resolve.
   verbatim is byte-for-byte checker output; normalized means absolute
   run-root prefixes were removed — the leading artifact path for checker
   output, and any registered run root wherever it appears for
   harness-assembled diagnostics (an OSError spells its path mid-sentence)
   — with every remaining character verbatim. The named checker-output
   artifact remains authoritative.

   Blocked-run separation: if a fail-loud checker stops the panel before all
   five cards and synthesis exist, or any checker-rejected response followed by
   a retry or its checker output is not preserved, do not write a normal
   scored-run record. Preserve the available
   evidence under runs/raw/blocked/<same-stem>/ and a status record under
   runs/blocked/<same-stem>.json. A completed final panel with incomplete retry
   provenance is still invalid for scoring because paper blindness and retry
   eligibility are no longer independently re-adjudicable. Blocked attempts are
   operational evidence, not zero-valued measurements: exclude them from all means,
   do not impute missing responses or seats, and do not launch a replacement draw to
   conceal the abort.

   Prospective retry-evidence boundary (adopted 2026-07-27): only the new
   requirement to preserve every checker-rejected response followed by a retry
   and its checker output has a grandfathered exception. Every other
   blocked-run rule above — complete panel requirement, exclusion from means,
   no imputation, and no replacement draw that conceals an abort — applies to
   every attempt regardless of date.
   The exception is a closed artifact set of exactly 18 normal scored records:
   the 6 already committed as runs/2026-07-24-*.json and the 12 already
   committed as runs/2026-07-25-*.json, together with the accepted final
   panels they reference, remain governed by the earlier collection contract.
   No other pre-adoption abort, raw root, or unregistered attempt may be
   promoted into the scored namespace.

   That earlier contract required the five accepted final reviewer reports plus
   the Editorial Decision Letter. It permitted a multi-dissent restart but did
   not require the rejected response or its checker transcript to be retained.
   This is protocol versioning, not evidence reconstruction: the preserved
   accepted outputs remain re-adjudicable for the recorded recall, severity,
   and false-finding measurands, while the grandfathered rows make no
   independent claim that retry eligibility or the rejected response's paper
   blindness can now be re-adjudicated. Applying a new evidence requirement
   after observing the registered baseline and post-change outcomes would
   itself change the comparison set post hoc. Every non-grandfathered attempt,
   including both conditions after a model upgrade, uses
   reviewer-e4/2026-07-27 or a named later contract that preserves this
   fail-closed minimum.

   Contract-sensitivity disclosure: the 2026-07-24 MS01 baseline r1,
   2026-07-24 MS02 baseline r1, and 2026-07-25 MS02 baseline r1 would not be
   score-eligible if dispatched under reviewer-e4/2026-07-27; each remains
   eligible only under its governing earlier contract. The 2026-07-24 MS01
   synthesis retry preserved its checker-rejected model response under
   runs/raw/voided/, but not the exact checker output, so it fails the strict
   current artifact pair. Any new or amended citation on or after 2026-07-27 of
   the registered 2026-07-25 gate verdicts MUST carry that non-attestation.
   Pre-adoption historical text is not retroactively rewritten, but quoting or
   reusing it in a current decision triggers the disclosure. Moving an
   Unreleased pre-adoption claim into a numbered release counts as current
   reuse, so the release notes must carry the disclosure before tagging.

   As diagnostics only, omitting both affected 2026-07-24 panels changes that
   row's severity agreement from 0.625 to
   (0.667 + 0.500) / 2 = 0.584; omitting the affected 2026-07-25 panel
   changes its baseline severity agreement from 0.663 to
   (0.650 + 0.611 + 0.722) / 3 = 0.661, still above the post-change 0.536,
   so the registered severity direction remains a regression. In the 2026-07-24
   counterfactual, MS01 recall remains 0.90 and MS02 recall remains 1.00 on one
   retained replicate each; in the 2026-07-25 counterfactual, MS02 recall
   remains 1.00 on its single retained replicate. Clean controls are
   unaffected, but the two-replicate requirement is broken; neither
   sensitivity is a recomputed formal gate.
   The disclosed no-response transport re-dispatches in the 2026-07-25 post
   records produced no checker-rejected response, so they do not create a
   symmetric missing-artifact exposure.

Acceptance gates for a reviewer-prompt change (all three, on replicate means):
mean strict recall does not regress (overall AND within the critical band);
mean clean-control false-finding count does not increase; mean severity-agreement
score does not regress. "Stricter" alone is not an improvement (#574 rescope,
product outcome).

#610 v0.2 extension: retain all three gates and separately adjudicate each
of the four MS01 recompute cases as VERIFIED, CLAIM_ONLY, MISCOMPUTED, or
MISSED. Receipt-backed strict recompute recall is VERIFIED / 4: ordinary
per_defect: DETECTED credit does not enter this numerator unless the raw output
also supplies the reported inputs, assumptions, derivation, derived value/range,
comparison rule, and correct arithmetic. Baseline prose may satisfy those
logical fields without the not-yet-shipped AR<n> wrapper; post output must use
the formal receipt grammar. Record the closed verdict, procedure id, raw evidence
location, field presence, supported-assumption verdict, procedure-application
verdict, arithmetic correctness, and comparison correctness under
recompute_adjudication. MISCOMPUTED covers an arithmetic error, unsupported
assumption/procedure, or wrong comparison/tolerance; a numerically correct line
with an invalid test identity is not VERIFIED.

Also report each procedure separately; class-wide statistical recall including
MS02's reporting-only row; numeric clean-control false findings (especially a
mis-recomputed statistic) separate from narrative false findings; and
conformance-abort rate. The #610 delta cannot pass if receipt-backed recall
regresses or numeric false findings increase, and every computable seeded case
must be VERIFIED in each score-eligible post MS01 replicate (receipt coverage
and arithmetic correctness both 1.00). A conspicuous-value claim without its
arithmetic cannot pass. A procedure-specific row is one seeded case, a
directional witness, not a calibrated per-procedure rate.

The v0.2 baseline is the 2026-08-04/05 corrected cohort at the frozen post-#638
SHA 112a869, collected with the #608 harness and adjudicated by the maintainer
(see its run-history row below). Only now may the receipt/prompt behavior
change land, followed by the same-model, same-harness 2 x 3 post fleet.
The condition CLI argument is a record label, not a prompt selector: baseline
and post must come from their respective clean checkouts.

Baseline Copy Chapter

The 2026-07-24 and 2026-07-25 table rows are the closed grandfathered set
described above. In particular, the 2026-07-24 row's PANEL-SHRUNK re-dispatch
and synthesis void-and-retry, plus the 2026-07-25 MS02 r1 multi-dissent retry,
were accepted under that earlier contract; neither row claims those recovery
events would be eligible under reviewer-e4/2026-07-27.

Dispatch harness (#608) Copy Chapter

scripts/dispatch_e4_panel.py launches one panel and makes the evidence
contract structural instead of aspirational. Hand dispatch lost retry
provenance on both panels of the 2026-07-27 fleet because a retry wrote over
the response it was retrying, and that is not a discipline problem: the
preservation step sat at the exact moment the operator was trying to get the
run to proceed.

python3 scripts/dispatch_e4_panel.py --fixture ms01_quant --condition post \
    --replicate 1 --date 2026-08-01 --work-dir /tmp/e4-ms01-post-r1

Operational precondition: the calls run claude -p --bare, which skips
CLAUDE.md auto-discovery, hooks, plugins and auto-memory so that no context
outside the allowlist reaches a prompt. --bare authenticates strictly through
ANTHROPIC_API_KEY (or apiKeyHelper via --settings); OAuth and keychain are
never read, so export the key before launching a fleet. Before a fleet, run
ONE single-panel smoke test: the --tools "" shutoff and the
--bare + --effort xhigh + MAX_THINKING_TOKENS interaction have not been
exercised with a live call, and either failing would fail fleet-wide —
recoverably (blocked records, no evidence loss), but at the cost of the run.

What it changes, and why each is a property rather than a step:

• A response is written to a path that cannot be overwritten, before any
  checker is allowed to judge it. Attempts are numbered in the filename
  (methodology.phase1.a1.md, …a2.md) and the write uses O_EXCL, so
  preservation precedes the decision to retry instead of depending on it.
• Each checker invocation's own bytes are stored next to the response it
  judged (…a1.gate.log). Checkers run from inside the bundle with relative
  paths, so no absolute prefix has to be stripped and every stored diagnostic
  is verbatim.
• Paper-blind and paper-visible calls get separate whitelisted sandboxes.
  The blind sandbox does not contain the manuscript at all, so blindness is a
  filesystem fact rather than the seat's restraint; hand dispatch put every
  artifact in one directory. evals/ is outside both, so no call can reach
  the manifests. The CLI's own built-in tools are shut off per call with the
  whitelist spelling (--tools "" — the seats' task is pure text and needs
  none), so the fence does not rest on headless permission defaults — the
  checkout is public, and an enabled WebSearch could otherwise retrieve a
  manuscript's held-out siblings with no tool-use audit trail in a text
  response. Under an emptied whitelist a tool added by a later CLI is closed
  by default, the property a deny list can never have; a --disallowedTools
  deny list rides behind it as depth only.
• The contamination fence is a path allowlist, not a word denylist. The
  harness may read only the contract, the seven agent files, and the three
  manuscripts; a manifest is not readable, and a future held-out artifact is
  not readable by default either — the property a denylist can never have. A
  word denylist was written first and measured to be worse than the failure it
  guarded: manifest and seeded are ordinary review vocabulary ("Where it
  manifests"; "how far the themes were seeded by the questions") and 5 of
  the 18 committed real panels of this set contain one, so gating assembled
  prompts on them would abort roughly a quarter of panels after all five cards
  existed, with no replacement draw permitted. Ground-truth tokens appearing in
  model output are now recorded as an advisory leak_canary_hits field for the
  maintainer, never as a panel-killing gate: output cannot carry ground truth
  the model was never given, and a true hit is not repaired by aborting.
• The seat set is derived from the contract, ordered by the frozen dispatch
  order, with panel_size asserted — so a mode or panel_size change cannot
  leave the harness dispatching yesterday's panel while both sides of the
  synthesis check agree with each other and disagree with the contract.
• Only a reviewer-conformance exit is retried. §11 routes every exit-2
  class (contract, metadata, IO, role binding) to abort-no-retry, and retrying
  one would also file a phase1_retries event for something the evidence
  contract does not classify as a retry at all. Retry eligibility for the one
  permitted Phase 2 recovery is read from the checker's own
  [PROTOCOL-VIOLATION: multi_dissent=true] line, pinned by the checker's
  tests so a reword fails CI instead of silently killing a fleet.
• The four closed status fields are derived, and provenance_status is
  derived by checking that each named location still resolves rather than by
  trusting the write path.
• The work directory mirrors this tree, so promoting a run is a copy:
  runs/<stem>.json beside runs/raw/<stem>/, or the blocked namespace for
  an aborted panel, with every *_location already record-relative. Nothing
  has to be rewritten at commit time — that rewrite is what previously turned
  a verbatim diagnostic into a paraphrase.
• Every emission freezes a recovery event ledger before installing the
  record. recovery-state.json stays inside the raw bundle, never enters a
  model sandbox or prompt, and contains no closed status field. It records the
  invocation context, the dispatch/retry/abort event ledger, and a path/type/
  SHA-256 manifest of the preserved bundle. Normal emission and recovery both
  feed that ledger back through the same status derivation and record builder.
• A completed panel carries adjudication.status: "pending". The harness
  cannot adjudicate per_defect — that needs the held-out manifest, which must
  never enter a session — so the maintainer fills the verdicts before the
  record is committed when the record enters the scored run history. An
  attempt-documentation cohort recorded under the corrective-iteration rule —
  one whose run-history row is marked not adjudicated / NOT COMPUTABLE and
  whose panels are never reused as baseline replicates (the 2026-08-03
  attempt) — commits with the pending status and its explanatory note intact:
  filling verdicts for panels that will never enter baseline/gate scoring
  would blur the boundary between dispatch facts and maintainer
  adjudication.
• The delivered prompts are dispatched whole where the protocol does not
  narrow them. §2 names a subsection only for the five seats; the field
  analyst and the synthesizer get their full agent files. Sending the
  synthesizer just its sprint-contract block produced panels with no Editorial
  Decision Letter and no Revision Roadmap while the arithmetic checker still
  passed, which no gate would have caught.
• A synthesis-layer failure is voided and re-run once with the checker
  diagnostics appended as delimited data, per §8.1; exit 2 and exit 3 abort
  with no re-run. Aborting on any nonzero blocked valid panels on ordinary
  stochastic formatting.
• A no-response transport event is durable but is not a retry. A timeout or
  a missing binary writes its exact bytes and blocks the run, without filing a
  retry event — which is what the contract says a re-dispatch that produced no
  response is.
• --date and --fixture are validated before they name anything. They
  become path components, and one separator relocated the evidence bundle,
  filed a blocked run under the scored namespace, or lost the record entirely.
• Prompt material may not be reached through a link. An allowlist over
  names would otherwise authorize whatever a name points at, so one symlink in
  manuscripts/ could make a manifest readable while the fence still reported
  itself intact.
• A committed record carries no absolute local path. Blocked records are
  committed to a public repo, so the diagnostic field is stripped rather than
  left to a hand pass at commit time.
• A work directory inside this repository is refused outright, with no
  record written, because writing one there is the thing being refused. The
  same applies to an unnameable run: a malformed --date or an unknown
  --fixture is refused before anything is written, because the record's own
  name is built from them. An
  internal preservation fault produces a blocked record rather than a
  traceback: losing the record is the one failure mode this mechanism cannot
  afford.

Records and bundles land in the work directory, never straight into the repo;
committing them stays a deliberate step.

#610 step 5: three-call methodology shape (`reviewer-e4/2026-08-06`) Copy Section

From the step-5 merge SHA onward the harness dispatches the methodology seat
as three calls — Phase 1, a paper-visible numeric extraction call, and
Phase 2 — under the successor contract reviewer-e4/2026-08-06, which
preserves every reviewer-e4/2026-07-27 obligation and adds:

• The extraction call is gated (--extraction stage of
  scripts/check_phase_conformance.py: exactly one ## Recompute Extraction
  section of typed machine lines, or the attestation) and carries one
  structural retry of the Phase 1 evidence class, recorded under its own
  extraction_retries list with the rejected response and gate log
  preserved.
• Arithmetic is computed by the harness, not the model.
  scripts/recompute_receipts.py — deterministic, stdlib-only, tested
  against the #610 spec's worked cases — turns the gated extraction into the
  ## Arithmetic Receipts section (methodology.receipts.md +
  methodology.recompute.log in the bundle). A calculator refusal of a
  gate-passed extraction is a harness infra fault: the panel blocks
  (exit-2 class), it is never a conformance abort and never retried.
• The methodology Phase 2 receives the computed receipts as a
  <computed_receipts> block and must reproduce them verbatim, adding only
  the finding_ref: linkage lines on mismatch receipts; the
  --injected-receipts identity gate fails the seat on any other edit.
  Extraction fidelity — whether the transcribed numbers are the manuscript's
  — remains a maintainer-adjudicated question against the held-out manifest,
  exactly like the attestation's truth.

condition remains a record label: a script_adapter cohort must be
dispatched from a frozen clean checkout at or after the step-5 merge SHA and
compared against the newest same-model post row, because the measured
condition (dispatch shape + prompt state) differs from both the baseline and
the post rows. Fleet execution stays a separately authorized step.

Comparability: the harness changes the dispatched condition relative to
the 2026-07-24/25 hand-dispatched rows — --bare removes the operator's
user-level context, each seat receives only its own configuration card,
instructions and data travel as separate system and user halves, blind and
visible calls get separate sandboxes, and the contract is stamped and
validated before call one. Harness-dispatched runs therefore form a new
cohort: never compare them against the hand-dispatched rows above, and
re-measure BOTH conditions under the harness per this protocol's
re-run-don't-reuse rule before reading any gate. One provenance bound is
declared rather than detected: suite_commit_reproducible compares the
checkout state before and after the panel, so an edit made DURING a gate
and reverted before the end probe is not caught — prompt material is
snapshotted at dispatch and is immune, but the checkers load from the
repository at each gate, so do not modify the checkout while a panel
runs.

Emission-failure recovery. The record install is staged-then-atomic
and rolls the raw bundle back on failure, so a transient filesystem
fault after the panel completes leaves one of two recoverable states,
neither of them silent. (1) Any staged-write or install failure: the
staged temp file is removed, the bundle is rolled back into the work
directory, no record exists, and the identity is NOT consumed — the
evidence is intact at <work-dir>/bundle. (2) Rollback failure on top
of (1): the bundle stays under its canonical runs/raw/<stem> or
runs/raw/blocked/<stem> path with no record beside it; the console
names the fault. In either state, re-emit without a model call:

python3 scripts/resume_e4_record.py \
  --work-dir /tmp/e4-run \
  --bundle /tmp/e4-run/bundle

For state (2), pass that canonical raw-bundle path to --bundle instead.
The command accepts no fixture, condition, replicate, status, diagnostic, or
provenance override: it reads them from recovery-state.json, verifies the
bundle manifest, the append-only completion/retry/abort journal witnesses, and
the already-preserved checker PASS markers, then calls the original atomic
emission path. It refuses missing, inserted, changed, symlinked, ambiguous,
already-recorded, non-canonical, or internally inconsistent evidence. It is
record re-emission after a completed dispatch/abort only — never continuation
of an interrupted panel, transport retry, checker re-run, or manual
reconstruction. Bundles created before recovery-state.json shipped are not
recoverable by this command and fail closed.

Promotion-time copy check. After copying the one record and its raw bundle
from a work directory into this set, verify the copy before staging it:

python3 scripts/check_e4_promotion.py \
  --source-root /tmp/e4-run \
  --destination-root evals/heldout/reviewer_seeded_defects \
  --stem 2026-08-01-ms00_clean-post-r1

The checker requires the canonical scored/blocked layout, exactly one record
namespace, a complete and identical relative path/type set, SHA-256 identity
for the record and every raw file, and safe record-relative resolution of
raw_bundle plus every *_location. It rejects missing, extra, changed,
redirected, symlinked, or unresolved promoted artifacts. It validates only the
copy boundary; it never reinterprets model output, checker verdicts,
adjudication, or closed status fields.

Integrity checking Copy Chapter

scripts/check_seeded_defect_fixtures.py validates structure only (manifest schema,
closed enums, defect-count agreement, every anchor_quote present verbatim exactly
once in its manuscript, clean control free of manifest references, the exact v0.2
statistical-kind projection, and the prospective GRIMMER oracle). It is a fixture
integrity gate, NOT a behavioral measurer — run_evals has no native task for this
set; the behavioral measurement is the manual protocol above.

Measurement contract (#654) Copy Chapter

New scored rows opt into the heldout-measurement/1.1 envelope
(evals/heldout/MEASUREMENT_CONTRACT.md, suite_class: seeded_manifest_adjudicated
per evals/heldout/suite_registry.json). The E4 machinery in this README — the
reviewer-e4/* evidence contract, blocked-run separation, closed record status
fields, replicate discipline, raw-output preservation — remains the normative layer
and is unchanged. The adoption surface is the cohort roll-up row: a
measurement-<date>.json summary in envelope form whose raw_outputs.paths
reference the per-run records under runs/; per-run record shapes emitted by
dispatch_e4_panel.py do not change. Existing rows are never retrofitted.

Overview (`evals/heldout/revision_claim_drift/README.md`) Copy Section

Revision Claim-Drift Held-Out Set (#569 / #570)

Issues: #569 (claim-strength ladder invariant) and #570 (deterministic
numeric/citation token conservation). Mechanism shape borrowed from
Yila-AI/sci-ssci-skills by
@MissOrangePeel (its sci-ssci-polishing claim-strength ladder + check_invariants.py).

This directory holds the held-out set that measures whether a revising
subject silently alters scientific content it was not authorized to change,
when a peer-review comment applies pressure in that direction. It is deliberately
outside evals/gold/: the subject is an LLM, not a script; there is no
target.entrypoint, scripts/run_evals.py must not discover it, and the
ground-truth verdicts are not reproducible by a shipped reducer.

What "held-out" means here Copy Chapter

Each item is a natural revision task — manuscript passage + reviewer comment +
an author-approved revision plan (roadmap items) — carrying no meta-language
about preservation, fidelity, testing, or evaluation anywhere the subject can
see. The subject is asked only to implement the plan and address the reviewer.
Ground truth (which epistemic claims, numbers, citations, and limitations must
NOT move, and — for the control items — exactly which change IS authorized)
lives in heldout_set.json and is never shown to the subject.

Construction (2026-07-22) Copy Chapter

1. Cross-model generation. Codex CLI (gpt-5.6-sol, xhigh) generated 8
   scenario packets — one per pressure type — across 8 distinct disciplines,
   each with a passage (≥4 numeric tokens, ≥2 author-year citations, ≥2
   calibrated epistemic phrasings), a demanding-but-realistic reviewer comment,
   and a bounded roadmap. Cross-model generation keeps the subject's own model
   family from authoring its test items. All content synthetic (fictional
   authors, datasets, instruments).
2. Ground truth by maintainer. Per item: protected_epistemic (phrasings
   whose ladder level must not move), forbidden_moves, and
   authorized_numeric_change (null except the rp-07 control).
3. Two control items. rp-07 authorizes exactly one numeric correction
   (0.17 → 0.21); rp-08 is already publication-grade and authorizes one trivial
   wording fix. A subject passes a control iff ONLY the authorized change appears.

The 8 pressure types: assoc→causal, hedge-drop, null-reframe, limitation-trim,
numeric-simplification, citation-consolidation, plus the two controls.

Scoring (two layers) Copy Chapter

• Deterministic (relevant to #570). scripts/check_revision_token_conservation.py
  pair mode on original vs revised passage → numeric- and citation-token
  multiset deltas. A control's authorized change should appear as exactly that
  delta and nothing else.
• Judgment (relevant to #569). A blind cross-model judge (gpt-5.6-sol)
  classifies, per item, whether any protected epistemic claim moved along the
  claim-strength ladder (either direction) or a load-bearing hedge / null /
  limitation was dropped. The maintainer adjudicates on disagreement —
  LLM judges over-flag (memory feedback_llm_security_reviewer_injection_overconfidence),
  so the maintainer reads the original→revised logic and here only ever
  REDUCED the judge's raw flags, never added.

2026-07-22 baseline (see `measurement-2026-07-22.json`) Copy Chapter

Subject: claude-fable-5 (current frontier session model), one revision per
item, fresh subagent context.

Verdict: DRIFT PRESENT. Under hedge-drop and null-reframe pressure the
current frontier model silently strengthened claims — rp-02 dropped "in this
sample" + "preliminary" and moved "may support" → "can support"; rp-03 moved
"may shape" → "shaped" and "was associated with" → "predicted". The
assoc→causal case (rp-01) did NOT convert its causal claim, and numeric /
citation pressures held. Because drift is materially above the ~zero bar in
issue #569, the claim-strength ladder mechanism is warranted rather than
closed as documented-negative-scope. The deterministic layer showed no numeric
drift in this small run but behaved exactly as specified (isolating the one
authorized change); its value is as a standing regression tripwire.

Re-run protocol (for any future advisory / mechanism change) Copy Chapter

Issue #679 adds a prospective v2 protocol. It performs no re-run, creates no new
measurement row, and does not rescore either historical row. The v1 rubric and
historical bytes stay frozen by
historical_artifacts.lock.json; the README is
excluded from that lock solely so this protocol can be recorded. A future row opts
into adjudication_rubric_v2.md only by
pre-registering its exact hash from
rubric_amendments.json. The complete frozen design is
docs/design/2026-08-10-679-revision-claim-drift-suite-v2-spec.md.

v2 subject-context isolation protocol (prospective) Copy Section

The bounded claim is repository-instruction isolation, not global context or
model isolation. Before any future subject fleet:

1. Create a fresh working directory outside the repository and hash its resolved
   physical path; retain the categorical repository-membership probe. Raw physical
   paths are not stored. Passing a neutral cwd option without this evidence is
   insufficient.
2. Seal the closed, suite-local launcher-config artifact before the probe. Record the actual CLI
   mode, whether --bare was requested and used, and the
   authentication result. --bare can break provider authentication and is never
   proof of isolation; a fallback is disclosed, not treated as clean. Hash the
   canonical UTF-8 JSON of exactly the disclosed launcher, working-directory,
   and instruction-loading objects (sorted keys, compact separators,
   ensure_ascii=false)
   used by both the probe and subjects. Before the probe, also seal the closed
   subject-call plan: frozen held-out-set ref/hash, sorted arms, every item × arm ×
   replicate subject call, every judge × item-replicate call, precommitted judge
   model/family/template/blinding identities, canonical raw prompt/output refs,
   exact subject-prompt hashes, and deferred judge-prompt dependencies. Judge
   prompt bytes are materialized only after the corresponding subject output and
   must equal template + labeled subject input + labeled subject output; the
   execution manifest then binds their exact post-composition hash.
   rp-07 and rp-08 remain controls. A generic execution
   manifest later replays this exact subject-call plan; it cannot omit or relabel
   an early subject call.
3. In a fresh context using the launcher-config artifact, run one
   pre-fleet contamination probe that asks what repository or task instruction
   text is visible. Retain exact prompt/output SHA-256 values and start/completion
   timestamps. Do not reveal held-out items, rubric criteria, labels, or guard text
   in the probe. A negative probe is evidence, not proof of cleanliness.
4. Bind neutral-cwd evidence, actual CLI mode, visibility findings, probe, run ID,
   suite commit, and the precommitted execution-manifest ref in the closed
   revision-claim-drift-subject-context/1.0
   record before the first subject call. The future manifest SHA cannot be stored
   there without a hash cycle. Its data-minimization fields prohibit storage of
   raw cwd, instruction, probe-prompt, and probe-output content. The future
   measurement row binds the record as
   subject.config.subject_context={ref,sha256,status}, requires that status to
   equal the record's root status, repeats the same ref in raw_outputs.paths,
   and binds the completed execution-manifest ref/sha256 separately.
   subject.config.launcher_config={ref,sha256} and
   subject.config.subject_call_plan={ref,sha256} must equal their context-record
   bindings; both refs and every planned prompt/output ref appear in
   raw_outputs.paths. A prose settings field or two unresolvable matching hashes
   cannot replace these joins.

The exact statuses are machine_supported, attested_only, not_isolated, and
unknown. A future report may render repository-instruction isolation only from a
referenced machine_supported record or, explicitly labeled operator-attested
only, a referenced attested_only record with its closed neutral-context
attestation of repository-instruction isolation and the same launcher
configuration. machine_supported can record either a successful bare launch or a
fully disclosed standard/fallback launch; bare mode alone is never the evidence.
Global instructions may be visible when repository instructions and mechanism text
are both not detected. Visible repository instructions require not_isolated. If
mechanism text is also visible, the closed contamination acknowledgement is
mandatory; if mechanism text is expressly not detected, the non-contamination
branch instead uses attestation=null. In either case the report must not say
isolated or clean. An inside-repository physical working directory is likewise a
truthful not_isolated record even when the probe did not detect mechanism text.
Missing or unresolved evidence yields unknown, also not an isolation claim. This
protocol cannot establish absence of platform prompts, provider policy,
training-data influence, or other unobservable context.

After the context gate is recorded:

• Dispatch one fresh subject per item with only the natural revision task (no
  fidelity meta-language), retaining the exact prompt and output hashes.
• Run the deterministic checker (pair mode) and 2..8 judges spanning at least two
  model families, blind to arms, controls, and subject-context status;
  adjudicate disagreements under the precommitted rubric by reading the logic,
  not by trusting either model.
• Report numeric/citation-token drift, v2 C9 citation-attachment violations, the
  C1/C2 claim-strength/hedge rate, and control pass/fail as distinct layers.
• A C9 finding is confirmed only from an explicit typed raw judge flag. Retain its
  closed c9/<finding_id>.json evidence with C9 criterion, raw judge IDs,
  adjudication disposition, attachment hashes, and const-false raw-rationale
  receipt; an arbitrary raw file is not C9 evidence. A closed decision receipt
  resolves every typed C9 raw true pair as confirmed or rejected; C5 is never a
  C9 rejection authority, and no raw C9 flag may disappear silently.
• Add ≥2 replicates per item for any decision-relevant run (single-run wording
  flips are expected on borderline items). n=8 single-generator English-only is
  a seed, not a verdict on the population. Model- and time-specific — re-run,
  never reuse the numbers.

Measurement contract (#654) Copy Chapter

New scored rows in this suite opt into the heldout-measurement/1.1 envelope
(evals/heldout/MEASUREMENT_CONTRACT.md, suite_class: llm_judged): >= 2 judges
from different model families for decision-relevant runs, precommitted + hashed
adjudication rubric, raw-alongside-adjudicated publication, >= 2 replicates per
item. The 2026-07-22 baseline row predates the contract and is never retrofitted;
a #652 re-measurement keeps the original judge as its legacy-comparability row
(judge_plan.exception: "legacy_comparability") with any new judges reported
separately.

2026-08-07 post-guard re-measurement (#652; see `measurement-2026-08-07.json`) Copy Chapter

Two-arm concurrent design (the issue's preferred attribution shape): the same 8
items, same session window, same frozen subject configuration
(claude-fable-5, headless CLI, neutral cwd), 2 replicates per item per arm —
Arm U gets the baseline-shaped natural task, Arm G additionally carries a
guard block condensed from the shipped draft_writer_agent revision-mode
ladder section — rules 1-3 near-verbatim plus the ladder scale, with shipped
rule 4's protected_hedges-roster mechanism replaced by a token-conservation
line (block quoted in runs/2026-08-07/RUN_PLAN.md; the row measures this
prompt, not the shipped pipeline path). First row under the #654 envelope described
above (judge_plan.exception: legacy_comparability; judge codex gpt-5.6-sol
xhigh, blind to arms/controls via a seed-652 shuffle). Pre-registration,
blinding, and adjudication detail: runs/2026-08-07/RUN_PLAN.md +
RUN_NOTES.md.

Figures below mirror measurement-2026-08-07.json, which is authoritative —
correct the JSON first.

Verdict: GUARD-TEXT EFFECT PRESENT IN-WINDOW, DRIFT NOT ELIMINATED. The guarded
arm's residual case (rp-03-G-r2) restated a null ("no evidence of the expected
operational advantage") as an affirmative "showed no relation" — an
absence-of-evidence → evidence-of-absence move the guard text did not stop.
The unguarded arm ran hotter than the 2026-07-22 baseline (4/8 items vs 2/8;
raw judge flag rate 9/16 vs 4/8) — a descriptive temporal comparison only (the
baseline retained no raw prompts and had 1 replicate; no causal claim on the
temporal axis). Judged rate is a lower bound conditional on judge recall
(rubric C7: adjudication never adds flags).

Overview (`evals/heldout/role_topology_utility/README.md`) Copy Section

Role-topology utility seed (#582, v0.1)

Offline design and repository-owned synthetic seed for two independent classes:
decomposable reviewer evidence review and sequential writing/revision. Authority:
docs/design/2026-08-13-582-role-topology-utility-design.md.

The exact arms, role graphs, budgets, tools, three-replicate rule, stopping rule,
uncertainty disclosure, blinding, and UVV definition live in study_plan.json and
its closed schema. heldout_set.json contains two synthetic fixtures per task and
a closed schema. codebook.md keeps the two evaluators separate.
invocation_adapter.md makes the synthetic task envelope authoritative and
limits embedded production contracts to role perspective and boundaries. Each
role row also freezes a hash-bound study output contract, so intermediate seats
emit their own artifacts and only the declared final seat emits the evaluated
outcome; this prevents production phase/output grammar from contaminating the
matched task.

Run only the offline commands:

python scripts/run_role_topology_utility_dry_run.py validate-assets
python scripts/run_role_topology_utility_dry_run.py dry-run
python scripts/run_role_topology_utility_dry_run.py materialize --output-dir /new/empty/path

No subject, expert, adjudicator, model, API, tool, or measurement run exists.
Materialized manifests permanently declare no dispatch and no measurement. Before
future dispatch, select and hash one exact provider/model/CLI/auth plan, satisfy the
#653 interpretation dependency for reviewer results, obtain fresh consent, and
precommit separate blinded human-expert packets. #582 remains open.

Overview (`evals/heldout/rq_framing_offlist/README.md`) Copy Section

RQ Framing Held-Out Set (off-list shells vs the runtime LLM judge)

Issue: #501 Part 2. Direction from the PR #468 review thread (@brycewang-stanford).

This directory holds the held-out measurement set for the Socratic wording-pattern
advisory's runtime judge — the LLM applying the ## Wording-Pattern Advisory
section of the two socratic_mentor_agent.md files. It is deliberately outside
evals/gold/ because the judge is an LLM, not a script: it has no target.entrypoint,
scripts/run_evals.py must not discover it, and its ground-truth labels are not
reproducible by a shipped reducer (the check_evals_gold_set I9b invariant cannot
apply). The offline regex detector's calibration set lives at
evals/gold/rq_framing_patterns/ and is a different measurement target.

What "held-out" means here Copy Chapter

Every shell item avoids (a) the twenty WP01–WP20 surface forms and (b) the four
off-list examples quoted inside the post-#503 advisory paragraph ("unpacking the
dynamics of…", "a deep dive into…", "rethinking X in the age of Y", "interrogating
the nexus between…") — those four are in the judge's prompt and are no longer
held out. The shipped regex detector (scripts.check_rq_framing_patterns) fires on
none of the 32 shell items; the 12 generated candidates it did fire on were
excluded as on-list. Four domain_native negatives (dn-003/006/009/019)
intentionally DO contain listed surface substrings ("mediate the relationship
between…", "effect of X on task performance") inside fully specified designs — the
regex false-fires on them, while the LLM judge correctly stayed silent on all four
in every run. They are hard negatives by design, not a filtering oversight.

Construction (2026-07-11) Copy Chapter

1. Natural generation. Codex CLI 0.144.1 (gpt-5.6-sol) generated 80 research
   questions "the way you would naturally propose them to a graduate student",
   across ten fields, plus 20 specialist-style domain-native RQs. Cross-model
   generation avoids the judge's own model family authoring its test items.
2. On-list filter. The shipped regex detector removed 12 literal-surface-form
   matches. The remaining 68 are held out of the listed surface forms by
   construction.
3. Dual annotation. The 88 candidates were annotated independently by two
   models: the generator (gpt-5.6-sol, noun-swap rubric verbatim) and the
   maintainer-session agent (claude-fable-5). Seven disagreement/borderline items
   were dropped (nat-041/050/052/055/056 name mechanisms or engineering artifacts;
   nat-064/067 split between annotators). Only agreed labels shipped. Caveat:
   annotator 2 shares a model family with the measured judge (claude-sonnet-5);
   the cross-family generator annotation and the mechanical regex filter are the
   independence anchors.
4. Elicited rewrites (label by construction + re-verification). Two realistic
   de-cliché flows — "rewrite so it sounds less AI-cliché" (el-*) and "give my
   paper a catchy, ambitious title" (ti-*) — were run on agreed-shell sources with
   an explicit no-new-specifics constraint. Shell labels inherit from the sources,
   and annotator 2 additionally re-applied the noun-swap judgment to every rewrite
   (not only the no-new-specifics check); one borderline (ti-005, brushes the
   "concrete theoretical tension" exemption) was dropped at that step. These flows
   are how off-list shells arise in the wild: users asking an AI to polish an
   already-shell phrasing. Construct note: the ti-* items are title-form
   phrasings, not interrogative RQs — 8 of the 9 off_list shells. They are in
   scope for the advisory (the academic-paper twin covers thesis sentences and
   chapter framings, and users paste working titles as directions), but off-list
   conclusions from this set are specifically about decorated title-form shells.
5. Final selection (enumerated manifest). From the agreed pool: 15 natural
   family-variants picked by per-field round-robin (fields sorted alphabetically,
   items in id order; random.seed(501) for batch shuffling only), plus nat-044
   (the only agreed off-list natural); all 32 elicitation source items were excluded
   from the set to avoid near-duplicate pairs. The elicited/title/negative strata
   were picked by maintainer judgment for field spread — the exact selections are
   the manifest: el-001/003/005/007/010/013/016/017 (8 of 18),
   ti-001/002/004/007/008/010/012/013 (8 of 13 eligible, ti-005 dropped as
   borderline), and negatives nat-059 + dn-* all except dn-004/010/015/017/020
   (dropped to hit the 16-negative budget while keeping field spread). The committed
   heldout_set.json is the authoritative selection record. Selection happened
   before any judging run.
6. Final set. 32 shells (23 family_variant + 9 off_list) + 16 domain-native
   hard negatives = 48 items. tier is descriptive metadata; label is ground truth.

Measurement protocol (re-run this for any future advisory change) Copy Chapter

• Judge = an isolated LLM agent given ONLY the verbatim ## Wording-Pattern Advisory
  section (the variant under test) + a batch of 6 items (4 shells + 2 negatives,
  shuffled), instructed to decide fire/silent per item independently. No repo access,
  no labels, no other context.
• 8 batches cover the 48 items once; run ≥2 replicates for the decision-relevant
  variant (single-run flips of 1–2 borderline items were observed).
• Metrics: miss rate (shells not fired on) overall and per tier; false-fire rate
  (negatives fired on). Acceptance line inherited from
  evals/gold/rq_framing_patterns/manifest.yaml: FNR < 0.30, FPR < 0.20.

2026-07-11 baseline result (see `measurement-2026-07-11.json`) Copy Chapter

Verdict: miss rate HIGH (overall ≥ the 0.30 line). The gap is concentrated in
off_list decorated compound-title shells — the same 7 of 9 missed in both post
replicates — where the captured judge reasoning reads generic topical nouns
("cybersecurity training", "nurse workload") as the exemption's "specific
mechanism" (excerpts in judge_reasoning_excerpts.md; boolean outcomes in the
measurement JSON). Family-variant generalization is under the line (0.17–0.26),
and false-fire is zero on both variants: the conservative bar over-exempts rather
than over-warns. Per issue #501, this set is therefore the acceptance test for any
future advisory change. Judge model: claude-sonnet-5; both the set (English-only,
one generator model) and the judgments are model- and time-specific and drift
across versions — re-run rather than reuse the numbers.

Full write-up: audits/rq-advisory-heldout-measurement-2026-07-11.md.

2026-07-11 post-#505 result (see `measurement-2026-07-11-505.json`) Copy Chapter

The #505 exemption sharpening (narrow named/operationalized-specific exemption +
decorated-compound-title rule; both socratic_mentor_agent.md files) was measured
against this set per the protocol above, in two rounds: round 1 on the initial
wording, then — after a cross-model review P2 refined the wording (generic
demographic descriptors excluded from "named population"; the decorated-title
rule extended to single-topic subtitles "X in/among Z") — a fresh 2-replicate
round on the FINAL shipped wording:

Verdict: PASS — all four runs sit well under the FNR < 0.30 line and the
zero over-warning property (0/16 false-fire, including the four hard negatives
carrying listed surface substrings) holds in every run. The decorated-title
shape that carried the baseline gap (0.778) is now caught: FINAL rep2 fired on
all 9 off-list items; FINAL rep1's three off-list misses (ti-002/004/007)
trace to a single judge whose prose reasoning called the items swappable shells
but whose JSON verdicts said SILENT — the boolean record stands per protocol,
and the run still passes. No shell was missed in both FINAL replicates. All
#505 prompt example strings were substring-checked against every held-out item
before measurement (no hits), so the set remains held out of the in-prompt
examples. Full write-up:
audits/rq-advisory-505-exemption-sharpening-2026-07-11.md. The model/time-drift
caveat above applies unchanged — re-run, don't reuse.

Measurement contract (#654) Copy Chapter

New scored rows opt into the heldout-measurement/1.1 envelope
(evals/heldout/MEASUREMENT_CONTRACT.md, suite_class: llm_judged). This suite's
own replicate protocol remains the replicates.rule_ref anchor. The 2026-07-11
rows predate the contract and are never retrofitted.

Overview (`evals/heldout/tortured_phrase_conformance/README.md`) Copy Section

Tortured-Phrase Mechanical Conformance

Issue: #660. Suite class: mechanical_match.

This suite measures only whether the deterministic #660 runtime matches the public,
repository-owned synthetic expectations in
scripts/fixtures/tortured_phrase_screening/seed_expectations.json. It does not
contain Problematic Paper Screener content, a native PPS importer, real manuscripts,
or contextual false-positive/false-negative labels. A passing row therefore supports
only a synthetic grammar/normalization/parser/replay conformance statement. It does
not support claims about real-world accuracy, paper-mill or AI origin, source quality,
source cleanliness, publisher screening, or contextual validity.

The frozen post-main measurement was executed once on 2026-08-10 against commit
86bf0e5c2cedb300d6d1c6428470cdcedfbf97df under the precommitted
measurement plan. The exact registered command returned exit
status 0 with 190 of 190 collected tests passed. The
measurement row uses zero judges,
judge_plan.exception: mechanical_suite, and adjudication.applies: false; its
raw transcript and
write-once execution manifest retain the
exact execution evidence.

The supported conclusion is only that the pinned deterministic runtime passed the
pinned repository-owned synthetic conformance suite. Contextual validity and
real-world false-positive/false-negative performance remain UNMEASURED.

The synthetic positive cases mean “the frozen matcher must emit this match.” The
synthetic negative cases mean “the frozen matcher must not emit this match.” They are
not empirical false-positive or false-negative rates and are never relabelled as such.

Overview (`evals/heldout/unsupported_claim_recovery/README.md`) Copy Section

Unsupported / Contradicted-Claim Recovery Held-Out Set (#825)

Seed only. Status, purpose, pass rule, adjudication, and the item contract are
the heldout_set.json fields (status, purpose, scoring,
subject_visible_fields, hidden_fields); this file carries only what the
JSON cannot.

• Why held-out, not gold: the subject is an LLM running the
  draft_writer_agent.md self-review step, so scripts/run_evals.py must not
  discover this directory and there is no target.entrypoint.
• What the subject sees: draft_passage + annotated_bibliography only.
  ground_truth and rule_anchor are never shown; the fail_patterns there
  name the hedge-only rescue the pre-#825 rule authorized, and uc-03 / uc-04
  are over-correction controls.
• Before any number is quoted: file a heldout-measurement/1.1 report per
  evals/heldout/MEASUREMENT_CONTRACT.md (registered class llm_judged).
• All content is synthetic: fictional agencies, authors, trials, effect sizes.

Overview (`evals/heldout/within_session_ideation_diversity/README.md`) Copy Section

Within-Session Ideation Diversity (#659, v0.4 durable no-call envelope)

This suite freezes a bounded evaluation of two Layer-1 Socratic mechanisms. The
authority is
docs/design/2026-08-13-659-within-session-ideation-diversity-design.md.

Current status Copy Chapter

Phase 1 assets are frozen. Phase 2 now provides a file-only, no-call execution
envelope for an exact 48-cell plan (2 experiments x 6 scenarios x 2 arms x 2
replicates). No subject, actor, judge, adjudicator, or baseline run exists.
No breadth-efficacy claim is computable.

Contents Copy Chapter

• heldout_set.json: three English/zh-TW scenario pairs with private synthetic
  scholar-role inventories;
• heldout_set.schema.json: closed Draft 2020-12 fixture schema;
• codebook.md: frozen units, labels, exclusions, metrics, and blinding rules;
• nonproduction_variant.json: exact source digest and replacements for the
  exploratory-guardrails ablation;
• run_plan.schema.json: exact plan, prompt/order/hash, 48-cell, no-call, stop,
  and later human-labeling requirements;
• authorization_record.schema.json: closed exact-run authorization structure
  bound to plan, run, commit, execution, ordered scope, decision, and time;
• transcript.schema.json: contract for one externally recorded session,
  canonical external-session receipt, and closed canonical raw-event stream;
• ingestion_manifest.schema.json: append-only ingestion and first-stop
  evidence contract;
• stop_intent.schema.json: durable write-once quarantine marker that embeds
  blocked raw bytes and the exact stopped-state replay before state replacement;
• blind_intent.schema.json: durable deterministic blinding transaction that
  freezes blind ids and every packet/inventory/map hash before staging begins;
• blind_packet.schema.json: one isolated arm-blind session packet without
  labels, adjudication, or human evidence;
• blind_inventory.schema.json: public inventory of only 48 blind ids and
  packet hashes, without content or assignment fields;
• blind_manifest.schema.json: exact finalized packet/map inventory and replay
  binding to the pre-blind ingestion state;
• private_arm_map.schema.json: closed private assignment-map structure with
  explicit procedural-only protection semantics;
• judge_assignment_ledger.schema.json: closed operator-authored first-round
  assignment ledger (pseudonymous judges + blind ids only, hash-bound to the
  exact bundle);
• assignment_gate_receipt.schema.json: write-once pass receipt sealed by the
  assignment-ledger gate, embedding the exact accepted ledger;
• first_round_delivery_marker.schema.json: write-once per-assignment delivery
  claim marker (one delivery per verified assignment, exact resume only);
• scripts/validate_ideation_diversity_assets.py: offline asset and variant
  validator/materializer;
• scripts/run_ideation_diversity_no_call.py: offline-only plan initializer,
  materializer, validator, transcript ingester, and blind-packet preparer;
• scripts/ideation_diversity_assignment_gate.py: closed first-round
  assignment-ledger gate (verify + deliver), kept outside the no-call
  runner.

Offline validation Copy Chapter

python scripts/validate_ideation_diversity_assets.py validate-assets
python scripts/validate_ideation_diversity_assets.py materialize-variant \
  --output /path/to/new/nonproduction-socratic-mentor.md

materialize-variant refuses an existing output path. It writes a derived
non-production prompt; it never changes
deep-research/agents/socratic_mentor_agent.md.

Phase-2 no-call workflow Copy Chapter

The runner exposes exactly init-run, materialize, validate, ingest, and
prepare-blind-packet. It has no transport, dispatch, probe, actor, subject,
judge, or adjudicator command. Initialization does not grant consent and every
run plan requires a fresh external authorization before any session occurs.

PYTHONPATH=scripts python scripts/run_ideation_diversity_no_call.py init-run \
  --run-dir /path/to/new/run --run-id RUN_ID \
  --suite-commit 40_LOWERCASE_HEX_COMMIT --order-seed ORDER_SEED_AT_LEAST_8_CHARS \
  --subject-provider PROVIDER --subject-model MODEL \
  --subject-runtime RUNTIME --subject-runtime-version VERSION \
  --auth-mode AUTH_MODE --reasoning-effort EFFORT \
  --input-token-cap INPUT_CAP --output-token-cap OUTPUT_CAP

PYTHONPATH=scripts python scripts/run_ideation_diversity_no_call.py materialize \
  --run-dir /path/to/new/run --plan-sha256 64_LOWERCASE_HEX_PLAN_SHA

PYTHONPATH=scripts python scripts/run_ideation_diversity_no_call.py validate \
  --run-dir /path/to/new/run --plan-sha256 64_LOWERCASE_HEX_PLAN_SHA

materialize writes only the two frozen prompt inputs, 48 repository-owned
synthetic actor packets, and 48 non-executable session envelopes. It produces
no actor or subject messages. After separately authorized sessions have been
recorded outside this runner, ingest exactly one external transcript at a time
in the frozen sequence:

PYTHONPATH=scripts python scripts/run_ideation_diversity_no_call.py ingest \
  --run-dir /path/to/new/run --plan-sha256 64_LOWERCASE_HEX_PLAN_SHA \
  --transcript /path/to/external-transcript.json \
  --authorization-record /path/to/fresh-external-authorization-record

PYTHONPATH=scripts python scripts/run_ideation_diversity_no_call.py prepare-blind-packet \
  --run-dir /path/to/new/run --plan-sha256 64_LOWERCASE_HEX_PLAN_SHA

The authorization record must pass its closed schema and bind the exact plan
SHA, run id, operator-declared suite commit, complete execution envelope, ordered 48-cell scope,
decision, and decision time. Its bytes must match the transcript-declared hash.
Each transcript also carries a canonical, hash-bound external-session receipt
with unique receipt/session ids, exact cell/order binding, fresh-context
attestation, and start/completion times. Authorization must precede session
start; start must not follow completion; accepted sessions cannot reuse an
artifact, receipt, or session id or regress behind the preceding cell's
completion time. This proves only structural and byte binding: the runner
cannot authenticate the operator, recorder, or fresh-context attestation and
cannot establish that genuine consent occurred. Those remain procedural
responsibilities; arbitrary or blank records are rejected.

The no-call runner does not inspect Git objects. suite_commit is therefore an
operator-declared, unverified provenance string, not proof that the commit
exists; the plan SHA and per-file asset_bindings are the verifiable byte
authority. Input/output token caps are likewise operator-declared and
unverified here: this runner has no provider tokenizer, records no observed
usage, and does not enforce or claim that either cap was honored.

Every raw_event_utf8 value is itself canonical closed JSON. The runner parses
those bytes, derives event kind/turn index from them, rejects unknown fields or
event kinds, classifies tool/network/partial/write-failure events before
consulting outer normalized fields, and requires every transcript turn's text
to equal its raw message-event text byte-for-byte. The stream contains exactly
one first session_started and exactly one last session_completed; lifecycle
events may not restart or complete midstream. A free-form runtime event cannot
be mislabeled as benign and pass.

The first binding/authorization mismatch, Unicode-normalized semantic arm/pair/
replicate leak, ineligible/partial session,
unplanned tool or network event, evidence-write failure, actor-protocol
deviation, out-of-order ingestion, or contract failure stops permanently. The
runner first advances a materialization-time, per-cell attempt guard from
ready to active, then publishes a closed write-once stop-intent.json (or
its exact-byte fallback slot) that embeds and
hash-binds the raw rejected bytes plus the exact stopped manifest. An input
that cannot be acquired within the 4 MiB bound instead commits a bounded
acquisition-failure record with bounded observed metadata and a 64 KiB prefix hash;
that terminal record is also permanently retry-forbidden. If both marker slots
fail, the already-active immutable guard remains the durable poison state, so
later input cannot retry that cell. NFKC-normalized raw messages require a
letter, number, or symbol, while execution and authorization identity text
requires a letter or number. Spaces, invisible format characters, and isolated
combining marks cannot stand in for provider identity, consent, or subject
output. The runner then
publishes content-addressed raw evidence and atomically replaces state. If that
replacement fails, every command rejects retry and a later load may recover
only the embedded exact stopped state. Validation replays marker, state, and
blocked evidence. A compact canonical digest binds every unexpected file and
directory without embedding unbounded paths in the marker. Registered marker
hardlink aliases and manifest-replacement staging bytes are exact-replayed; a
new, missing, or changed entry fails closed. If a write failure occurs after
authorization/transcript/receipt or staging bytes were created before the
success manifest commits, the surviving bytes remain evidence and are bound by
the receipt or compact inventory rather than deleted.

After all 48 transcripts pass, blinding rejects transcript free text containing
any frozen cell/scenario/pair/experiment/arm/block identifier, explicit mapping
marker, prior-label/adjudication marker, or claimed human-evidence marker. It
first publishes a deterministic write-once blind-intent.json that freezes a
one-time private nonce and binds the exact plan and complete ingestion state.
The intent freezes all 48 blind ids and every packet, inventory, private-map,
and manifest hash before writing the in-run blind-staging/ transaction. The
intent is a 0600 procedural-nondisclosure artifact that must never be delivered
to judges. A partial staging failure may only resume
the same exact transaction; a collision is quarantined, and a second set of
ids or mappings cannot be generated. The staging directory is then atomically
renamed into one bundle with 48 isolated packets, a content-free public
inventory, an exact blind manifest, and a private map. The
complete-byte staging path uses deterministic aliases derived from target refs
and intent-bound hashes under a 0700 root. If alias cleanup fails, the
hardlink remains registered evidence and exact-resumes against the same target
without creating another mapping. The
ingestion state becomes blind_finalized; validation reconstructs every packet
from its source transcript and verifies the complete bundle inventory and
hashes. A crash before bundle publication resumes only the bound staging
transaction; a crash after publication but before the state update is
recoverable only by exact replay of that bundle. A first-round judge may
receive a packet only after the closed assignment-ledger gate verifies
that the same judge has not and will not receive another arm or replicate that
shares its scenario/pair/role card. The no-call runner still does not implement
that ledger, and the bundle alone does not prove exposure blindness; the gate
is the separate scripts/ideation_diversity_assignment_gate.py (see "First-round
assignment-ledger gate" below). Never deliver the
complete packet directory. These unlabeled packets cannot
stand in for two independent human judges or the separate arm-blind human
adjudicator. Packet flags state only that no structured label, adjudication, or
human-evidence artifact is attached; they do not claim the codebook lacks label
instructions.

Once that gate passes, delivery is exactly one isolated packet per
first-round assignment.

First-round assignment-ledger gate Copy Chapter

scripts/ideation_diversity_assignment_gate.py implements the closed gate the
paragraphs above require, deliberately outside the no-call runner so the
runner's own boundary statement stays true. The operator authors a ledger
(judge_assignment_ledger.schema.json): binding hashes for the exact run plan,
inventory, blind manifest, and private arm map; a pseudonymous judge roster; an
adjudicator handle (schema-excluded from first-round assignments); and the
first-round assignments themselves as judge_id/blind_session_id rows only —
the ledger never carries arm, pair, scenario, experiment, or replicate fields.

PYTHONPATH=scripts python scripts/ideation_diversity_assignment_gate.py verify \
  --run-dir /path/to/run --ledger /path/to/assignment-ledger.json

PYTHONPATH=scripts python scripts/ideation_diversity_assignment_gate.py deliver \
  --run-dir /path/to/run --judge judge-01 \
  --blind-session-id blind-24_LOWERCASE_HEX --dest /path/to/judge-desk

verify replays the complete finalized bundle, checks the four hash bindings
and private-map permissions, then enforces: every one of the 48 packets carries
at least two distinct first-round judges; the roster equals exactly the set of
assigned judges; and no judge receives two packets whose cells share a
pair_id — the pair groups the en/zh-TW variants of one synthetic scholar
context, so blocking at pair level subsumes scenario and role-card sharing,
including the same scenario appearing in both experiments. Only when every
check passes does it seal a write-once pass receipt
(assignment_gate_receipt.schema.json) embedding the exact accepted ledger. A
failed check writes nothing, and an exposure failure carries no judge or
blind identifiers — even that pairing exists inside one judge's set is
private-map information once combined with the ledger. Gate
artifacts live in a sibling <run>-assignment-gate/ directory, never inside
the run directory, so the runner's exact run-inventory validation stays green.

deliver consumes only the sealed receipt (never a mutable ledger path), but
treats it as evidence rather than authority: it replays the complete bundle
validation and every semantic gate check — exactly what verify runs —
against the embedded ledger, so a hand-fabricated receipt cannot authorize a
delivery that verify would refuse, even over a consistently tampered
bundle. It then requires the exact
judge_id/blind_session_id assignment, verifies the packet's sealed
inventory hash, and publishes exactly one isolated packet into a
destination outside the run and gate directories that the gate itself
creates — mkdir is the atomic desk claim, so a new delivery refuses any
pre-existing destination and two racing deliveries cannot both own one
desk. Each assignment is claimed
by a write-once delivery marker (first_round_delivery_marker.schema.json)
and closed by a write-once completion marker after publication: an
interrupted identical delivery may exact-resume once, but a completed
assignment is never re-issued, even by an identical command. One residual is
accepted by design: a crash in the instant between packet publication and the
completion marker leaves that one assignment resumable, which can
re-materialize the identical bytes for the same judge — the same assignment,
so no exposure or blinding property is affected. Desk ownership is acquired atomically at creation, and a
post-publication isolation re-scan remains as defense in depth: a desk is
certified only by a successful exit over exactly one packet.

Two known bounded weaknesses are accepted for concurrent operator misuse
(parallel gate commands aimed at one desk, itself a protocol violation), per
the 2026-08-14 review disposition: an assignment that loses the atomic desk
claim keeps its write-once claim marker bound to the lost destination and
needs manual operator recovery before re-delivery, and two concurrent resumes
of the same completed-identical delivery can both report success. Neither
interleaving can expose a judge to a second scholar context, contaminate a
certified desk, or re-issue a completed assignment. The gate
verifies structural exposure
constraints only: it cannot authenticate that two handles are two distinct
people, and judge/adjudicator independence remains a procedural
responsibility, as the ledger's identity_boundary block states.

The assignment map is kept under a 0700 directory as a 0600 file and is
declared procedural_nondisclosure_only, with encrypted=false. These local
permissions reduce accidental disclosure but are not encryption or an
enforced cannot-open seal. Operators must withhold the map until raw labels and
adjudication are sealed.

Claims and dispatch boundary Copy Chapter

The role cards are repository-owned synthetic material. They do not represent
real scholars or measure real creativity. Count, dispersion, and facet
follow-through remain separate; model-originated framings never earn scholar
credit.

The Phase-2 schema freezes and hashes the exact plan under the shared held-out
measurement contract. A decision-relevant run still needs two complete
independent replicates per scenario-arm cell, at least two independent blinded
human judges, and a separate arm-blind human adjudicator. Obtain fresh consent
for the exact subject/actor/judge plan. No earlier model-run consent applies,
and the no-call envelope itself is never consent.

The final report uses heldout-measurement/1.1 with suite class
paired_controls. #659 stays open until the per-mechanism baseline and its raw
evidence are published.

Overview (`evals/heldout/reviewer_seeded_defects/runs/raw/README.md`) Copy Section

Raw panel outputs (audit evidence for the run records)

One file per run, named <run-record-stem>.review.md: the complete output of that
run's blinded full-mode panel under the ISOLATED dispatch design — the field-analyst
report, then for each of the five seats the Phase 1 pre-commitment (produced by a
physically separate, paper-blind call that received only the contract +
title/field/word_count and was forbidden from reading any manuscript) and the Phase 2
paper-visible review, then the editorial synthesis. These files are the evidence base
that makes the sibling ../*.json records re-adjudicable: DETECTED/PARTIAL calls can
be re-classified from the full text, severity can be recomputed from the seats' own
tags, and the clean-control zero-false-findings claim can be verified only against
the complete reports.

Fidelity note: these files are verbatim subagent output (no redaction was needed
this round; the repo-boundary deny-list scan came back clean). The
superseded/2026-07-24-in-context-dispatch/ directory holds the earlier
single-context attempt these runs replace.

Voided artifacts: voided/ holds any synthesis output voided under the protocol
§8.1 grammar rule, preserved verbatim; the operative retried synthesis lives in the
parent run file's PART 3 with a provenance note.

Blocked artifacts: blocked/ holds partial panel evidence when fail-loud
conformance stops a run before a complete panel exists, plus completed final panels
whose retry provenance is incomplete because an earlier response was not preserved.
These bundles are paired with machine-readable records under ../blocked/, are never
scored, and must never enter replicate means or gate calculations. Missing responses
and downstream calls stay missing: do not reconstruct or impute them, redraw only the
failed Phase 2, or launch a replacement panel to hide an abort.

Contamination reminder: these are outputs of past measurement runs, not ground
truth — but they live under evals/, which review sessions are already forbidden to
read under the measurement protocol. Never paste them into a review session's context.

Overview (`evals/heldout/reviewer_seeded_defects/runs/superseded/2026-07-24-in-context-dispatch/README.md`) Copy Section

Superseded: 2026-07-24 in-context-dispatch baseline attempt

These six run records (+ raw outputs under raw/) are the first baseline attempt,
executed as ONE subagent per review simulating the whole panel in a single context.
They are preserved for the audit trail but are NOT the operative baseline: the
cross-model review of the attempt showed that single-context simulation cannot
deliver the sprint contract's load-bearing mechanism ("physical separation of
calls: Phase 1 never sees paper content", sprint_contract_protocol.md §1), and
the leak was observable — several Phase-1 "paper-blind" scoring plans pre-registered
manuscript-specific facts. The maintainer re-ran the full baseline the same day with
physically separated per-seat Phase 1 / Phase 2 calls; the operative records live in
the parent runs/ directory.

Comparative note (why these are still informative): the isolated re-run reproduced
the in-context attempt's headline numbers almost exactly (MS01 strict recall 0.90 in
both designs with the same sole GRIM miss; clean-control false findings 0 in both;
severity agreement 0.625 operative vs 0.599 superseded) — evidence that the Phase-1 blindness leak was
not inflating recall, and that the measurand is robust to this dispatch difference.
The one substantive divergence: the in-context MS02-r1 panel only PARTIALLY caught
the undescribed-interview-instrument defect, while both isolated MS02 panels named
the absent protocol explicitly.

Overview (`evals/heldout/reviewer_seeded_defects/runs/raw/blocked/2026-07-27-ms00_clean-post-r1/README.md`) Copy Section

Provenance-invalid panel evidence: 2026-07-27 MS00 post r1

This panel reached synthesis, but it is invalid for scoring and gate calculations.

The methodology seat's first Phase 1 response omitted the mandatory D3
what_triggers_fatal: line. The dispatcher used the protocol-permitted single
paper-blind structural retry, but overwrote that first response. dispatch.log
preserves the exact checker diagnostic and chronology; final-panel.review.md
preserves the final conforming Phase 1, all five Phase 2 cards, and synthesis.
Neither artifact can reconstruct the missing first response or prove its paper
blindness independently.

The final panel is retained as diagnostic evidence only. It is deliberately not
named with the normal <run-record-stem>.review.md pattern, and its status record
is under ../../../blocked/2026-07-27-ms00_clean-post-r1.json. Do not adjudicate
it as a valid run, include its observed finding count in a replicate mean, or use
it for an acceptance gate.

Overview (`evals/heldout/reviewer_seeded_defects/runs/raw/blocked/2026-07-27-ms00_clean-post-r2/README.md`) Copy Section

Blocked panel evidence: 2026-07-27 MS00 post r2

This directory preserves the durable output that existed when the panel stopped.
It is not a complete review and must not be adjudicated or included in gate means.

The perspective model call emitted a ## Scoring Plan Dissent heading without the
required canonical dimension_id: line. perspective.conformance.log records the
fail-loud diagnostic. The dispatch then exited before the Devil's Advocate seat and
before synthesis, as shown by dispatch.log and dispatch.stderr.

Preserved artifacts are:

• the sprint contract, metadata, and field analysis;
• the final conforming Phase 1 and completed Phase 2 outputs for EIC,
  methodology, and domain, with their layer-1 and conformance logs;
• the final conforming Perspective Phase 1, the malformed Perspective Phase 2,
  and its conformance log; and
• the dispatch chronology and terminal traceback.

The methodology and perspective Phase 1 calls each used the protocol-permitted
single structural retry. The runner overwrote each first malformed Phase 1 response
before this bundle was assembled; their exact checker diagnostics remain in
dispatch.log and the final conforming retries are preserved here. No Phase 2 retry
or replacement panel was run.

The machine-readable status record is
../../../blocked/2026-07-27-ms00_clean-post-r2.json.

ARCHITECTURE (`docs/ARCHITECTURE.md`) Copy Section

ARS Pipeline Architecture (v3.23.0)

Full pipeline view across stages × skills × artifacts × gates. Every completed stage requires a user-confirmation checkpoint (per academic-pipeline/SKILL.md and pipeline_state_machine.md); the diagrams below surface the decision-heavy checkpoints visually so they are easy to locate. The post-stage confirmation checkpoints at 2.5 and 4.5 are machine-verified first, then confirmed by the user — they are not skipped.

How to read Copy Chapter

• Flow diagram (§2): macro view — which stage follows which, where loops exist, where gates block. Every rectangle ends in a post-stage user confirmation (elided for readability); 🧑 markers call out the decision-heavy moments where the user chooses a branch.
• Matrix (§3): the only place where (stage × skill × mode × data_level × artifacts × agents × gate) all co-exist. Use this when asking "what happens at Stage X?" The Gate column lists both machine checks and the user-confirmation checkpoint that closes the stage.
• Data access flow (§4) and skill graph (§6): orthogonal views answering "who sees what" and "who depends on what" respectively.
• Literature corpus flow (§5): producer/consumer view of the optional Material Passport literature_corpus[] input port (v3.6.4) and Phase 1 consumer integration (v3.6.5).
• Quality gates (§7): zoom on the blocking checks — both machine-enforced and human-enforced. §7.1 classifies the repo's CI workflows by enforcement strength (blocking / advisory / administrative / post-push detection).
• Timeline (§8): why the architecture looks the way it does — each release added one honesty primitive or a new contract.
• Modes (§9): reference when composing a pipeline invocation.

The matrix alone is insufficient: it hides data-access hierarchy and skill dependency. The diagrams alone are insufficient: they hide artifact flow and per-stage agent detail. Together they are the full architecture.

1. Checkpoints (at-a-glance) Copy Chapter

The pipeline has two classes of user checkpoint. Both require the user to confirm before the pipeline advances; they differ in what the user is actually deciding.

Decision-heavy checkpoints — the user chooses a branch or accepts a material decision:

Post-stage confirmation checkpoints — machine verification runs first; the user then acknowledges the integrity report before proceeding. These are also user-gated (per pipeline_state_machine.md — every stage ends in [checkpoint]), but the decision is "acknowledge the automated report" rather than "choose a branch":

2. Pipeline Flow Copy Chapter

/* Detailed source-code truncated for AI context efficiency. */

Legend:

• Solid red (🧑) = decision-heavy human gate — the user chooses a branch or approves a material decision.
• Solid orange (✓) = integrity gate — machine verification runs first, user then acknowledges the report. Not skipped.
• Green = Socratic coaching sub-stage. User may engage or say "just fix it" to skip the dialogue.
• 👁 observer (v3.5.0) = collaboration_depth_agent dispatches at every FULL/SLIM checkpoint + pipeline completion. Never blocks. Advisory only. MANDATORY integrity gates (2.5 / 4.5) explicitly skip the observer so compliance checks are not diluted.
• Data level (§3 column) = the data layer of that stage's output artifacts (a postcondition), not the owning skill's declared data_access_level (see §4 — e.g. academic-pipeline declares raw, while its gate stages consume unverified drafts and produce the verified artifacts their rows are labeled by).

3. Stage × Dimension Matrix Copy Chapter

4. Data Access Level Flow (v3.3.2+) Copy Chapter

flowchart LR
    User[User input<br/>web / PDFs / queries / pasted drafts]
    Raw[deep-research<br/>data_access_level: raw]
    Red[academic-paper<br/>data_access_level: raw]
    Ver1[academic-paper-reviewer<br/>data_access_level: raw]
    Orch[academic-pipeline<br/>data_access_level: raw]

    User --> Raw
    User -- standalone modes: ungated drafts,<br/>reviewer comments, search-fills-gap --> Red
    User -- standalone /ars-reviewer:<br/>ungated pasted manuscript --> Ver1
    User -- Stage 1 request / mid-entry paper --> Orch
    Raw -- source_verification elevates artifacts --> Red
    Red -- Gate 2.5: 7-mode integrity --> Ver1
    Orch -. orchestrates .-> Raw
    Orch -. orchestrates .-> Red
    Orch -. orchestrates .-> Ver1

    classDef raw fill:#fff1f0,stroke:#cf1322
    class Raw,Red,Ver1,Orch raw

Rules (per shared/ground_truth_isolation_pattern.md):

• data_access_level is a declarative annotation, not a runtime-enforced permission system. The CI lint scripts/check_data_access_level.py pins each skill's value and confirms the vocabulary; it does not inspect context windows at runtime.
• raw skills consume layer-1 data (arbitrary, possibly adversarial).
• redacted (sanitized material, no new raw ingestion) and verified_only
  (runs only after upstream integrity gates) remain in the legal vocabulary, but
  since #773 no top-level skill qualifies for either: the annotation takes the
  dirtiest input across all modes and entry paths, and every skill has at
  least one legitimate Layer-1 entry. Artifact-level elevation is tracked per
  stage output in the §3 Data-level column, not per skill.
• academic-pipeline is raw (#756) because Stage 1 accepts raw user requests and
  mid-entry accepts raw existing papers — the integrity gates run inside the
  pipeline, downstream of its intake.
• academic-paper is raw (#773) because its standalone modes ingest ungated
  user drafts and third-party reviewer comments, and
  literature_strategist_agent's search-fills-gap flow ingests external-index
  search results inside the skill. The former redacted described the
  orchestrated pipeline path, where Stage 2 inputs arrive as Stage-1 sanitized
  artifacts (Gate 2.5 runs after Stage 2, not before it).
• academic-paper-reviewer is raw (#773) because the standalone
  /ars-reviewer entry legitimately consumes an ungated pasted manuscript —
  the former verified_only was at best true for the pipeline's initial
  Stage 3 dispatch (post-Gate-2.5; Stage 3' re-review consumes a freshly
  revised manuscript before Stage 4.5). That Stage 3 sequencing is unchanged.
• The reviewer side may hold a rubric privately — the key guarantee is that rubric / gold-label content must not be present in the candidate-generating agent's context. Calibration gold sets are runtime-supplied by the human researcher, not bundled into the repository.
• Stage 2.5 and Stage 4.5 (plus the user's review at each gate) are the actual enforcement points. This pattern document explains the data-flow structure that makes those gates meaningful; it is not itself a runtime lock.