AI-Research-SKILLs Documentation

Comprehensive open-source library of AI research and engineering skills for any AI model. Package the skills and your claude code/codex/gemini agent will be...

13,836 stars TeX 22 files ยท ~50,918 tokens #ai#ai-research#claude#claude-code#claude-skills#codex#gemini#gpt-5
RAW Doc

Technical Documentation: Orchestra-Research/AI-Research-SKILLs

โ„น๏ธ Provenance: Multi-Tier Ingestion ยท CodeWiki Reference

Overview (`README.md`)

AI Research `Skills` Library

The most comprehensive open-source skills library enabling AI agents to autonomously conduct AI research โ€” from idea to paper

View All 23 Categories

Table of Contents

Our Mission

We enable AI agents to autonomously conduct AI research โ€” from literature survey and idea generation through experiment execution to paper writing. The library provides both the research orchestration layer (autoresearch, ideation, paper writing) and the engineering skills (training, evaluation, deployment) needed at each stage.

Path Towards AI Research Agent

Modern AI research requires mastering dozens of specialized tools and frameworks.
AI Researchers spend more time debugging infrastructure than testing hypotheses โ€” slowing the pace of scientific discovery.
We provide a comprehensive skills library that enables AI agents to autonomously conduct the full research lifecycle โ€” from brainstorming ideas to writing the paper.

  • Autonomous Research - The autoresearch skill orchestrates the entire research workflow using a two-loop architecture, routing to domain skills as needed
  • Specialized Expertise - Each domain skill provides deep, production-ready knowledge of a specific framework (Megatron-LM, vLLM, TRL, etc.)
  • End-to-End Coverage - 98 skills spanning the full AI research lifecycle, from ideation and literature survey to experiments and paper writing
  • Research-Grade Quality - Documentation sourced from official repos, real GitHub issues, and battle-tested production workflows

Available AI Research Engineering Skills

Quality over quantity: Each skill provides comprehensive, expert-level guidance with real code examples, troubleshooting guides, and production-ready workflows.

For humans โ€” interactive installer with one command:

bash
npx @orchestra-research/ai-research-skills

For AI agents โ€” point your agent to the welcome doc and it handles the rest:

text
Read https://www.orchestra-research.com/ai-research-skills/welcome.md and follow the instructions to install and use AI Research Skills.

This installs all 98 skills, loads the autoresearch orchestration layer, and starts autonomous research.

What the installer does

  • Auto-detects your installed coding agents (Claude Code, Hermes Agent, OpenCode, Qoder, Cursor, Gemini CLI, etc.)
  • Installs skills to ~/.orchestra/skills/ with symlinks to each agent (falls back to copy on Windows)
  • Offers everything, quickstart bundle, by category, or individual skills
  • Updates installed skills with latest versions
  • Uninstalls all or selected skills

CLI Commands

bash
# Interactive installer (recommended)
npx @orchestra-research/ai-research-skills

# Direct commands
npx @orchestra-research/ai-research-skills list      # View installed skills
npx @orchestra-research/ai-research-skills update    # Update installed skills

Claude Code Marketplace (Alternative)

Install skill categories directly using the Claude Code CLI:

bash
# Add the marketplace
/plugin marketplace add orchestra-research/AI-research-SKILLs

# Install by category (23 categories available)
/plugin install fine-tuning@ai-research-skills        # Axolotl, LLaMA-Factory, PEFT, Unsloth
/plugin install post-training@ai-research-skills      # TRL, GRPO, OpenRLHF, SimPO, verl, slime, miles, torchforge
/plugin install inference-serving@ai-research-skills  # vLLM, TensorRT-LLM, llama.cpp, SGLang
/plugin install distributed-training@ai-research-skills
/plugin install optimization@ai-research-skills

All 23 Categories (98 Skills)

Category Skills Included
Autoresearch 1 Autonomous research orchestration โ€” central layer that manages the full lifecycle and routes to all other skills
Ideation 2 Research Brainstorming, Creative Thinking
ML Paper Writing 2 ML Paper Writing (LaTeX templates, citation verification), Academic Plotting
Model Architecture 5 LitGPT, Mamba, NanoGPT, RWKV, TorchTitan
Tokenization 2 HuggingFace Tokenizers, SentencePiece
Fine-Tuning 4 Axolotl, LLaMA-Factory, PEFT, Unsloth
Mech Interp 4 TransformerLens, SAELens, pyvene, nnsight
Data Processing 2 NeMo Curator, Ray Data
Post-Training 8 TRL, GRPO, OpenRLHF, SimPO, verl, slime, miles, torchforge
Safety 4 Constitutional AI, LlamaGuard, NeMo Guardrails, Prompt Guard
Distributed 6 DeepSpeed, FSDP, Accelerate, Megatron-Core, Lightning, Ray Train
Infrastructure 3 Modal, Lambda Labs, SkyPilot
Optimization 6 Flash Attention, bitsandbytes, GPTQ, AWQ, HQQ, GGUF
Evaluation 3 lm-eval-harness, BigCode, NeMo Evaluator
Inference 4 vLLM, TensorRT-LLM, llama.cpp, SGLang
MLOps 3 W&B, MLflow, TensorBoard
Agents 4 LangChain, LlamaIndex, CrewAI, AutoGPT
RAG 5 Chroma, FAISS, Pinecone, Qdrant, Sentence Transformers
Prompt Eng 4 DSPy, Instructor, Guidance, Outlines
Observability 2 LangSmith, Phoenix
Multimodal 7 CLIP, Whisper, LLaVA, BLIP-2, SAM, Stable Diffusion, AudioCraft
Emerging 6 MoE, Model Merging, Long Context, Speculative Decoding, Distillation, Pruning
Agent-Native Research Artifact 3 ARA Compiler, Research Manager, Rigor Reviewer

View All 98 Skills in Details

๐Ÿ”ฌ Autoresearch (1 skill) โ€” Central Orchestration Layer

  • Autoresearch - Autonomous research orchestration using a two-loop architecture (inner optimization + outer synthesis). Manages the full lifecycle from literature survey to paper writing, routing to all domain-specific skills. Supports Claude Code /loop and OpenClaw heartbeat for continuous operation (390 lines + 3 refs)

๐Ÿ—๏ธ Model Architecture (5 skills)

  • LitGPT - Lightning AI's 20+ clean LLM implementations with production training recipes (462 lines + 4 refs)
  • Mamba - State-space models with O(n) complexity, 5ร— faster than Transformers (253 lines + 3 refs)
  • RWKV - RNN+Transformer hybrid, infinite context, Linux Foundation project (253 lines + 3 refs)
  • NanoGPT - Educational GPT in ~300 lines by Karpathy (283 lines + 3 refs)
  • TorchTitan - PyTorch-native distributed training for Llama 3.1 with 4D parallelism

๐Ÿ”ค Tokenization (2 skills)

  • HuggingFace Tokenizers - Rust-based, <20s/GB, BPE/WordPiece/Unigram algorithms (486 lines + 4 refs)
  • SentencePiece - Language-independent, 50k sentences/sec, used by T5/ALBERT (228 lines + 2 refs)

๐ŸŽฏ Fine-Tuning (4 skills)

  • Axolotl - YAML-based fine-tuning with 100+ models (156 lines + 4 refs)
  • LLaMA-Factory - WebUI no-code fine-tuning (78 lines + 5 refs)
  • Unsloth - 2x faster QLoRA fine-tuning (75 lines + 4 refs)
  • PEFT - Parameter-efficient fine-tuning with LoRA, QLoRA, DoRA, 25+ methods (431 lines + 2 refs)

๐Ÿ”ฌ Mechanistic Interpretability (4 skills)

  • TransformerLens - Neel Nanda's library for mech interp with HookPoints, activation caching (346 lines + 3 refs)
  • SAELens - Sparse Autoencoder training and analysis for feature discovery (386 lines + 3 refs)
  • pyvene - Stanford's causal intervention library with declarative configs (473 lines + 3 refs)
  • nnsight - Remote interpretability via NDIF, run experiments on 70B+ models (436 lines + 3 refs)

๐Ÿ“Š Data Processing (2 skills)

  • Ray Data - Distributed ML data processing, streaming execution, GPU support (318 lines + 2 refs)
  • NeMo Curator - GPU-accelerated data curation, 16ร— faster deduplication (375 lines + 2 refs)

๐ŸŽ“ Post-Training (8 skills)

  • TRL Fine-Tuning - Transformer Reinforcement Learning (447 lines + 4 refs)
  • GRPO-RL-Training (TRL) - Group Relative Policy Optimization with TRL (569 lines, gold standard)
  • OpenRLHF - Full RLHF pipeline with Ray + vLLM (241 lines + 4 refs)
  • SimPO - Simple Preference Optimization, no reference model needed (211 lines + 3 refs)
  • verl - ByteDance's HybridFlow RL framework, FSDP/Megatron + vLLM/SGLang backends (389 lines + 2 refs)
  • slime - THUDM's Megatron+SGLang framework powering GLM-4.x models (464 lines + 2 refs)
  • miles - Enterprise fork of slime with FP8, INT4, speculative RL for MoE training (315 lines + 2 refs)
  • torchforge - Meta's PyTorch-native RL with Monarch+TorchTitan+vLLM (380 lines + 2 refs)

๐Ÿ›ก๏ธ Safety & Alignment (4 skills)

  • Constitutional AI - AI-driven self-improvement via principles (282 lines)
  • LlamaGuard - Safety classifier for LLM inputs/outputs (329 lines)
  • NeMo Guardrails - Programmable guardrails with Colang (289 lines)
  • Prompt Guard - Meta's 86M prompt injection & jailbreak detector, 99%+ TPR, <2ms GPU (313 lines)

โšก Distributed Training (6 skills)

  • Megatron-Core - NVIDIA's framework for training 2B-462B param models with 47% MFU on H100 (359 lines + 4 refs)
  • DeepSpeed - Microsoft's ZeRO optimization (137 lines + 9 refs)
  • PyTorch FSDP2 - Fully Sharded Data Parallel v2 with fully_shard and DTensor (231 lines + 12 refs)
  • Accelerate - HuggingFace's 4-line distributed training API (324 lines + 3 refs)
  • PyTorch Lightning - High-level training framework with Trainer class (339 lines + 3 refs)
  • Ray Train - Multi-node orchestration and hyperparameter tuning (399 lines + 1 ref)

๐Ÿš€ Optimization (6 skills)

  • Flash Attention - 2-4x faster attention with memory efficiency (359 lines + 2 refs)
  • bitsandbytes - 8-bit/4-bit quantization for 50-75% memory reduction (403 lines + 3 refs)
  • GPTQ - 4-bit post-training quantization, 4ร— memory reduction, <2% accuracy loss (443 lines + 3 refs)
  • AWQ - Activation-aware weight quantization, 4-bit with minimal accuracy loss (310 lines + 2 refs)
  • HQQ - Half-Quadratic Quantization, no calibration data needed, multi-backend (370 lines + 2 refs)
  • GGUF - llama.cpp quantization format, K-quant methods, CPU/Metal inference (380 lines + 2 refs)

๐Ÿ“Š Evaluation (3 skills)

  • lm-evaluation-harness - EleutherAI's standard for benchmarking LLMs across 60+ tasks (482 lines + 4 refs)
  • BigCode Evaluation Harness - Code model benchmarking with HumanEval, MBPP, MultiPL-E, pass@k metrics (406 lines + 3 refs)
  • NeMo Evaluator - NVIDIA's enterprise platform for 100+ benchmarks across 18+ harnesses with multi-backend execution (454 lines + 4 refs)

โ˜๏ธ Infrastructure (3 skills)

  • Modal - Serverless GPU cloud with Python-native API, T4-H200 on-demand (342 lines + 2 refs)
  • SkyPilot - Multi-cloud orchestration across 20+ providers with spot recovery (390 lines + 2 refs)
  • Lambda Labs - Reserved/on-demand GPU cloud with H100/A100, persistent filesystems (390 lines + 2 refs)

๐Ÿ”ฅ Inference & Serving (4 skills)

  • vLLM - High-throughput LLM serving with PagedAttention (356 lines + 4 refs, production-ready)
  • TensorRT-LLM - NVIDIA's fastest inference, 24k tok/s, FP8/INT4 quantization (180 lines + 3 refs)
  • llama.cpp - CPU/Apple Silicon inference, GGUF quantization (251 lines + 3 refs)
  • SGLang - Structured generation with RadixAttention, 5-10ร— faster for agents (435 lines + 3 refs)

๐Ÿค– Agents (4 skills)

  • LangChain - Most popular agent framework, 500+ integrations, ReAct pattern (658 lines + 3 refs, production-ready)
  • LlamaIndex - Data framework for LLM apps, 300+ connectors, RAG-focused (535 lines + 3 refs)
  • CrewAI - Multi-agent orchestration, role-based collaboration, autonomous workflows (498 lines + 3 refs)
  • AutoGPT - Autonomous AI agent platform, visual workflow builder, continuous execution (400 lines + 2 refs)

๐Ÿ” RAG (5 skills)

  • Chroma - Open-source embedding database, local/cloud, 24k stars (385 lines + 1 ref)
  • FAISS - Facebook's similarity search, billion-scale, GPU acceleration (295 lines)
  • Sentence Transformers - 5000+ embedding models, multilingual, 15k stars (370 lines)
  • Pinecone - Managed vector database, auto-scaling, <100ms latency (410 lines)
  • Qdrant - High-performance vector search, Rust-powered, hybrid search with filtering (493 lines + 2 refs)

๐ŸŽจ Multimodal (7 skills)

  • CLIP - OpenAI's vision-language model, zero-shot classification, 25k stars (320 lines)
  • Whisper - Robust speech recognition, 99 languages, 73k stars (395 lines)
  • LLaVA - Vision-language assistant, image chat, GPT-4V level (360 lines)
  • Stable Diffusion - Text-to-image generation via HuggingFace Diffusers, SDXL, ControlNet (380 lines + 2 refs)
  • Segment Anything - Meta's SAM for zero-shot image segmentation with points/boxes (500 lines + 2 refs)
  • BLIP-2 - Vision-language pretraining with Q-Former, image captioning, VQA (500 lines + 2 refs)
  • AudioCraft - Meta's MusicGen/AudioGen for text-to-music and text-to-sound (470 lines + 2 refs)

๐ŸŽฏ Prompt Engineering (4 skills)

  • DSPy - Declarative prompt programming with optimizers, Stanford NLP, 22k stars (438 lines + 3 refs)
  • Instructor - Structured LLM outputs with Pydantic validation, 15k stars (726 lines + 3 refs)
  • Guidance - Constrained generation with regex/grammars, Microsoft Research, 18k stars (485 lines + 3 refs)
  • Outlines - Structured text with FSM, zero-overhead, 8k stars (601 lines + 3 refs)

๐Ÿ“Š MLOps (3 skills)

  • Weights & Biases - Experiment tracking, sweeps, artifacts, model registry (427 lines + 3 refs)
  • MLflow - Model registry, tracking, deployment, autologging (514 lines + 3 refs)
  • TensorBoard - Visualization, profiling, embeddings, scalars/images (538 lines + 3 refs)

๐Ÿ‘๏ธ Observability (2 skills)

  • LangSmith - LLM observability, tracing, evaluation, monitoring for AI apps (422 lines + 2 refs)
  • Phoenix - Open-source AI observability with OpenTelemetry tracing and LLM evaluation (380 lines + 2 refs)

๐Ÿ”ฌ Emerging Techniques (6 skills)

  • MoE Training - Mixture of Experts training with DeepSpeed, Mixtral 8x7B, 5ร— cost reduction (515 lines + 3 refs)
  • Model Merging - Combine models with TIES, DARE, SLERP using mergekit (528 lines + 3 refs)
  • Long Context - Extend context windows with RoPE, YaRN, ALiBi, 32k-128k tokens (624 lines + 3 refs)
  • Speculative Decoding - 1.5-3.6ร— faster inference with Medusa, Lookahead (379 lines)
  • Knowledge Distillation - Compress models 70Bโ†’7B with MiniLLM, temperature scaling (424 lines)
  • Model Pruning - 50% sparsity with Wanda, SparseGPT, <1% accuracy loss (417 lines)

๐Ÿ“ ML Paper Writing (2 skills)

  • ML Paper Writing - Write publication-ready papers for NeurIPS, ICML, ICLR, ACL, AAAI, COLM with LaTeX templates, citation verification, and writing best practices (532 lines + 5 refs)
  • Academic Plotting - Generate publication-quality figures for ML papers: architecture diagrams via Gemini AI and data-driven charts via matplotlib/seaborn with venue-specific styling (479 lines + 3 refs)

๐Ÿ’ก Ideation (2 skills)

  • Research Brainstorming - Structured ideation frameworks for discovering high-impact research directions with 10 complementary lenses (384 lines)
  • Creative Thinking - Cognitive science frameworks (bisociation, structure-mapping, constraint manipulation) for genuinely novel research ideas (366 lines)

๐Ÿงฌ Agent-Native Research Artifact (3 skills)

  • ARA Compiler - Compiles any research input (PDF papers, repos, experiment logs, raw notes) into a complete Agent-Native Research Artifact with claims, exploration graph, evidence, and code stubs (245 lines + 3 refs)
  • ARA Research Manager - Post-task research recorder that runs at session end to extract decisions, experiments, dead ends, and pivots from conversation history into the ara/ directory with user-vs-AI provenance tags (324 lines + 3 refs)
  • ARA Rigor Reviewer - ARA Seal Level 2 semantic epistemic review scoring six dimensions of research rigor (evidence relevance, falsifiability, scope, coherence, exploration integrity, methodology) with severity-ranked findings (322 lines + 1 ref)

Demos

All 98 skills in this repo are automatically synced to Orchestra Research, where you can add them to your projects with one click and use them with AI research agents.

See skills in action โ†’ demos/

We maintain a curated collection of demo repositories showing how to use skills for real AI research tasks:

Demo Skills Used What It Does
Norm Heterogeneity โ†’ LoRA Brittleness Autoresearch, ML Paper Writing, Ideation Agent autonomously discovered norm heterogeneity predicts fine-tuning difficulty (r=-0.99), pivoting from a null result on ETF overlaps
RL Algorithm Brain Scan Autoresearch, GRPO, TRL, SAELens, TransformerLens, ML Paper Writing Agent found DPO is a rank-1 perturbation (95.6% recovery from one SVD direction) while online RL is distributed and structure-preserving
NeMo Eval: GPQA Benchmark NeMo Evaluator Compare Llama 8B/70B/405B on graduate-level science questions
LoRA Without Regret Reproduction GRPO, TRL Reproduce SFT + GRPO RL experiments via prompting
Layer-Wise Quantization Experiment llama.cpp, GGUF Investigate optimal layer precision allocationโ€”early layers at Q8 achieve 1.9ร— compression with 1.3% perplexity loss
Cross-Lingual Alignment Analysis FAISS Quantify how well multilingual embeddings align semantic concepts across 8 languages using FAISS similarity search
Scientific Plotting Demo Academic Plotting Generate publication-quality figures for the Andes QoE-aware LLM serving paper โ€” Gemini AI architecture diagrams + matplotlib data charts (CDF, multi-panel grids, bar charts)

Featured Demos: Two papers produced entirely by AI agents using the autoresearch skill. The Norm Heterogeneity paper demonstrates autonomous research pivoting โ€” the agent refuted its own hypothesis and discovered a stronger finding. The RL Brain Scan paper demonstrates multi-skill orchestration โ€” the agent trained RL models, analyzed internals with interpretability tools, and synthesized the insight that "DPO is rank-1 alignment." Both papers written end-to-end by the agent.

Skill Structure

Each skill follows a battle-tested format for maximum usefulness:

text
skill-name/
โ”œโ”€โ”€ SKILL.md                    # Quick reference (50-150 lines)
โ”‚   โ”œโ”€โ”€ Metadata (name, description, version)
โ”‚   โ”œโ”€โ”€ When to use this skill
โ”‚   โ”œโ”€โ”€ Quick patterns & examples
โ”‚   โ””โ”€โ”€ Links to references
โ”‚
โ”œโ”€โ”€ references/                 # Deep documentation (300KB+)
โ”‚   โ”œโ”€โ”€ README.md              # From GitHub/official docs
โ”‚   โ”œโ”€โ”€ api.md                 # API reference
โ”‚   โ”œโ”€โ”€ tutorials.md           # Step-by-step guides
โ”‚   โ”œโ”€โ”€ issues.md              # Real GitHub issues & solutions
โ”‚   โ”œโ”€โ”€ releases.md            # Version history & breaking changes
โ”‚   โ””โ”€โ”€ file_structure.md      # Codebase navigation
โ”‚
โ”œโ”€โ”€ scripts/                    # Helper scripts (optional)
โ””โ”€โ”€ assets/                     # Templates & examples (optional)

Quality Standards

  • 300KB+ documentation from official sources
  • Real GitHub issues & solutions (when available)
  • Code examples with language detection
  • Version history & breaking changes
  • Links to official docs

Roadmap

The library spans 98 comprehensive skills across the full AI research lifecycle. See our detailed roadmap for the complete development plan.

View Full Roadmap โ†’

View Detailed Statistics

Metric Current Target
Skills 87 (high-quality, standardized YAML) 80 โœ…
Avg Lines/Skill 420 lines (focused + progressive disclosure) 200-600 lines
Documentation ~130,000 lines total (SKILL.md + references) 100,000+ lines
Gold Standard Skills 65 with comprehensive references 50+
Contributors 1 100+
Coverage Architecture, Tokenization, Fine-Tuning, Mechanistic Interpretability, Data Processing, Post-Training, Safety, Distributed, Optimization, Evaluation, Infrastructure, Inference, Agents, RAG, Multimodal, Prompt Engineering, MLOps, Observability, ML Paper Writing, Ideation, Autoresearch Full Lifecycle โœ…

Recent Progress: npm package @orchestra-research/ai-research-skills for one-command installation across all coding agents

Philosophy: Quality > Quantity. Following Anthropic official best practices - each skill provides 200-500 lines of focused, actionable guidance with progressive disclosure.

Repository Structure

text
/* Detailed source-code truncated for AI context efficiency. */

Use Cases

For Researchers

"I need to fine-tune Llama 3 with custom data"
โ†’ 03-fine-tuning/axolotl/ - YAML configs, 100+ model support

For ML Engineers

"How do I optimize inference latency?"
โ†’ 12-inference-serving/vllm/ - PagedAttention, batching

For Students

"I want to learn how transformers work"
โ†’ 01-model-architecture/litgpt/ - Clean implementations

For Teams

"We need to scale training to 100 GPUs"
โ†’ 08-distributed-training/deepspeed/ - ZeRO stages, 3D parallelism

License

MIT License - See LICENSE for details.

Note: Individual skills may reference libraries with different licenses. Please check each project's license before use.

Citation

If you use AI Research Skills in your work or find it helpful for a publication, we'd appreciate a citation:

BibTeX

bibtex
@software{ai_research_skills,
  title     = {AI Research Skills Library},
  author    = {{Orchestra Research}},
  year      = {2025},
  url       = {https://github.com/orchestra-research/AI-research-SKILLs},
  note      = {Open-source skills library enabling AI agents to autonomously conduct AI research}
}

APA

Orchestra Research. (2025). AI Research Skills Library [Computer software]. https://github.com/orchestra-research/AI-research-SKILLs

Chicago

Orchestra Research. "AI Research Skills Library." GitHub, 2025. https://github.com/orchestra-research/AI-research-SKILLs.

IEEE

Orchestra Research, "AI Research Skills Library," 2025. [Online]. Available: https://github.com/orchestra-research/AI-research-SKILLs

Tip: You can also click "Cite this repository" in the GitHub sidebar for auto-formatted citations.

Acknowledgments

Built with:

  • Claude Code - AI pair programming
  • Skill Seeker - Automated doc scraping
  • Open Source AI Community - For amazing tools and docs

Special thanks to:

  • EleutherAI, HuggingFace, NVIDIA, Lightning AI, Meta AI, Anthropic
  • All researchers who maintain excellent documentation

Contributors

Thanks to all the people who have contributed to the AI Research Skills Library:

We welcome contributions from the AI research community! See CONTRIBUTING.md for detailed guidelines on:

  • Adding new skills
  • Improving existing skills
  • Quality standards and best practices
  • Submission process

Recent Updates

June 2026 - v1.7.1 ๐Ÿงน Inventory Consistency, Drift Guard & Security Hardening

  • ๐Ÿ“Š Repo-wide inventory reconciled to 98 skills / 23 categories โ€” corrected stale counts that had drifted apart across files: CLAUDE.md (said 90), CONTRIBUTING.md (said 86/22), the README sync line (said 87), WELCOME.md and the npm package README (said 86/22). Also fixed wrong per-category listings (TorchTitan, SwanLab, A-Evolve, ML Training Recipes, Cosmos Policy/OpenPI/OpenVLA-OFT, the paper-writing skills)
  • ๐Ÿ›ก๏ธ New CI drift guard โ€” scripts/check-inventory.sh + check-inventory.yml fail CI whenever the documented skill/category counts diverge from the actual SKILL.md count on disk, so the inventory can't silently drift again
  • ๐Ÿ“ฆ Marketplace sync hardening โ€” sync-skills.yml now prunes build artifacts (node_modules/__pycache__/*.pyc/.ipynb_checkpoints) before zipping and fails loudly above 190 files instead of hitting the marketplace's 200-file rejection
  • ๐Ÿ”’ Security โ€” pinned CLI dependencies (chalk/inquirer/ora) to exact, patched versions with a regenerated lockfile ([email protected] clears the tmp path-traversal advisory; npm audit now reports 0 vulnerabilities)
  • ๐Ÿงน Full open PR/issue triage pass against scope + contribution standards

April 2026 - v1.6.0 ๐Ÿงฌ Agent-Native Research Artifact (ARA) โ€” 23rd Category, 98 Skills

  • ๐Ÿงฌ NEW CATEGORY: 22-agent-native-research-artifact/ (the 23rd category) โ€” three skills that turn research outputs into a falsifiable, agent-traversable artifact:
    • ๐Ÿ› ๏ธ ARA Compiler โ€” compiles any input (PDF papers, GitHub repos, experiment logs, raw notes) into a structured ARA with cognitive layer (claims, concepts, heuristics), physical layer (configs, code stubs), exploration graph (research DAG), and grounded evidence
    • ๐Ÿ“‹ ARA Research Manager โ€” post-task epilogue that scans conversation history at session end and writes decisions, experiments, dead ends, claims, heuristics, and pivots into the ara/ directory with user / ai-suggested / ai-executed / user-revised provenance tags
    • ๐Ÿ” ARA Rigor Reviewer โ€” Seal Level 2 semantic epistemic review scoring six dimensions of research rigor (evidence relevance, falsifiability, scope calibration, argument coherence, exploration integrity, methodological rigor) and emitting a severity-ranked report with a Strong Accept-to-Reject recommendation
  • ๐Ÿ”— Sourced from the Agent-Native-Research-Artifact-Init reference repo, restructured to AI-research-SKILLs standards (kebab-case names, third-person descriptions, Title-Case tags, one-level-deep references)
  • ๐Ÿงฉ Plugin entry agent-native-research-artifact added to .claude-plugin/marketplace.json; CLI category registered as 22-agent-native-research-artifact with three individual skill entries in the npm installer
  • ๐Ÿ”„ Auto-syncs to Orchestra marketplace via sync-skills.yml on push; npm package republished as @orchestra-research/[email protected] via publish-npm.yml on version bump
  • ๐Ÿ“Š 98 total skills across 23 categories โ€” full lifecycle from idea โ†’ paper โ†’ falsifiable, auditable artifact

March 2026 - v1.4.0 ๐Ÿ”ฌ Autoresearch & 86 Skills โ€” Full Research Lifecycle

  • ๐Ÿ”ฌ NEW SKILL: Autoresearch โ€” autonomous research orchestration using a two-loop architecture (inner optimization loop + outer synthesis loop)
  • ๐Ÿง  Manages the full research lifecycle: literature survey โ†’ ideation โ†’ experiments โ†’ synthesis โ†’ paper writing
  • ๐Ÿ”„ Routes to all 86 domain skills automatically โ€” agents don't need to know which skill to use
  • โฐ Mandatory /loop (Claude Code) and cron job (OpenClaw) for continuous autonomous operation
  • ๐Ÿ“Š Generates research presentations (HTML/PDF) with optimization trajectory plots for human review
  • ๐Ÿ“ Findings.md as persistent project memory across sessions with "Lessons and Constraints" tracking
  • ๐Ÿ—‚๏ธ Structured workspace: research-state.yaml, findings.md, research-log.md, literature/, experiments/, src/, data/, to_human/
  • ๐Ÿ“„ Two demo papers produced by autoresearch: Norm Heterogeneity โ†’ LoRA Brittleness and RL Algorithm Brain Scan
  • ๐Ÿš€ WELCOME.md for cold-start agent bootstrap โ€” one URL to go from zero to autonomous research
  • ๐Ÿ“ฆ npm v1.4.x with Windows symlink fallback, all 22 categories installable
  • ๐Ÿค– Supported agents: Claude Code, Hermes Agent, OpenCode, OpenClaw, Qoder, Cursor, Codex, Gemini CLI, Qwen Code
  • ๐Ÿ“Š 87 total skills across 22 categories โ€” complete research lifecycle coverage

February 2026 - v0.15.0 ๐Ÿ›ก๏ธ Prompt Guard & 83 Skills

  • ๐Ÿ›ก๏ธ NEW SKILL: Prompt Guard - Meta's 86M prompt injection & jailbreak detector
  • โšก 99%+ TPR, <1% FPR, <2ms GPU latency, multilingual (8 languages)
  • ๐Ÿ”’ 3 workflows: user input filtering, third-party data filtering, batch RAG processing
  • ๐Ÿ“Š 83 total skills across 20 categories

January 2026 - v0.14.0 ๐Ÿ“ฆ npm Package & 82 Skills

  • ๐Ÿ“ฆ NEW: npx @orchestra-research/ai-research-skills - One-command installation for all coding agents
  • ๐Ÿค– Supported agents: Claude Code, OpenCode, Cursor, Codex, Gemini CLI, Qwen Code
  • โœจ Interactive installer with category/individual skill selection
  • ๐Ÿ”„ Update installed skills, selective uninstall
  • ๐Ÿ“Š 82 total skills (5 new post-training skills: verl, slime, miles, torchforge + TorchTitan)
  • ๐Ÿ—๏ธ Megatron-Core moved to Distributed Training category

January 2026 - v0.13.0 ๐Ÿ“ ML Paper Writing & Demos Gallery

  • ๐Ÿ“ NEW CATEGORY: ML Paper Writing (20th category, 77th skill)
  • ๐ŸŽฏ Write publication-ready papers for NeurIPS, ICML, ICLR, ACL, AAAI, COLM
  • ๐Ÿ“š Writing philosophy from top researchers (Neel Nanda, Farquhar, Gopen & Swan, Lipton, Perez)
  • ๐Ÿ”ฌ Citation verification workflow - never hallucinate references
  • ๐Ÿ“„ LaTeX templates for 6 major conferences
  • ๐ŸŽช NEW: Curated demos gallery (demos/) showcasing skills in action
  • ๐Ÿ”— Demo repos: NeMo Evaluator benchmark, LoRA Without Regret reproduction
  • ๐Ÿ“– 936-line comprehensive SKILL.md with 4 workflows

January 2026 - v0.12.0 ๐Ÿ“Š NeMo Evaluator SDK

  • ๐Ÿ“Š NEW SKILL: NeMo Evaluator SDK for enterprise LLM benchmarking
  • ๐Ÿ”ง NVIDIA's evaluation platform with 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM)
  • โšก Multi-backend execution: local Docker, Slurm HPC, Lepton cloud
  • ๐Ÿ“ฆ Container-first architecture for reproducible evaluation
  • ๐Ÿ“ 454 lines SKILL.md + 4 comprehensive reference files (~48KB documentation)

December 2025 - v0.11.0 ๐Ÿ”ฌ Mechanistic Interpretability

  • ๐Ÿ”ฌ NEW CATEGORY: Mechanistic Interpretability (4 skills)
  • ๐Ÿ” TransformerLens skill: Neel Nanda's library for mech interp with HookPoints, activation caching, circuit analysis
  • ๐Ÿง  SAELens skill: Sparse Autoencoder training and analysis for feature discovery, monosemanticity research
  • โšก pyvene skill: Stanford's causal intervention library with declarative configs, DAS, activation patching
  • ๐ŸŒ nnsight skill: Remote interpretability via NDIF, run experiments on 70B+ models without local GPUs
  • ๐Ÿ“ ~6,500 new lines of documentation across 16 files
  • 76 total skills (filling the missing 04 category slot)

November 25, 2025 - v0.10.0 ๐ŸŽ‰ 70 Skills Complete!

  • ๐ŸŽ‰ ROADMAP COMPLETE: Reached 70-skill milestone!
  • ๐Ÿš€ Added 4 skills: Lambda Labs, Segment Anything (SAM), BLIP-2, AudioCraft
  • โ˜๏ธ Lambda Labs skill: Reserved/on-demand GPU cloud with H100/A100, persistent filesystems, 1-Click Clusters
  • ๐Ÿ–ผ๏ธ SAM skill: Meta's Segment Anything for zero-shot image segmentation with points/boxes/masks
  • ๐Ÿ‘๏ธ BLIP-2 skill: Vision-language pretraining with Q-Former, image captioning, VQA
  • ๐ŸŽต AudioCraft skill: Meta's MusicGen/AudioGen for text-to-music and text-to-sound generation
  • ๐Ÿ“ ~10,000 new lines of documentation across 12 files
  • 70 total skills (100% roadmap complete!)

November 25, 2025 - v0.9.0

  • ๐Ÿš€ Added 2 infrastructure skills: Modal, SkyPilot
  • โ˜๏ธ Modal skill: Serverless GPU cloud with Python-native API, T4-H200 on-demand, auto-scaling
  • ๐ŸŒ SkyPilot skill: Multi-cloud orchestration across 20+ providers with spot recovery
  • โœจ New Infrastructure category (2 skills - serverless GPU and multi-cloud orchestration)
  • ๐Ÿ“ ~2,500 new lines of documentation across 6 files
  • 66 total skills (94% towards 70-skill target)

November 25, 2025 - v0.8.0

  • ๐Ÿš€ Added 5 high-priority skills: HQQ, GGUF, Phoenix, AutoGPT, Stable Diffusion
  • โšก HQQ skill: Half-Quadratic Quantization without calibration data, multi-backend support
  • ๐Ÿ“ฆ GGUF skill: llama.cpp quantization format, K-quant methods, CPU/Metal inference
  • ๐Ÿ‘๏ธ Phoenix skill: Open-source AI observability with OpenTelemetry tracing and LLM evaluation
  • ๐Ÿค– AutoGPT skill: Autonomous AI agent platform with visual workflow builder
  • ๐ŸŽจ Stable Diffusion skill: Text-to-image generation via Diffusers, SDXL, ControlNet, LoRA
  • ๐Ÿ“ ~9,000 new lines of documentation across 15 files
  • 64 total skills (91% towards 70-skill target)

November 25, 2025 - v0.7.0

  • ๐Ÿš€ Added 5 high-priority skills: PEFT, CrewAI, Qdrant, AWQ, LangSmith
  • โœจ New Observability category with LangSmith for LLM tracing and evaluation
  • ๐ŸŽฏ PEFT skill: Parameter-efficient fine-tuning with LoRA, QLoRA, DoRA, 25+ methods
  • ๐Ÿค– CrewAI skill: Multi-agent orchestration with role-based collaboration
  • ๐Ÿ” Qdrant skill: High-performance Rust vector search with hybrid filtering
  • โšก AWQ skill: Activation-aware 4-bit quantization with minimal accuracy loss
  • ๐Ÿ“ ~8,000 new lines of documentation across 15 files
  • 59 total skills (84% towards 70-skill target)

November 15, 2025 - v0.6.0

  • ๐Ÿ“Š Added 3 comprehensive MLOps skills: Weights & Biases, MLflow, TensorBoard
  • โœจ New MLOps category (3 skills - experiment tracking, model registry, visualization)
  • ๐Ÿ“ ~10,000 new lines of documentation across 13 files
  • ๐Ÿ”ง Comprehensive coverage: experiment tracking, hyperparameter sweeps, model registry, profiling, embeddings visualization
  • 54 total skills (77% towards 70-skill target)

November 12, 2025 - v0.5.0

  • ๐ŸŽฏ Added 4 comprehensive prompt engineering skills: DSPy, Instructor, Guidance, Outlines
  • โœจ New Prompt Engineering category (4 skills - DSPy, Instructor, Guidance, Outlines)
  • ๐Ÿ“ ~10,000 new lines of documentation across 16 files
  • ๐Ÿ”ง Comprehensive coverage: declarative programming, structured outputs, constrained generation, FSM-based generation
  • 47 total skills (67% towards 70-skill target)

November 9, 2025 - v0.4.0

  • ๐Ÿค– Added 11 comprehensive skills: LangChain, LlamaIndex, Chroma, FAISS, Sentence Transformers, Pinecone, CLIP, Whisper, LLaVA
  • โœจ New Agents category (2 skills - LangChain, LlamaIndex)
  • ๐Ÿ” New RAG category (4 skills - Chroma, FAISS, Sentence Transformers, Pinecone)
  • ๐ŸŽจ New Multimodal category (3 skills - CLIP, Whisper, LLaVA)
  • ๐Ÿ“ ~15,000 new lines of documentation
  • 43 total skills (61% towards 70-skill target)

November 8, 2025 - v0.3.0

  • ๐Ÿš€ Added 8 comprehensive skills: TensorRT-LLM, llama.cpp, SGLang, GPTQ, HuggingFace Tokenizers, SentencePiece, Ray Data, NeMo Curator
  • โšก Completed Inference & Serving category (4/4 skills)
  • ๐Ÿ”ค New Tokenization category (2 skills)
  • ๐Ÿ“Š New Data Processing category (2 skills)
  • ๐Ÿ“ 9,617 new lines of documentation across 30 files
  • 32 total skills (45% towards 70-skill target)

November 6, 2025 - v0.2.0

  • Added 10 skills from GitHub (Megatron-Core, Lightning, Ray Train, etc.)
  • Improved skill structure with comprehensive references
  • Created strategic roadmap to 70 skills
  • Added contribution guidelines

November 3, 2025 - v0.1.0

  • ๐ŸŽ‰ Initial release with 5 fine-tuning skills

Community

Join our community to stay updated, ask questions, and connect with other AI researchers:

  • SkillEvolve Meta-Skill - Connect your agent to the collective intelligence of the community. Captures techniques discovered during sessions and shares them back as curated skills.
  • Slack Community - Chat with the team and other users
  • Twitter/X - Follow for updates and announcements
  • LinkedIn - Connect professionally

Star History

Overview (`demos/README.md`)

Curated collection of demo repositories showcasing skills in action

Each demo is a standalone repository demonstrating how to use specific skills from this library to accomplish real AI research tasks. Demos include complete code, results, analysis, and documentation.

Available Demos

1. NeMo Evaluator: GPQA Diamond Benchmark

Repository: zechenzhangAGI/Nemo-Eval-Skill-Demo

Skills Used: NeMo Evaluator

What It Does:
Compares Llama models (8B, 70B, 405B) on the GPQA Diamond benchmarkโ€”198 graduate-level science questions. Demonstrates end-to-end evaluation workflow using NVIDIA NeMo Evaluator.

Key Results:

Model Accuracy Notes
Llama-3.1-8B-Instruct 27.3% 20.7% extraction failures
Llama-3.3-70B-Instruct 48.0% Clean extraction
Llama-3.1-405B-Instruct 53.0% Best performance

What You'll Learn:

  • Setting up NeMo Evaluator with NVIDIA Build API
  • Writing evaluation configs for different models
  • Analyzing benchmark results across model scales
  • Creating visualizations (accuracy plots, Venn diagrams, failure taxonomy)

Repository Contents:

text
โ”œโ”€โ”€ configs/           # YAML configs for each model
โ”œโ”€โ”€ results/           # Raw evaluation outputs
โ”œโ”€โ”€ analysis/          # Analysis scripts and visualizations
โ”‚   โ”œโ”€โ”€ model_accuracy.png
โ”‚   โ”œโ”€โ”€ failure_taxonomy_plot.png
โ”‚   โ””โ”€โ”€ venn_diagrams.png
โ””โ”€โ”€ README.md          # Full documentation

2. Reproducing "LoRA Without Regret" with AI Agents

Repository: Featured on Orchestra Research Blog

Skills Used: GRPO RL Training, TRL Fine-Tuning

What It Does:
Reproduces Thinking Machines Lab's "LoRA Without Regret" paper findings entirely through prompting an AI agent. The agent autonomously:

  • Writes training code for both SFT and GRPO reinforcement learning
  • Provisions H100 GPUs and runs experiments overnight
  • Performs LoRA rank ablation studies (rank 1 through 256)
  • Generates publication-ready analysis and visualizations

Why It's Impressive:
A researcher simply described the paper they wanted to reproduce, and the AI agent handled everythingโ€”from understanding the methodology to executing multi-day GPU experiments to analyzing results. No manual coding required.

What You'll Learn:

  • How to prompt AI agents for autonomous research reproduction
  • End-to-end SFT and GRPO training pipelines
  • LoRA vs full fine-tuning experimental design
  • Automated analysis and reporting

Resources:

3. Layer-Wise Quantization Experiment

Repository: AmberLJC/llama-quantization-experiment

Skills Used: llama.cpp, GGUF

What It Does:
Investigates optimal layer precision allocation for quantized LLMs. Demonstrates that early layers at Q8 achieve 1.9ร— compression with only 1.3% perplexity lossโ€”showing not all layers are created equal when it comes to quantization.

What You'll Learn:

  • Layer-wise quantization strategies for LLMs
  • Measuring perplexity impact of different precision levels per layer
  • Using llama.cpp and GGUF for quantization experiments
  • Identifying which layers are most sensitive to reduced precision

4. Cross-Lingual Alignment Analysis

Repository: AmberLJC/faiss-demo

Skills Used: FAISS

What It Does:
Quantifies how well multilingual embeddings align semantic concepts across 8 languages using FAISS similarity search. Reveals the structure of cross-lingual representations and where alignment breaks down.

What You'll Learn:

  • Building and querying FAISS indexes for multilingual embeddings
  • Measuring cross-lingual semantic alignment quality
  • Analyzing embedding space structure across languages
  • Using similarity search to evaluate multilingual models

5. Autoresearch: Embedding Norm Heterogeneity Drives LoRA Brittleness

Paper: autoresearch-norm-heterogeneity/

Skills Used: Autoresearch, ML Paper Writing, Research Ideation

What It Does:
An AI agent ran the full autoresearch workflow autonomously. Starting from a hypothesis about ETF crystallization, the agent discovered a null result โ€” ETF overlaps do NOT predict fine-tuning difficulty โ€” then pivoted to identify embedding norm heterogeneity as the actual causal predictor (r=-0.99 at 1.4B scale). The agent wrote the paper end-to-end.

Why It's Impressive:
The research pivot was autonomous. The agent refuted its own starting hypothesis, identified a better predictor, validated it causally (equalizing norms improves fine-tunability by 79%), and wrote a paper with a stronger finding than the original plan.

6. Autoresearch: The RL Algorithm Brain Scan

Paper: autoresearch-rl-brain-scan/

Skills Used: Autoresearch, GRPO RL Training, TRL, SAELens, TransformerLens, ML Paper Writing

What It Does:
An AI agent systematically compared what RLOO, GRPO, and DPO do to model internals using SVD analysis of weight deltas and SAE feature overlap. Key discovery: DPO is a rank-1 perturbation (one SVD direction recovers 95.6% of its behavioral effect), while online RL methods produce distributed, structure-preserving changes.

Why It's Impressive:
The agent orchestrated multiple domain skills (RL training, mechanistic interpretability, paper writing) across the full research lifecycle. The insight that "DPO is rank-1 alignment" is a conceptual contribution that emerged from the outer synthesis loop โ€” not just metric optimization.

7. Scientific Plotting: Publication-Quality Figures

Demo: scientific-plotting-demo/

Skills Used: Academic Plotting

What It Does:
Generates all key figures for the Andes QoE-aware LLM serving paper using both workflows from the academic-plotting skill:

  • Workflow 1 (Gemini AI): System architecture diagram using gemini-3-pro-image-preview with 6-section prompt structure, Style B "Modern Minimal", and Nord palette โ€” 3 non-deterministic attempts with best-of-3 selection
  • Workflow 2 (matplotlib): Five data-driven figures โ€” QoE definition illustration, 3-panel CDF comparison, 4x3 multi-panel burst intensity grid, summary bar charts โ€” all with publication rcParams, colorblind-safe palette, and PDF+PNG export

Key Results:

Metric Result
QoE improvement over vLLM 4.7x
GPU resource savings 61%
Gemini text accuracy 100% (all labels spelled correctly)
Figures generated 6 (1 AI diagram + 5 data charts)

What You'll Learn:

  • Crafting 6-section Gemini prompts for architecture diagrams
  • Multi-attempt generation with evaluation rubric
  • Publication-quality matplotlib figures with venue-specific styling
  • Colorblind-safe palettes, multi-panel layouts, and dual PDF/PNG export

Repository Contents:

text
scientific-plotting-demo/
โ”œโ”€โ”€ README.md                                # Full demo documentation with all figures
โ””โ”€โ”€ figures/
    โ”œโ”€โ”€ gen_fig_andes_architecture_gemini.py  # Gemini AI diagram script
    โ”œโ”€โ”€ gen_fig_andes_workflow.py             # matplotlib architecture alternative
    โ”œโ”€โ”€ gen_fig_experiment_results.py         # Data charts (CDF, grid, bars, QoE)
    โ”œโ”€โ”€ fig_andes_architecture*.png           # Gemini outputs (best + 3 attempts)
    โ”œโ”€โ”€ fig_cdf_comparison.{pdf,png}          # 3-panel CDF
    โ”œโ”€โ”€ fig_burst_intensity.{pdf,png}         # 4x3 multi-panel grid
    โ”œโ”€โ”€ fig_qoe_definition.{pdf,png}          # QoE metric illustration
    โ””โ”€โ”€ fig_summary_improvements.{pdf,png}    # Summary bar charts

Coming Soon

ML Paper Writing: From Repo to Publication

Skills Used: ML Paper Writing

What It Will Do:
Transform a research repository with experimental results into a publication-ready paper for top ML conferences (NeurIPS, ICML, ICLR).

Status: In development

How Demos Are Organized

Each demo repository follows a consistent structure:

text
demo-name/
โ”œโ”€โ”€ README.md              # Overview, results summary, how to run
โ”œโ”€โ”€ configs/               # Configuration files
โ”œโ”€โ”€ results/               # Raw outputs and data
โ”œโ”€โ”€ analysis/              # Scripts and visualizations
โ”œโ”€โ”€ .env.example           # Required environment variables
โ””โ”€โ”€ requirements.txt       # Python dependencies (if applicable)

Design Principles:

  • Self-contained: Clone and run without external dependencies (except API keys)
  • Reproducible: Clear instructions to replicate results
  • Educational: Explains the "why" not just the "how"
  • Real results: Actual outputs, not mock data

Contributing a Demo

Want to showcase a skill? We welcome demo contributions!

Requirements:

  1. Uses one or more skills from this library
  2. Produces meaningful, reproducible results
  3. Includes clear documentation
  4. Has visual outputs (plots, tables, reports)

To contribute:

  1. Create your demo repository
  2. Follow the structure above
  3. Open an issue or PR to add it to this index

Overview (`06-post-training/grpo-rl-training/README.md`)

GRPO/RL Training Skill

Expert-level guidance for Group Relative Policy Optimization with TRL

๐Ÿ“ Skill Structure

text
grpo-rl-training/
โ”œโ”€โ”€ SKILL.md                              # Main skill documentation (READ THIS FIRST)
โ”œโ”€โ”€ README.md                             # This file
โ”œโ”€โ”€ templates/
โ”‚   โ””โ”€โ”€ basic_grpo_training.py            # Production-ready training template
โ””โ”€โ”€ examples/
    โ””โ”€โ”€ reward_functions_library.py       # 20+ reward function examples

๐Ÿš€ Quick Start

  1. Read SKILL.md - Comprehensive guide with all concepts and patterns
  2. Copy templates/basic_grpo_training.py - Start with working code
  3. Browse examples/reward_functions_library.py - Pick reward functions for your task
  4. Modify for your use case - Adapt dataset, rewards, and config

๐Ÿ’ก What's Inside

SKILL.md (Main Documentation)

  • Core GRPO concepts and algorithm fundamentals
  • Complete implementation workflow (dataset โ†’ rewards โ†’ training โ†’ deployment)
  • 10+ reward function examples with code
  • Hyperparameter tuning guide
  • Training insights (loss behavior, metrics, debugging)
  • Troubleshooting guide
  • Production best practices

Templates

  • basic_grpo_training.py: Minimal, production-ready training script
    • Uses Qwen 2.5 1.5B Instruct
    • 3 reward functions (format + correctness)
    • LoRA for efficient training
    • Fully documented and ready to run

Examples

  • reward_functions_library.py: 20+ battle-tested reward functions
    • Correctness rewards (exact match, fuzzy match, numeric, code execution)
    • Format rewards (XML, JSON, strict/soft)
    • Length rewards (ideal length, min/max)
    • Style rewards (reasoning quality, citations, repetition penalty)
    • Combined rewards (multi-objective optimization)
    • Preset collections for common tasks

๐Ÿ“– Usage for Agents

When this skill is loaded in your agent's context:

  1. Always read SKILL.md first before implementing
  2. Start simple - Use length-based reward to validate setup
  3. Build incrementally - Add one reward function at a time
  4. Reference examples - Copy patterns from reward_functions_library.py
  5. Monitor training - Watch reward metrics (not loss!)

๐ŸŽฏ Common Use Cases

Task Type Recommended Rewards Template
Math reasoning MATH_REASONING_REWARDS preset basic_grpo_training.py
Code generation CODE_GENERATION_REWARDS preset Modify dataset in template
Summarization SUMMARIZATION_REWARDS preset Adjust prompts + rewards
Q&A QA_REWARDS preset Use fuzzy match + citations

โš ๏ธ Critical Reminders

  • Loss goes UP during training - This is normal (it's KL divergence)
  • Use 3-5 reward functions - Single rewards often fail
  • Test rewards before training - Debug each function independently
  • Monitor reward_std - Should stay > 0.1 (avoid mode collapse)
  • Start with num_generations=4-8 - Scale up if GPU allows

๐Ÿ”— External Resources

๐Ÿ“ Version

v1.0.0 - Initial release (January 2025)

๐Ÿ‘จโ€๐Ÿ’ป Maintained By

Orchestra Research
For questions or improvements, see https://orchestra.com

License: MIT
Last Updated: January 2025

Overview (`demos/autoresearch-norm-heterogeneity/README.md`)

Autoresearch Demo: Embedding Norm Heterogeneity Drives LoRA Fine-Tuning Brittleness

Paper: norm-heterogeneity-lora-brittleness.pdf

Skills Used: Autoresearch, ML Paper Writing, Research Ideation

What Happened

An AI agent ran the full autoresearch workflow autonomously โ€” from literature survey through experiments to paper writing. Starting from the hypothesis that ETF crystallization drives LoRA fine-tuning brittleness in overtrained models, the agent:

  1. Surveyed literature connecting two recent papers: NeurIPS 2025 Best Paper Runner-Up on superposition/ETF structure and ICML 2025 on catastrophic overtraining
  2. Ran inner loop experiments across Pythia-410M and Pythia-1.4B checkpoints, computing ETF overlap metrics and norm statistics at each checkpoint, then applying LoRA fine-tuning
  3. Discovered a null result โ€” ETF overlap geometry does NOT predict fine-tuning difficulty (r=0.14), refuting the starting hypothesis
  4. Pivoted โ€” identified embedding norm heterogeneity (coefficient of variation) as the actual causal predictor (r=-0.84 at 410M, r=-0.99 at 1.4B)
  5. Deepened with causal experiments โ€” equalizing norms before LoRA increases fine-tunability by up to 79%
  6. Wrote the paper using the ml-paper-writing skill

Key Findings

  • ETF overlap metrics show no correlation with LoRA fine-tuning difficulty โ€” a clear negative result
  • Norm CV of LM head rows strongly predicts deconfounded fine-tunability (r=-0.99 at 1.4B)
  • Equalizing norms before LoRA increases relative fine-tunability by up to 79%
  • The effect is rank-independent โ€” increasing LoRA rank does not mitigate it
  • Norms encode semantic specificity, creating an impedance mismatch with LoRA's uniform low-rank updates

Why This Demo Matters

This demonstrates the autoresearch two-loop architecture working as designed:

  • Inner loop ran constrained experiments (checkpoint analysis, LoRA fine-tuning, metric computation)
  • Outer loop synthesized a null result into a pivot, leading to a stronger finding than the original hypothesis
  • The agent autonomously went from "ETF predicts brittleness" to "actually no, norm heterogeneity does" โ€” a genuine research pivot that produced a more interesting paper

Overview (`demos/autoresearch-rl-brain-scan/README.md`)

Autoresearch Demo: The RL Algorithm Brain Scan

Paper: rl_algorithm_brain_scan.pdf

Skills Used: Autoresearch, ML Paper Writing, GRPO RL Training, TRL, SAELens, TransformerLens

What Happened

An AI agent autonomously investigated what RL alignment algorithms actually do to model internals โ€” a question no prior work had systematically addressed. The agent:

  1. Surveyed literature on RLOO, GRPO, and DPO, identifying the gap: nobody had compared what these algorithms do at the weight and feature level on the same base model
  2. Ran inner loop experiments training GPT-2 Small with RLOO, GRPO, and DPO on sentiment and toxicity tasks, then analyzing weight deltas via SVD and feature changes via SAELens
  3. Discovered three key findings through outer loop synthesis:
    • DPO is a rank-1 perturbation (top-1 SVD direction recovers 95.6% of behavioral effect)
    • Online RL (RLOO/GRPO) produces distributed, structure-preserving modifications (effective rank 200 vs 119)
    • DPO creates a "concentrated perturbation cascade" disrupting 2x more SAE features in later layers
  4. Validated causally with SVD ablation experiments โ€” not just correlation but causal evidence
  5. Wrote the paper in ICML format using the ml-paper-writing skill

Key Findings

  • DPO is rank-1 alignment: A single SVD direction per weight matrix recovers 95.6% of DPO's behavioral effect. GRPO needs 50+ directions for equivalent recovery.
  • Online RL preserves structure: RLOO and GRPO maintain higher effective rank (200 vs 119) and better preserve the base model's SAE feature structure (Jaccard 0.83 vs 0.69)
  • DPO's concentrated perturbation cascade: Despite lower-rank changes, DPO disrupts 2x more SAE features in later layers (1619 vs 527-870), amplifying perturbations through the network
  • Results hold across sentiment and toxicity tasks with statistical significance (n=3 seeds, non-overlapping CIs)

Why This Demo Matters

This demonstrates autoresearch orchestrating multiple domain skills together:

  • Post-training skills (TRL, GRPO) for training the RL models
  • Interpretability skills (SAELens, TransformerLens) for analyzing what changed
  • Paper writing skill for producing the ICML submission
  • The two-loop architecture enabled the agent to both run experiments AND synthesize them into mechanistic understanding โ€” "DPO is a rank-1 perturbation" is a conceptual insight, not just a metric

Overview (`demos/scientific-plotting-demo/README.md`)

Academic Plotting Skill Demo

Publication-quality figures generated using the academic-plotting skill from the AI Research Skills library. Demonstrates both Gemini AI diagram generation (Workflow 1) and matplotlib/seaborn data charts (Workflow 2).

Source Paper

Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services

Jiachen Liu, Jae-Won Chung, Zhiyu Wu, Fan Lai, Myungjin Lee, Mosharaf Chowdhury

Andes is a QoE-aware LLM serving system that enhances user experience by ensuring users receive tokens promptly and at a smooth, digestible pace. Its preemptive token-level request scheduler dynamically prioritizes requests based on expected QoE gain and GPU resource usage, achieving up to 4.7x QoE improvement or 61% GPU savings compared to existing systems.

Metric Result
QoE improvement over vLLM 4.7x
GPU resource savings 61%
Peak queue length reduction 85%

1. System Architecture Workflow (Gemini AI)

Core contribution diagram showing Andes' co-design of the inference server (Token-Level Request Scheduler + Overhead-Aware Refiner) and client (Token Pacer). Generated using the updated academic-plotting skill:

  • Model: gemini-3-pro-image-preview
  • Style: Style B "Modern Minimal" โ€” ultra-clean, spacious, authoritative
  • Palette: "Nord" โ€” desaturated section fills, Aurora Yellow accents for Andes components
  • Prompt: 6-section structure (Framing, Visual Style, Colors, Layout, Connections, Constraints)
  • Attempts: 3 non-deterministic, best selected

Selected Result (Attempt 1)

Figure 1: Andes QoE-Aware LLM Serving System Architecture

AI-generated diagram showing the full request lifecycle: (1) User submits request, (2) Client enqueues with QoE parameters, (3) Request Tracker feeds state to scheduler, (4) Token-Level Scheduler admits/resumes/preempts at token granularity, (5) Executor streams tokens, (6) Token Pacer delivers smoothly at reading speed. Yellow-accented components are Andes' novel contributions.

gemini-3-pro-image-preview | Style B: Modern Minimal | Nord Palette | Best of 3

All 3 Gemini Attempts (for comparison)

Attempt 1 (Selected) Attempt 2 Attempt 3
Best spacing, color accents, arrow routing Good, slightly tighter spacing Good separation, dashed preempt

All 3 attempts have 100% text accuracy โ€” every label spelled correctly (Token Pacer, Overhead Refiner, KV Cache, etc.). This is a major improvement over the previous generation which had misspellings in all attempts.

2. QoE Metric Definition

Four foundational cases illustrating how the QoE metric captures different types of user experience degradation in text streaming services.

Figure 2: User Experience Cases and QoE Definition

(a) Perfect experience: actual delivery matches ideal consumption timeline. (b) Long initial delay: head-of-line blocking inflates TTFT. (c) Slow streaming: token generation slower than consumption speed. (d) Pause in middle: preemption causes mid-stream pause. The shaded area represents QoE degradation (S_delay).

matplotlib | Line Plot

3. CDF Comparison: QoE, TTFT, TDS

Three-panel CDF comparison on real-world BurstGPT traces showing Andes' improvements across all key metrics. Follows the multi-panel figure pattern with shared styling and colorblind-safe colors.

Figure 3: CDF of QoE, TTFT, and TDS on BurstGPT Trace

Andes (orange) achieves near-perfect QoE for 97% of requests (QoE >= 0.95), compared to only 75% for vLLM (blue). TTFT is reduced from 10.5s to 1.8s average. TDS remains comparable, showing Andes doesn't sacrifice throughput.

matplotlib | CDF Plot | PDF + PNG

4. Multi-Panel: Varying Burst Intensity

4x3 grid showing average QoE across 4 models and 3 datasets under varying burst intensities. This demonstrates the academic plotting skill's ability to create complex multi-panel figures with shared axes, model labels, and a unified legend.

Figure 4: Average QoE Under Varying Burst Intensity

Across all 12 model-dataset combinations, Andes (orange) consistently maintains higher QoE than all baselines as burst intensity increases. vLLM (blue), LQSF (green), and Sarathi-Serve (red) degrade significantly under heavy bursts due to FCFS scheduling and head-of-line blocking. Andes achieves up to 4.7x improvement at the highest burst intensity.

matplotlib | Multi-Panel Grid | 4 Methods x 4 Models x 3 Datasets

5. Summary: Key Improvements

Three-panel bar chart summarizing the headline results from the paper. Each panel uses a distinct color to represent different aspects of improvement.

Figure 5: Summary of Key Improvements

(a) Andes achieves 0.99 average QoE vs 0.88 for vLLM on real-world traces. (b) QoE improvement ranges from 3.2x to 4.7x across different model architectures. (c) Andes saves 61% GPU resources, reduces peak queue by 85%, and handles 2.6x more concurrent requests.

matplotlib | Grouped Bar | PDF + PNG

6. How These Figures Were Generated

All figures follow the academic-plotting skill's two workflows and publication standards.

Workflow 1: Gemini AI Diagram

The system architecture (Figure 1) uses gemini-3-pro-image-preview with the skill's 6-section prompt structure and Style B: Modern Minimal visual style. Key elements:

  1. Framing โ€” Sets the tone: "ultra-clean, modern, authoritative, like Apple docs meets Nature paper"
  2. Visual Style โ€” Full Modern Minimal style block: floating boxes with shadow, no borders, thin gray arrows
  3. Color Palette โ€” Nord palette with exact hex codes for every element
  4. Layout โ€” Every box named, spatially positioned, with nested sub-components
  5. Connections โ€” Every arrow individually specified: source, target, style, color, label, routing
  6. Constraints โ€” What NOT to include, adapted for the Modern Minimal style
python
from google import genai
client = genai.Client(api_key=API_KEY)

# 6-section prompt: Framing + Style + Colors + Layout + Connections + Constraints
PROMPT = """
SECTION 1 โ€” FRAMING:
Create an ultra-clean, modern technical architecture diagram for an OSDI paper.
Think: Apple developer docs meets Nature paper...

SECTION 2 โ€” VISUAL STYLE (Modern Minimal):
Ultra-clean geometric shapes, floating boxes with shadow, thin gray arrows...

SECTION 3 โ€” COLOR PALETTE (Nord):
Deep text: #2E3440, Andes accent: Aurora Yellow #EBCB8B, Executor: Frost #5E81AC...

SECTION 4 โ€” LAYOUT:
Two zones: CLIENT (#EEF1F6) and SERVER (#EDF3ED), each with floating boxes...

SECTION 5 โ€” CONNECTIONS:
8 arrows with step numbers, dashed red preempt path, green delivery flow...

SECTION 6 โ€” CONSTRAINTS:
ZERO decoration, generous whitespace, CRITICAL TEXT ACCURACY...
"""

# Generate 3 non-deterministic attempts
for i in range(1, 4):
    response = client.models.generate_content(
        model="gemini-3-pro-image-preview",
        contents=PROMPT,
        config=genai.types.GenerateContentConfig(
            response_modalities=["IMAGE", "TEXT"]))

Workflow 2: Data-Driven Charts

Experiment figures use matplotlib with publication defaults: serif fonts, colorblind-safe palette, 300 DPI export, venue-appropriate sizing. Each figure exports both PDF (vector for LaTeX) and PNG (raster).

python
# Publication defaults
plt.rcParams.update({
    "font.family": "serif",
    "font.size": 10,
    "axes.spines.top": False,
    "savefig.dpi": 300,
})

# Colorblind-safe palette
COLORS = {
    "blue": "#4C72B0",
    "orange": "#DD8452",
    "green": "#55A868",
    "red": "#C44E52",
}

7. Generated Files

text
demo/
โ”œโ”€โ”€ README.md                                # This demo page
โ””โ”€โ”€ figures/
    โ”œโ”€โ”€ gen_fig_andes_architecture_gemini.py  # Gemini diagram script (Workflow 1)
    โ”œโ”€โ”€ gen_fig_andes_workflow.py             # matplotlib diagram (alternative)
    โ”œโ”€โ”€ gen_fig_experiment_results.py         # Data charts script (Workflow 2)
    โ”œโ”€โ”€ fig_andes_architecture.png            # Gemini best attempt (selected)
    โ”œโ”€โ”€ fig_andes_architecture_attempt1.png   # Gemini attempt 1
    โ”œโ”€โ”€ fig_andes_architecture_attempt2.png   # Gemini attempt 2
    โ”œโ”€โ”€ fig_andes_architecture_attempt3.png   # Gemini attempt 3
    โ”œโ”€โ”€ fig_andes_workflow.pdf                # matplotlib vector diagram
    โ”œโ”€โ”€ fig_andes_workflow.png                # matplotlib raster diagram
    โ”œโ”€โ”€ fig_cdf_comparison.pdf               # CDF panels (vector)
    โ”œโ”€โ”€ fig_cdf_comparison.png               # CDF panels (raster)
    โ”œโ”€โ”€ fig_burst_intensity.pdf              # Multi-panel grid (vector)
    โ”œโ”€โ”€ fig_burst_intensity.png              # Multi-panel grid (raster)
    โ”œโ”€โ”€ fig_qoe_definition.pdf              # QoE illustration (vector)
    โ”œโ”€โ”€ fig_qoe_definition.png              # QoE illustration (raster)
    โ”œโ”€โ”€ fig_summary_improvements.pdf         # Summary bars (vector)
    โ””โ”€โ”€ fig_summary_improvements.png         # Summary bars (raster)

Generated using the academic-plotting skill from AI Research Skills. Paper: arXiv:2404.16283. Figures use synthetic data matching paper-reported distributions.

Overview (`packages/ai-research-skills/README.md`)

@orchestra-research/ai-research-skills

Install AI research engineering skills to your coding agents (Claude Code, Hermes Agent, OpenCode, Cursor, Gemini CLI, and more).

bash
npx @orchestra-research/ai-research-skills

Features

  • 98 skills across 23 categories for AI research engineering
  • Auto-detects installed coding agents
  • Interactive installer with guided experience
  • Global or local install โ€” install globally with symlinks, or per-project with --local for version-controlled, project-specific skill sets
  • Works with 9 agents: Claude Code, Hermes Agent, OpenCode, OpenClaw, Cursor, Codex, Gemini CLI, Qwen Code, and shared .agents/

Quick Start

Run the interactive installer:

bash
npx @orchestra-research/ai-research-skills

This will:

  1. Detect your installed coding agents
  2. Let you choose what to install (everything, categories, or quick start bundle)
  3. Download skills from GitHub
  4. Create symlinks to each agent's skills directory

Commands

bash
# Interactive mode (recommended)
npx @orchestra-research/ai-research-skills

# Install everything (global)
npx @orchestra-research/ai-research-skills install --all

# Install a specific category
npx @orchestra-research/ai-research-skills install post-training

# List installed skills
npx @orchestra-research/ai-research-skills list

# Update all skills
npx @orchestra-research/ai-research-skills update

Local Installation (per-project)

Install skills directly into your project directory so different projects can have different skill sets:

bash
# Install all skills locally to the current project
npx @orchestra-research/ai-research-skills install --all --local

# Install a category locally
npx @orchestra-research/ai-research-skills install --category post-training --local

# List locally installed skills
npx @orchestra-research/ai-research-skills list --local

# Update local skills
npx @orchestra-research/ai-research-skills update --local

# Uninstall local skills
npx @orchestra-research/ai-research-skills uninstall --local

Local installation copies skills (not symlinks) into agent directories within your project:

text
my-project/
โ”œโ”€โ”€ .claude/skills/        # Claude Code picks these up
โ”‚   โ”œโ”€โ”€ grpo-rl-training/
โ”‚   โ””โ”€โ”€ vllm/
โ”œโ”€โ”€ .cursor/skills/        # Cursor picks these up
โ”‚   โ”œโ”€โ”€ grpo-rl-training/
โ”‚   โ””โ”€โ”€ vllm/
โ”œโ”€โ”€ .orchestra-skills.json # Tracks installed skills
โ””โ”€โ”€ ...

Benefits:

  • Per-project skills: Each project gets only the skills it needs
  • Version control: Commit skills to your repo so the whole team has them
  • Reproducible: Lock file (.orchestra-skills.json) tracks what's installed

Categories

Category Skills Description
Autoresearch 1 Central orchestration โ€” manages full research lifecycle, routes to all other skills
Model Architecture 6 LitGPT, Mamba, TorchTitan, Megatron...
Post-Training 8 GRPO, verl, slime, miles, torchforge...
Fine-Tuning 5 Axolotl, Unsloth, PEFT, Torchtune...
Distributed Training 6 DeepSpeed, FSDP, Megatron...
Inference Serving 4 vLLM, TensorRT-LLM, SGLang...
Optimization 6 Flash Attention, GPTQ, AWQ...
And 15 more... Ideation, Paper Writing, RAG, Agents, Multimodal...

How It Works

Global Install (default)

  1. Canonical Storage: Skills are stored once at ~/.orchestra/skills/
  2. Symlinks: Each agent gets symlinks pointing to the canonical copy
  3. Auto-activation: Skills activate when you discuss relevant topics
text
~/.orchestra/skills/       # Single source of truth
โ”œโ”€โ”€ 06-post-training/
โ”‚   โ”œโ”€โ”€ verl/
โ”‚   โ””โ”€โ”€ grpo-rl-training/
โ””โ”€โ”€ ...

~/.claude/skills/          # Symlinks for Claude Code
โ”œโ”€โ”€ verl โ†’ ~/.orchestra/skills/.../verl
โ””โ”€โ”€ grpo-rl-training โ†’ ...

~/.cursor/skills/          # Symlinks for Cursor
โ””โ”€โ”€ (same links)

Local Install (`--local`)

  1. Direct Copy: Skills are copied into agent directories within your project
  2. Version Control: Files can be committed to git for team sharing
  3. Lock File: .orchestra-skills.json tracks what's installed
text
my-project/
โ”œโ”€โ”€ .claude/skills/verl/           # Copied for Claude Code
โ”œโ”€โ”€ .cursor/skills/verl/           # Copied for Cursor
โ”œโ”€โ”€ .codex/skills/verl/            # Copied for Codex
โ””โ”€โ”€ .orchestra-skills.json         # Lock file

Supported Agents

Agent Config Directory
Claude Code ~/.claude
OpenCode ~/.config/opencode
OpenClaw ~/.openclaw
Cursor ~/.cursor
Codex (OpenAI) ~/.codex
Gemini CLI ~/.gemini
Qwen Code ~/.qwen
Shared Agents ~/.agents
Hermes Agent ~/.hermes

License

MIT - Orchestra Research

Overview (`04-mechanistic-interpretability/nnsight/references/README.md`)

nnsight Reference Documentation

This directory contains comprehensive reference materials for nnsight.

Contents

  • api.md - Complete API reference for LanguageModel, tracing, and proxy objects
  • tutorials.md - Step-by-step tutorials for local and remote interpretability

Installation

bash
# Basic installation
pip install nnsight

# For vLLM support
pip install "nnsight[vllm]"

Basic Usage

python
from nnsight import LanguageModel

# Load model
model = LanguageModel("openai-community/gpt2", device_map="auto")

# Trace and access internals
with model.trace("The Eiffel Tower is in") as tracer:
    # Access layer output
    hidden = model.transformer.h[5].output[0].save()

    # Modify activations
    model.transformer.h[8].output[0][:] *= 0.5

    # Get final output
    logits = model.output.save()

# Access saved values outside context
print(hidden.shape)

Key Concepts

Tracing

The trace() context enables deferred execution - operations are recorded and executed together.

Proxy Objects

Inside trace, module accesses return Proxies. Call .save() to retrieve values after execution.

Remote Execution (NDIF)

Run the same code on massive models (70B+) without local GPUs:

python
# Same code, just add remote=True
with model.trace("Hello", remote=True):
    hidden = model.model.layers[40].output[0].save()

NDIF Setup

  1. Sign up at https://login.ndif.us/
  2. Get API key
  3. Set environment variable: export NDIF_API_KEY=your_key

Available Remote Models

Overview (`04-mechanistic-interpretability/pyvene/references/README.md`)

pyvene Reference Documentation

This directory contains comprehensive reference materials for pyvene.

Contents

  • api.md - Complete API reference for IntervenableModel, intervention types, and configurations
  • tutorials.md - Step-by-step tutorials for causal tracing, activation patching, and trainable interventions

Installation

bash
pip install pyvene

Basic Usage

python
import pyvene as pv
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model
model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

# Define intervention
config = pv.IntervenableConfig(
    representations=[
        pv.RepresentationConfig(
            layer=5,
            component="block_output",
            intervention_type=pv.VanillaIntervention,
        )
    ]
)

# Create intervenable model
intervenable = pv.IntervenableModel(config, model)

# Run intervention (swap activations from source to base)
base_inputs = tokenizer("The cat sat on the", return_tensors="pt")
source_inputs = tokenizer("The dog ran through the", return_tensors="pt")

_, outputs = intervenable(
    base=base_inputs,
    sources=[source_inputs],
)

Key Concepts

Intervention Types

  • VanillaIntervention: Swap activations between runs
  • AdditionIntervention: Add source to base activations
  • ZeroIntervention: Zero out activations (ablation)
  • CollectIntervention: Collect activations without modifying
  • RotatedSpaceIntervention: Trainable intervention for causal discovery

Components

Target specific parts of the model:

  • block_input, block_output
  • mlp_input, mlp_output, mlp_activation
  • attention_input, attention_output
  • query_output, key_output, value_output

HuggingFace Integration

Save and load interventions via HuggingFace Hub for reproducibility.

Overview (`04-mechanistic-interpretability/saelens/references/README.md`)

SAELens Reference Documentation

This directory contains comprehensive reference materials for SAELens.

Contents

  • api.md - Complete API reference for SAE, TrainingSAE, and configuration classes
  • tutorials.md - Step-by-step tutorials for training and analyzing SAEs
  • papers.md - Key research papers on sparse autoencoders

Installation

bash
pip install sae-lens

Requirements: Python 3.10+, transformer-lens>=2.0.0

Basic Usage

python
from transformer_lens import HookedTransformer
from sae_lens import SAE

# Load model and SAE
model = HookedTransformer.from_pretrained("gpt2-small", device="cuda")
sae, cfg_dict, sparsity = SAE.from_pretrained(
    release="gpt2-small-res-jb",
    sae_id="blocks.8.hook_resid_pre",
    device="cuda"
)

# Encode activations to sparse features
tokens = model.to_tokens("Hello world")
_, cache = model.run_with_cache(tokens)
activations = cache["resid_pre", 8]

features = sae.encode(activations)  # Sparse feature activations
reconstructed = sae.decode(features)  # Reconstructed activations

Key Concepts

Sparse Autoencoders

SAEs decompose dense neural activations into sparse, interpretable features:

  • Encoder: Maps d_model โ†’ d_sae (typically 4-16x expansion)
  • ReLU/TopK: Enforces sparsity
  • Decoder: Reconstructs original activations

Training Loss

Loss = MSE(original, reconstructed) + L1_coefficient ร— L1(features)

Key Metrics

  • L0: Average number of active features (target: 50-200)
  • CE Loss Score: Cross-entropy recovered vs original model (target: 80-95%)
  • Dead Features: Features that never activate (target: <5%)

Available Pre-trained SAEs

Release Model Description
gpt2-small-res-jb GPT-2 Small Residual stream SAEs
gemma-2b-res Gemma 2B Residual stream SAEs
Various Search HuggingFace Community-trained SAEs

Overview (`04-mechanistic-interpretability/transformer-lens/references/README.md`)

TransformerLens Reference Documentation

This directory contains comprehensive reference materials for TransformerLens.

Contents

  • api.md - Complete API reference for HookedTransformer, ActivationCache, and HookPoints
  • tutorials.md - Step-by-step tutorials for common interpretability workflows
  • papers.md - Key research papers and foundational concepts

Installation

bash
pip install transformer-lens

Basic Usage

python
from transformer_lens import HookedTransformer

# Load model
model = HookedTransformer.from_pretrained("gpt2-small")

# Run with activation caching
tokens = model.to_tokens("Hello world")
logits, cache = model.run_with_cache(tokens)

# Access activations
residual = cache["resid_post", 5]  # Layer 5 residual stream
attention = cache["pattern", 3]    # Layer 3 attention patterns

Key Concepts

HookPoints

Every activation in the transformer has a HookPoint wrapper, enabling:

  • Reading activations via run_with_cache()
  • Modifying activations via run_with_hooks()

Activation Cache

The ActivationCache stores all intermediate activations with helper methods for:

  • Residual stream decomposition
  • Logit attribution
  • Layer-wise analysis

Supported Models (50+)

GPT-2, LLaMA, Mistral, Pythia, GPT-Neo, OPT, Gemma, Phi, and more.

Overview (`14-agents/a-evolve/references/README.md`)

A-Evolve Official Documentation Reference

This document consolidates key information from the official A-Evolve documentation
at github.com/A-EVO-Lab/a-evolve.

Table of Contents

Project Overview

A-Evolve is the universal infrastructure for evolving AI agents through self-improvement. It enables automatic, data-driven optimization of agents across any domain using any evolution algorithm.

Design Principles

  1. File-system as contract: All evolvable agent state lives as plain files in a workspace directory. No databases, no learned weights, no opaque parameters. Every mutation is an explicit edit to a text file.

  2. Pluggable everything: Three interfaces โ€” BaseAgent, BenchmarkAdapter, EvolutionEngine โ€” enable any combination of agent, benchmark, and algorithm.

  3. Git for versioning: Every evolution cycle creates git snapshots. Changes are diffable, rollbackable, and human-readable.

  4. LLM-in-the-loop: The default evolution engine uses an LLM with bash tools to analyze observations and directly mutate workspace files. The evolver is itself an AI agent improving other AI agents.

  5. Zero manual engineering: Once configured, evolution runs autonomously. The loop handles solving, evaluation, mutation, gating, and convergence detection.

Key Results

Using Claude Opus 4.6 as both the solver and evolver model:

Benchmark Score Leaderboard Position
MCP-Atlas 79.4% #1
SWE-bench Verified 76.8% ~#5
Terminal-Bench 2.0 76.5% ~#7
SkillsBench 34.9% #2

These results demonstrate that LLM-driven evolution of prompts, skills, and memory can produce state-of-the-art agent performance across diverse domains.

Installation Guide

Requirements

  • Python >= 3.11
  • Git (for workspace versioning)
  • An LLM API key (Anthropic, OpenAI, or AWS Bedrock credentials)

Installation Options

bash
# Core package (matplotlib, pyyaml)
pip install a-evolve

# With specific LLM provider support
pip install a-evolve[anthropic]     # Anthropic Claude API
pip install a-evolve[openai]        # OpenAI API
pip install a-evolve[bedrock]       # AWS Bedrock (boto3)
pip install a-evolve[litellm]       # Multi-provider via LiteLLM

# With domain-specific dependencies
pip install a-evolve[swe]           # SWE-bench (strands-agents, datasets, swebench)
pip install a-evolve[mcp]           # MCP-Atlas (mcp, strands-agents, litellm)
pip install a-evolve[skillbench]    # SkillsBench (strands-agents)

# Everything
pip install a-evolve[all]

# Development
pip install a-evolve[dev]           # pytest, ruff, hypothesis

From Source

bash
git clone https://github.com/A-EVO-Lab/a-evolve.git
cd a-evolve
pip install -e ".[all,dev]"

Verifying Installation

python
import agent_evolve as ae
print(ae.__version__)  # Should print version
print(ae.Evolver)      # Should print class reference

Quick Start Guide

3-Line Evolution

python
import agent_evolve as ae

evolver = ae.Evolver(agent="swe", benchmark="swe-verified")
results = evolver.run(cycles=10)
print(f"Final score: {results.final_score}")

This:

  1. Copies the built-in SWE seed workspace to a working directory
  2. Instantiates SweAgent from the workspace manifest
  3. Runs 10 evolution cycles against SWE-bench Verified
  4. Returns EvolutionResult with scores, convergence status, and details

With Custom Configuration

python
import agent_evolve as ae

config = ae.EvolveConfig(
    batch_size=15,              # 15 tasks per cycle
    max_cycles=25,              # Up to 25 evolution rounds
    evolve_prompts=True,        # Mutate system prompt
    evolve_skills=True,         # Discover and refine skills
    evolve_memory=True,         # Build episodic memory
    holdout_ratio=0.2,          # 20% held out for validation
    evolver_model="us.anthropic.claude-opus-4-6-v1",
    egl_threshold=0.02,         # Stop if < 2% improvement
    egl_window=5,               # Over 5 consecutive cycles
)

evolver = ae.Evolver(
    agent="swe",
    benchmark="swe-verified",
    config=config,
)
results = evolver.run()

# Inspect results
print(f"Cycles: {results.cycles_completed}")
print(f"Score: {results.final_score:.3f}")
print(f"Converged: {results.converged}")
print(f"Score history: {results.score_history}")

Architecture Overview

System Diagram

text
User Code (3 lines)
    โ”‚
    โ–ผ
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚            Evolver API               โ”‚
โ”‚  - Resolves agent, benchmark, config โ”‚
โ”‚  - Creates EvolutionLoop             โ”‚
โ”‚  - Returns EvolutionResult           โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
               โ”‚
    โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ–ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
    โ”‚   EvolutionLoop     โ”‚
    โ”‚  For each cycle:    โ”‚
    โ”‚  1. Solve           โ”‚
    โ”‚  2. Observe         โ”‚
    โ”‚  3. Snapshot        โ”‚
    โ”‚  4. Evolve          โ”‚
    โ”‚  5. Snapshot        โ”‚
    โ”‚  6. Record          โ”‚
    โ”‚  7. Reload          โ”‚
    โ”‚  8. Converge?       โ”‚
    โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
               โ”‚
    โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
    โ”‚          โ”‚          โ”‚
    โ–ผ          โ–ผ          โ–ผ
 Agent    Benchmark    Engine
solve()  evaluate()   step()
    โ”‚          โ”‚          โ”‚
    โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
               โ”‚
               โ–ผ
       Agent Workspace
       (filesystem + git)

Component Interactions

Forward flow (solve):

  1. EvolutionLoop calls benchmark.get_tasks() to get a batch of tasks
  2. For each task, calls agent.solve(task) to get a Trajectory
  3. Calls benchmark.evaluate(task, trajectory) to get Feedback
  4. Bundles into Observation(task, trajectory, feedback) triples

Evolution flow (mutate):

  1. EvolutionLoop passes observations to engine.step()
  2. Engine reads workspace files, analyzes observations
  3. Engine mutates workspace files (prompts, skills, memory)
  4. Returns StepResult(mutated, summary, metadata)

Reload flow (sync):

  1. EvolutionLoop calls agent.reload_from_fs()
  2. Agent re-reads prompts, skills, memory from workspace
  3. Next cycle uses evolved state

Agent Protocol

BaseAgent Abstract Class

All evolvable agents inherit from BaseAgent:

python
from agent_evolve.protocol.base_agent import BaseAgent
from agent_evolve.types import Task, Trajectory

class MyAgent(BaseAgent):
    def __init__(self, workspace_dir: str):
        super().__init__(workspace_dir)
        # Initialize your LLM client, tools, etc.

    def solve(self, task: Task) -> Trajectory:
        """Solve a single task and return the trajectory.

        This is the only method you MUST override.
        """
        # Your solving logic here
        return Trajectory(
            task_id=task.id,
            output="solution",
            steps=[{"tool": "llm", "action": "generate"}],
        )

Agent Lifecycle

  1. Construction: __init__(workspace_dir) โ€” set up LLM client, load initial state
  2. State loading: reload_from_fs() โ€” read prompts, skills, memory from workspace
  3. Solving: solve(task) โ€” process one task, return trajectory
  4. Memory buffering: remember(content, category) โ€” store lessons during solve
  5. State export: export_to_fs() โ€” flush buffered memories and skill proposals
  6. Hot reload: reload_from_fs() โ€” re-read after evolution mutates files

Agent Properties

Property Type Description
system_prompt str Content of prompts/system.md
skills list[SkillMeta] Available skills from skills/ directory
memories list[dict] Loaded episodic/semantic memories

Agent Best Practices

  1. Always use self.system_prompt โ€” don't hardcode prompts
  2. Inject skills into LLM context โ€” they're the primary evolution mechanism
  3. Call remember() for reusable lessons โ€” not for task-specific notes
  4. Keep solve() deterministic when possible (temperature=0 for reproducibility)
  5. Truncate trajectories โ€” don't store full conversation if not needed for evolution

Benchmark Adapters

BenchmarkAdapter Abstract Class

python
from agent_evolve.benchmarks.base import BenchmarkAdapter
from agent_evolve.types import Task, Trajectory, Feedback

class MyBenchmark(BenchmarkAdapter):
    def get_tasks(self, split="train", limit=10):
        """Return tasks from the benchmark dataset.

        Args:
            split: "train" or "test" (for holdout evaluation)
            limit: Maximum number of tasks to return (default 10)
        """
        return [Task(id="1", input="task description")]

    def evaluate(self, task, trajectory):
        """Evaluate an agent's trajectory on a task.

        Returns Feedback with:
        - success: bool (binary pass/fail)
        - score: float (0.0 to 1.0 continuous)
        - detail: str (human-readable explanation)
        """
        return Feedback(success=True, score=0.9, detail="Passed 9/10 tests")

Benchmark Best Practices

  1. Rich feedback details โ€” the evolver reads feedback.detail to decide what to mutate
  2. Deterministic evaluation โ€” same input should produce same score
  3. Diverse task coverage โ€” include easy, medium, and hard tasks
  4. Strict train/test split โ€” no overlap between splits
  5. Score granularity โ€” continuous scores (0.0-1.0) are more useful than binary pass/fail

Evolution Engines

EvolutionEngine Abstract Class

python
from agent_evolve.engine.base import EvolutionEngine
from agent_evolve.types import StepResult

class MyEngine(EvolutionEngine):
    def step(self, workspace, observations, history, trial):
        """Mutate the workspace based on observations.

        Args:
            workspace: AgentWorkspace โ€” typed I/O for agent files
            observations: list[Observation] โ€” recent (task, trajectory, feedback) triples
            history: EvolutionHistory โ€” query past cycles and workspace versions
            trial: TrialRunner โ€” optional live evaluation runner

        Returns:
            StepResult with mutated flag, summary, and metadata
        """
        # Analyze observations, mutate workspace
        return StepResult(mutated=True, summary="Updated prompts")

    def on_cycle_end(self, accepted: bool, score: float):
        """Optional callback after gating decision."""
        pass

Engine Selection Guide

Engine When to Use Compute Cost
AEvolveEngine (default) General-purpose, diverse domains High (full LLM call)
GuidedSynthesisEngine Skill discovery focus Medium
AdaptiveEvolutionEngine Noisy evaluation, fine control Medium
AdaptiveSkillEngine Skill-heavy domains Medium
Custom Domain-specific mutation logic Variable

Workspace Contract

Directory Structure

text
workspace/
โ”œโ”€โ”€ manifest.yaml              # Required: agent metadata
โ”œโ”€โ”€ prompts/
โ”‚   โ”œโ”€โ”€ system.md              # Main system prompt
โ”‚   โ””โ”€โ”€ fragments/             # Modular prompt pieces
โ”‚       โ”œโ”€โ”€ reasoning.md
โ”‚       โ””โ”€โ”€ output_format.md
โ”œโ”€โ”€ skills/
โ”‚   โ”œโ”€โ”€ _drafts/               # Proposed skills pending review
โ”‚   โ”‚   โ””โ”€โ”€ new-skill.md
โ”‚   โ””โ”€โ”€ verify-solution/       # Accepted skills
โ”‚       โ””โ”€โ”€ SKILL.md
โ”œโ”€โ”€ tools/
โ”‚   โ”œโ”€โ”€ registry.yaml          # Tool manifest
โ”‚   โ””โ”€โ”€ custom_tool.py         # Tool implementations
โ”œโ”€โ”€ memory/
โ”‚   โ”œโ”€โ”€ episodic.jsonl         # Failure lessons
โ”‚   โ””โ”€โ”€ semantic.jsonl         # Domain knowledge
โ””โ”€โ”€ evolution/                 # Managed by loop
    โ”œโ”€โ”€ observations/
    โ”‚   โ”œโ”€โ”€ batch_0001.jsonl
    โ”‚   โ””โ”€โ”€ batch_0002.jsonl
    โ”œโ”€โ”€ history.jsonl
    โ””โ”€โ”€ metrics.json

Manifest Format

yaml
agent:
  type: reference                                    # Must be "reference"
  entrypoint: my_package.agents.MyAgent              # Dotted Python path

evolvable_layers:                                    # Which directories can be mutated
  - prompts                                          # System prompt + fragments
  - skills                                           # Skill library
  - memory                                           # Episodic/semantic memory
  # - tools                                          # Tool implementations (optional)

reload_strategy: hot                                 # "hot" (re-read files) or "cold" (restart)

AgentWorkspace API

The AgentWorkspace class provides typed read/write access:

Prompts:

  • read_prompt() -> str โ€” reads prompts/system.md
  • write_prompt(content: str) โ€” writes prompts/system.md
  • read_fragment(name: str) -> str โ€” reads prompts/fragments/{name}
  • write_fragment(name: str, content: str) โ€” writes a fragment
  • list_fragments() -> list[str] โ€” lists fragment filenames

Skills:

  • list_skills() -> list[SkillMeta] โ€” lists skills with name, description, path
  • read_skill(name: str) -> str โ€” reads skill content (frontmatter stripped)
  • write_skill(name: str, content: str) โ€” writes or updates a skill
  • delete_skill(name: str) โ€” removes a skill directory

Drafts:

  • list_drafts() -> list[dict] โ€” lists pending skill proposals
  • write_draft(name: str, content: str) โ€” writes a draft proposal
  • clear_drafts() โ€” removes all pending drafts

Memory:

  • add_memory(entry: dict, category: str = "episodic") โ€” appends to category JSONL
  • read_memories(category: str = "episodic", limit: int = 100) -> list[dict]
  • read_all_memories(limit: int = 100) -> list[dict] โ€” all categories combined

Tools:

  • read_tool_registry() -> list[dict] โ€” reads tools/registry.yaml
  • write_tool_registry(tools: list[dict]) โ€” writes tool manifest
  • read_tool(name: str) -> str โ€” reads tool source code
  • write_tool(name: str, content: str) โ€” writes tool implementation

Evolution Metadata:

  • read_evolution_history() -> list[dict] โ€” reads evolution/history.jsonl
  • read_evolution_metrics() -> dict โ€” reads evolution/metrics.json

Configuration Reference

EvolveConfig Fields

Field Type Default Description
batch_size int 10 Tasks per solve round
max_cycles int 20 Maximum evolution iterations
holdout_ratio float 0.2 Fraction held out for validation
evolve_prompts bool True Allow prompt mutation
evolve_skills bool True Allow skill creation/modification
evolve_memory bool True Allow memory writes
evolve_tools bool False Allow tool implementation changes
trajectory_only bool False Hide scores from evolver
evolver_model str "us.anthropic.claude-opus-4-6-v1" LLM for evolution engine
evolver_max_tokens int 16384 Max tokens for evolver calls
egl_threshold float 0.05 Convergence epsilon
egl_window int 3 Cycles for plateau detection
extra dict {} Extension point for custom params

Loading from YAML

yaml
# config.yaml
batch_size: 15
max_cycles: 30
evolve_prompts: true
evolve_skills: true
evolve_memory: false
evolver_model: us.anthropic.claude-opus-4-6-v1
egl_threshold: 0.03
egl_window: 5
extra:
  solver_proposed: true
  merge_threshold: 0.7
python
config = ae.EvolveConfig.from_yaml("config.yaml")

Configuration Strategies

Conservative (stable improvement):

python
config = ae.EvolveConfig(
    batch_size=10,
    max_cycles=10,
    evolve_prompts=True,
    evolve_skills=False,
    evolve_memory=False,
    egl_threshold=0.05,
)

Aggressive (maximum exploration):

python
config = ae.EvolveConfig(
    batch_size=20,
    max_cycles=50,
    evolve_prompts=True,
    evolve_skills=True,
    evolve_memory=True,
    evolve_tools=True,
    egl_threshold=0.01,
    egl_window=7,
)

Skill-focused (procedure discovery):

python
config = ae.EvolveConfig(
    batch_size=10,
    max_cycles=25,
    evolve_prompts=False,
    evolve_skills=True,
    evolve_memory=True,
)

Built-in Agents

SWE Agent (`seed_workspaces/swe/`)

Domain: SWE-bench code patching
Model: Claude Opus 4.6 via AWS Bedrock
Framework: Strands-agents (CodeDojo-compatible)

Key features:

  • Verify-fix loop: runs tests before and after each edit
  • Hypothesis-first approach: form theory before exploring
  • Skill proposal generation: agent reflects on verification process
  • Conversation capture with per-turn token tracking
  • Dynamic tool loading from workspace tools/registry.yaml

Tools available: bash, submit, text_editor, python_exec

Terminal Agent (`seed_workspaces/terminal/`)

Domain: Terminal-Bench 2.0 shell challenges
Model: Claude Sonnet 4 via AWS Bedrock
Framework: Strands-agents

Key features:

  • Concurrent timeout enforcement via ThreadPoolExecutor
  • Test file copying only during evaluation (prevents cheating)
  • Pre-built skills: self-verification, environment-discovery, scientific-computing, debug-and-fix
  • Memory injection disabled (time-sensitive tasks)
  • Graceful timeout fallback

Tools available: bash, python, submit

MCP Agent (`seed_workspaces/mcp/`)

Domain: MCP-Atlas tool calling
Model: Claude Opus 4.6 via AWS Bedrock
Framework: Strands-agents with MCP integration

Key features:

  • MCP server connection management
  • Tool discovery and invocation
  • Multi-provider support via LiteLLM

Built-in Benchmarks

SWE-bench Verified

Module: agent_evolve.benchmarks.swe_verified
Tasks: Real GitHub issues from popular Python repositories
Evaluation: Runs test suite, checks if agent's patch fixes the issue
Metric: Pass rate (0.0 to 1.0)

MCP-Atlas

Module: agent_evolve.benchmarks.mcp_atlas
Tasks: Tool calling scenarios with MCP servers
Evaluation: Checks correct tool selection and parameter passing
Metric: Accuracy (0.0 to 1.0)

Terminal-Bench 2.0

Module: agent_evolve.benchmarks.terminal2
Tasks: Shell command challenges (file manipulation, system admin, scripting)
Evaluation: Runs test scripts to verify terminal state
Metric: Pass rate (0.0 to 1.0)

SkillsBench

Module: agent_evolve.benchmarks.skill_bench
Tasks: Multi-step procedural tasks
Evaluation: Checks step-by-step correctness
Metric: Accuracy (0.0 to 1.0)

ARC-AGI-3

Module: agent_evolve.benchmarks.arc_agi3
Tasks: Interactive game levels (25 games, 181 levels)
Evaluation: RHAE score (ratio of human to agent actions, squared)
Metric: Average RHAE across levels (0.0 to 1.0)

Evolution Algorithms

AEvolveEngine (SkillForge)

Module: agent_evolve.algorithms.skillforge.engine
Strategy: LLM-driven workspace mutation

The default engine gives an LLM full bash tool access to the workspace and asks it to improve the agent based on observations. This is the most flexible engine โ€” it can make arbitrary changes to any workspace file.

Context provided to the LLM:

  • Recent observations (task inputs, agent outputs, feedback)
  • Current system prompt
  • Current skill library
  • Pending draft proposals
  • Score history

Mutation capabilities:

  • Edit system prompt (refine, consolidate, extend)
  • Create new skills from observed patterns
  • Merge overlapping skills
  • Write episodic memory entries
  • Review and curate draft proposals

GuidedSynthesisEngine

Module: agent_evolve.algorithms.guided_synth
Strategy: Memory-first, curated skills

Emphasizes learning from failures before creating skills. Conservative approach that prevents skill bloat.

Process:

  1. Extract lessons from failed tasks
  2. Write episodic memory entries
  3. After accumulating patterns, synthesize skill proposals
  4. Curate proposals: ACCEPT, MERGE, or SKIP

AdaptiveEvolutionEngine

Module: agent_evolve.algorithms.adaptive
Strategy: Reward tracking + observation filtering

Adjusts intervention intensity based on score trends. Makes smaller changes when improving, larger changes when plateaued.

AdaptiveSkillEngine

Module: agent_evolve.algorithms.adaptive_skill
Strategy: Skill-centric discovery

Focuses exclusively on building the skill library. Identifies task categories where the agent fails and creates targeted skills.

Skill System

Skill File Format

markdown
---
name: verify-edge-cases
description: "TRIGGER when: checking boundary conditions. DO NOT TRIGGER: for happy-path tests."
---

## Pattern
Test all falsy-but-valid values: 0, False, "", [], {}

## Process
1. List all input boundaries
2. Run each against the implementation
3. Check both output AND side effects

Skill Discovery Process

  1. Agent proposes: During solve(), agent writes draft to skills/_drafts/
  2. Engine reviews: During step(), engine reads drafts and decides:
    • ACCEPT: Move to skills/{name}/SKILL.md
    • MERGE: Combine with existing similar skill
    • SKIP: Discard (too narrow, redundant, or incorrect)
  3. Engine creates: Engine can also create skills directly from observation analysis
  4. Refinement: Existing skills are updated based on new observations

Skill Library Management

Target: 5-10 broad, reusable skills per workspace. Avoid:

  • 30+ narrow skills (library bloat)
  • Skills that duplicate system prompt content
  • Skills with no TRIGGER condition (always-on = should be in prompt)

Memory System

Episodic Memory

Records specific lessons from task attempts:

json
{"content": "pytest --no-header flag needed for clean output", "category": "episodic", "task_id": "django-16379"}
{"content": "Off-by-one errors common in range() with len()", "category": "episodic", "task_id": "numpy-8823"}

Semantic Memory

General domain knowledge:

json
{"content": "Django uses reverse URL resolution via urlpatterns", "category": "semantic"}
{"content": "NumPy broadcasting rules: dimensions must match or be 1", "category": "semantic"}

Memory Limits

  • BaseAgent.reload_from_fs() loads up to 200 memory entries by default
  • AgentWorkspace.read_memories() defaults to limit=100
  • Old memories should be pruned or consolidated during evolution

Version Control

Git Tagging Convention

Tag When Created Purpose
pre-evo-1 Before cycle 1 evolution Snapshot of solve-only state
evo-1 After cycle 1 evolution Snapshot of evolved state
pre-evo-2 Before cycle 2 evolution Snapshot before next mutation
evo-2 After cycle 2 evolution Snapshot of evolved state

Useful Git Commands

bash
# See all evolution checkpoints
git tag -l "evo-*"

# Compare two evolution stages
git diff evo-1 evo-10

# See what changed in a specific cycle
git diff pre-evo-5 evo-5

# Read a file at a specific point in time
git show evo-3:prompts/system.md

# Revert to a known good state
git checkout evo-5 -- .

Observation Pipeline

JSONL Format

Each observation is stored in evolution/observations/batch_{label}.jsonl:

json
{
  "task_id": "django__django-16379",
  "task_input": "Fix FileBasedCache has_key method...",
  "task_metadata": {},
  "agent_output": "--- a/django/core/cache/backends/filebased.py\n+++ ...",
  "steps": [
    {"tool": "bash", "action": "read_file", "file": "django/core/cache/backends/filebased.py"},
    {"tool": "text_editor", "action": "edit", "file": "django/core/cache/backends/filebased.py"}
  ],
  "success": true,
  "score": 1.0,
  "feedback_detail": "All 24 tests passed"
}

Querying Observations

python
history = EvolutionHistory("./my-workspace")

# All observations from last 3 cycles
recent = history.get_observations(last_n_cycles=3)

# Only failures
failures = history.get_observations(only_failures=True)

# Score curve
scores = history.get_score_curve()  # [(1, 0.62), (2, 0.68), ...]

FAQ

Can I use A-Evolve with any LLM?

Yes. The agent can use any LLM for solving. The evolver model is configurable via EvolveConfig.evolver_model. Supported providers: Anthropic (direct API), OpenAI, AWS Bedrock, LiteLLM (multi-provider).

Does evolution require training data?

No in the traditional ML sense. You need a BenchmarkAdapter that provides tasks and evaluation, but there are no training/gradient steps. Evolution is purely file-system mutation guided by LLM reasoning.

How many cycles should I run?

Start with 10 cycles and check convergence. If score is still improving, run more. Default convergence detection (egl_threshold=0.05, egl_window=3) stops automatically when improvement plateaus.

Can I resume evolution after stopping?

Yes. The workspace retains its evolved state. Create a new Evolver pointing to the same workspace and call run() again.

Is evolution deterministic?

No. LLM calls are inherently non-deterministic. Running the same config twice may produce different evolved agents with similar final scores.

Can I evolve multiple agents simultaneously?

Yes, but each must have its own workspace directory. The evolution loop modifies workspace files directly, so concurrent access to the same workspace is not safe.

What's the cost per evolution cycle?

Each cycle involves: (batch_size) agent solve calls + 1 evolver call. For batch_size=10 with Claude, expect ~$5-20 per cycle depending on task complexity and model used.

Can I use A-Evolve without a benchmark?

Not directly. The evolution loop requires BenchmarkAdapter.evaluate() to produce Feedback. However, you can implement a custom benchmark that uses human evaluation, LLM-as-judge, or any other scoring mechanism.

Overview (`20-ml-paper-writing/ml-paper-writing/templates/README.md`)

LaTeX Templates for ML/AI Conferences

This directory contains official LaTeX templates for major machine learning and AI conferences.

Systems conference templates (OSDI, NSDI, ASPLOS, SOSP) have moved to the systems-paper-writing skill.

Compiling LaTeX to PDF

Setup:

  1. Install TeX Live (full distribution recommended)

    • macOS: brew install --cask mactex
    • Ubuntu: sudo apt install texlive-full
    • Windows: Download from tug.org/texlive

  2. Install VS Code extension: LaTeX Workshop by James Yu

    • Open VS Code โ†’ Extensions (Cmd/Ctrl+Shift+X) โ†’ Search "LaTeX Workshop" โ†’ Install

Usage:

  • Open any .tex file in VS Code
  • Save the file (Cmd/Ctrl+S) โ†’ Auto-compiles to PDF
  • Click the green play button or use Cmd/Ctrl+Alt+B to build
  • View PDF: Click "View LaTeX PDF" icon or Cmd/Ctrl+Alt+V
  • Side-by-side view: Cmd/Ctrl+Alt+V then drag tab

Settings (add to VS Code settings.json):

json
{
  "latex-workshop.latex.autoBuild.run": "onSave",
  "latex-workshop.view.pdf.viewer": "tab",
  "latex-workshop.latex.recipes": [
    {
      "name": "pdflatex โ†’ bibtex โ†’ pdflatex ร— 2",
      "tools": ["pdflatex", "bibtex", "pdflatex", "pdflatex"]
    }
  ]
}

Option 2: Command Line

bash
# Basic compilation
pdflatex main.tex

# With bibliography (full workflow)
pdflatex main.tex
bibtex main
pdflatex main.tex
pdflatex main.tex

# Using latexmk (handles dependencies automatically)
latexmk -pdf main.tex

# Continuous compilation (watches for changes)
latexmk -pdf -pvc main.tex

Option 3: Overleaf (Online)

  1. Go to overleaf.com
  2. New Project โ†’ Upload Project โ†’ Upload the template folder as ZIP
  3. Edit online with real-time PDF preview
  4. No local installation needed

Option 4: Other IDEs

IDE Extension/Plugin Notes
Cursor LaTeX Workshop Same as VS Code
Sublime Text LaTeXTools Popular, well-maintained
Vim/Neovim VimTeX Powerful, keyboard-driven
Emacs AUCTeX Comprehensive LaTeX environment
TeXstudio Built-in Dedicated LaTeX IDE
Texmaker Built-in Cross-platform LaTeX editor

Troubleshooting Compilation

"File not found" errors:

bash
# Ensure you're in the template directory
cd templates/icml2026
pdflatex example_paper.tex

Bibliography not appearing:

bash
# Run bibtex after first pdflatex
pdflatex main.tex
bibtex main        # Uses main.aux to find citations
pdflatex main.tex  # Incorporates bibliography
pdflatex main.tex  # Resolves references

Missing packages:

bash
# TeX Live package manager
tlmgr install <package-name>

# Or install full distribution to avoid this

Available Templates

ML/AI Conferences / ML/AI

Conference Directory Year Source
ICML icml2026/ 2026 Official ICML
ICLR iclr2026/ 2026 Official GitHub
NeurIPS neurips2025/ 2025 Community template
ACL acl/ 2025+ Official ACL
AAAI aaai2026/ 2026 AAAI Author Kit
COLM colm2025/ 2025 Official COLM

Systems Conferences

Systems conference templates (OSDI, NSDI, ASPLOS, SOSP) are now in the systems-paper-writing skill.

Usage

ICML 2026

latex
\documentclass{article}
\usepackage{icml2026}  % For submission
% \usepackage[accepted]{icml2026}  % For camera-ready

\begin{document}
% Your paper content
\end{document}

Key files:

  • icml2026.sty - Style file
  • icml2026.bst - Bibliography style
  • example_paper.tex - Example document

ICLR 2026

latex
\documentclass{article}
\usepackage[submission]{iclr2026_conference}  % For submission
% \usepackage[final]{iclr2026_conference}  % For camera-ready

\begin{document}
% Your paper content
\end{document}

Key files:

  • iclr2026_conference.sty - Style file
  • iclr2026_conference.bst - Bibliography style
  • iclr2026_conference.tex - Example document

ACL Venues (ACL, EMNLP, NAACL)

latex
\documentclass[11pt]{article}
\usepackage[review]{acl}  % For review
% \usepackage{acl}  % For camera-ready

\begin{document}
% Your paper content
\end{document}

Key files:

  • acl.sty - Style file
  • acl_natbib.bst - Bibliography style
  • acl_latex.tex - Example document

AAAI 2026

latex
\documentclass[letterpaper]{article}
\usepackage[submission]{aaai2026}  % For submission
% \usepackage{aaai2026}  % For camera-ready

\begin{document}
% Your paper content
\end{document}

Key files:

  • aaai2026.sty - Style file
  • aaai2026.bst - Bibliography style

COLM 2025

latex
\documentclass{article}
\usepackage[submission]{colm2025_conference}  % For submission
% \usepackage[final]{colm2025_conference}  % For camera-ready

\begin{document}
% Your paper content
\end{document}

Key files:

  • colm2025_conference.sty - Style file
  • colm2025_conference.bst - Bibliography style

Page Limits Summary

Conference Submission Camera-Ready Notes
ICML 2026 8 pages 9 pages +unlimited refs/appendix
ICLR 2026 9 pages 10 pages +unlimited refs/appendix
NeurIPS 2025 9 pages 9 pages +checklist outside limit
ACL 2025 8 pages (long) varies +unlimited refs/appendix
AAAI 2026 7 pages 8 pages +unlimited refs/appendix
COLM 2025 9 pages 10 pages +unlimited refs/appendix

Systems conferences (OSDI, NSDI, ASPLOS, SOSP): See the systems-paper-writing skill for page limits and templates.

Common Issues

Compilation Errors

  1. Missing packages: Install full TeX distribution (TeX Live Full or MikTeX)
  2. Bibliography errors: Use the provided .bst file with \bibliographystyle{}
  3. Font warnings: Install cm-super or use \usepackage{lmodern}

Anonymization

For submission, ensure:

  • No author names in \author{}
  • No acknowledgments section
  • No grant numbers
  • Use anonymous repositories
  • Cite own work in third person

Common LaTeX Packages

latex
% Recommended packages (check compatibility with venue style)
\usepackage{amsmath,amsthm,amssymb}  % Math
\usepackage{graphicx}                 % Figures
\usepackage{booktabs}                 % Tables
\usepackage{hyperref}                 % Links
\usepackage{algorithm,algorithmic}    % Algorithms
\usepackage{natbib}                   % Citations

Updating Templates

Templates are updated annually. Check official sources before each submission:
ML/AI:

Systems: See the systems-paper-writing skill for OSDI, NSDI, ASPLOS, SOSP template sources

Overview (`20-ml-paper-writing/ml-paper-writing/templates/aaai2026/README.md`)

AAAI 2026 ็ปŸไธ€LaTeXๆจกๆฟไฝฟ็”จ่ฏดๆ˜Ž / AAAI 2026 Unified LaTeX Template Guide

๐Ÿ“ ้‡่ฆ่ฏดๆ˜Ž / Important Notice: ๆœฌไป“ๅบ“ๅ€ŸๅŠฉCursorๅœจAAAI 2026ๅฎ˜ๆ–นๆจกๆฟๅŸบ็ก€ไธŠๆ”น่ฟ›ๅพ—ๅˆฐใ€‚ๅฆ‚ๆžœ้‡ๅˆฐไธๆปก่ถณๆˆ–ๆœ‰ๅ†ฒ็ช็š„ๆƒ…ๅ†ต๏ผŒ่ฏท็งฏๆžๆissuesใ€‚

๐Ÿ“ Important Notice: This repository is improved based on the official AAAI 2026 template with the assistance of Cursor. If you encounter any issues or conflicts, please actively submit issues.

ไธญๆ–‡ | English

๐ŸŒ ๅœจ็บฟๆŸฅ็œ‹ / Online Access

๐Ÿ“– ๅœจ็บฟ้˜…่ฏปๅ’Œๆต‹่ฏ•ๆจกๆฟ: https://cn.overleaf.com/read/wyhcnvcrtpyt#cd4a07

๐Ÿ“– Online View and Test Template: https://cn.overleaf.com/read/wyhcnvcrtpyt#cd4a07

๐Ÿ’ก ๆ็คบ / Tips:

  • ไธญๆ–‡๏ผšๆ‚จๅฏไปฅ้€š่ฟ‡ไธŠ่ฟฐ้“พๆŽฅๅœจOverleafไธญ็›ดๆŽฅๆŸฅ็œ‹ใ€็ผ–่พ‘ๅ’Œ็ผ–่ฏ‘ๆจกๆฟ๏ผŒๆ— ้œ€ๆœฌๅœฐๅฎ‰่ฃ…LaTeX็Žฏๅขƒ
  • English: You can view, edit, and compile the template directly in Overleaf using the link above, without needing a local LaTeX installation

ไธญๆ–‡็‰ˆๆœฌ

ๆฆ‚่ฟฐ โœ…

ๆˆ‘ๅทฒ็ปๅฐ†AAAI 2026็š„ไธคไธช็‰ˆๆœฌ๏ผˆๅŒฟๅๆŠ•็จฟ็‰ˆๆœฌๅ’Œcamera-ready็‰ˆๆœฌ๏ผ‰ๅฎŒๆ•ดๅˆๅนถๆˆไธ€ไธช็ปŸไธ€็š„ๆจกๆฟๆ–‡ไปถ aaai2026-unified-template.texใ€‚

่ฏฅๆจกๆฟๅŒ…ๅซไบ†ๅŽŸๅง‹ไธคไธชๆจกๆฟ็š„ๆ‰€ๆœ‰ๅฎŒๆ•ดๅ†…ๅฎน๏ผˆๅ…ฑ886่กŒ๏ผŒๆฏ”ๅŽŸๅง‹ๆ–‡ไปถๆ›ดๅ…จ้ข๏ผ‰๏ผŒๅŒ…ๆ‹ฌ๏ผš

  • ๆ‰€ๆœ‰ๆ ผๅผๅŒ–่ฏดๆ˜Žๅ’Œ่ฆๆฑ‚
  • ๅฎŒๆ•ด็š„็คบไพ‹ไปฃ็ ๅ’Œ่กจๆ ผ
  • ๅ›พ็‰‡ๅค„็†ๆŒ‡ๅ—
  • ๅ‚่€ƒๆ–‡็Œฎๆ ผๅผ่ฆๆฑ‚
  • ๆ‰€ๆœ‰็ซ ่Š‚ๅ’Œ้™„ๅฝ•ๅ†…ๅฎน
  • ็‰ˆๆœฌ็‰นๅฎš็š„Acknowledgments้ƒจๅˆ†

ไธป่ฆๅทฎๅผ‚ๅˆ†ๆž

้€š่ฟ‡ๆฏ”่พƒๅŽŸๅง‹็š„ไธคไธชๆจกๆฟ๏ผŒๆˆ‘ๅ‘็Žฐไธป่ฆๅทฎๅผ‚ๅœจไบŽ๏ผš

1. ๅŒ…็š„ๅŠ ่ฝฝๆ–นๅผ

  • ๅŒฟๅ็‰ˆๆœฌ: \usepackage[submission]{aaai2026}
  • Camera-ready็‰ˆๆœฌ: \usepackage{aaai2026}

2. ๆ ‡้ข˜ๅทฎๅผ‚

  • ๅŒฟๅ็‰ˆๆœฌ: "AAAI Press Anonymous Submission Instructions for Authors Using LaTeX"
  • Camera-ready็‰ˆๆœฌ: "AAAI Press Formatting Instructions for Authors Using LaTeX --- A Guide"
  • ๅŒฟๅ็‰ˆๆœฌ: Links็Žฏๅขƒ่ขซๆณจ้‡ŠๆŽ‰๏ผŒ้˜ฒๆญขๆณ„้œฒไฝœ่€…่บซไปฝ
  • Camera-ready็‰ˆๆœฌ: Links็Žฏๅขƒๆญฃๅธธๆ˜พ็คบ

4. ๅ†…ๅฎน้ƒจๅˆ†ๅทฎๅผ‚

  • ๅŒฟๅ็‰ˆๆœฌ: ๅŒ…ๅซ"Preparing an Anonymous Submission"้ƒจๅˆ†็š„็‰นๆฎŠ่ฏดๆ˜Ž
  • Camera-ready็‰ˆๆœฌ: ๅŒ…ๅซๅฎŒๆ•ด็š„ๆ ผๅผ่ฏดๆ˜Žๅ’Œ็‰ˆๆƒไฟกๆฏ

ไพ่ต–ๆ–‡ไปถๆฃ€ๆŸฅ็ป“ๆžœ

โœ… ๅทฒ้ชŒ่ฏๅนถๅคๅˆถๅˆฐไธป็›ฎๅฝ•็š„ๆ–‡ไปถ๏ผš

  • aaai2026.sty - AAAI 2026 ๆ ทๅผๆ–‡ไปถ๏ผˆไธคไธช็‰ˆๆœฌๅฎŒๅ…จ็›ธๅŒ๏ผ‰
  • aaai2026.bst - ๅ‚่€ƒๆ–‡็Œฎๆ ทๅผๆ–‡ไปถ๏ผˆไธคไธช็‰ˆๆœฌๅฎŒๅ…จ็›ธๅŒ๏ผ‰
  • aaai2026.bib - ็คบไพ‹ๅ‚่€ƒๆ–‡็Œฎๆ–‡ไปถ
  • figure1.pdf ๅ’Œ figure2.pdf - ็คบไพ‹ๅ›พ็‰‡ๆ–‡ไปถ

ๆ‰€ๆœ‰่ฟ™ไบ›ๆ–‡ไปถๅœจไธคไธช็‰ˆๆœฌไธญ้ƒฝๆ˜ฏ็›ธๅŒ็š„๏ผŒๅ› ๆญค็ปŸไธ€ๆจกๆฟๅฏไปฅๆญฃๅธธๅทฅไฝœใ€‚

ๅฆ‚ไฝ•ไฝฟ็”จ็ปŸไธ€ๆจกๆฟ

ๅˆ‡ๆขๅˆฐๅŒฟๅๆŠ•็จฟ็‰ˆๆœฌ

ๅœจๆจกๆฟๆ–‡ไปถ็ฌฌ11่กŒ๏ผŒๅ–ๆถˆๆณจ้‡Š่ฟ™ไธ€่กŒ๏ผš

latex
\def\aaaianonymous{true}

ๅˆ‡ๆขๅˆฐCamera-ready็‰ˆๆœฌ

ๅœจๆจกๆฟๆ–‡ไปถ็ฌฌ11่กŒ๏ผŒๆณจ้‡ŠๆŽ‰ๆˆ–ๅˆ ้™ค่ฟ™ไธ€่กŒ๏ผš

latex
% \def\aaaianonymous{true}

ไธ€้”ฎๅˆ‡ๆข็š„ๆ ธๅฟƒๆœบๅˆถ

็ปŸไธ€ๆจกๆฟไฝฟ็”จไบ†LaTeX็š„ๆกไปถ็ผ–่ฏ‘ๅŠŸ่ƒฝ๏ผš

latex
% ๆกไปถๅŒ…ๅŠ ่ฝฝ
\ifdefined\aaaianonymous
    \usepackage[submission]{aaai2026}  % ๅŒฟๅ็‰ˆๆœฌ
\else
    \usepackage{aaai2026}              % Camera-ready็‰ˆๆœฌ
\fi

% ๆกไปถๆ ‡้ข˜่ฎพ็ฝฎ
\ifdefined\aaaianonymous
    \title{AAAI Press Anonymous Submission\\Instructions for Authors Using \LaTeX{}}
\else
    \title{AAAI Press Formatting Instructions \\for Authors Using \LaTeX{} --- A Guide}
\fi

% ๆกไปถๅ†…ๅฎนๆ˜พ็คบ
\ifdefined\aaaianonymous
    % ๅŒฟๅ็‰ˆๆœฌ็‰นๆœ‰ๅ†…ๅฎน
\else
    % Camera-ready็‰ˆๆœฌ็‰นๆœ‰ๅ†…ๅฎน
\fi

ๆ–‡ไปถๆธ…ๅ•

ไธป็›ฎๅฝ•็ŽฐๅœจๅŒ…ๅซไปฅไธ‹ๆ–‡ไปถ๏ผš

  • aaai2026-unified-template.tex - ็ปŸไธ€ไธป่ฎบๆ–‡ๆจกๆฟๆ–‡ไปถ
  • aaai2026-unified-supp.tex - ็ปŸไธ€่กฅๅ……ๆๆ–™ๆจกๆฟๆ–‡ไปถ
  • aaai2026.sty - AAAI 2026 LaTeX ๆ ทๅผๆ–‡ไปถ
  • aaai2026.bst - ๅ‚่€ƒๆ–‡็Œฎๆ ทๅผๆ–‡ไปถ
  • aaai2026.bib - ็คบไพ‹ๅ‚่€ƒๆ–‡็Œฎๆ–‡ไปถ
  • figure1.pdf - ็คบไพ‹ๅ›พ็‰‡1
  • figure2.pdf - ็คบไพ‹ๅ›พ็‰‡2
  • README.md - ๆœฌ่ฏดๆ˜Žๆ–‡ๆกฃ

่กฅๅ……ๆๆ–™ๆจกๆฟ (Supplementary Material Template)

ๆฆ‚่ฟฐ

aaai2026-unified-supp.tex ๆ˜ฏไธ“้—จไธบAAAI 2026่กฅๅ……ๆๆ–™่ฎพ่ฎก็š„็ปŸไธ€ๆจกๆฟ๏ผŒไธŽไธป่ฎบๆ–‡ๆจกๆฟไฝฟ็”จ็›ธๅŒ็š„็‰ˆๆœฌๅˆ‡ๆขๆœบๅˆถใ€‚

ไธป่ฆๅŠŸ่ƒฝ

  • ็‰ˆๆœฌๅˆ‡ๆข: ้€š่ฟ‡ไฟฎๆ”นไธ€่กŒไปฃ็ ๅœจๅŒฟๅๆŠ•็จฟๅ’Œcamera-ready็‰ˆๆœฌ้—ดๅˆ‡ๆข
  • ่กฅๅ……ๅ†…ๅฎนๆ”ฏๆŒ: ๆ”ฏๆŒ้ขๅค–็š„ๅฎž้ชŒใ€ๆŽจๅฏผใ€ๆ•ฐๆฎใ€ๅ›พ่กจใ€็ฎ—ๆณ•็ญ‰
  • ๆ ผๅผไธ€่‡ดๆ€ง: ไธŽไธป่ฎบๆ–‡ๆจกๆฟไฟๆŒๅฎŒๅ…จไธ€่‡ด็š„ๆ ผๅผ่ฆๆฑ‚
  • ไปฃ็ ็คบไพ‹: ๅŒ…ๅซ็ฎ—ๆณ•ใ€ไปฃ็ ๅˆ—่กจ็ญ‰่กฅๅ……ๆๆ–™็š„็คบไพ‹

ไฝฟ็”จๆ–นๆณ•

ไธŽไธป่ฎบๆ–‡ๆจกๆฟ็›ธๅŒ๏ผŒๅช้œ€ไฟฎๆ”น็ฌฌ11่กŒ๏ผš

latex
% ๅŒฟๅๆŠ•็จฟ็‰ˆๆœฌ
\def\aaaianonymous{true}

% Camera-ready็‰ˆๆœฌ  
% \def\aaaianonymous{true}

่กฅๅ……ๆๆ–™ๅ†…ๅฎนๅปบ่ฎฎ

  • ้ขๅค–็š„ๅฎž้ชŒ็ป“ๆžœๅ’Œๆถˆ่ž็ ”็ฉถ
  • ่ฏฆ็ป†็š„ๆ•ฐๅญฆๆŽจๅฏผๅ’Œ่ฏๆ˜Ž
  • ๆ›ดๅคš็š„ๅ›พ่กจๅ’Œๅฏ่ง†ๅŒ–
  • ็ฎ—ๆณ•ไผชไปฃ็ ๅ’Œๅฎž็Žฐ็ป†่Š‚
  • ๆ•ฐๆฎ้›†ๆ่ฟฐๅ’Œ้ข„ๅค„็†ๆญฅ้ชค
  • ่ถ…ๅ‚ๆ•ฐ่ฎพ็ฝฎๅ’Œๅฎž้ชŒ้…็ฝฎ
  • ๅคฑ่ดฅๆกˆไพ‹ๅˆ†ๆž
  • ่ฎก็ฎ—ๅคๆ‚ๅบฆๅˆ†ๆž

ไฝฟ็”จๆฃ€ๆŸฅๆธ…ๅ• (Usage Checklist)

๐Ÿ“‹ ๆŠ•็จฟๅ‰ๆฃ€ๆŸฅๆธ…ๅ• (Pre-Submission Checklist)

็‰ˆๆœฌ่ฎพ็ฝฎ:

  • ๅทฒ่ฎพ็ฝฎ \def\aaaianonymous{true} (ๅŒฟๅๆŠ•็จฟ)
  • ๅทฒๆณจ้‡ŠๆŽ‰ๆ‰€ๆœ‰ๅฏ่ƒฝๆšด้œฒ่บซไปฝ็š„ไฟกๆฏ
  • ๅทฒๅŒฟๅๅŒ–ๅ‚่€ƒๆ–‡็Œฎ๏ผˆ็งป้™คไฝœ่€…ๅง“ๅ๏ผ‰

ๅ†…ๅฎนๅฎŒๆ•ดๆ€ง:

  • ๆ ‡้ข˜ใ€ๆ‘˜่ฆใ€ๅ…ณ้”ฎ่ฏๅทฒๅกซๅ†™
  • ๆ‰€ๆœ‰็ซ ่Š‚ๅ†…ๅฎนๅฎŒๆ•ด
  • ๅ›พ่กจ็ผ–ๅท่ฟž็ปญไธ”ๆญฃ็กฎ
  • ๅ‚่€ƒๆ–‡็Œฎๆ ผๅผๆญฃ็กฎ
  • ่กฅๅ……ๆๆ–™๏ผˆๅฆ‚ๆœ‰๏ผ‰ๅทฒๅ‡†ๅค‡

ๆ ผๅผๆฃ€ๆŸฅ:

  • ้กต้ข่พน่ท็ฌฆๅˆ่ฆๆฑ‚
  • ๅญ—ไฝ“ๅ’Œๅญ—ๅทๆญฃ็กฎ
  • ่กŒ้—ด่ท็ฌฆๅˆๆ ‡ๅ‡†
  • ๅ›พ่กจไฝ็ฝฎๅ’Œๅคงๅฐๅˆ้€‚
  • ๆ•ฐๅญฆๅ…ฌๅผๆ ผๅผๆญฃ็กฎ

ๆŠ€ๆœฏๆฃ€ๆŸฅ:

  • LaTeX็ผ–่ฏ‘ๆ— ้”™่ฏฏ
  • ๅ‚่€ƒๆ–‡็Œฎๆญฃ็กฎ็”Ÿๆˆ
  • PDF่พ“ๅ‡บๆญฃๅธธ
  • ๆ–‡ไปถๅคงๅฐๅœจ้™ๅˆถ่Œƒๅ›ดๅ†…

๐Ÿ“‹ ๅฝ•็”จๅŽๆฃ€ๆŸฅๆธ…ๅ• (Post-Acceptance Checklist)

็‰ˆๆœฌๅˆ‡ๆข:

  • ๅทฒๆณจ้‡ŠๆŽ‰ \def\aaaianonymous{true} (camera-ready)
  • ๅทฒๆทปๅŠ ๅฎŒๆ•ด็š„ไฝœ่€…ไฟกๆฏ
  • ๅทฒๆทปๅŠ ๆ‰€ๆœ‰ไฝœ่€…ๅ•ไฝไฟกๆฏ
  • ๅทฒๆขๅคๆ‰€ๆœ‰่ขซๆณจ้‡Š็š„ๅ†…ๅฎน

ๅ†…ๅฎนๆ›ดๆ–ฐ:

  • ๅทฒๆ นๆฎๅฎก็จฟๆ„่งไฟฎๆ”นๅ†…ๅฎน
  • ๅทฒๆ›ดๆ–ฐๆ‰€ๆœ‰ๅ›พ่กจๅ’Œๅฎž้ชŒ
  • ๅทฒๅฎŒๅ–„่กฅๅ……ๆๆ–™
  • ๅทฒๆฃ€ๆŸฅๆ‰€ๆœ‰้“พๆŽฅๅ’Œๅผ•็”จ

ๆœ€็ปˆๆฃ€ๆŸฅ:

  • ๆœ€็ปˆPDF่ดจ้‡ๆฃ€ๆŸฅ
  • ๆ‰€ๆœ‰ๆ–‡ไปถๅทฒๅค‡ไปฝ
  • ็ฌฆๅˆไผš่ฎฎๆœ€็ปˆๆไบค่ฆๆฑ‚
  • ่กฅๅ……ๆๆ–™ๅทฒๅ•็‹ฌๆไบค๏ผˆๅฆ‚้œ€่ฆ๏ผ‰

๐Ÿ“‹ ่กฅๅ……ๆๆ–™ๆฃ€ๆŸฅๆธ…ๅ• (Supplementary Material Checklist)

ๅ†…ๅฎน็ป„็ป‡:

  • ่กฅๅ……ๆๆ–™ไธŽไธป่ฎบๆ–‡ๅ†…ๅฎนๅฏนๅบ”
  • ็ซ ่Š‚็ป“ๆž„ๆธ…ๆ™ฐๅˆ็†
  • ๅ›พ่กจ็ผ–ๅทไธŽไธป่ฎบๆ–‡ไธๅ†ฒ็ช
  • ๅ‚่€ƒๆ–‡็Œฎๆ ผๅผไธ€่‡ด

ๆŠ€ๆœฏ็ป†่Š‚:

  • ็ฎ—ๆณ•ไผชไปฃ็ ๆธ…ๆ™ฐๅฎŒๆ•ด
  • ๅฎž้ชŒ่ฎพ็ฝฎ่ฏฆ็ป†่ฏดๆ˜Ž
  • ๆ•ฐๆฎ้ข„ๅค„็†ๆญฅ้ชคๆ˜Ž็กฎ
  • ่ถ…ๅ‚ๆ•ฐ้…็ฝฎๅฎŒๆ•ด

ๆ ผๅผ่ฆๆฑ‚:

  • ไฝฟ็”จ็ปŸไธ€็š„suppๆจกๆฟ
  • ้กต้ข่ฎพ็ฝฎไธŽไธป่ฎบๆ–‡ไธ€่‡ด
  • ๅญ—ไฝ“ๅ’Œๆ ผๅผ็ฌฆๅˆ่ฆๆฑ‚
  • ๆ–‡ไปถๅคงๅฐๅœจ้™ๅˆถ่Œƒๅ›ดๅ†…

ๅฎž้™…ไฝฟ็”จๅปบ่ฎฎ

  1. ๆŠ•็จฟ้˜ถๆฎต:

    • ๅ–ๆถˆๆณจ้‡Š \def\aaaianonymous{true}
    • ็กฎไฟไธๅŒ…ๅซไปปไฝ•ๅฏ่ƒฝๆšด้œฒ่บซไปฝ็š„ไฟกๆฏ
    • ๆฃ€ๆŸฅๅ‚่€ƒๆ–‡็Œฎๆ˜ฏๅฆๅทฒๅŒฟๅๅŒ–

  2. ๅฝ•็”จๅŽๅ‡†ๅค‡final็‰ˆๆœฌ:

    • ๆณจ้‡ŠๆŽ‰ๆˆ–ๅˆ ้™ค \def\aaaianonymous{true} ่ฟ™ไธ€่กŒ
    • ๆทปๅŠ ๅฎŒๆ•ด็š„ไฝœ่€…ไฟกๆฏๅ’Œaffiliations
    • ๅ–ๆถˆๆณจ้‡Šlinks็Žฏๅขƒ๏ผˆๅฆ‚ๆžœ้œ€่ฆ๏ผ‰

  3. ็ผ–่ฏ‘ๆต‹่ฏ•:

    • ๅˆ†ๅˆซๅœจไธค็งๆจกๅผไธ‹็ผ–่ฏ‘๏ผŒ็กฎไฟ้ƒฝ่ƒฝๆญฃๅธธๅทฅไฝœ
    • ๆฃ€ๆŸฅ่พ“ๅ‡บ็š„PDFๆ˜ฏๅฆ็ฌฆๅˆ่ฆๆฑ‚
    • ้ชŒ่ฏๅ‚่€ƒๆ–‡็Œฎๆ ผๅผๆ˜ฏๅฆๆญฃ็กฎ

  4. ไพ่ต–ๆ–‡ไปถ็กฎ่ฎค:

    • ็กฎไฟๆ‰€ๆœ‰ไพ่ต–ๆ–‡ไปถ้ƒฝๅœจๅŒไธ€็›ฎๅฝ•ไธ‹
    • ๅฆ‚ๆžœ็งปๅŠจๆจกๆฟๆ–‡ไปถ๏ผŒ่ฎฐๅพ—ๅŒๆ—ถ็งปๅŠจไพ่ต–ๆ–‡ไปถ

้‡่ฆๆณจๆ„ไบ‹้กน

โš ๏ธ ๅ…ณไบŽBibliography Style:

  • aaai2026.styๆ–‡ไปถๅทฒ็ป่‡ชๅŠจ่ฎพ็ฝฎไบ†\bibliographystyle{aaai2026}
  • ไธ่ฆๅœจๆ–‡ๆกฃไธญๅ†ๆฌกๆทปๅŠ \bibliographystyle{aaai2026}ๅ‘ฝไปค
  • ๅฆๅˆ™ไผšๅ‡บ็Žฐ"Illegal, another \bibstyle command"้”™่ฏฏ
  • ๅช้œ€่ฆไฝฟ็”จ\bibliography{aaai2026}ๅ‘ฝไปคๅณๅฏ

็ผ–่ฏ‘ๅ‘ฝไปค็คบไพ‹

bash
# ็ผ–่ฏ‘LaTeXๆ–‡ๆกฃ
pdflatex aaai2026-unified-template.tex
bibtex aaai2026-unified-template
pdflatex aaai2026-unified-template.tex
pdflatex aaai2026-unified-template.tex

ๅธธ่ง้—ฎ้ข˜่งฃๅ†ณ

1. "Illegal, another \bibstyle command"้”™่ฏฏ

ๅŽŸๅ› : ้‡ๅค่ฎพ็ฝฎไบ†bibliography style
่งฃๅ†ณๆ–นๆกˆ: ๅˆ ้™คๆ–‡ๆกฃไธญ็š„\bibliographystyle{aaai2026}ๅ‘ฝไปค๏ผŒaaai2026.styไผš่‡ชๅŠจๅค„็†

2. ๅ‚่€ƒๆ–‡็Œฎๆ ผๅผไธๆญฃ็กฎ

ๅŽŸๅ› : ๅฏ่ƒฝ็ผบๅฐ‘natbibๅŒ…ๆˆ–่€…BibTeXๆ–‡ไปถ้—ฎ้ข˜
่งฃๅ†ณๆ–นๆกˆ: ็กฎไฟๆŒ‰็…งๆ ‡ๅ‡†็š„LaTeX็ผ–่ฏ‘ๆต็จ‹๏ผšpdflatex โ†’ bibtex โ†’ pdflatex โ†’ pdflatex

English Version

Overview โœ…

I have completely merged the two AAAI 2026 versions (anonymous submission and camera-ready) into a single unified template file aaai2026-unified-template.tex.

This template contains all complete content from both original templates (886 lines total, more comprehensive than the original files), including:

  • All formatting instructions and requirements
  • Complete example codes and tables
  • Image processing guidelines
  • Reference formatting requirements
  • All sections and appendix content
  • Version-specific Acknowledgments sections

Key Differences Analysis

By comparing the two original templates, the main differences are:

1. Package Loading Method

  • Anonymous version: \usepackage[submission]{aaai2026}
  • Camera-ready version: \usepackage{aaai2026}

2. Title Differences

  • Anonymous version: "AAAI Press Anonymous Submission Instructions for Authors Using LaTeX"
  • Camera-ready version: "AAAI Press Formatting Instructions for Authors Using LaTeX --- A Guide"
  • Anonymous version: Links environment commented out to prevent identity disclosure
  • Camera-ready version: Links environment displayed normally

4. Content Section Differences

  • Anonymous version: Contains special instructions in "Preparing an Anonymous Submission" section
  • Camera-ready version: Contains complete formatting instructions and copyright information

Dependency Files Verification

โœ… Files verified and copied to main directory:

  • aaai2026.sty - AAAI 2026 style file (identical in both versions)
  • aaai2026.bst - Bibliography style file (identical in both versions)
  • aaai2026.bib - Sample bibliography file
  • figure1.pdf and figure2.pdf - Sample image files

All these files are identical in both versions, so the unified template works properly.

How to Use the Unified Template

Switch to Anonymous Submission Version

On line 11 of the template file, uncomment this line:

latex
\def\aaaianonymous{true}

Switch to Camera-ready Version

On line 11 of the template file, comment out or delete this line:

latex
% \def\aaaianonymous{true}

Core Mechanism of One-Click Switching

The unified template uses LaTeX conditional compilation:

latex
% Conditional package loading
\ifdefined\aaaianonymous
    \usepackage[submission]{aaai2026}  % Anonymous version
\else
    \usepackage{aaai2026}              % Camera-ready version
\fi

% Conditional title setting
\ifdefined\aaaianonymous
    \title{AAAI Press Anonymous Submission\\Instructions for Authors Using \LaTeX{}}
\else
    \title{AAAI Press Formatting Instructions \\for Authors Using \LaTeX{} --- A Guide}
\fi

% Conditional content display
\ifdefined\aaaianonymous
    % Anonymous version specific content
\else
    % Camera-ready version specific content
\fi

File List

The main directory now contains the following files:

  • aaai2026-unified-template.tex - Unified main paper template file
  • aaai2026-unified-supp.tex - Unified supplementary material template file
  • aaai2026.sty - AAAI 2026 LaTeX style file
  • aaai2026.bst - Bibliography style file
  • aaai2026.bib - Sample bibliography file
  • figure1.pdf - Sample image 1
  • figure2.pdf - Sample image 2
  • README.md - This documentation

Supplementary Material Template

Overview

aaai2026-unified-supp.tex is a unified template specifically designed for AAAI 2026 supplementary materials, using the same version switching mechanism as the main paper template.

Key Features

  • Version Switching: Switch between anonymous submission and camera-ready versions by modifying one line of code
  • Supplementary Content Support: Supports additional experiments, derivations, data, figures, algorithms, etc.
  • Format Consistency: Maintains complete format consistency with the main paper template
  • Code Examples: Includes examples for algorithms, code listings, and other supplementary materials

Usage

Same as the main paper template, just modify line 11:

latex
% Anonymous submission version
\def\aaaianonymous{true}

% Camera-ready version
% \def\aaaianonymous{true}

Supplementary Material Content Suggestions

  • Additional experimental results and ablation studies
  • Detailed mathematical derivations and proofs
  • More figures and visualizations
  • Algorithm pseudocode and implementation details
  • Dataset descriptions and preprocessing steps
  • Hyperparameter settings and experimental configurations
  • Failure case analysis
  • Computational complexity analysis

Usage Checklist

๐Ÿ“‹ Pre-Submission Checklist

Version Setup:

  • Set \def\aaaianonymous{true} (anonymous submission)
  • Commented out all information that could reveal identity
  • Anonymized references (removed author names)

Content Completeness:

  • Title, abstract, and keywords filled
  • All sections complete
  • Figure and table numbers consecutive and correct
  • Reference format correct
  • Supplementary materials prepared (if any)

Format Check:

  • Page margins meet requirements
  • Font and font size correct
  • Line spacing meets standards
  • Figure and table positions and sizes appropriate
  • Mathematical formula format correct

Technical Check:

  • LaTeX compilation error-free
  • References generated correctly
  • PDF output normal
  • File size within limits

๐Ÿ“‹ Post-Acceptance Checklist

Version Switch:

  • Commented out \def\aaaianonymous{true} (camera-ready)
  • Added complete author information
  • Added all author affiliation information
  • Restored all commented content

Content Updates:

  • Modified content according to reviewer comments
  • Updated all figures and experiments
  • Completed supplementary materials
  • Checked all links and citations

Final Check:

  • Final PDF quality check
  • All files backed up
  • Meets conference final submission requirements
  • Supplementary materials submitted separately (if needed)

๐Ÿ“‹ Supplementary Material Checklist

Content Organization:

  • Supplementary materials correspond to main paper content
  • Chapter structure clear and reasonable
  • Figure and table numbers don't conflict with main paper
  • Reference format consistent

Technical Details:

  • Algorithm pseudocode clear and complete
  • Experimental setup explained in detail
  • Data preprocessing steps clear
  • Hyperparameter configuration complete

Format Requirements:

  • Using unified supp template
  • Page settings consistent with main paper
  • Font and format meet requirements
  • File size within limits

Practical Usage Recommendations

  1. Submission Stage:

    • Uncomment \def\aaaianonymous{true}
    • Ensure no information that could reveal identity is included
    • Check that references are anonymized

  2. Preparing final version after acceptance:

    • Comment out or delete the \def\aaaianonymous{true} line
    • Add complete author information and affiliations
    • Uncomment links environment (if needed)

  3. Compilation Testing:

    • Compile in both modes to ensure proper functionality
    • Check if the output PDF meets requirements
    • Verify reference formatting is correct

  4. Dependency File Confirmation:

    • Ensure all dependency files are in the same directory
    • Remember to move dependency files when moving the template file

Important Notes

โš ๏ธ About Bibliography Style:

  • The aaai2026.sty file automatically sets \bibliographystyle{aaai2026}
  • Do NOT add \bibliographystyle{aaai2026} command again in your document
  • Otherwise you'll get "Illegal, another \bibstyle command" error
  • Just use the \bibliography{aaai2026} command

Compilation Commands Example

bash
# Compile LaTeX document
pdflatex aaai2026-unified-template.tex
bibtex aaai2026-unified-template
pdflatex aaai2026-unified-template.tex
pdflatex aaai2026-unified-template.tex

Common Issues and Solutions

1. "Illegal, another \bibstyle command" Error

Cause: Duplicate bibliography style setting
Solution: Remove the \bibliographystyle{aaai2026} command from your document, aaai2026.sty handles it automatically

2. Incorrect Reference Format

Cause: Missing natbib package or BibTeX file issues
Solution: Follow the standard LaTeX compilation process: pdflatex โ†’ bibtex โ†’ pdflatex โ†’ pdflatex

็‰ˆๆœฌไฟกๆฏ / Version Information

  • ๆจกๆฟ็‰ˆๆœฌ / Template Version: AAAI 2026 Unified (Main + Supplementary)
  • ๅˆ›ๅปบๆ—ฅๆœŸ / Created: 2024ๅนด12ๆœˆ
  • ๆ”ฏๆŒๆ ผๅผ / Supported Formats: Anonymous Submission & Camera-Ready
  • ๆจกๆฟ็ฑปๅž‹ / Template Types: Main Paper Template & Supplementary Material Template
  • ๅ…ผๅฎนๆ€ง / Compatibility: LaTeX 2020+ / TeXLive 2024+

๐ŸŽ‰ ็Žฐๅœจๆ‚จๅช้œ€่ฆไฟฎๆ”นไธ€่กŒไปฃ็ ๅฐฑๅฏไปฅๅœจไธคไธช็‰ˆๆœฌไน‹้—ดๅˆ‡ๆข๏ผŒๅŒๆ—ถๆ‰€ๆœ‰ๅฟ…่ฆ็š„ไพ่ต–ๆ–‡ไปถ้ƒฝๅทฒ็ปๅ‡†ๅค‡ๅฐฑ็ปช๏ผ
๐ŸŽ‰ Now you only need to modify one line of code to switch between the two versions, with all necessary dependency files ready to use!

Overview (`20-ml-paper-writing/ml-paper-writing/templates/acl/README.md`)

*ACL Paper Styles

This directory contains the latest LaTeX templates for *ACL conferences.

Instructions for authors

Paper submissions to *ACL conferences must use the official ACL style
templates.

The LaTeX style files are available

Please see acl_latex.tex for an example.

Please follow the paper formatting guidelines general to *ACL
conferences:

Authors may not modify these style files or use templates designed for
other conferences.

Instructions for publications chairs

To adapt the style files for your conference, please fork this repository and
make necessary changes. Minimally, you'll need to update the name of
the conference and rename the files.

If you make improvements to the templates that should be propagated to
future conferences, please submit a pull request. Thank you in
advance!

In older versions of the templates, authors were asked to fill in the
START submission ID so that it would be stamped at the top of each
page of the anonymized version. This is no longer needed, because it
is now possible to do this stamping automatically within
START. Currently, the way to do this is for the program chair to email
[email protected] and request it.

Instructions for making changes to style files

Overview (`20-ml-paper-writing/ml-paper-writing/templates/colm2025/README.md`)

Template

Template and style files for CoLM 2025

ROADMAP (`docs/ROADMAP.md`)

๐Ÿ—บ๏ธ Roadmap

Vision

Build the most comprehensive open-source library of AI research skills, enabling AI agents to autonomously conduct experiments from hypothesis to deployment.

Target: 86 comprehensive skills โ€” achieved โœ…

Progress Overview

Metric Current Target
Skills 86 (high-quality, standardized YAML) 86 โœ…
Avg Lines/Skill 420 lines (focused + progressive disclosure) 200-500 lines
Documentation ~130,000 lines total (SKILL.md + references) 100,000+ lines
Gold Standard Skills 65 with comprehensive references 50+ โœ…
Coverage Autoresearch, Ideation, Paper Writing, Architecture, Tokenization, Fine-Tuning, Data Processing, Post-Training, Safety, Distributed, Infrastructure, Optimization, Evaluation, Inference, Agents, RAG, Multimodal, MLOps, Observability, Prompt Engineering, Emerging Techniques Full Lifecycle โœ…

Development Phases

โœ… Phase 1: Model Architecture (COMPLETE - 5 skills)

Status: Core model architectures covered

Completed Skills:

  • โœ… Megatron-Core - NVIDIA's framework for training 2B-462B param models
  • โœ… LitGPT - Lightning AI's 20+ clean LLM implementations
  • โœ… Mamba - State-space models with O(n) complexity
  • โœ… RWKV - RNN+Transformer hybrid, infinite context
  • โœ… NanoGPT - Educational GPT in ~300 lines by Karpathy

โœ… Phase 2: Tokenization (COMPLETE - 2 skills)

Status: Essential tokenization frameworks covered

Completed Skills:

  • โœ… HuggingFace Tokenizers - Rust-based, BPE/WordPiece/Unigram
  • โœ… SentencePiece - Language-independent tokenization

โœ… Phase 3: Fine-Tuning (COMPLETE - 4 skills)

Status: Core fine-tuning frameworks covered

Completed Skills:

  • โœ… Axolotl - YAML-based fine-tuning with 100+ models
  • โœ… LLaMA-Factory - WebUI no-code fine-tuning
  • โœ… Unsloth - 2x faster QLoRA fine-tuning
  • โœ… PEFT - Parameter-efficient fine-tuning with LoRA, QLoRA, DoRA, 25+ methods

โœ… Phase 4: Data Processing (COMPLETE - 2 skills)

Status: Distributed data processing covered

Completed Skills:

  • โœ… Ray Data - Distributed ML data processing
  • โœ… NeMo Curator - GPU-accelerated data curation

โœ… Phase 5: Post-Training (COMPLETE - 4 skills)

Status: RLHF and alignment techniques covered

Completed Skills:

  • โœ… TRL Fine-Tuning - Transformer Reinforcement Learning
  • โœ… GRPO-RL-Training - Group Relative Policy Optimization (gold standard)
  • โœ… OpenRLHF - Full RLHF pipeline with Ray + vLLM
  • โœ… SimPO - Simple Preference Optimization

โœ… Phase 6: Safety & Alignment (COMPLETE - 4 skills)

Status: Core safety frameworks covered

Completed Skills:

  • โœ… Constitutional AI - AI-driven self-improvement via principles
  • โœ… LlamaGuard - Safety classifier for LLM inputs/outputs
  • โœ… NeMo Guardrails - Programmable guardrails with Colang
  • โœ… Prompt Guard - Meta's 86M prompt injection & jailbreak detector

โœ… Phase 7: Distributed Training (COMPLETE - 5 skills)

Status: Major distributed training frameworks covered

Completed Skills:

  • โœ… DeepSpeed - Microsoft's ZeRO optimization
  • โœ… PyTorch FSDP - Fully Sharded Data Parallel
  • โœ… Accelerate - HuggingFace's distributed training API
  • โœ… PyTorch Lightning - High-level training framework
  • โœ… Ray Train - Multi-node orchestration

โœ… Phase 8: Optimization (COMPLETE - 6 skills)

Status: Core optimization techniques covered

Completed Skills:

  • โœ… Flash Attention - 2-4x faster attention with memory efficiency
  • โœ… bitsandbytes - 8-bit/4-bit quantization
  • โœ… GPTQ - 4-bit post-training quantization
  • โœ… AWQ - Activation-aware weight quantization
  • โœ… HQQ - Half-Quadratic Quantization without calibration data
  • โœ… GGUF - llama.cpp quantization format for CPU/Metal inference

โœ… Phase 9: Evaluation (COMPLETE - 1 skill)

Status: Standard benchmarking framework available

Completed Skills:

  • โœ… lm-evaluation-harness - EleutherAI's standard for benchmarking LLMs

โœ… Phase 10: Inference & Serving (COMPLETE - 4 skills)

Status: Production inference frameworks covered

Completed Skills:

  • โœ… vLLM - High-throughput LLM serving with PagedAttention
  • โœ… TensorRT-LLM - NVIDIA's fastest inference
  • โœ… llama.cpp - CPU/Apple Silicon inference
  • โœ… SGLang - Structured generation with RadixAttention

โœ… Phase 10.5: Infrastructure (COMPLETE - 3 skills)

Status: Cloud infrastructure and orchestration covered

Completed Skills:

  • โœ… Modal - Serverless GPU cloud with Python-native API, T4-H200 on-demand
  • โœ… SkyPilot - Multi-cloud orchestration across 20+ providers with spot recovery
  • โœ… Lambda Labs - Reserved/on-demand GPU cloud with H100/A100, persistent filesystems

โœ… Phase 11: Agents (COMPLETE - 4 skills)

Status: Major agent frameworks covered

Completed Skills:

  • โœ… LangChain - Most popular agent framework, 500+ integrations
  • โœ… LlamaIndex - Data framework for LLM apps, 300+ connectors
  • โœ… CrewAI - Multi-agent orchestration with role-based collaboration
  • โœ… AutoGPT - Autonomous AI agent platform with visual workflow builder

โœ… Phase 12: RAG (COMPLETE - 5 skills)

Status: Core RAG and vector database skills covered

Completed Skills:

  • โœ… Chroma - Open-source embedding database
  • โœ… FAISS - Facebook's similarity search, billion-scale
  • โœ… Sentence Transformers - 5000+ embedding models
  • โœ… Pinecone - Managed vector database
  • โœ… Qdrant - High-performance Rust vector search with hybrid filtering

โœ… Phase 13: Multimodal (COMPLETE - 7 skills)

Status: Comprehensive multimodal frameworks covered

Completed Skills:

  • โœ… CLIP - OpenAI's vision-language model
  • โœ… Whisper - Robust speech recognition, 99 languages
  • โœ… LLaVA - Vision-language assistant, GPT-4V level
  • โœ… Stable Diffusion - Text-to-image generation via HuggingFace Diffusers
  • โœ… Segment Anything (SAM) - Meta's zero-shot image segmentation with points/boxes/masks
  • โœ… BLIP-2 - Vision-language pretraining with Q-Former, image captioning, VQA
  • โœ… AudioCraft - Meta's MusicGen/AudioGen for text-to-music and text-to-sound

โœ… Phase 14: Advanced Optimization (COMPLETE)

Status: Advanced optimization techniques covered (merged into Phase 8)

Note: HQQ and GGUF skills have been completed and merged into Phase 8: Optimization.

โœ… Phase 15: MLOps & Observability (COMPLETE - 5 skills)

Status: Core MLOps and LLM observability covered

Completed Skills:

  • โœ… MLflow - Open-source MLOps platform for tracking experiments
  • โœ… TensorBoard - Visualization and experiment tracking
  • โœ… Weights & Biases - Experiment tracking and collaboration
  • โœ… LangSmith - LLM observability, tracing, evaluation
  • โœ… Phoenix - Open-source AI observability with OpenTelemetry tracing

โœ… Phase 16: Prompt Engineering & Advanced Applications (COMPLETE - 6 skills)

Status: Core prompt engineering and multi-agent tools covered

Completed Skills:

  • โœ… DSPy - Declarative prompt optimization and LM programming
  • โœ… Guidance - Constrained generation and structured prompting
  • โœ… Instructor - Structured output with Pydantic models
  • โœ… Outlines - Structured text generation with regex and grammars
  • โœ… CrewAI - Multi-agent orchestration (completed in Phase 11)
  • โœ… AutoGPT - Autonomous agents (completed in Phase 11)

โœ… Phase 17: Extended Multimodal (COMPLETE)

Status: All extended multimodal skills complete, merged into Phase 13

Note: BLIP-2, SAM, and AudioCraft have been completed and merged into Phase 13: Multimodal.

โœ… Phase 18: Emerging Techniques (COMPLETE - 6 skills)

Status: Core emerging techniques covered

Completed Skills:

  • โœ… MoE Training - Mixture of Experts with DeepSpeed/HuggingFace
  • โœ… Model Merging - mergekit, SLERP, and model composition
  • โœ… Long Context - RoPE extensions, ALiBi, and context scaling
  • โœ… Speculative Decoding - Medusa, Lookahead, and draft models for faster inference
  • โœ… Knowledge Distillation - MiniLLM, reverse KLD, teacher-student training
  • โœ… Model Pruning - Wanda, SparseGPT, and structured pruning

Contributing to the Roadmap

Want to help us achieve these goals?

  1. Pick a skill from the roadmap - Comment on GitHub Discussions to claim it
  2. Follow the contribution guide - Use our template and quality standards
  3. Submit your PR - We review within 48 hours

๐ŸŽ‰ Roadmap Complete!

All 70 skills have been completed! The library now covers the full AI research lifecycle:

  1. โœ… Phase 1-10: Core ML infrastructure (Architecture, Tokenization, Fine-Tuning, Data Processing, Post-Training, Safety, Distributed Training, Optimization, Evaluation, Inference)
  2. โœ… Phase 10.5: Infrastructure (Modal, SkyPilot, Lambda Labs)
  3. โœ… Phase 11-12: Applications (Agents, RAG)
  4. โœ… Phase 13: Multimodal (CLIP, Whisper, LLaVA, Stable Diffusion, SAM, BLIP-2, AudioCraft)
  5. โœ… Phase 14-16: Advanced (Optimization, MLOps & Observability, Prompt Engineering)
  6. โœ… Phase 17-18: Extended (Extended Multimodal, Emerging Techniques)

Future Directions

While the 70-skill roadmap is complete, the library will continue to evolve with:

  • Updates: Keeping existing skills current with latest versions
  • Community contributions: Additional skills from contributors
  • Emerging tools: New frameworks and techniques as they mature

Philosophy

Quality over Quantity: Each skill must provide real value with comprehensive guidance, not just links to docs. We aim for 300+ lines of expert-level content per skill, with real code examples, troubleshooting guides, and production-ready workflows.

SKILL CREATION GUIDE (`docs/SKILL_CREATION_GUIDE.md`)

Skill Creation Guide

Based on: Anthropic Official Best Practices
Last Updated: November 6, 2025

Core Principles (from Anthropic)

1. Concise is Key

The context window is a public good. Your skill shares it with system prompts, conversation history, and other skills.

Default assumption: Claude is already smart

Only add context Claude doesn't already have. Challenge each piece of information:

  • "Does Claude really need this explanation?"
  • "Can I assume Claude knows this?"
  • "Does this paragraph justify its token cost?"

Good (50 tokens):

markdown
## Extract PDF text

Use pdfplumber for text extraction:

```python
import pdfplumber
with pdfplumber.open("file.pdf") as pdf:
    text = pdf.pages[0].extract_text()
text
**Bad** (150 tokens):
```markdown
## Extract PDF text

PDF (Portable Document Format) files are a common file format that contains
text, images, and other content. To extract text from a PDF, you'll need to
use a library. There are many libraries available for PDF processing, but we
recommend pdfplumber because it's easy to use and handles most cases well.
First, you'll need to install it using pip. Then you can use the code below...

2. Progressive Disclosure

SKILL.md serves as an overview that points Claude to detailed materials as needed.

  • Keep SKILL.md body under 500 lines for optimal performance
  • Aim for 200-300 lines in practice
  • Split content into separate reference files
  • Keep references ONE LEVEL DEEP from SKILL.md (no nested references)

Structure:

text
skill-name/
โ”œโ”€โ”€ SKILL.md              # Main overview (200-300 lines)
โ”œโ”€โ”€ server-deployment.md  # Specific topic (loaded as needed)
โ”œโ”€โ”€ offline-inference.md  # Another topic (loaded as needed)
โ”œโ”€โ”€ optimization.md       # Advanced topic (loaded as needed)
โ””โ”€โ”€ scripts/
    โ”œโ”€โ”€ validate.py       # Utility script (executed, not loaded)
    โ””โ”€โ”€ helper.py         # Another script

3. Use Workflows with Checklists

For multi-step tasks, provide copy-paste checklists:

markdown
## Deployment workflow

Copy this checklist and track progress:

Task Progress:

  • Step 1: Configure server settings
  • Step 2: Validate configuration
  • Step 3: Deploy to production
  • Step 4: Verify deployment
text
**Step 1: Configure server settings**

Edit `config.yaml` with production values.

**Step 2: Validate configuration**

Run validator and fix errors:
```bash
python validate.py config.yaml
# If errors: fix โ†’ validate again โ†’ continue

Step 3: Deploy to production

[Specific deployment command]

Step 4: Verify deployment

[Verification steps]

text
### 4. Feedback Loops for Quality

**Common pattern**: Run validator โ†’ fix errors โ†’ repeat

```markdown
## Document editing process

1. Make your edits to `document.xml`
2. **Validate immediately**: `python validate.py document.xml`
3. If validation fails:
   - Review the error message carefully
   - Fix the issues
   - Run validation again
4. **Only proceed when validation passes**
5. Export final document

YAML Frontmatter Requirements

All SKILL.md files must include properly formatted YAML frontmatter with the following fields:

yaml
---
name: skill-name-here
description: Third-person description of what this does and when to use it. Include key terms and triggers. Maximum 1024 characters.
version: 1.0.0
author: Orchestra Research
license: MIT
tags: [Tag One, Tag Two, Tag Three]
dependencies: [package1>=1.0.0, package2>=2.0.0]
---

Field Requirements

Field Required Format Notes
name โœ… Yes kebab-case No quotes, lowercase with hyphens
description โœ… Yes Plain text No quotes, concise explanation
version โœ… Yes Semantic version Format: MAJOR.MINOR.PATCH
author โœ… Yes Plain text Use "Orchestra Research"
license โœ… Yes License identifier Typically MIT
tags โœ… Yes Array Capitalized words, no quotes
dependencies โš ๏ธ Optional Array Include version constraints

name field:

  • Maximum 64 characters
  • Lowercase letters, numbers, hyphens only
  • No XML tags
  • No reserved words: "anthropic", "claude"
  • Recommended: Use gerund form (e.g., serving-llms, processing-pdfs, analyzing-data)

description field:

  • Maximum 1024 characters
  • Non-empty
  • No XML tags
  • No quotes around the text
  • MUST be third person: "Processes files..." not "I can help you..."
  • Include what it does AND when to use it
  • Include key terms for discovery

tags field:

  • Use Title Case for all tags (capitalize first letter of each word)
  • Keep acronyms UPPERCASE (e.g., GRPO, TRL, RLHF, DPO, MLOps, RAG)
  • Use descriptive, searchable terms
  • Include 5-10 relevant tags
  • No quotes around tags

dependencies field:

  • Only include direct dependencies needed to use the skill
  • Include minimum version constraints using >=
  • No quotes around package names
  • List core packages first, optional packages last

Examples:

โœ… Good:

yaml
---
name: serving-llms
description: Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency, or serving models with limited GPU memory.
version: 1.0.0
author: Orchestra Research
license: MIT
tags: [Inference, Serving, vLLM, PagedAttention, Production Deployment, High Throughput]
dependencies: [vllm>=0.2.0, torch>=2.0.0, transformers>=4.35.0]
---

โœ… Good:

yaml
---
name: processing-pdfs
description: Extracts text and tables from PDF files, fills forms, merges documents. Use when working with PDF files or when the user mentions PDFs, forms, or document extraction.
version: 1.0.0
author: Orchestra Research
license: MIT
tags: [PDF Processing, Text Extraction, Document Processing, Forms]
dependencies: [pdfplumber>=0.9.0, PyPDF2>=3.0.0]
---

โŒ Bad (quotes and missing fields):

yaml
---
name: "skill-name"
description: "I can help you process PDF files"
---

โŒ Bad (first person, vague):

yaml
---
name: docs-helper
description: Helps with documents
version: 1.0.0
author: Orchestra Research
license: MIT
tags: [documents]
---

Skill Structure Best Practices

File Organization

Simple skill (just SKILL.md):

text
skill-name/
โ””โ”€โ”€ SKILL.md

Complex skill (with references):

text
skill-name/
โ”œโ”€โ”€ SKILL.md                  # Overview, points to references
โ”œโ”€โ”€ server-deployment.md      # Topic-specific guide
โ”œโ”€โ”€ offline-inference.md      # Another topic
โ”œโ”€โ”€ optimization.md           # Advanced features
โ”œโ”€โ”€ troubleshooting.md        # Common issues
โ””โ”€โ”€ scripts/
    โ”œโ”€โ”€ validate.py           # Utility script
    โ””โ”€โ”€ setup.sh              # Setup script

Domain-specific organization (for Skills with multiple domains):

text
bigquery-skill/
โ”œโ”€โ”€ SKILL.md                  # Overview and navigation
โ””โ”€โ”€ reference/
    โ”œโ”€โ”€ finance.md            # Revenue, billing metrics
    โ”œโ”€โ”€ sales.md              # Opportunities, pipeline
    โ”œโ”€โ”€ product.md            # API usage, features
    โ””โ”€โ”€ marketing.md          # Campaigns, attribution

Reference Files

One level deep: All reference files should link directly from SKILL.md

โœ… Good:

markdown
# SKILL.md

**Server deployment**: See [server-deployment.md](server-deployment.md)
**Offline inference**: See [offline-inference.md](offline-inference.md)
**API reference**: See [api-reference.md](api-reference.md)

โŒ Bad (nested references):

markdown
# SKILL.md
See [advanced.md](advanced.md)...

# advanced.md
See [details.md](details.md)...

# details.md
Here's the actual information...

Table of contents: For reference files >100 lines, include table of contents at top

markdown
# API Reference

## Contents
- Authentication and setup
- Core methods (create, read, update, delete)
- Advanced features (batch operations, webhooks)
- Error handling patterns
- Code examples

## Authentication and setup
...

Content Guidelines

Assume Claude is Smart

Don't explain basics. Assume Claude knows:

  • What PDFs are
  • How libraries work
  • What APIs are
  • Common programming concepts
  • Standard ML/AI terminology

Only explain:

  • Domain-specific concepts unique to this tool
  • Non-obvious gotchas
  • Best practices from community experience

Consistent Terminology

Choose one term and use it throughout:

โœ… Good:

  • Always "API endpoint"
  • Always "field"
  • Always "extract"

โŒ Bad:

  • Mix "API endpoint", "URL", "API route", "path"
  • Mix "field", "box", "element", "control"
  • Mix "extract", "pull", "get", "retrieve"

Avoid Time-Sensitive Information

โŒ Bad:

markdown
If you're doing this before August 2025, use the old API.
After August 2025, use the new API.

โœ… Good:

markdown
## Current method

Use the v2 API endpoint: `api.example.com/v2/messages`

## Old patterns

<details>
<summary>Legacy v1 API (deprecated 2025-08)</summary>

The v1 API used: `api.example.com/v1/messages`

This endpoint is no longer supported.
</details>

Provide Examples (Input/Output Pairs)

For skills where output quality depends on seeing examples:

markdown
## Commit message format

Generate commit messages following these examples:

**Example 1:**
Input: Added user authentication with JWT tokens
Output:

feat(auth): implement JWT-based authentication

Add login endpoint and token validation middleware

text
**Example 2:**
Input: Fixed bug where dates displayed incorrectly in reports
Output:

fix(reports): correct date formatting in timezone conversion

Use UTC timestamps consistently across report generation

text
Follow this style: type(scope): brief description, then detailed explanation.

Common Patterns

Template Pattern

Provide templates for output format. Match strictness to needs.

For strict requirements:

markdown
## Report structure

ALWAYS use this exact template structure:

```markdown
# [Analysis Title]

## Executive summary
[One-paragraph overview of key findings]

## Key findings
- Finding 1 with supporting data
- Finding 2 with supporting data
- Finding 3 with supporting data

## Recommendations
1. Specific actionable recommendation
2. Specific actionable recommendation
```

For flexible guidance:

markdown
## Report structure

Here is a sensible default format, but use your best judgment:

```markdown
# [Analysis Title]

## Executive summary
[Overview]

## Key findings
[Adapt sections based on what you discover]

## Recommendations
[Tailor to the specific context]
```

Adjust sections as needed for the specific analysis type.

Conditional Workflow Pattern

Guide Claude through decision points:

markdown
## Document modification workflow

1. Determine the modification type:

   **Creating new content?** โ†’ Follow "Creation workflow" below
   **Editing existing content?** โ†’ Follow "Editing workflow" below

2. Creation workflow:
   - Use docx-js library
   - Build document from scratch
   - Export to .docx format

3. Editing workflow:
   - Unpack existing document
   - Modify XML directly
   - Validate after each change
   - Repack when complete

Anti-Patterns to Avoid

โŒ Windows-Style Paths

Always use forward slashes:

โœ… Good: scripts/helper.py, reference/guide.md
โŒ Bad: scripts\helper.py, reference\guide.md

โŒ Too Many Options

Don't present multiple approaches unless necessary:

โŒ Bad:
"You can use pypdf, or pdfplumber, or PyMuPDF, or pdf2image, or..."

โœ… Good:
"Use pdfplumber for text extraction:

python
import pdfplumber

For scanned PDFs requiring OCR, use pdf2image with pytesseract instead."

โŒ Nested References

โŒ Bad: SKILL.md โ†’ advanced.md โ†’ details.md โ†’ actual info
โœ… Good: SKILL.md โ†’ [topic].md (all references one level deep)

โŒ Over-Explaining Basics

โŒ Bad (150 tokens):
"PDF files are a common format. They contain text and images. To process them, you need a library. Python has many PDF libraries. We recommend pdfplumber because..."

โœ… Good (30 tokens):
"Use pdfplumber for PDF text extraction:

python
import pdfplumber
with pdfplumber.open("file.pdf") as pdf:
    text = pdf.pages[0].extract_text()
```"

---

## Quality Checklist

Before submitting a skill:

### Core Quality
- [ ] Description is specific and includes key terms
- [ ] Description includes both what it does and when to use it
- [ ] SKILL.md body is under 500 lines (aim for 200-300)
- [ ] Additional details in separate files (if needed)
- [ ] No time-sensitive information (or in "old patterns" section)
- [ ] Consistent terminology throughout
- [ ] Examples are concrete, not abstract
- [ ] File references are one level deep
- [ ] Progressive disclosure used appropriately
- [ ] Workflows have clear steps with checklists

### Code and Scripts
- [ ] Scripts solve problems rather than punt to Claude
- [ ] Error handling is explicit and helpful
- [ ] No "magic numbers" (all values justified)
- [ ] Required packages listed in instructions
- [ ] No Windows-style paths (all forward slashes)
- [ ] Validation/verification steps for critical operations
- [ ] Feedback loops included for quality-critical tasks

### Content Quality
- [ ] Assumes Claude is smart (no over-explaining basics)
- [ ] Third person description
- [ ] Gerund naming (e.g., "serving-llms" not "llm-server")
- [ ] Clear when to use vs alternatives
- [ ] Concrete examples with input/output pairs
- [ ] Troubleshooting section with common issues

---

## Recommended Process

### 1. Research Phase

- Read official documentation thoroughly
- Analyze real-world usage (blog posts, Stack Overflow, GitHub issues)
- Identify key concepts and common gotchas
- Find production code examples

### 2. Outline Phase

Create structure outline:
1. Quick start (20-30 lines)
2. Common workflows with checklists (80-120 lines)
3. When to use vs alternatives (20-30 lines)
4. Common issues (30-50 lines)
5. Advanced topics with links to reference files (10-20 lines)

**Target**: 200-300 lines for SKILL.md

### 3. Writing Phase

Use SKILL_TEMPLATE.md as starting point:
- Fill in YAML frontmatter (name, description)
- Write concise quick start
- Create 2-3 workflows with copy-paste checklists
- Add common issues section
- Link to reference files for advanced topics

### 4. Reference Files Phase

Create separate markdown files for:
- Detailed API documentation
- Advanced features
- Troubleshooting guides
- Configuration references
- Domain-specific content

Each file:
- Has clear purpose
- Links directly from SKILL.md
- Includes table of contents if >100 lines
- Focuses on one topic

### 5. Testing Phase

Test with Claude:
- Activate the skill
- Try common workflows
- Verify checklist format works
- Test progressive disclosure (does Claude load right files?)
- Check cross-references work

### 6. Iteration Phase

Based on testing:
- Simplify over-explained sections
- Add missing common issues
- Improve workflow clarity
- Reorganize reference files if needed

---

## Examples of Good Skills

**For structure reference**, see official Anthropic examples in `anthropic_official_docs/best_practices.md`:
- PDF Processing skill (lines 286-307)
- BigQuery skill (lines 316-344)
- Git Commit Helper (lines 229-233)

**From this project**:
- Reference GRPO-RL-Training skill for comprehensive workflows
- But make it MORE CONCISE following Anthropic guidelines

---

## Common Mistakes to Avoid

1. **Making SKILL.md too long** (>500 lines is RED FLAG)
2. **Over-explaining basics** (assume Claude knows ML/programming)
3. **No workflows with checklists** (makes complex tasks hard)
4. **Nested references** (keep one level deep)
5. **First-person descriptions** (use third person!)
6. **Vague skill names** (use gerund form with specific terms)
7. **No "when to use vs alternatives"** (critical for skill selection)
8. **Missing validation steps** (add feedback loops)
9. **Too many options** (provide default with escape hatch)
10. **Time-sensitive info** (use "old patterns" section instead)

---

## Resources

- **Anthropic Official Best Practices**: [anthropic_official_docs/best_practices.md](anthropic_official_docs/best_practices.md)
- **Skill Template**: [SKILL_TEMPLATE.md](SKILL_TEMPLATE.md)
- **Contributing Guide**: [CONTRIBUTING.md](CONTRIBUTING.md)

---

### SKILL TEMPLATE (`docs/SKILL_TEMPLATE.md`)

---
name: example-skill-name
version: 1.0.0
author: Orchestra Research
license: MIT
tags: [Tag One, Tag Two, Tag Three, Key Concept, Use Case]
dependencies: [package1>=1.0.0, package2>=2.0.0]
---

# [Skill Title]

## Quick start

[One paragraph overview of what this skill provides]

**Basic usage**:
```[language]
# Minimal working example (5-10 lines)
import library
result = library.function(input)
print(result)

Common workflows

Workflow 1: [Primary Use Case]

Copy this checklist and track progress:

text
Task Progress:
- [ ] Step 1: [First action]
- [ ] Step 2: [Second action]
- [ ] Step 3: [Validation step]
- [ ] Step 4: [Completion step]

Step 1: [First action]

[Brief instruction - assume Claude knows basics]

[language]
# Code example
[concise code]

Step 2: [Second action]

[Brief instruction]

[language]
# Code example
[concise code]

Step 3: [Validation step]

Run validator and fix errors if found:

bash
validate_script.py input.json
# If errors: fix โ†’ validate again โ†’ continue

Step 4: [Completion step]

[Final action]

Workflow 2: [Secondary Use Case]

[Similar structure with checklist]

When to use vs alternatives

Use this when:

  • [Specific scenario 1]
  • [Specific scenario 2]

Use [Alternative] instead when:

  • [Different scenario]

Common issues

Issue: [Error message or problem]

Fix by adjusting [parameter]:

[language]
# Solution code
[concise fix]

Issue: [Another common problem]

Check [specific requirement], then [action].

Advanced topics

[Advanced feature 1]: See references/advanced-features.md
[Advanced feature 2]: See references/optimization.md
[API reference]: See references/api-reference.md

Resources

  • Official docs: [URL]
  • GitHub: [URL]

Npm Package Plan (`docs/npm-package-plan.md`)

NPM Package Plan: @orchestra-research/skills

Overview

Create an npm/npx package that allows users to easily install AI research skills to their preferred coding agents (Claude Code, Cursor, Codex, Windsurf, etc.).

Package Name Options

  • @orchestra-research/skills (recommended - scoped, professional)
  • ai-research-skills (simple, may conflict)
  • orchestra-skills (short, brandable)

Architecture

Inspired By

Based on research of existing solutions:

  • vercel-labs/skills: Canonical storage + symlinks, 27 agent support, lock file
  • openskills: Universal loader, AGENTS.md generation
  • add-skill: Zero-dependency, auto-detection

Core Components

text
@orchestra-research/skills/
โ”œโ”€โ”€ bin/
โ”‚   โ””โ”€โ”€ cli.js                 # CLI entry point
โ”œโ”€โ”€ src/
โ”‚   โ”œโ”€โ”€ agents/                # Agent-specific handlers
โ”‚   โ”‚   โ”œโ”€โ”€ claude.js          # .claude/skills/
โ”‚   โ”‚   โ”œโ”€โ”€ cursor.js          # .cursor/skills/
โ”‚   โ”‚   โ”œโ”€โ”€ codex.js           # .codex/skills/
โ”‚   โ”‚   โ”œโ”€โ”€ windsurf.js        # .windsurf/skills/
โ”‚   โ”‚   โ”œโ”€โ”€ copilot.js         # .github/copilot-instructions.md
โ”‚   โ”‚   โ””โ”€โ”€ index.js           # Agent registry
โ”‚   โ”œโ”€โ”€ commands/
โ”‚   โ”‚   โ”œโ”€โ”€ install.js         # Install skills
โ”‚   โ”‚   โ”œโ”€โ”€ list.js            # List available/installed skills
โ”‚   โ”‚   โ”œโ”€โ”€ update.js          # Update skills
โ”‚   โ”‚   โ”œโ”€โ”€ remove.js          # Remove skills
โ”‚   โ”‚   โ””โ”€โ”€ detect.js          # Detect installed agents
โ”‚   โ”œโ”€โ”€ storage/
โ”‚   โ”‚   โ”œโ”€โ”€ canonical.js       # ~/.orchestra-skills/ management
โ”‚   โ”‚   โ”œโ”€โ”€ lock.js            # Lock file management
โ”‚   โ”‚   โ””โ”€โ”€ symlink.js         # Symlink utilities
โ”‚   โ”œโ”€โ”€ registry/
โ”‚   โ”‚   โ””โ”€โ”€ skills.json        # Skill manifest (or fetch from GitHub)
โ”‚   โ””โ”€โ”€ utils/
โ”‚       โ”œโ”€โ”€ fetch.js           # Download skills from GitHub
โ”‚       โ””โ”€โ”€ format.js          # Format for different agents
โ”œโ”€โ”€ package.json
โ””โ”€โ”€ README.md

Supported Agents (7 Verified)

All agents below have been verified to support SKILL.md files with the same format.

Agent Config Location Skills Location Source
Claude Code ~/.claude/ .claude/skills/ Verified locally
Cursor ~/.cursor/ .cursor/skills/ DeepWiki
Codex (OpenAI) ~/.codex/ .codex/skills/ DeepWiki
Windsurf ~/.windsurf/ .windsurf/skills/ Windsurf Docs
Gemini CLI ~/.gemini/ .gemini/skills/ DeepWiki
Kilo Code ~/.kilocode/ .kilocode/skills/ Kilo Docs
Qwen Code ~/.qwen/ .qwen/skills/ Qwen Docs

CLI Commands

Installation

bash
# Install globally
npm install @orchestra-research/skills

# Or use npx (recommended)
npx @orchestra-research/skills <command>

Commands

bash
# Detect installed coding agents
npx @orchestra-research/skills detect

# List all available skills
npx @orchestra-research/skills list

# List skills by category
npx @orchestra-research/skills list --category post-training

# List available categories
npx @orchestra-research/skills categories

# Install all skills for detected agents
npx @orchestra-research/skills install --all

# Install specific category (user selects from list)
npx @orchestra-research/skills install --category post-training

# Install multiple categories
npx @orchestra-research/skills install --category post-training,fine-tuning,inference

# Install specific skill
npx @orchestra-research/skills install verl

# Install for specific agent only
npx @orchestra-research/skills install verl --agent claude

# Install to project scope (current directory)
npx @orchestra-research/skills install verl --scope project

# Install to global scope (home directory)
npx @orchestra-research/skills install verl --scope global

# Interactive mode - prompts user to select categories/skills
npx @orchestra-research/skills install --interactive

# Update all skills
npx @orchestra-research/skills update

# Remove a skill
npx @orchestra-research/skills remove verl

# Show skill info
npx @orchestra-research/skills info verl

Interactive Installation Flow

When running npx @orchestra-research/skills install --interactive:

text
? What would you like to install?
  โ—‹ All skills (86 skills)
  โ—‹ Select by category
  โ—‹ Select individual skills

? Select categories to install: (Space to select, Enter to confirm)
  โ—‰ 01-model-architecture (6 skills)
  โ—ฏ 02-tokenization (2 skills)
  โ—ฏ 03-fine-tuning (5 skills)
  โ—‰ 06-post-training (8 skills)
  โ—ฏ 20-ml-paper-writing (1 skill)
  ...

? Confirm installation of 14 skills to Claude Code, Cursor, Gemini CLI? (Y/n)

Storage Strategy

Single source of truth with symlinks:

text
~/.orchestra-skills/                    # Canonical storage
โ”œโ”€โ”€ .lock.json                          # Lock file for versioning
โ”œโ”€โ”€ 01-model-architecture/
โ”‚   โ”œโ”€โ”€ megatron-core/
โ”‚   โ”‚   โ””โ”€โ”€ SKILL.md
โ”‚   โ””โ”€โ”€ litgpt/
โ”‚       โ””โ”€โ”€ SKILL.md
โ”œโ”€โ”€ 06-post-training/
โ”‚   โ”œโ”€โ”€ verl/
โ”‚   โ”‚   โ”œโ”€โ”€ SKILL.md
โ”‚   โ”‚   โ””โ”€โ”€ references/
โ”‚   โ”œโ”€โ”€ slime/
โ”‚   โ””โ”€โ”€ ...
โ””โ”€โ”€ ...

~/.claude/skills/                       # Symlinks to canonical
โ”œโ”€โ”€ verl -> ~/.orchestra-skills/06-post-training/verl
โ”œโ”€โ”€ slime -> ~/.orchestra-skills/06-post-training/slime
โ””โ”€โ”€ ...

~/.cursor/skills/                       # Same symlinks
โ”œโ”€โ”€ verl -> ~/.orchestra-skills/06-post-training/verl
โ””โ”€โ”€ ...

Lock File Format

json
{
  "version": "1.0.0",
  "lastUpdated": "2025-01-28T00:00:00Z",
  "skills": {
    "verl": {
      "version": "1.0.0",
      "category": "06-post-training",
      "installedAt": "2025-01-28T00:00:00Z",
      "agents": ["claude", "cursor"]
    }
  },
  "agents": {
    "claude": {
      "detected": true,
      "scope": "global",
      "path": "~/.claude/skills"
    }
  }
}

Skill Structure Patterns

The repository has two skill organization patterns:

Pattern 1: Nested Skills (Most Categories)

text
XX-category/
โ”œโ”€โ”€ skill-name-1/
โ”‚   โ”œโ”€โ”€ SKILL.md
โ”‚   โ””โ”€โ”€ references/
โ”œโ”€โ”€ skill-name-2/
โ”‚   โ””โ”€โ”€ SKILL.md
โ””โ”€โ”€ ...

Example: 06-post-training/verl/SKILL.md

Pattern 2: Standalone Skills (Single Skill = Category)

text
XX-category-name/
โ”œโ”€โ”€ SKILL.md
โ”œโ”€โ”€ references/
โ””โ”€โ”€ templates/

Example: 20-ml-paper-writing/SKILL.md (the category IS the skill)

The npm package must handle both patterns when fetching skills.

Skill Registry

Option A: Embedded (Simpler)

Include skill manifest in npm package, update with releases:

json
// src/registry/skills.json
{
  "version": "1.0.0",
  "categories": {
    "01-model-architecture": {
      "name": "Model Architecture",
      "skills": ["megatron-core", "litgpt", "mamba", "rwkv", "nanogpt"]
    },
    "06-post-training": {
      "name": "Post-Training (RLHF/DPO/GRPO)",
      "skills": ["trl", "grpo", "openrlhf", "simpo", "verl", "slime", "miles", "torchforge"]
    }
  },
  "skills": {
    "verl": {
      "name": "verl",
      "category": "06-post-training",
      "description": "Volcano Engine RL for LLM post-training",
      "tags": ["Reinforcement Learning", "RLHF", "GRPO", "PPO"]
    }
  }
}

Option B: Remote Fetch (More Flexible)

Fetch skill manifest from GitHub API on each run:

javascript
const REPO = 'orchestra-research/AI-research-SKILLs';
const MANIFEST_URL = `https://api.github.com/repos/${REPO}/contents/skill-manifest.json`;

async function fetchSkillManifest() {
  const response = await fetch(MANIFEST_URL);
  return JSON.parse(atob(response.content));
}

Recommendation: Start with embedded, add remote fetch as update mechanism.

Installation Flow

text
/* Detailed source-code truncated for AI context efficiency. */

Agent-Specific Handling

All 7 Verified Agents

All agents use the same SKILL.md format and symlink pattern:

javascript
// src/agents/index.js
export const agents = {
  claude: {
    name: 'Claude Code',
    configDir: '~/.claude',
    skillsDir: '~/.claude/skills',
    projectSkillsDir: '.claude/skills',
  },
  cursor: {
    name: 'Cursor',
    configDir: '~/.cursor',
    skillsDir: '~/.cursor/skills',
    projectSkillsDir: '.cursor/skills',
  },
  codex: {
    name: 'Codex (OpenAI)',
    configDir: '~/.codex',
    skillsDir: '~/.codex/skills',
    projectSkillsDir: '.codex/skills',
  },
  windsurf: {
    name: 'Windsurf',
    configDir: '~/.windsurf',
    skillsDir: '~/.windsurf/skills',
    projectSkillsDir: '.windsurf/skills',
  },
  gemini: {
    name: 'Gemini CLI',
    configDir: '~/.gemini',
    skillsDir: '~/.gemini/skills',
    projectSkillsDir: '.gemini/skills',
  },
  kilo: {
    name: 'Kilo Code',
    configDir: '~/.kilocode',
    skillsDir: '~/.kilocode/skills',
    projectSkillsDir: '.kilocode/skills',
  },
  qwen: {
    name: 'Qwen Code',
    configDir: '~/.qwen',
    skillsDir: '~/.qwen/skills',
    projectSkillsDir: '.qwen/skills',
  },
};

// Common install function for all agents
function installSkill(agent, skillName, canonicalPath, scope) {
  const targetDir = scope === 'project'
    ? agent.projectSkillsDir
    : expandHome(agent.skillsDir);

  fs.ensureDirSync(targetDir);
  fs.symlinkSync(canonicalPath, path.join(targetDir, skillName));
}

User Experience

First Run

bash
$ npx @orchestra-research/skills detect

๐Ÿ” Detecting installed coding agents...

โœ“ Claude Code     ~/.claude
โœ“ Cursor          ~/.cursor
โœ— Codex           not found
โœ— Windsurf        not found
โœ“ GitHub Copilot  available for projects

Found 2 global agents, 1 project-only agent.
Run 'npx @orchestra-research/skills install --all' to install all skills.

Installing Skills

bash
$ npx @orchestra-research/skills install post-training

๐Ÿ“ฆ Installing post-training skills...

Downloading skills from GitHub...
  โœ“ trl (1.2 KB)
  โœ“ grpo (15.3 KB)
  โœ“ openrlhf (8.7 KB)
  โœ“ simpo (4.2 KB)
  โœ“ verl (12.1 KB)
  โœ“ slime (18.4 KB)
  โœ“ miles (9.8 KB)
  โœ“ torchforge (11.2 KB)

Creating symlinks...
  โœ“ Claude Code: 8 skills installed
  โœ“ Cursor: 8 skills installed

โœจ Done! Installed 8 skills for 2 agents.

Skills are stored in: ~/.orchestra-skills/06-post-training/
Symlinks created in: ~/.claude/skills/, ~/.cursor/skills/

Listing Skills

bash
$ npx @orchestra-research/skills list

๐Ÿ“š AI Research Skills (81 total)

Model Architecture (5)
  โ—‹ megatron-core    Megatron-Core for large-scale model training
  โ—‹ litgpt           LitGPT for efficient LLM development
  โ—‹ mamba            Mamba state space models
  โ—‹ rwkv             RWKV linear attention models
  โ—‹ nanogpt          NanoGPT for learning/prototyping

Post-Training (8)
  โ— verl             Volcano Engine RL for LLM post-training
  โ— slime            Megatron-SGLang RL training framework
  โ— miles            Enterprise-grade RL for large MoE models
  โ— torchforge       PyTorch-native agentic RL library
  โ—‹ trl              Transformer Reinforcement Learning
  โ—‹ grpo             Group Relative Policy Optimization
  โ—‹ openrlhf         OpenRLHF training framework
  โ—‹ simpo            Simple Preference Optimization

โ— = installed, โ—‹ = available

Implementation Phases

Phase 1: MVP (Week 1)

  • Basic CLI structure with commander.js
  • Agent detection (Claude, Cursor, Codex)
  • Download skills from GitHub
  • Symlink installation to detected agents
  • Basic list and install commands

Phase 2: Full Features (Week 2)

  • Canonical storage with lock file
  • Update and remove commands
  • Category filtering
  • Project vs global scope
  • Copilot special handling

Phase 3: Polish (Week 3)

  • Interactive mode (inquirer.js prompts)
  • Progress bars and better UX
  • Error handling and recovery
  • Documentation and README
  • npm publish and GitHub Actions for releases

Dependencies

json
{
  "dependencies": {
    "commander": "^12.0.0",      // CLI framework
    "chalk": "^5.3.0",           // Colored output
    "ora": "^8.0.0",             // Spinners
    "fs-extra": "^11.2.0",       // File utilities
    "node-fetch": "^3.3.0",      // HTTP requests
    "inquirer": "^9.2.0"         // Interactive prompts (optional)
  }
}

Publishing

bash
# Login to npm
npm login

# Publish scoped package (public)
npm publish --access public

Alternatives Considered

1. Shell Script (Rejected)

PR #6 approach - too limited, no cross-platform support, poor UX.

2. Python Package (Possible Alternative)

Could work with pipx install orchestra-skills, but npm/npx is more common for dev tools.

3. Homebrew Formula (Future)

Could add brew install orchestra-skills later for Mac users.

Next Steps

  1. Create new repository or directory for npm package
  2. Implement Phase 1 MVP
  3. Test with Claude Code and Cursor
  4. Publish to npm
  5. Update main README with installation instructions
  6. Close PR #6 with reference to new approach

References

Writing Assets ML Paper Guide (`docs/writing-assets/ML_paper_guide.md`)

The Complete Guide to Writing Top-Quality ML Academic Papers

Writing successful ML papers for venues like NeurIPS, ICML, and ICLR demands mastery of a specific craft: translating rigorous technical work into a compelling narrative that busy reviewers can quickly evaluate. The single most critical insight across all expert sources: your paper is not a collection of experimentsโ€”it's a story with one clear contribution supported by evidence. This guide synthesizes advice from prominent researchers including Neel Nanda, Andrej Karpathy, Jacob Steinhardt, and Sebastian Farquhar, alongside official conference guidelines and practical tools for citation management.

The stakes are high: top ML conferences maintain ~25% acceptance rates, and reviewers spend limited time per paper. Seminal work like Adam and Knowledge Distillation faced initial rejections. Success requires not just strong research but strategic communicationโ€”front-loading value, maintaining precision, and providing reproducibility details that build reviewer confidence.

The narrative principle that separates accepted papers

Every successful ML paper centers on what Neel Nanda calls "the narrative": a short, rigorous, evidence-based technical story with a takeaway readers care about. This narrative rests on three pillars that must be crystal clear by the end of your introduction.

The "What" consists of one to three specific novel claims fitting within a cohesive theme. Vague contributions like "we study X" fail immediatelyโ€”reviewers need precise, falsifiable claims. The "Why" provides rigorous empirical evidence that convincingly supports those claims, including strong baselines honestly tuned and experiments that distinguish between competing hypotheses rather than merely showing "decent results." The "So What" answers why readers should care, connecting your contribution to problems the community recognizes as important.

Andrej Karpathy reinforces this: "A paper is not a random collection of experiments you report on. The paper sells a single thing that was not obvious or present before. The entire paper is organized around this core contribution with surgical precision." This applies whether you're presenting a new architecture, a theoretical result, or improved understanding of existing methodsโ€”NeurIPS explicitly notes that "originality does not necessarily require an entirely new method."

The practical implication is severe: if you cannot state your contribution in one sentence, you don't yet have a paper. Everything elseโ€”experiments, related work, discussionโ€”exists only to support that core claim.

Front-load value: the title-to-methods pipeline

Readers encounter your paper in a predictable pattern: title โ†’ abstract โ†’ introduction โ†’ figures โ†’ maybe the rest. Nanda advises spending "about the same amount of time on each of: the abstract, the intro, the figures, and everything else." This isn't hyperboleโ€”most reviewers form preliminary judgments before reaching your methods section.

The abstract follows a tight five-sentence structure perfected by Sebastian Farquhar: (1) What you achieved ("We introduce...", "We prove...", "We demonstrate..."), (2) Why this is hard and important, (3) How you do it with specialist keywords, (4-5) What evidence you have, including your most remarkable number. Generic openings like "Large language models have achieved remarkable success" waste precious spaceโ€”Zachary Lipton's rule: "If the first sentence can be pre-pended to any ML paper, delete it."

The introduction should not exceed 1-1.5 pages and must include a bullet-point contribution list of 2-4 items (max 1-2 lines each in two-column format). Farquhar emphasizes: "Methods should start by page 2-3 maximum"โ€”if your introduction runs longer, you're burying the actual contribution.

Figure 1 deserves special attention because many readers skip directly to it. It should convey your core idea, approach, or most compelling result. Use vector graphics, ensure readability in black-and-white, and write captions that stand alone without requiring the main text.

Section-by-section execution matters

Related Work should be organized methodologically, not paper-by-paper. Good: "One line of work uses Floogledoodle's assumption [refs] whereas we use Doobersnoddle's assumption because..." Bad: "Snap et al. introduced X while Crackle et al. introduced Y." Cite generouslyโ€”reviewers likely authored relevant papersโ€”and distribute citations throughout your paper rather than confining them to one section.

Methods must enable reimplementation. ICML's checklist requires: conceptual outline or pseudocode, clearly stated algorithms, all hyperparameters listed, and architectural details sufficient for reproduction. Present your final design decisions here; comparative ablations belong in experiments or appendix.

Experiments require explicit structure. For each experiment, state: what claim it supports, how it connects to your main contribution, the experimental setting (with details in appendix), and explicit guidance on what to observe in figures ("the blue line shows X, which demonstrates Y"). The ICML checklist mandates: error bars with methodology specified (standard deviation vs. standard error), hyperparameter search ranges, compute infrastructure (GPU type, total hours), and seed-setting methods.

Limitations deserve their own sectionโ€”NeurIPS and ICML require this. Counter-intuitively, honesty helps: reviewers are explicitly instructed not to penalize papers for acknowledging limitations. Pre-empt criticisms by identifying weaknesses before reviewers do and explaining why they don't undermine your core claims.

Writing style that signals quality

Gopen and Swan's "Science of Scientific Writing" establishes principles that ML papers routinely violate. Place emphasis at sentence ends (the "stress position"): readers naturally weight final words more heavily. Put context first: establish familiar information before introducing new concepts. Keep subject and verb close together: anything intervening reads as interruption. One unit, one function: each paragraph should make exactly one point.

Specific style rules from multiple sources converge on these practices. Minimize pronounsโ€”if you must use "this" or "those," use them as adjectives ("this result") to provide clarity. Place verbs early in sentences for easier parsing. Use minimal-syllable words. Eliminate hedging unless genuine uncertainty existsโ€”"may" and "can" should almost always be dropped. Lipton notes that "provides very tight approximation" drips with insecurity compared to "provides tight approximation."

Jacob Steinhardt emphasizes precision over brevity: replace "performance" with "accuracy" or "speed" depending on meaning. Use consistent phrasingโ€”referring to the same concept with different terms creates confusion. Avoid vocabulary that signals incremental work: never "combine," "modify," or "expand"; instead "develop" or "propose."

For mathematical writing, state all assumptions formally, provide intuitive explanations alongside proofs, and use consistent notation. Ethan Perez's practical tip: unfold apostrophes ("X's Y" โ†’ "The Y of X") for clarity.

Tables, figures, and visual communication

Tables should use the booktabs LaTeX package for professional appearanceโ€”avoid vertical lines, use horizontal rules sparingly. Bold the best value per metric and include symbols indicating direction (โ†‘ higher is better, โ†“ lower is better). Right-align numerical columns and maintain consistent decimal precision across all values.

Figures must be vector graphics (PDF, EPS) for plots and diagrams; raster formats (PNG at 600 DPI) only for photographs or dense visualizations. Critical accessibility requirement: 8% of men have color vision deficiency. Use the Okabe-Ito or Paul Tol palettes, avoid red-green combinations, and verify your figures work in grayscale. The SciencePlots Python package provides publication-ready styles with a single line: plt.style.use(['science', 'ieee']).

Architecture diagrams benefit from TikZ via PlotNeuralNet (GitHub: HarisIqbal88/PlotNeuralNet), which generates LaTeX code from Python. For training visualizations, include shaded regions showing variance across runs and use log scale when values span multiple orders of magnitude.

Captions should be self-containedโ€”readers must understand figures without consulting main text. ICML explicitly states: "Do not include a title inside the figure; the caption should serve this function."

Conference requirements every submission must meet

NeurIPS 2025: 9 content pages plus unlimited references; mandatory paper checklist covering reproducibility, ethics, and societal impact (desk rejection if missing); 6-point scoring system; lay summaries required for accepted papers. Reviews of accepted papers become public.

ICML 2025: 8 content pages plus one additional page allowed for camera-ready; Broader Impact Statement required at end before references (doesn't count toward limit); reciprocal reviewing requirementโ€”all submissions need a designated reviewer from authors.

ICLR 2025-2026: 10 pages plus unlimited appendices; double-blind via OpenReview; new LLM policy requiring disclosure of AI use in writing (violations result in desk rejection).

All three conferences evaluate papers on four core dimensions: quality (technical soundness, well-supported claims), clarity (clear writing, reproducible by experts), significance (community impact, advances understanding), and originality (new insights, clear differentiation from prior work). Reviewers separate concerns into major issues (essential for publication) and minor issues (not essential), and strong reviews follow Daniel Dennett's rules: first re-express the position fairly, then list agreements and what you learned, only then critique.

Citation APIs that prevent hallucination

AI-generated citations have a documented ~40% error rate, including fabricated papers with real author names and fake titles. A reliable workflow requires programmatic verification through multiple APIs.

Semantic Scholar (api.semanticscholar.org) covers 214M papers with 2.49B citations. Rate limit: 1 RPS with free API key. Python library: pip install semanticscholar. Search, retrieve metadata, and access citation graphsโ€”ideal for ML papers specifically.

CrossRef (api.crossref.org) is the primary source for DOI metadata and offers direct BibTeX retrieval via content negotiation:

python
import requests
def doi_to_bibtex(doi):
    return requests.get(f"https://doi.org/{doi}", 
                       headers={"Accept": "application/x-bibtex"}).text

arXiv API (export.arxiv.org/api) provides metadata for preprints. Python library: pip install arxiv. No authentication required, but maintain 3-second delays between requests.

OpenAlex (api.openalex.org) offers 240M+ works under CC0 licenseโ€”the open successor to Microsoft Academic Graph. 100K requests/day, 10 RPS with email in query string.

Google Scholar has no official APIโ€”scraping violates ToS. Use SerpApi ($75-275/month) only if Semantic Scholar coverage is insufficient.

The verified citation workflow for AI assistants

For any AI-assisted paper writing, implement this verification pipeline to eliminate hallucinated citations:

  1. Search using Semantic Scholar or OpenAlex APIs with specific queries
  2. Verify existence by confirming the paper appears in at least two sources (Semantic Scholar + CrossRef, or DOI resolution + arXiv)
  3. Retrieve BibTeX via DOI content negotiation for guaranteed accuracy
  4. Verify claims by accessing the actual paper (via DOI link or Semantic Scholar PDF) and confirming the attributed claim appears in the source
  5. Maintain clean .bib files using Zotero with Better BibTeX plugin for auto-export and consistent citation keys

Tools like Citely (citely.ai) and CiteSure provide batch verification of reference lists. ReciteWorks checks that in-text citations match your reference list.

For LaTeX, prefer BibLaTeX with Biber backend over legacy BibTeXโ€”it provides full Unicode support, extended entry types (@online, @dataset), and flexible customization. Use \citep{} for parenthetical citations and \citet{} for textual citations.

Conclusion

Writing top-quality ML papers is fundamentally about reducing cognitive load for reviewers while maximizing evidence density for your claims. The hierarchy of importance is clear: narrative clarity beats methodological complexity, front-loaded value beats comprehensive coverage, and verified reproducibility beats impressive numbers.

The most actionable insight: treat writing as iterative design. Nanda recommends paper swaps for mutual feedback; Karpathy suggests submitting a 5-page draft with all experiments two weeks before deadline to reveal critical gaps. Sebastian Farquhar captures the modern reality: "If you are a good writer, you are better than LLMs. If you are a bad writer, you need the practice"โ€”but LLMs excel at identifying unclear passages through misinterpretation and simulating harsh reviewer feedback.

For citation workflows specifically, the combination of Semantic Scholar search โ†’ DOI content negotiation โ†’ BibLaTeX management provides a reliable, hallucination-resistant pipeline suitable for integration into AI writing assistants. Every citation must be verified to exist before inclusionโ€”the alternative is contributing to the documented problem of fabricated references that has affected even NeurIPS accepted papers.

Writing Assets Ml Paper Writing Sources (`docs/writing-assets/ml_paper_writing_sources.md`)

Comprehensive Source List: Building an ML Paper Writing Skill

This document compiles authoritative sources for creating a Claude skill that writes high-quality ML/AI papers for venues like ICLR, NeurIPS, and ICML.

Part 1: Writing Philosophy & Guides from ML Researchers

Primary Sources (Must-Read)

Source Author URL Key Value
Highly Opinionated Advice on How to Write ML Papers Neel Nanda https://www.alignmentforum.org/posts/eJGptPbbFPZGLpjsp/highly-opinionated-advice-on-how-to-write-ml-papers Core narrative philosophy, "what/why/so what" framework, figure-first approach
How to Write ML Papers Sebastian Farquhar (DeepMind) https://sebastianfarquhar.com/on-research/2024/11/04/how_to_write_ml_papers/ 5-sentence abstract formula, structure templates, reader expectations
A Survival Guide to a PhD Andrej Karpathy http://karpathy.github.io/2016/09/07/phd/ Paper structure recipe, importance of reviewing bad papers, contribution framing
Heuristics for Scientific Writing (ML Perspective) Zachary Lipton (CMU) https://www.approximatelycorrect.com/2018/01/29/heuristics-technical-scientific-writing-machine-learning-perspective/ Snappy maxims for clear prose, vacuous intensifier warnings, section balance
Advice for Authors Jacob Steinhardt (UC Berkeley) https://jsteinhardt.stat.berkeley.edu/blog/advice-for-authors Precision over brevity, consistent terminology, reader-centric writing
Easy Paper Writing Tips Ethan Perez (Anthropic) https://ethanperez.net/easy-paper-writing-tips/ Practical micro-level tips, apostrophe unfolding, clarity tricks

Foundational Scientific Writing

Source Author URL Key Value
The Science of Scientific Writing Gopen & Swan https://cseweb.ucsd.edu/~swanson/papers/science-of-writing.pdf Topic/stress positions, old-before-new principle, sentence-level clarity
Summary of Science of Scientific Writing Lawrence Crowl https://www.crowl.org/Lawrence/writing/GopenSwan90.html Condensed version of Gopen & Swan principles

Additional Researcher Perspectives

Source URL Key Value
How To Write A Research Paper In Machine Learning https://grigorisg9gr.github.io/machine%20learning/research%20paper/how-to-write-a-research-paper-in-machine-learning/ Practical walkthrough
A Recipe for Training Neural Networks (Karpathy) http://karpathy.github.io/2019/04/25/recipe/ Debugging methodology that translates to paper structure

Part 2: Official Conference Guidelines

NeurIPS

Document URL Purpose
Paper Checklist Guidelines https://neurips.cc/public/guides/PaperChecklist Mandatory checklist items, reproducibility requirements
2025 Reviewer Guidelines https://neurips.cc/Conferences/2025/ReviewerGuidelines What reviewers look for, scoring criteria
Formatting Instructions https://arxiv.org/html/2505.10292v1 LaTeX template, page limits, style requirements

ICML

Document URL Purpose
Paper Guidelines https://icml.cc/Conferences/2024/PaperGuidelines Submission requirements, ethics policy
Style & Author Instructions https://icml.cc/Conferences/2022/StyleAuthorInstructions Formatting specifications
Reviewer Tutorial https://icml.cc/Conferences/2022/ReviewerTutorial Evaluation criteria from reviewer perspective
Reviewer Guidelines (2020) https://icml.cc/Conferences/2020/ReviewerGuidelines Detailed review criteria
ICML 2025 LaTeX Template https://www.overleaf.com/latex/templates/icml2025-template/dhxrkcgkvnkt Official Overleaf template

ICLR

Document URL Purpose
Author Guide 2026 https://iclr.cc/Conferences/2026/AuthorGuide Submission requirements, LLM disclosure policy
LLM Disclosure Policy https://eu.36kr.com/en/p/3443306502428032 NEW: Mandatory AI use disclosure (desk rejection if missing)

Part 3: Citation APIs & Tools (Hallucination Prevention)

Primary APIs for Paper Search & Metadata

API Documentation URL Key Features Rate Limits
Semantic Scholar API https://api.semanticscholar.org/api-docs/ 214M papers, citation graphs, AI-trained search 1 RPS with API key
Semantic Scholar Tutorial https://www.semanticscholar.org/product/api/tutorial Step-by-step usage guide -
CrossRef REST API https://www.crossref.org/documentation/retrieve-metadata/rest-api/ DOI metadata, direct BibTeX via content negotiation Polite pool with mailto
arXiv API https://info.arxiv.org/help/api/basics.html Preprint metadata, full-text access 3-second delays
OpenAlex API https://docs.openalex.org/api-entities/works 240M+ works, CC0 license, open successor to MAG 100K/day, 10 RPS

Python Libraries

Library Install Documentation Purpose
semanticscholar pip install semanticscholar https://semanticscholar.readthedocs.io/ Official-ish Python wrapper
arxiv pip install arxiv https://pypi.org/project/arxiv/ arXiv search and download
habanero pip install habanero https://github.com/sckott/habanero CrossRef Python client

BibTeX Retrieval Code Pattern

python
import requests

def doi_to_bibtex(doi: str) -> str:
    """Get BibTeX directly from DOI via CrossRef content negotiation."""
    response = requests.get(
        f"https://doi.org/{doi}",
        headers={"Accept": "application/x-bibtex"}
    )
    response.raise_for_status()
    return response.text

# Example: Get verified BibTeX for "Attention Is All You Need"
bibtex = doi_to_bibtex("10.48550/arXiv.1706.03762")

Citation Verification Tools

Tool URL Purpose
Citely https://citely.ai/citation-checker Batch verification of AI-generated citations
ReciteWorks https://reciteworks.com/ Check in-text citations match reference list

LaTeX Citation Management

Resource URL Key Info
BibTeX vs BibLaTeX Guide https://electricalvoice.com/latex-vs-bibtex-vs-biblatex/ When to use which system
BibLaTeX Comprehensive Guide https://latextutorial.net/latex-vs-bibtex-vs-biblatex/ Modern citation management

Part 4: The Verified Citation Workflow

text
1. SEARCH: User specifies topic โ†’ Query Semantic Scholar API
   โ””โ”€ Use paper/search endpoint with specific keywords
   
2. VERIFY EXISTENCE: For each candidate paper:
   โ””โ”€ Confirm paper exists in 2+ sources (Semantic Scholar + CrossRef/arXiv)
   โ””โ”€ Verify DOI resolves correctly
   
3. GET BIBTEX: Use DOI content negotiation
   โ””โ”€ requests.get(f"https://doi.org/{doi}", headers={"Accept": "application/x-bibtex"})
   โ””โ”€ NEVER generate BibTeX from memory - always fetch
   
4. VERIFY CLAIMS: Before citing paper for specific claim:
   โ””โ”€ Retrieve paper abstract/full-text via Semantic Scholar
   โ””โ”€ Confirm the attributed claim actually appears in source
   
5. BUILD BIBLIOGRAPHY:
   โ””โ”€ Maintain .bib file with only verified entries
   โ””โ”€ Use consistent citation keys (e.g., author_year_firstword)

Why This Matters

From research on AI citation hallucination:

  • ~40% of AI-generated citations contain errors (Enago Academy research)
  • NeurIPS 2025 found 100+ hallucinated citations slipped through review
  • Common errors: fabricated titles, wrong authors, non-existent papers with plausible metadata

Part 5: Visualization & Formatting Resources

Figure Creation

Tool URL Purpose
PlotNeuralNet https://github.com/HarisIqbal88/PlotNeuralNet TikZ neural network diagrams
SciencePlots https://github.com/garrettj403/SciencePlots Publication-ready matplotlib styles
Okabe-Ito Palette https://jfly.uni-koeln.de/color/ Colorblind-safe color scheme

LaTeX Templates

Venue Template URL
NeurIPS https://neurips.cc/Conferences/2025/PaperInformation/StyleFiles
ICML https://www.overleaf.com/latex/templates/icml2025-template/dhxrkcgkvnkt
ICLR https://iclr.cc/Conferences/2026/AuthorGuide (links to template)

Part 6: Key Principles Summary (For Skill Encoding)

From Neel Nanda

  1. Paper = short, rigorous, evidence-based technical story with a takeaway readers care about
  2. Spend equal time on: abstract, intro, figures, everything else
  3. Every experiment must support a specific claim connected to contribution
  4. "If you can't state your contribution in one sentence, you don't have a paper yet"

From Karpathy

  1. "The paper sells a single thing that was not obvious before"
  2. Default structure: Intro โ†’ Related Work โ†’ Model โ†’ Experiments โ†’ Conclusions
  3. Review bad papers to learn what NOT to do (binary classifier training)

From Zachary Lipton

  1. "If the first sentence can be pre-pended to any ML paper, delete it"
  2. Figures should tell coherent story even if reader skips text
  3. Sections should be balanced like bullets on slides
  4. "provides very tight approximation" drips with insecurity โ†’ "provides tight approximation"

From Sebastian Farquhar

  1. Methods should start by page 2-3 maximum
  2. Abstract formula: (1) What achieved, (2) Why hard/important, (3) How with keywords, (4-5) Evidence + best number
  3. Introduction must have 2-4 bullet contribution list (max 1-2 lines each)

From Gopen & Swan

  1. Place emphasis at sentence ends (stress position)
  2. Put context (old info) before new information
  3. Keep subject and verb close together
  4. One unit = one function (each paragraph = one point)

Part 7: Additional Resources

Hallucination & AI Writing Concerns

Source URL
AI Hallucinations in Research Citations https://www.enago.com/academy/ai-hallucinations-research-citations/
Hallucination in AI-Generated Writing (PMC) https://pmc.ncbi.nlm.nih.gov/articles/PMC10726751/
NeurIPS 2025 AI Hallucination Report https://byteiota.com/neurips-2025-100-ai-hallucinations-slip-through-review/

ML Conference Review System Analysis

Source URL
Position: ML Conferences Should Have Refutations Track https://arxiv.org/html/2506.19882v1

Usage Notes for Skill Development

  1. For paper structure: Start with Nanda + Farquhar for high-level philosophy, use conference guidelines for specifics

  2. For writing style: Combine Lipton's heuristics + Gopen & Swan's principles + Ethan Perez's micro-tips

  3. For citation workflow: Implement Semantic Scholar โ†’ DOI verification โ†’ CrossRef BibTeX pipeline; NEVER generate citations from model memory

  4. For figures/tables: Reference booktabs for tables, SciencePlots for figures, always use colorblind-safe palettes

  5. For reviewer simulation: Study reviewer guidelines from all three venues to anticipate criticisms

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories โ†’

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#ai Browse ai catalog โ†’ #ai-research Browse ai-research catalog โ†’ #claude Browse claude catalog โ†’ #claude-code Browse claude-code catalog โ†’ #claude-skills Browse claude-skills catalog โ†’ #codex Browse codex catalog โ†’ โšก MCP Servers Model Context Protocol โ†’ ๐Ÿง  Agent Skills Prompt & rule files โ†’ ๐Ÿ”Œ OpenAPI Specs REST & Swagger schemas โ†’