# Technical Documentation: Orchestra-Research/AI-Research-SKILLs
> ℹ️ **Provenance:** Hybrid Fusion: `Orchestra-Research/AI-Research-SKILLs` (README + 9 In-Tree Chapters) · [CodeWiki Reference](https://codewiki.google/github.com/Orchestra-Research/AI-Research-SKILLs) · Recency: Active (< 180 days)
## 1. Project Overview & Quickstart (Orchestra-Research/AI-Research-SKILLs)
# AI Research `Skills` Library
> **The most comprehensive open-source skills library enabling AI agents to autonomously conduct AI research — from idea to paper**
**View All 23 Categories**
---
## Table of Contents
- [Our Mission](#our-mission)
- [Path Towards AI Research Agent](#path-towards-ai-research-agent)
- [Available AI Research Engineering Skills](#available-ai-research-engineering-skills)
- [Demos](#demos)
- [Skill Structure](#skill-structure)
- [Roadmap](#roadmap)
- [Repository Structure](#repository-structure)
- [Use Cases](#use-cases)
- [Contributors](#contributors)
- [Citation](#citation)
- [Community](#community)
## Our Mission
We enable AI agents to **autonomously conduct AI research** — from literature survey and idea generation through experiment execution to paper writing. The library provides both the **research orchestration layer** (autoresearch, ideation, paper writing) and the **engineering skills** (training, evaluation, deployment) needed at each stage.
## Path Towards AI Research Agent
Modern AI research requires mastering dozens of specialized tools and frameworks.
AI Researchers spend more time debugging infrastructure than testing hypotheses — slowing the pace of scientific discovery.
We provide a comprehensive skills library that enables AI agents to autonomously conduct the full research lifecycle — from brainstorming ideas to writing the paper.
- Autonomous Research - The **autoresearch** skill orchestrates the entire research workflow using a two-loop architecture, routing to domain skills as needed
- Specialized Expertise - Each domain skill provides deep, production-ready knowledge of a specific framework (Megatron-LM, vLLM, TRL, etc.)
- End-to-End Coverage - 98 skills spanning the full AI research lifecycle, from ideation and literature survey to experiments and paper writing
- Research-Grade Quality - Documentation sourced from official repos, real GitHub issues, and battle-tested production workflows
## Available AI Research Engineering Skills
**Quality over quantity**: Each skill provides comprehensive, expert-level guidance with real code examples, troubleshooting guides, and production-ready workflows.
### 📦 Quick Install (Recommended)
**For humans** — interactive installer with one command:
```bash
npx @orchestra-research/ai-research-skills
```
**For AI agents** — point your agent to the welcome doc and it handles the rest:
```
Read https://www.orchestra-research.com/ai-research-skills/welcome.md and follow the instructions to install and use AI Research Skills.
```
This installs all 98 skills, loads the **autoresearch** orchestration layer, and starts autonomous research.
**What the installer does**
- **Auto-detects** your installed coding agents (Claude Code, Hermes Agent, OpenCode, Qoder, Cursor, Gemini CLI, etc.)
- **Installs** skills to `~/.orchestra/skills/` with symlinks to each agent (falls back to copy on Windows)
- **Offers** everything, quickstart bundle, by category, or individual skills
- **Updates** installed skills with latest versions
- **Uninstalls** all or selected skills
**CLI Commands**
```bash
# Interactive installer (recommended)
npx @orchestra-research/ai-research-skills
# Direct commands
npx @orchestra-research/ai-research-skills list # View installed skills
npx @orchestra-research/ai-research-skills update # Update installed skills
```
**Claude Code Marketplace (Alternative)**
Install skill categories directly using the **Claude Code CLI**:
```bash
# Add the marketplace
/plugin marketplace add orchestra-research/AI-research-SKILLs
# Install by category (23 categories available)
/plugin install fine-tuning@ai-research-skills # Axolotl, LLaMA-Factory, PEFT, Unsloth
/plugin install post-training@ai-research-skills # TRL, GRPO, OpenRLHF, SimPO, verl, slime, miles, torchforge
/plugin install inference-serving@ai-research-skills # vLLM, TensorRT-LLM, llama.cpp, SGLang
/plugin install distributed-training@ai-research-skills
/plugin install optimization@ai-research-skills
```
### All 23 Categories (98 Skills)
| Category | Skills | Included |
|----------|--------|----------|
| **Autoresearch** | **1** | **Autonomous research orchestration — central layer that manages the full lifecycle and routes to all other skills** |
| Ideation | 2 | Research Brainstorming, Creative Thinking |
| ML Paper Writing | 2 | ML Paper Writing (LaTeX templates, citation verification), Academic Plotting |
| Model Architecture | 5 | LitGPT, Mamba, NanoGPT, RWKV, TorchTitan |
| Tokenization | 2 | HuggingFace Tokenizers, SentencePiece |
| Fine-Tuning | 4 | Axolotl, LLaMA-Factory, PEFT, Unsloth |
| Mech Interp | 4 | TransformerLens, SAELens, pyvene, nnsight |
| Data Processing | 2 | NeMo Curator, Ray Data |
| Post-Training | 8 | TRL, GRPO, OpenRLHF, SimPO, verl, slime, miles, torchforge |
| Safety | 4 | Constitutional AI, LlamaGuard, NeMo Guardrails, Prompt Guard |
| Distributed | 6 | DeepSpeed, FSDP, Accelerate, Megatron-Core, Lightning, Ray Train |
| Infrastructure | 3 | Modal, Lambda Labs, SkyPilot |
| Optimization | 6 | Flash Attention, bitsandbytes, GPTQ, AWQ, HQQ, GGUF |
| Evaluation | 3 | lm-eval-harness, BigCode, NeMo Evaluator |
| Inference | 4 | vLLM, TensorRT-LLM, llama.cpp, SGLang |
| MLOps | 3 | W&B, MLflow, TensorBoard |
| Agents | 4 | LangChain, LlamaIndex, CrewAI, AutoGPT |
| RAG | 5 | Chroma, FAISS, Pinecone, Qdrant, Sentence Transformers |
| Prompt Eng | 4 | DSPy, Instructor, Guidance, Outlines |
| Observability | 2 | LangSmith, Phoenix |
| Multimodal | 7 | CLIP, Whisper, LLaVA, BLIP-2, SAM, Stable Diffusion, AudioCraft |
| Emerging | 6 | MoE, Model Merging, Long Context, Speculative Decoding, Distillation, Pruning |
| Agent-Native Research Artifact | 3 | ARA Compiler, Research Manager, Rigor Reviewer |
**View All 98 Skills in Details**
### 🔬 Autoresearch (1 skill) — Central Orchestration Layer
- **[Autoresearch](0-autoresearch-skill/)** - Autonomous research orchestration using a two-loop architecture (inner optimization + outer synthesis). Manages the full lifecycle from literature survey to paper writing, routing to all domain-specific skills. Supports Claude Code /loop and OpenClaw heartbeat for continuous operation (390 lines + 3 refs)
### 🏗️ Model Architecture (5 skills)
- **[LitGPT](01-model-architecture/litgpt/)** - Lightning AI's 20+ clean LLM implementations with production training recipes (462 lines + 4 refs)
- **[Mamba](01-model-architecture/mamba/)** - State-space models with O(n) complexity, 5× faster than Transformers (253 lines + 3 refs)
- **[RWKV](01-model-architecture/rwkv/)** - RNN+Transformer hybrid, infinite context, Linux Foundation project (253 lines + 3 refs)
- **[NanoGPT](01-model-architecture/nanogpt/)** - Educational GPT in ~300 lines by Karpathy (283 lines + 3 refs)
- **[TorchTitan](01-model-architecture/torchtitan/)** - PyTorch-native distributed training for Llama 3.1 with 4D parallelism
### 🔤 Tokenization (2 skills)
- **[HuggingFace Tokenizers](02-tokenization/huggingface-tokenizers/)** - Rust-based, <20s/GB, BPE/WordPiece/Unigram algorithms (486 lines + 4 refs)
- **[SentencePiece](02-tokenization/sentencepiece/)** - Language-independent, 50k sentences/sec, used by T5/ALBERT (228 lines + 2 refs)
### 🎯 Fine-Tuning (4 skills)
- **[Axolotl](03-fine-tuning/axolotl/)** - YAML-based fine-tuning with 100+ models (156 lines + 4 refs)
- **[LLaMA-Factory](03-fine-tuning/llama-factory/)** - WebUI no-code fine-tuning (78 lines + 5 refs)
- **[Unsloth](03-fine-tuning/unsloth/)** - 2x faster QLoRA fine-tuning (75 lines + 4 refs)
- **[PEFT](03-fine-tuning/peft/)** - Parameter-efficient fine-tuning with LoRA, QLoRA, DoRA, 25+ methods (431 lines + 2 refs)
### 🔬 Mechanistic Interpretability (4 skills)
- **[TransformerLens](04-mechanistic-interpretability/transformer-lens/)** - Neel Nanda's library for mech interp with HookPoints, activation caching (346 lines + 3 refs)
- **[SAELens](04-mechanistic-interpretability/saelens/)** - Sparse Autoencoder training and analysis for feature discovery (386 lines + 3 refs)
- **[pyvene](04-mechanistic-interpretability/pyvene/)** - Stanford's causal intervention library with declarative configs (473 lines + 3 refs)
- **[nnsight](04-mechanistic-interpretability/nnsight/)** - Remote interpretability via NDIF, run experiments on 70B+ models (436 lines + 3 refs)
### 📊 Data Processing (2 skills)
- **[Ray Data](05-data-processing/ray-data/)** - Distributed ML data processing, streaming execution, GPU support (318 lines + 2 refs)
- **[NeMo Curator](05-data-processing/nemo-curator/)** - GPU-accelerated data curation, 16× faster deduplication (375 lines + 2 refs)
### 🎓 Post-Training (8 skills)
- **[TRL Fine-Tuning](06-post-training/trl-fine-tuning/)** - Transformer Reinforcement Learning (447 lines + 4 refs)
- **[GRPO-RL-Training](06-post-training/grpo-rl-training/)** (TRL) - Group Relative Policy Optimization with TRL (569 lines, **gold standard**)
- **[OpenRLHF](06-post-training/openrlhf/)** - Full RLHF pipeline with Ray + vLLM (241 lines + 4 refs)
- **[SimPO](06-post-training/simpo/)** - Simple Preference Optimization, no reference model needed (211 lines + 3 refs)
- **[verl](06-post-training/verl/)** - ByteDance's HybridFlow RL framework, FSDP/Megatron + vLLM/SGLang backends (389 lines + 2 refs)
- **[slime](06-post-training/slime/)** - THUDM's Megatron+SGLang framework powering GLM-4.x models (464 lines + 2 refs)
- **[miles](06-post-training/miles/)** - Enterprise fork of slime with FP8, INT4, speculative RL for MoE training (315 lines + 2 refs)
- **[torchforge](06-post-training/torchforge/)** - Meta's PyTorch-native RL with Monarch+TorchTitan+vLLM (380 lines + 2 refs)
### 🛡️ Safety & Alignment (4 skills)
- **[Constitutional AI](07-safety-alignment/constitutional-ai/)** - AI-driven self-improvement via principles (282 lines)
- **[LlamaGuard](07-safety-alignment/llamaguard/)** - Safety classifier for LLM inputs/outputs (329 lines)
- **[NeMo Guardrails](07-safety-alignment/nemo-guardrails/)** - Programmable guardrails with Colang (289 lines)
- **[Prompt Guard](07-safety-alignment/prompt-guard/)** - Meta's 86M prompt injection & jailbreak detector, 99%+ TPR, <2ms GPU (313 lines)
### ⚡ Distributed Training (6 skills)
- **[Megatron-Core](08-distributed-training/megatron-core/)** - NVIDIA's framework for training 2B-462B param models with 47% MFU on H100 (359 lines + 4 refs)
- **[DeepSpeed](08-distributed-training/deepspeed/)** - Microsoft's ZeRO optimization (137 lines + 9 refs)
- **[PyTorch FSDP2](08-distributed-training/pytorch-fsdp2/)** - Fully Sharded Data Parallel v2 with `fully_shard` and DTensor (231 lines + 12 refs)
- **[Accelerate](08-distributed-training/accelerate/)** - HuggingFace's 4-line distributed training API (324 lines + 3 refs)
- **[PyTorch Lightning](08-distributed-training/pytorch-lightning/)** - High-level training framework with Trainer class (339 lines + 3 refs)
- **[Ray Train](08-distributed-training/ray-train/)** - Multi-node orchestration and hyperparameter tuning (399 lines + 1 ref)
### 🚀 Optimization (6 skills)
- **[Flash Attention](10-optimization/flash-attention/)** - 2-4x faster attention with memory efficiency (359 lines + 2 refs)
- **[bitsandbytes](10-optimization/bitsandbytes/)** - 8-bit/4-bit quantization for 50-75% memory reduction (403 lines + 3 refs)
- **[GPTQ](10-optimization/gptq/)** - 4-bit post-training quantization, 4× memory reduction, <2% accuracy loss (443 lines + 3 refs)
- **[AWQ](10-optimization/awq/)** - Activation-aware weight quantization, 4-bit with minimal accuracy loss (310 lines + 2 refs)
- **[HQQ](10-optimization/hqq/)** - Half-Quadratic Quantization, no calibration data needed, multi-backend (370 lines + 2 refs)
- **[GGUF](10-optimization/gguf/)** - llama.cpp quantization format, K-quant methods, CPU/Metal inference (380 lines + 2 refs)
### 📊 Evaluation (3 skills)
- **[lm-evaluation-harness](11-evaluation/lm-evaluation-harness/)** - EleutherAI's standard for benchmarking LLMs across 60+ tasks (482 lines + 4 refs)
- **[BigCode Evaluation Harness](11-evaluation/bigcode-evaluation-harness/)** - Code model benchmarking with HumanEval, MBPP, MultiPL-E, pass@k metrics (406 lines + 3 refs)
- **[NeMo Evaluator](11-evaluation/nemo-evaluator/)** - NVIDIA's enterprise platform for 100+ benchmarks across 18+ harnesses with multi-backend execution (454 lines + 4 refs)
### ☁️ Infrastructure (3 skills)
- **[Modal](09-infrastructure/modal/)** - Serverless GPU cloud with Python-native API, T4-H200 on-demand (342 lines + 2 refs)
- **[SkyPilot](09-infrastructure/skypilot/)** - Multi-cloud orchestration across 20+ providers with spot recovery (390 lines + 2 refs)
- **[Lambda Labs](09-infrastructure/lambda-labs/)** - Reserved/on-demand GPU cloud with H100/A100, persistent filesystems (390 lines + 2 refs)
### 🔥 Inference & Serving (4 skills)
- **[vLLM](12-inference-serving/vllm/)** - High-throughput LLM serving with PagedAttention (356 lines + 4 refs, **production-ready**)
- **[TensorRT-LLM](12-inference-serving/tensorrt-llm/)** - NVIDIA's fastest inference, 24k tok/s, FP8/INT4 quantization (180 lines + 3 refs)
- **[llama.cpp](12-inference-serving/llama-cpp/)** - CPU/Apple Silicon inference, GGUF quantization (251 lines + 3 refs)
- **[SGLang](12-inference-serving/sglang/)** - Structured generation with RadixAttention, 5-10× faster for agents (435 lines + 3 refs)
### 🤖 Agents (4 skills)
- **[LangChain](14-agents/langchain/)** - Most popular agent framework, 500+ integrations, ReAct pattern (658 lines + 3 refs, **production-ready**)
- **[LlamaIndex](14-agents/llamaindex/)** - Data framework for LLM apps, 300+ connectors, RAG-focused (535 lines + 3 refs)
- **[CrewAI](14-agents/crewai/)** - Multi-agent orchestration, role-based collaboration, autonomous workflows (498 lines + 3 refs)
- **[AutoGPT](14-agents/autogpt/)** - Autonomous AI agent platform, visual workflow builder, continuous execution (400 lines + 2 refs)
### 🔍 RAG (5 skills)
- **[Chroma](15-rag/chroma/)** - Open-source embedding database, local/cloud, 24k stars (385 lines + 1 ref)
- **[FAISS](15-rag/faiss/)** - Facebook's similarity search, billion-scale, GPU acceleration (295 lines)
- **[Sentence Transformers](15-rag/sentence-transformers/)** - 5000+ embedding models, multilingual, 15k stars (370 lines)
- **[Pinecone](15-rag/pinecone/)** - Managed vector database, auto-scaling, <100ms latency (410 lines)
- **[Qdrant](15-rag/qdrant/)** - High-performance vector search, Rust-powered, hybrid search with filtering (493 lines + 2 refs)
### 🎨 Multimodal (7 skills)
- **[CLIP](18-multimodal/clip/)** - OpenAI's vision-language model, zero-shot classification, 25k stars (320 lines)
- **[Whisper](18-multimodal/whisper/)** - Robust speech recognition, 99 languages, 73k stars (395 lines)
- **[LLaVA](18-multimodal/llava/)** - Vision-language assistant, image chat, GPT-4V level (360 lines)
- **[Stable Diffusion](18-multimodal/stable-diffusion/)** - Text-to-image generation via HuggingFace Diffusers, SDXL, ControlNet (380 lines + 2 refs)
- **[Segment Anything](18-multimodal/segment-anything/)** - Meta's SAM for zero-shot image segmentation with points/boxes (500 lines + 2 refs)
- **[BLIP-2](18-multimodal/blip-2/)** - Vision-language pretraining with Q-Former, image captioning, VQA (500 lines + 2 refs)
- **[AudioCraft](18-multimodal/audiocraft/)** - Meta's MusicGen/AudioGen for text-to-music and text-to-sound (470 lines + 2 refs)
### 🎯 Prompt Engineering (4 skills)
- **[DSPy](16-prompt-engineering/dspy/)** - Declarative prompt programming with optimizers, Stanford NLP, 22k stars (438 lines + 3 refs)
- **[Instructor](16-prompt-engineering/instructor/)** - Structured LLM outputs with Pydantic validation, 15k stars (726 lines + 3 refs)
- **[Guidance](16-prompt-engineering/guidance/)** - Constrained generation with regex/grammars, Microsoft Research, 18k stars (485 lines + 3 refs)
- **[Outlines](16-prompt-engineering/outlines/)** - Structured text with FSM, zero-overhead, 8k stars (601 lines + 3 refs)
### 📊 MLOps (3 skills)
- **[Weights & Biases](13-mlops/weights-and-biases/)** - Experiment tracking, sweeps, artifacts, model registry (427 lines + 3 refs)
- **[MLflow](13-mlops/mlflow/)** - Model registry, tracking, deployment, autologging (514 lines + 3 refs)
- **[TensorBoard](13-mlops/tensorboard/)** - Visualization, profiling, embeddings, scalars/images (538 lines + 3 refs)
### 👁️ Observability (2 skills)
- **[LangSmith](17-observability/langsmith/)** - LLM observability, tracing, evaluation, monitoring for AI apps (422 lines + 2 refs)
- **[Phoenix](17-observability/phoenix/)** - Open-source AI observability with OpenTelemetry tracing and LLM evaluation (380 lines + 2 refs)
### 🔬 Emerging Techniques (6 skills)
- **[MoE Training](19-emerging-techniques/moe-training/)** - Mixture of Experts training with DeepSpeed, Mixtral 8x7B, 5× cost reduction (515 lines + 3 refs)
- **[Model Merging](19-emerging-techniques/model-merging/)** - Combine models with TIES, DARE, SLERP using mergekit (528 lines + 3 refs)
- **[Long Context](19-emerging-techniques/long-context/)** - Extend context windows with RoPE, YaRN, ALiBi, 32k-128k tokens (624 lines + 3 refs)
- **[Speculative Decoding](19-emerging-techniques/speculative-decoding/)** - 1.5-3.6× faster inference with Medusa, Lookahead (379 lines)
- **[Knowledge Distillation](19-emerging-techniques/knowledge-distillation/)** - Compress models 70B→7B with MiniLLM, temperature scaling (424 lines)
- **[Model Pruning](19-emerging-techniques/model-pruning/)** - 50% sparsity with Wanda, SparseGPT, <1% accuracy loss (417 lines)
### 📝 ML Paper Writing (2 skills)
- **[ML Paper Writing](20-ml-paper-writing/)** - Write publication-ready papers for NeurIPS, ICML, ICLR, ACL, AAAI, COLM with LaTeX templates, citation verification, and writing best practices (532 lines + 5 refs)
- **[Academic Plotting](20-ml-paper-writing/academic-plotting/)** - Generate publication-quality figures for ML papers: architecture diagrams via Gemini AI and data-driven charts via matplotlib/seaborn with venue-specific styling (479 lines + 3 refs)
### 💡 Ideation (2 skills)
- **[Research Brainstorming](21-research-ideation/brainstorming-research-ideas/)** - Structured ideation frameworks for discovering high-impact research directions with 10 complementary lenses (384 lines)
- **[Creative Thinking](21-research-ideation/creative-thinking-for-research/)** - Cognitive science frameworks (bisociation, structure-mapping, constraint manipulation) for genuinely novel research ideas (366 lines)
### 🧬 Agent-Native Research Artifact (3 skills)
- **[ARA Compiler](22-agent-native-research-artifact/compiler/)** - Compiles any research input (PDF papers, repos, experiment logs, raw notes) into a complete Agent-Native Research Artifact with claims, exploration graph, evidence, and code stubs (245 lines + 3 refs)
- **[ARA Research Manager](22-agent-native-research-artifact/research-manager/)** - Post-task research recorder that runs at session end to extract decisions, experiments, dead ends, and pivots from conversation history into the `ara/` directory with user-vs-AI provenance tags (324 lines + 3 refs)
- **[ARA Rigor Reviewer](22-agent-native-research-artifact/rigor-reviewer/)** - ARA Seal Level 2 semantic epistemic review scoring six dimensions of research rigor (evidence relevance, falsifiability, scope, coherence, exploration integrity, methodology) with severity-ranked findings (322 lines + 1 ref)
## Demos
All 98 skills in this repo are automatically synced to [Orchestra Research](https://www.orchestra-research.com/research-skills), where you can add them to your projects with one click and use them with AI research agents.
**See skills in action → [demos/](demos/README.md)**
We maintain a curated collection of demo repositories showing how to use skills for real AI research tasks:
| Demo | Skills Used | What It Does |
|------|-------------|--------------|
| **[Norm Heterogeneity → LoRA Brittleness](demos/autoresearch-norm-heterogeneity/)** | Autoresearch, ML Paper Writing, Ideation | Agent autonomously discovered norm heterogeneity predicts fine-tuning difficulty (r=-0.99), pivoting from a null result on ETF overlaps |
| **[RL Algorithm Brain Scan](demos/autoresearch-rl-brain-scan/)** | Autoresearch, GRPO, TRL, SAELens, TransformerLens, ML Paper Writing | Agent found DPO is a rank-1 perturbation (95.6% recovery from one SVD direction) while online RL is distributed and structure-preserving |
| **[NeMo Eval: GPQA Benchmark](https://github.com/zechenzhangAGI/Nemo-Eval-Skill-Demo)** | NeMo Evaluator | Compare Llama 8B/70B/405B on graduate-level science questions |
| **[LoRA Without Regret Reproduction](https://www.orchestra-research.com/perspectives/LLM-with-Orchestra)** | GRPO, TRL | Reproduce SFT + GRPO RL experiments via prompting |
| **[Layer-Wise Quantization Experiment](https://github.com/AmberLJC/llama-quantization-experiment)** | llama.cpp, GGUF | Investigate optimal layer precision allocation—early layers at Q8 achieve 1.9× compression with 1.3% perplexity loss |
| **[Cross-Lingual Alignment Analysis](https://github.com/AmberLJC/faiss-demo)** | FAISS | Quantify how well multilingual embeddings align semantic concepts across 8 languages using FAISS similarity search |
| **[Scientific Plotting Demo](demos/scientific-plotting-demo/)** | Academic Plotting | Generate publication-quality figures for the Andes QoE-aware LLM serving paper — Gemini AI architecture diagrams + matplotlib data charts (CDF, multi-panel grids, bar charts) |
**Featured Demos**: Two papers produced entirely by AI agents using the **autoresearch** skill. The [Norm Heterogeneity paper](demos/autoresearch-norm-heterogeneity/) demonstrates autonomous research pivoting — the agent refuted its own hypothesis and discovered a stronger finding. The [RL Brain Scan paper](demos/autoresearch-rl-brain-scan/) demonstrates multi-skill orchestration — the agent trained RL models, analyzed internals with interpretability tools, and synthesized the insight that "DPO is rank-1 alignment." Both papers written end-to-end by the agent.
## Skill Structure
Each skill follows a battle-tested format for maximum usefulness:
```
skill-name/
├── SKILL.md # Quick reference (50-150 lines)
│ ├── Metadata (name, description, version)
│ ├── When to use this skill
│ ├── Quick patterns & examples
│ └── Links to references
│
├── references/ # Deep documentation (300KB+)
│ ├── README.md # From GitHub/official docs
│ ├── api.md # API reference
│ ├── tutorials.md # Step-by-step guides
│ ├── issues.md # Real GitHub issues & solutions
│ ├── releases.md # Version history & breaking changes
│ └── file_structure.md # Codebase navigation
│
├── scripts/ # Helper scripts (optional)
└── assets/ # Templates & examples (optional)
```
**Quality Standards**
- 300KB+ documentation from official sources
- Real GitHub issues & solutions (when available)
- Code examples with language detection
- Version history & breaking changes
- Links to official docs
## Roadmap
The library spans 98 comprehensive skills across the full AI research lifecycle. See our [detailed roadmap](docs/ROADMAP.md) for the complete development plan.
[View Full Roadmap →](docs/ROADMAP.md)
**View Detailed Statistics**
| Metric | Current | Target |
|--------|---------|--------|
| **Skills** | **87** (high-quality, standardized YAML) | 80 ✅ |
| **Avg Lines/Skill** | **420 lines** (focused + progressive disclosure) | 200-600 lines |
| **Documentation** | **~130,000 lines** total (SKILL.md + references) | 100,000+ lines |
| **Gold Standard Skills** | **65** with comprehensive references | 50+ |
| **Contributors** | 1 | 100+ |
| **Coverage** | Architecture, Tokenization, Fine-Tuning, Mechanistic Interpretability, Data Processing, Post-Training, Safety, Distributed, Optimization, Evaluation, Infrastructure, Inference, Agents, RAG, Multimodal, Prompt Engineering, MLOps, Observability, ML Paper Writing, Ideation, Autoresearch | Full Lifecycle ✅ |
**Recent Progress**: npm package `@orchestra-research/ai-research-skills` for one-command installation across all coding agents
**Philosophy**: Quality > Quantity. Following [Anthropic official best practices](anthropic_official_docs/best_practices.md) - each skill provides 200-500 lines of focused, actionable guidance with progressive disclosure.
## Repository Structure
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Use Cases
### For Researchers
"I need to fine-tune Llama 3 with custom data"
→ **03-fine-tuning/axolotl/** - YAML configs, 100+ model support
### For ML Engineers
"How do I optimize inference latency?"
→ **12-inference-serving/vllm/** - PagedAttention, batching
### For Students
"I want to learn how transformers work"
→ **01-model-architecture/litgpt/** - Clean implementations
### For Teams
"We need to scale training to 100 GPUs"
→ **08-distributed-training/deepspeed/** - ZeRO stages, 3D parallelism
## License
MIT License - See [LICENSE](LICENSE) for details.
**Note**: Individual skills may reference libraries with different licenses. Please check each project's license before use.
## Citation
If you use AI Research Skills in your work or find it helpful for a publication, we'd appreciate a citation:
**BibTeX**
```bibtex
@software{ai_research_skills,
title = {AI Research Skills Library},
author = {{Orchestra Research}},
year = {2025},
url = {https://github.com/orchestra-research/AI-research-SKILLs},
note = {Open-source skills library enabling AI agents to autonomously conduct AI research}
}
```
**APA**
> Orchestra Research. (2025). *AI Research Skills Library* [Computer software]. https://github.com/orchestra-research/AI-research-SKILLs
**Chicago**
> Orchestra Research. "AI Research Skills Library." GitHub, 2025. https://github.com/orchestra-research/AI-research-SKILLs.
**IEEE**
> Orchestra Research, "AI Research Skills Library," 2025. [Online]. Available: https://github.com/orchestra-research/AI-research-SKILLs
> **Tip**: You can also click **"Cite this repository"** in the GitHub sidebar for auto-formatted citations.
## Acknowledgments
Built with:
- **[Claude Code](https://www.claude.com/product/claude-code)** - AI pair programming
- **[Skill Seeker](https://github.com/yusufkaraaslan/Skill_Seekers)** - Automated doc scraping
- **Open Source AI Community** - For amazing tools and docs
Special thanks to:
- EleutherAI, HuggingFace, NVIDIA, Lightning AI, Meta AI, Anthropic
- All researchers who maintain excellent documentation
## Contributors
Thanks to all the people who have contributed to the AI Research Skills Library:
We welcome contributions from the AI research community! See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines on:
- Adding new skills
- Improving existing skills
- Quality standards and best practices
- Submission process
## Recent Updates
**June 2026 - v1.7.1 🧹 Inventory Consistency, Drift Guard & Security Hardening**
- 📊 **Repo-wide inventory reconciled to 98 skills / 23 categories** — corrected stale counts that had drifted apart across files: CLAUDE.md (said 90), CONTRIBUTING.md (said 86/22), the README sync line (said 87), WELCOME.md and the npm package README (said 86/22). Also fixed wrong per-category listings (TorchTitan, SwanLab, A-Evolve, ML Training Recipes, Cosmos Policy/OpenPI/OpenVLA-OFT, the paper-writing skills)
- 🛡️ **New CI drift guard** — `scripts/check-inventory.sh` + `check-inventory.yml` fail CI whenever the documented skill/category counts diverge from the actual `SKILL.md` count on disk, so the inventory can't silently drift again
- 📦 **Marketplace sync hardening** — `sync-skills.yml` now prunes build artifacts (`node_modules`/`__pycache__`/`*.pyc`/`.ipynb_checkpoints`) before zipping and fails loudly above 190 files instead of hitting the marketplace's 200-file rejection
- 🔒 **Security** — pinned CLI dependencies (`chalk`/`inquirer`/`ora`) to exact, patched versions with a regenerated lockfile (`inquirer@9.3.8` clears the `tmp` path-traversal advisory; `npm audit` now reports 0 vulnerabilities)
- 🧹 Full open PR/issue triage pass against scope + contribution standards
**April 2026 - v1.6.0 🧬 Agent-Native Research Artifact (ARA) — 23rd Category, 98 Skills**
- 🧬 **NEW CATEGORY**: `22-agent-native-research-artifact/` (the 23rd category) — three skills that turn research outputs into a falsifiable, agent-traversable artifact:
- 🛠️ **[ARA Compiler](22-agent-native-research-artifact/compiler/)** — compiles any input (PDF papers, GitHub repos, experiment logs, raw notes) into a structured ARA with cognitive layer (claims, concepts, heuristics), physical layer (configs, code stubs), exploration graph (research DAG), and grounded evidence
- 📋 **[ARA Research Manager](22-agent-native-research-artifact/research-manager/)** — post-task epilogue that scans conversation history at session end and writes decisions, experiments, dead ends, claims, heuristics, and pivots into the `ara/` directory with `user` / `ai-suggested` / `ai-executed` / `user-revised` provenance tags
- 🔍 **[ARA Rigor Reviewer](22-agent-native-research-artifact/rigor-reviewer/)** — Seal Level 2 semantic epistemic review scoring six dimensions of research rigor (evidence relevance, falsifiability, scope calibration, argument coherence, exploration integrity, methodological rigor) and emitting a severity-ranked report with a Strong Accept-to-Reject recommendation
- 🔗 Sourced from the [Agent-Native-Research-Artifact-Init](https://github.com/Orchestra-Research/Agent-Native-Research-Artifact-Init) reference repo, restructured to AI-research-SKILLs standards (kebab-case names, third-person descriptions, Title-Case tags, one-level-deep references)
- 🧩 Plugin entry `agent-native-research-artifact` added to `.claude-plugin/marketplace.json`; CLI category registered as `22-agent-native-research-artifact` with three individual skill entries in the npm installer
- 🔄 Auto-syncs to Orchestra marketplace via `sync-skills.yml` on push; npm package republished as `@orchestra-research/ai-research-skills@1.6.0` via `publish-npm.yml` on version bump
- 📊 **98 total skills** across **23 categories** — full lifecycle from idea → paper → falsifiable, auditable artifact
**March 2026 - v1.4.0 🔬 Autoresearch & 86 Skills — Full Research Lifecycle**
- 🔬 **NEW SKILL**: **Autoresearch** — autonomous research orchestration using a two-loop architecture (inner optimization loop + outer synthesis loop)
- 🧠 Manages the full research lifecycle: literature survey → ideation → experiments → synthesis → paper writing
- 🔄 Routes to all 86 domain skills automatically — agents don't need to know which skill to use
- ⏰ Mandatory `/loop` (Claude Code) and cron job (OpenClaw) for continuous autonomous operation
- 📊 Generates research presentations (HTML/PDF) with optimization trajectory plots for human review
- 📝 Findings.md as persistent project memory across sessions with "Lessons and Constraints" tracking
- 🗂️ Structured workspace: research-state.yaml, findings.md, research-log.md, literature/, experiments/, src/, data/, to_human/
- 📄 **Two demo papers produced by autoresearch**: [Norm Heterogeneity → LoRA Brittleness](demos/autoresearch-norm-heterogeneity/) and [RL Algorithm Brain Scan](demos/autoresearch-rl-brain-scan/)
- 🚀 WELCOME.md for cold-start agent bootstrap — one URL to go from zero to autonomous research
- 📦 npm v1.4.x with Windows symlink fallback, all 22 categories installable
- 🤖 **Supported agents**: Claude Code, Hermes Agent, OpenCode, OpenClaw, Qoder, Cursor, Codex, Gemini CLI, Qwen Code
- 📊 **87 total skills** across **22 categories** — complete research lifecycle coverage
**February 2026 - v0.15.0 🛡️ Prompt Guard & 83 Skills**
- 🛡️ **NEW SKILL**: Prompt Guard - Meta's 86M prompt injection & jailbreak detector
- ⚡ 99%+ TPR, <1% FPR, <2ms GPU latency, multilingual (8 languages)
- 🔒 3 workflows: user input filtering, third-party data filtering, batch RAG processing
- 📊 **83 total skills** across 20 categories
**January 2026 - v0.14.0 📦 npm Package & 82 Skills**
- 📦 **NEW**: `npx @orchestra-research/ai-research-skills` - One-command installation for all coding agents
- 🤖 **Supported agents**: Claude Code, OpenCode, Cursor, Codex, Gemini CLI, Qwen Code
- ✨ Interactive installer with category/individual skill selection
- 🔄 Update installed skills, selective uninstall
- 📊 **82 total skills** (5 new post-training skills: verl, slime, miles, torchforge + TorchTitan)
- 🏗️ Megatron-Core moved to Distributed Training category
**January 2026 - v0.13.0 📝 ML Paper Writing & Demos Gallery**
- 📝 **NEW CATEGORY**: ML Paper Writing (20th category, 77th skill)
- 🎯 Write publication-ready papers for NeurIPS, ICML, ICLR, ACL, AAAI, COLM
- 📚 Writing philosophy from top researchers (Neel Nanda, Farquhar, Gopen & Swan, Lipton, Perez)
- 🔬 Citation verification workflow - never hallucinate references
- 📄 LaTeX templates for 6 major conferences
- 🎪 **NEW**: Curated demos gallery (`demos/`) showcasing skills in action
- 🔗 Demo repos: NeMo Evaluator benchmark, LoRA Without Regret reproduction
- 📖 936-line comprehensive SKILL.md with 4 workflows
**January 2026 - v0.12.0 📊 NeMo Evaluator SDK**
- 📊 **NEW SKILL**: NeMo Evaluator SDK for enterprise LLM benchmarking
- 🔧 NVIDIA's evaluation platform with 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM)
- ⚡ Multi-backend execution: local Docker, Slurm HPC, Lepton cloud
- 📦 Container-first architecture for reproducible evaluation
- 📝 454 lines SKILL.md + 4 comprehensive reference files (~48KB documentation)
**December 2025 - v0.11.0 🔬 Mechanistic Interpretability**
- 🔬 **NEW CATEGORY**: Mechanistic Interpretability (4 skills)
- 🔍 TransformerLens skill: Neel Nanda's library for mech interp with HookPoints, activation caching, circuit analysis
- 🧠 SAELens skill: Sparse Autoencoder training and analysis for feature discovery, monosemanticity research
- ⚡ pyvene skill: Stanford's causal intervention library with declarative configs, DAS, activation patching
- 🌐 nnsight skill: Remote interpretability via NDIF, run experiments on 70B+ models without local GPUs
- 📝 ~6,500 new lines of documentation across 16 files
- **76 total skills** (filling the missing 04 category slot)
**November 25, 2025 - v0.10.0 🎉 70 Skills Complete!**
- 🎉 **ROADMAP COMPLETE**: Reached 70-skill milestone!
- 🚀 Added 4 skills: Lambda Labs, Segment Anything (SAM), BLIP-2, AudioCraft
- ☁️ Lambda Labs skill: Reserved/on-demand GPU cloud with H100/A100, persistent filesystems, 1-Click Clusters
- 🖼️ SAM skill: Meta's Segment Anything for zero-shot image segmentation with points/boxes/masks
- 👁️ BLIP-2 skill: Vision-language pretraining with Q-Former, image captioning, VQA
- 🎵 AudioCraft skill: Meta's MusicGen/AudioGen for text-to-music and text-to-sound generation
- 📝 ~10,000 new lines of documentation across 12 files
- **70 total skills** (100% roadmap complete!)
**November 25, 2025 - v0.9.0**
- 🚀 Added 2 infrastructure skills: Modal, SkyPilot
- ☁️ Modal skill: Serverless GPU cloud with Python-native API, T4-H200 on-demand, auto-scaling
- 🌐 SkyPilot skill: Multi-cloud orchestration across 20+ providers with spot recovery
- ✨ New Infrastructure category (2 skills - serverless GPU and multi-cloud orchestration)
- 📝 ~2,500 new lines of documentation across 6 files
- **66 total skills** (94% towards 70-skill target)
**November 25, 2025 - v0.8.0**
- 🚀 Added 5 high-priority skills: HQQ, GGUF, Phoenix, AutoGPT, Stable Diffusion
- ⚡ HQQ skill: Half-Quadratic Quantization without calibration data, multi-backend support
- 📦 GGUF skill: llama.cpp quantization format, K-quant methods, CPU/Metal inference
- 👁️ Phoenix skill: Open-source AI observability with OpenTelemetry tracing and LLM evaluation
- 🤖 AutoGPT skill: Autonomous AI agent platform with visual workflow builder
- 🎨 Stable Diffusion skill: Text-to-image generation via Diffusers, SDXL, ControlNet, LoRA
- 📝 ~9,000 new lines of documentation across 15 files
- **64 total skills** (91% towards 70-skill target)
**November 25, 2025 - v0.7.0**
- 🚀 Added 5 high-priority skills: PEFT, CrewAI, Qdrant, AWQ, LangSmith
- ✨ New Observability category with LangSmith for LLM tracing and evaluation
- 🎯 PEFT skill: Parameter-efficient fine-tuning with LoRA, QLoRA, DoRA, 25+ methods
- 🤖 CrewAI skill: Multi-agent orchestration with role-based collaboration
- 🔍 Qdrant skill: High-performance Rust vector search with hybrid filtering
- ⚡ AWQ skill: Activation-aware 4-bit quantization with minimal accuracy loss
- 📝 ~8,000 new lines of documentation across 15 files
- **59 total skills** (84% towards 70-skill target)
**November 15, 2025 - v0.6.0**
- 📊 Added 3 comprehensive MLOps skills: Weights & Biases, MLflow, TensorBoard
- ✨ New MLOps category (3 skills - experiment tracking, model registry, visualization)
- 📝 ~10,000 new lines of documentation across 13 files
- 🔧 Comprehensive coverage: experiment tracking, hyperparameter sweeps, model registry, profiling, embeddings visualization
- **54 total skills** (77% towards 70-skill target)
**November 12, 2025 - v0.5.0**
- 🎯 Added 4 comprehensive prompt engineering skills: DSPy, Instructor, Guidance, Outlines
- ✨ New Prompt Engineering category (4 skills - DSPy, Instructor, Guidance, Outlines)
- 📝 ~10,000 new lines of documentation across 16 files
- 🔧 Comprehensive coverage: declarative programming, structured outputs, constrained generation, FSM-based generation
- **47 total skills** (67% towards 70-skill target)
**November 9, 2025 - v0.4.0**
- 🤖 Added 11 comprehensive skills: LangChain, LlamaIndex, Chroma, FAISS, Sentence Transformers, Pinecone, CLIP, Whisper, LLaVA
- ✨ New Agents category (2 skills - LangChain, LlamaIndex)
- 🔍 New RAG category (4 skills - Chroma, FAISS, Sentence Transformers, Pinecone)
- 🎨 New Multimodal category (3 skills - CLIP, Whisper, LLaVA)
- 📝 ~15,000 new lines of documentation
- **43 total skills** (61% towards 70-skill target)
**November 8, 2025 - v0.3.0**
- 🚀 Added 8 comprehensive skills: TensorRT-LLM, llama.cpp, SGLang, GPTQ, HuggingFace Tokenizers, SentencePiece, Ray Data, NeMo Curator
- ⚡ Completed Inference & Serving category (4/4 skills)
- 🔤 New Tokenization category (2 skills)
- 📊 New Data Processing category (2 skills)
- 📝 9,617 new lines of documentation across 30 files
- **32 total skills** (45% towards 70-skill target)
**November 6, 2025 - v0.2.0**
- Added 10 skills from GitHub (Megatron-Core, Lightning, Ray Train, etc.)
- Improved skill structure with comprehensive references
- Created strategic roadmap to 70 skills
- Added contribution guidelines
**November 3, 2025 - v0.1.0**
- 🎉 Initial release with 5 fine-tuning skills
## Community
Join our community to stay updated, ask questions, and connect with other AI researchers:
- **[SkillEvolve Meta-Skill](https://github.com/Skill-Evolve/meta-skill)** - Connect your agent to the collective intelligence of the community. Captures techniques discovered during sessions and shares them back as curated skills.
- **[Slack Community](https://join.slack.com/t/orchestrarese-efu1990/shared_invite/zt-3iu6gr8io-zJvpkZTPToEviQ9KFZvNSg)** - Chat with the team and other users
- **[Twitter/X](https://x.com/orch_research)** - Follow for updates and announcements
- **[LinkedIn](https://www.linkedin.com/company/orchestra-research/)** - Connect professionally
## Star History
## 2. In-Tree Documentation Chapters (Orchestra-Research/AI-Research-SKILLs)
## File: README.md
# AI Research `Skills` Library
> **The most comprehensive open-source skills library enabling AI agents to autonomously conduct AI research — from idea to paper**
**View All 23 Categories**
---
## Table of Contents
- [Our Mission](#our-mission)
- [Path Towards AI Research Agent](#path-towards-ai-research-agent)
- [Available AI Research Engineering Skills](#available-ai-research-engineering-skills)
- [Demos](#demos)
- [Skill Structure](#skill-structure)
- [Roadmap](#roadmap)
- [Repository Structure](#repository-structure)
- [Use Cases](#use-cases)
- [Contributors](#contributors)
- [Citation](#citation)
- [Community](#community)
## Our Mission
We enable AI agents to **autonomously conduct AI research** — from literature survey and idea generation through experiment execution to paper writing. The library provides both the **research orchestration layer** (autoresearch, ideation, paper writing) and the **engineering skills** (training, evaluation, deployment) needed at each stage.
## Path Towards AI Research Agent
Modern AI research requires mastering dozens of specialized tools and frameworks.
AI Researchers spend more time debugging infrastructure than testing hypotheses — slowing the pace of scientific discovery.
We provide a comprehensive skills library that enables AI agents to autonomously conduct the full research lifecycle — from brainstorming ideas to writing the paper.
- Autonomous Research - The **autoresearch** skill orchestrates the entire research workflow using a two-loop architecture, routing to domain skills as needed
- Specialized Expertise - Each domain skill provides deep, production-ready knowledge of a specific framework (Megatron-LM, vLLM, TRL, etc.)
- End-to-End Coverage - 98 skills spanning the full AI research lifecycle, from ideation and literature survey to experiments and paper writing
- Research-Grade Quality - Documentation sourced from official repos, real GitHub issues, and battle-tested production workflows
## Available AI Research Engineering Skills
**Quality over quantity**: Each skill provides comprehensive, expert-level guidance with real code examples, troubleshooting guides, and production-ready workflows.
### 📦 Quick Install (Recommended)
**For humans** — interactive installer with one command:
```bash
npx @orchestra-research/ai-research-skills
```
**For AI agents** — point your agent to the welcome doc and it handles the rest:
```
Read https://www.orchestra-research.com/ai-research-skills/welcome.md and follow the instructions to install and use AI Research Skills.
```
This installs all 98 skills, loads the **autoresearch** orchestration layer, and starts autonomous research.
**What the installer does**
- **Auto-detects** your installed coding agents (Claude Code, Hermes Agent, OpenCode, Qoder, Cursor, Gemini CLI, etc.)
- **Installs** skills to `~/.orchestra/skills/` with symlinks to each agent (falls back to copy on Windows)
- **Offers** everything, quickstart bundle, by category, or individual skills
- **Updates** installed skills with latest versions
- **Uninstalls** all or selected skills
**CLI Commands**
```bash
# Interactive installer (recommended)
npx @orchestra-research/ai-research-skills
# Direct commands
npx @orchestra-research/ai-research-skills list # View installed skills
npx @orchestra-research/ai-research-skills update # Update installed skills
```
**Claude Code Marketplace (Alternative)**
Install skill categories directly using the **Claude Code CLI**:
```bash
# Add the marketplace
/plugin marketplace add orchestra-research/AI-research-SKILLs
# Install by category (23 categories available)
/plugin install fine-tuning@ai-research-skills # Axolotl, LLaMA-Factory, PEFT, Unsloth
/plugin install post-training@ai-research-skills # TRL, GRPO, OpenRLHF, SimPO, verl, slime, miles, torchforge
/plugin install inference-serving@ai-research-skills # vLLM, TensorRT-LLM, llama.cpp, SGLang
/plugin install distributed-training@ai-research-skills
/plugin install optimization@ai-research-skills
```
### All 23 Categories (98 Skills)
| Category | Skills | Included |
|----------|--------|----------|
| **Autoresearch** | **1** | **Autonomous research orchestration — central layer that manages the full lifecycle and routes to all other skills** |
| Ideation | 2 | Research Brainstorming, Creative Thinking |
| ML Paper Writing | 2 | ML Paper Writing (LaTeX templates, citation verification), Academic Plotting |
| Model Architecture | 5 | LitGPT, Mamba, NanoGPT, RWKV, TorchTitan |
| Tokenization | 2 | HuggingFace Tokenizers, SentencePiece |
| Fine-Tuning | 4 | Axolotl, LLaMA-Factory, PEFT, Unsloth |
| Mech Interp | 4 | TransformerLens, SAELens, pyvene, nnsight |
| Data Processing | 2 | NeMo Curator, Ray Data |
| Post-Training | 8 | TRL, GRPO, OpenRLHF, SimPO, verl, slime, miles, torchforge |
| Safety | 4 | Constitutional AI, LlamaGuard, NeMo Guardrails, Prompt Guard |
| Distributed | 6 | DeepSpeed, FSDP, Accelerate, Megatron-Core, Lightning, Ray Train |
| Infrastructure | 3 | Modal, Lambda Labs, SkyPilot |
| Optimization | 6 | Flash Attention, bitsandbytes, GPTQ, AWQ, HQQ, GGUF |
| Evaluation | 3 | lm-eval-harness, BigCode, NeMo Evaluator |
| Inference | 4 | vLLM, TensorRT-LLM, llama.cpp, SGLang |
| MLOps | 3 | W&B, MLflow, TensorBoard |
| Agents | 4 | LangChain, LlamaIndex, CrewAI, AutoGPT |
| RAG | 5 | Chroma, FAISS, Pinecone, Qdrant, Sentence Transformers |
| Prompt Eng | 4 | DSPy, Instructor, Guidance, Outlines |
| Observability | 2 | LangSmith, Phoenix |
| Multimodal | 7 | CLIP, Whisper, LLaVA, BLIP-2, SAM, Stable Diffusion, AudioCraft |
| Emerging | 6 | MoE, Model Merging, Long Context, Speculative Decoding, Distillation, Pruning |
| Agent-Native Research Artifact | 3 | ARA Compiler, Research Manager, Rigor Reviewer |
**View All 98 Skills in Details**
### 🔬 Autoresearch (1 skill) — Central Orchestration Layer
- **[Autoresearch](0-autoresearch-skill/)** - Autonomous research orchestration using a two-loop architecture (inner optimization + outer synthesis). Manages the full lifecycle from literature survey to paper writing, routing to all domain-specific skills. Supports Claude Code /loop and OpenClaw heartbeat for continuous operation (390 lines + 3 refs)
### 🏗️ Model Architecture (5 skills)
- **[LitGPT](01-model-architecture/litgpt/)** - Lightning AI's 20+ clean LLM implementations with production training recipes (462 lines + 4 refs)
- **[Mamba](01-model-architecture/mamba/)** - State-space models with O(n) complexity, 5× faster than Transformers (253 lines + 3 refs)
- **[RWKV](01-model-architecture/rwkv/)** - RNN+Transformer hybrid, infinite context, Linux Foundation project (253 lines + 3 refs)
- **[NanoGPT](01-model-architecture/nanogpt/)** - Educational GPT in ~300 lines by Karpathy (283 lines + 3 refs)
- **[TorchTitan](01-model-architecture/torchtitan/)** - PyTorch-native distributed training for Llama 3.1 with 4D parallelism
### 🔤 Tokenization (2 skills)
- **[HuggingFace Tokenizers](02-tokenization/huggingface-tokenizers/)** - Rust-based, <20s/GB, BPE/WordPiece/Unigram algorithms (486 lines + 4 refs)
- **[SentencePiece](02-tokenization/sentencepiece/)** - Language-independent, 50k sentences/sec, used by T5/ALBERT (228 lines + 2 refs)
### 🎯 Fine-Tuning (4 skills)
- **[Axolotl](03-fine-tuning/axolotl/)** - YAML-based fine-tuning with 100+ models (156 lines + 4 refs)
- **[LLaMA-Factory](03-fine-tuning/llama-factory/)** - WebUI no-code fine-tuning (78 lines + 5 refs)
- **[Unsloth](03-fine-tuning/unsloth/)** - 2x faster QLoRA fine-tuning (75 lines + 4 refs)
- **[PEFT](03-fine-tuning/peft/)** - Parameter-efficient fine-tuning with LoRA, QLoRA, DoRA, 25+ methods (431 lines + 2 refs)
### 🔬 Mechanistic Interpretability (4 skills)
- **[TransformerLens](04-mechanistic-interpretability/transformer-lens/)** - Neel Nanda's library for mech interp with HookPoints, activation caching (346 lines + 3 refs)
- **[SAELens](04-mechanistic-interpretability/saelens/)** - Sparse Autoencoder training and analysis for feature discovery (386 lines + 3 refs)
- **[pyvene](04-mechanistic-interpretability/pyvene/)** - Stanford's causal intervention library with declarative configs (473 lines + 3 refs)
- **[nnsight](04-mechanistic-interpretability/nnsight/)** - Remote interpretability via NDIF, run experiments on 70B+ models (436 lines + 3 refs)
### 📊 Data Processing (2 skills)
- **[Ray Data](05-data-processing/ray-data/)** - Distributed ML data processing, streaming execution, GPU support (318 lines + 2 refs)
- **[NeMo Curator](05-data-processing/nemo-curator/)** - GPU-accelerated data curation, 16× faster deduplication (375 lines + 2 refs)
### 🎓 Post-Training (8 skills)
- **[TRL Fine-Tuning](06-post-training/trl-fine-tuning/)** - Transformer Reinforcement Learning (447 lines + 4 refs)
- **[GRPO-RL-Training](06-post-training/grpo-rl-training/)** (TRL) - Group Relative Policy Optimization with TRL (569 lines, **gold standard**)
- **[OpenRLHF](06-post-training/openrlhf/)** - Full RLHF pipeline with Ray + vLLM (241 lines + 4 refs)
- **[SimPO](06-post-training/simpo/)** - Simple Preference Optimization, no reference model needed (211 lines + 3 refs)
- **[verl](06-post-training/verl/)** - ByteDance's HybridFlow RL framework, FSDP/Megatron + vLLM/SGLang backends (389 lines + 2 refs)
- **[slime](06-post-training/slime/)** - THUDM's Megatron+SGLang framework powering GLM-4.x models (464 lines + 2 refs)
- **[miles](06-post-training/miles/)** - Enterprise fork of slime with FP8, INT4, speculative RL for MoE training (315 lines + 2 refs)
- **[torchforge](06-post-training/torchforge/)** - Meta's PyTorch-native RL with Monarch+TorchTitan+vLLM (380 lines + 2 refs)
### 🛡️ Safety & Alignment (4 skills)
- **[Constitutional AI](07-safety-alignment/constitutional-ai/)** - AI-driven self-improvement via principles (282 lines)
- **[LlamaGuard](07-safety-alignment/llamaguard/)** - Safety classifier for LLM inputs/outputs (329 lines)
- **[NeMo Guardrails](07-safety-alignment/nemo-guardrails/)** - Programmable guardrails with Colang (289 lines)
- **[Prompt Guard](07-safety-alignment/prompt-guard/)** - Meta's 86M prompt injection & jailbreak detector, 99%+ TPR, <2ms GPU (313 lines)
### ⚡ Distributed Training (6 skills)
- **[Megatron-Core](08-distributed-training/megatron-core/)** - NVIDIA's framework for training 2B-462B param models with 47% MFU on H100 (359 lines + 4 refs)
- **[DeepSpeed](08-distributed-training/deepspeed/)** - Microsoft's ZeRO optimization (137 lines + 9 refs)
- **[PyTorch FSDP2](08-distributed-training/pytorch-fsdp2/)** - Fully Sharded Data Parallel v2 with `fully_shard` and DTensor (231 lines + 12 refs)
- **[Accelerate](08-distributed-training/accelerate/)** - HuggingFace's 4-line distributed training API (324 lines + 3 refs)
- **[PyTorch Lightning](08-distributed-training/pytorch-lightning/)** - High-level training framework with Trainer class (339 lines + 3 refs)
- **[Ray Train](08-distributed-training/ray-train/)** - Multi-node orchestration and hyperparameter tuning (399 lines + 1 ref)
### 🚀 Optimization (6 skills)
- **[Flash Attention](10-optimization/flash-attention/)** - 2-4x faster attention with memory efficiency (359 lines + 2 refs)
- **[bitsandbytes](10-optimization/bitsandbytes/)** - 8-bit/4-bit quantization for 50-75% memory reduction (403 lines + 3 refs)
- **[GPTQ](10-optimization/gptq/)** - 4-bit post-training quantization, 4× memory reduction, <2% accuracy loss (443 lines + 3 refs)
- **[AWQ](10-optimization/awq/)** - Activation-aware weight quantization, 4-bit with minimal accuracy loss (310 lines + 2 refs)
- **[HQQ](10-optimization/hqq/)** - Half-Quadratic Quantization, no calibration data needed, multi-backend (370 lines + 2 refs)
- **[GGUF](10-optimization/gguf/)** - llama.cpp quantization format, K-quant methods, CPU/Metal inference (380 lines + 2 refs)
### 📊 Evaluation (3 skills)
- **[lm-evaluation-harness](11-evaluation/lm-evaluation-harness/)** - EleutherAI's standard for benchmarking LLMs across 60+ tasks (482 lines + 4 refs)
- **[BigCode Evaluation Harness](11-evaluation/bigcode-evaluation-harness/)** - Code model benchmarking with HumanEval, MBPP, MultiPL-E, pass@k metrics (406 lines + 3 refs)
- **[NeMo Evaluator](11-evaluation/nemo-evaluator/)** - NVIDIA's enterprise platform for 100+ benchmarks across 18+ harnesses with multi-backend execution (454 lines + 4 refs)
### ☁️ Infrastructure (3 skills)
- **[Modal](09-infrastructure/modal/)** - Serverless GPU cloud with Python-native API, T4-H200 on-demand (342 lines + 2 refs)
- **[SkyPilot](09-infrastructure/skypilot/)** - Multi-cloud orchestration across 20+ providers with spot recovery (390 lines + 2 refs)
- **[Lambda Labs](09-infrastructure/lambda-labs/)** - Reserved/on-demand GPU cloud with H100/A100, persistent filesystems (390 lines + 2 refs)
### 🔥 Inference & Serving (4 skills)
- **[vLLM](12-inference-serving/vllm/)** - High-throughput LLM serving with PagedAttention (356 lines + 4 refs, **production-ready**)
- **[TensorRT-LLM](12-inference-serving/tensorrt-llm/)** - NVIDIA's fastest inference, 24k tok/s, FP8/INT4 quantization (180 lines + 3 refs)
- **[llama.cpp](12-inference-serving/llama-cpp/)** - CPU/Apple Silicon inference, GGUF quantization (251 lines + 3 refs)
- **[SGLang](12-inference-serving/sglang/)** - Structured generation with RadixAttention, 5-10× faster for agents (435 lines + 3 refs)
### 🤖 Agents (4 skills)
- **[LangChain](14-agents/langchain/)** - Most popular agent framework, 500+ integrations, ReAct pattern (658 lines + 3 refs, **production-ready**)
- **[LlamaIndex](14-agents/llamaindex/)** - Data framework for LLM apps, 300+ connectors, RAG-focused (535 lines + 3 refs)
- **[CrewAI](14-agents/crewai/)** - Multi-agent orchestration, role-based collaboration, autonomous workflows (498 lines + 3 refs)
- **[AutoGPT](14-agents/autogpt/)** - Autonomous AI agent platform, visual workflow builder, continuous execution (400 lines + 2 refs)
### 🔍 RAG (5 skills)
- **[Chroma](15-rag/chroma/)** - Open-source embedding database, local/cloud, 24k stars (385 lines + 1 ref)
- **[FAISS](15-rag/faiss/)** - Facebook's similarity search, billion-scale, GPU acceleration (295 lines)
- **[Sentence Transformers](15-rag/sentence-transformers/)** - 5000+ embedding models, multilingual, 15k stars (370 lines)
- **[Pinecone](15-rag/pinecone/)** - Managed vector database, auto-scaling, <100ms latency (410 lines)
- **[Qdrant](15-rag/qdrant/)** - High-performance vector search, Rust-powered, hybrid search with filtering (493 lines + 2 refs)
### 🎨 Multimodal (7 skills)
- **[CLIP](18-multimodal/clip/)** - OpenAI's vision-language model, zero-shot classification, 25k stars (320 lines)
- **[Whisper](18-multimodal/whisper/)** - Robust speech recognition, 99 languages, 73k stars (395 lines)
- **[LLaVA](18-multimodal/llava/)** - Vision-language assistant, image chat, GPT-4V level (360 lines)
- **[Stable Diffusion](18-multimodal/stable-diffusion/)** - Text-to-image generation via HuggingFace Diffusers, SDXL, ControlNet (380 lines + 2 refs)
- **[Segment Anything](18-multimodal/segment-anything/)** - Meta's SAM for zero-shot image segmentation with points/boxes (500 lines + 2 refs)
- **[BLIP-2](18-multimodal/blip-2/)** - Vision-language pretraining with Q-Former, image captioning, VQA (500 lines + 2 refs)
- **[AudioCraft](18-multimodal/audiocraft/)** - Meta's MusicGen/AudioGen for text-to-music and text-to-sound (470 lines + 2 refs)
### 🎯 Prompt Engineering (4 skills)
- **[DSPy](16-prompt-engineering/dspy/)** - Declarative prompt programming with optimizers, Stanford NLP, 22k stars (438 lines + 3 refs)
- **[Instructor](16-prompt-engineering/instructor/)** - Structured LLM outputs with Pydantic validation, 15k stars (726 lines + 3 refs)
- **[Guidance](16-prompt-engineering/guidance/)** - Constrained generation with regex/grammars, Microsoft Research, 18k stars (485 lines + 3 refs)
- **[Outlines](16-prompt-engineering/outlines/)** - Structured text with FSM, zero-overhead, 8k stars (601 lines + 3 refs)
### 📊 MLOps (3 skills)
- **[Weights & Biases](13-mlops/weights-and-biases/)** - Experiment tracking, sweeps, artifacts, model registry (427 lines + 3 refs)
- **[MLflow](13-mlops/mlflow/)** - Model registry, tracking, deployment, autologging (514 lines + 3 refs)
- **[TensorBoard](13-mlops/tensorboard/)** - Visualization, profiling, embeddings, scalars/images (538 lines + 3 refs)
### 👁️ Observability (2 skills)
- **[LangSmith](17-observability/langsmith/)** - LLM observability, tracing, evaluation, monitoring for AI apps (422 lines + 2 refs)
- **[Phoenix](17-observability/phoenix/)** - Open-source AI observability with OpenTelemetry tracing and LLM evaluation (380 lines + 2 refs)
### 🔬 Emerging Techniques (6 skills)
- **[MoE Training](19-emerging-techniques/moe-training/)** - Mixture of Experts training with DeepSpeed, Mixtral 8x7B, 5× cost reduction (515 lines + 3 refs)
- **[Model Merging](19-emerging-techniques/model-merging/)** - Combine models with TIES, DARE, SLERP using mergekit (528 lines + 3 refs)
- **[Long Context](19-emerging-techniques/long-context/)** - Extend context windows with RoPE, YaRN, ALiBi, 32k-128k tokens (624 lines + 3 refs)
- **[Speculative Decoding](19-emerging-techniques/speculative-decoding/)** - 1.5-3.6× faster inference with Medusa, Lookahead (379 lines)
- **[Knowledge Distillation](19-emerging-techniques/knowledge-distillation/)** - Compress models 70B→7B with MiniLLM, temperature scaling (424 lines)
- **[Model Pruning](19-emerging-techniques/model-pruning/)** - 50% sparsity with Wanda, SparseGPT, <1% accuracy loss (417 lines)
### 📝 ML Paper Writing (2 skills)
- **[ML Paper Writing](20-ml-paper-writing/)** - Write publication-ready papers for NeurIPS, ICML, ICLR, ACL, AAAI, COLM with LaTeX templates, citation verification, and writing best practices (532 lines + 5 refs)
- **[Academic Plotting](20-ml-paper-writing/academic-plotting/)** - Generate publication-quality figures for ML papers: architecture diagrams via Gemini AI and data-driven charts via matplotlib/seaborn with venue-specific styling (479 lines + 3 refs)
### 💡 Ideation (2 skills)
- **[Research Brainstorming](21-research-ideation/brainstorming-research-ideas/)** - Structured ideation frameworks for discovering high-impact research directions with 10 complementary lenses (384 lines)
- **[Creative Thinking](21-research-ideation/creative-thinking-for-research/)** - Cognitive science frameworks (bisociation, structure-mapping, constraint manipulation) for genuinely novel research ideas (366 lines)
### 🧬 Agent-Native Research Artifact (3 skills)
- **[ARA Compiler](22-agent-native-research-artifact/compiler/)** - Compiles any research input (PDF papers, repos, experiment logs, raw notes) into a complete Agent-Native Research Artifact with claims, exploration graph, evidence, and code stubs (245 lines + 3 refs)
- **[ARA Research Manager](22-agent-native-research-artifact/research-manager/)** - Post-task research recorder that runs at session end to extract decisions, experiments, dead ends, and pivots from conversation history into the `ara/` directory with user-vs-AI provenance tags (324 lines + 3 refs)
- **[ARA Rigor Reviewer](22-agent-native-research-artifact/rigor-reviewer/)** - ARA Seal Level 2 semantic epistemic review scoring six dimensions of research rigor (evidence relevance, falsifiability, scope, coherence, exploration integrity, methodology) with severity-ranked findings (322 lines + 1 ref)
## Demos
All 98 skills in this repo are automatically synced to [Orchestra Research](https://www.orchestra-research.com/research-skills), where you can add them to your projects with one click and use them with AI research agents.
**See skills in action → [demos/](demos/README.md)**
We maintain a curated collection of demo repositories showing how to use skills for real AI research tasks:
| Demo | Skills Used | What It Does |
|------|-------------|--------------|
| **[Norm Heterogeneity → LoRA Brittleness](demos/autoresearch-norm-heterogeneity/)** | Autoresearch, ML Paper Writing, Ideation | Agent autonomously discovered norm heterogeneity predicts fine-tuning difficulty (r=-0.99), pivoting from a null result on ETF overlaps |
| **[RL Algorithm Brain Scan](demos/autoresearch-rl-brain-scan/)** | Autoresearch, GRPO, TRL, SAELens, TransformerLens, ML Paper Writing | Agent found DPO is a rank-1 perturbation (95.6% recovery from one SVD direction) while online RL is distributed and structure-preserving |
| **[NeMo Eval: GPQA Benchmark](https://github.com/zechenzhangAGI/Nemo-Eval-Skill-Demo)** | NeMo Evaluator | Compare Llama 8B/70B/405B on graduate-level science questions |
| **[LoRA Without Regret Reproduction](https://www.orchestra-research.com/perspectives/LLM-with-Orchestra)** | GRPO, TRL | Reproduce SFT + GRPO RL experiments via prompting |
| **[Layer-Wise Quantization Experiment](https://github.com/AmberLJC/llama-quantization-experiment)** | llama.cpp, GGUF | Investigate optimal layer precision allocation—early layers at Q8 achieve 1.9× compression with 1.3% perplexity loss |
| **[Cross-Lingual Alignment Analysis](https://github.com/AmberLJC/faiss-demo)** | FAISS | Quantify how well multilingual embeddings align semantic concepts across 8 languages using FAISS similarity search |
| **[Scientific Plotting Demo](demos/scientific-plotting-demo/)** | Academic Plotting | Generate publication-quality figures for the Andes QoE-aware LLM serving paper — Gemini AI architecture diagrams + matplotlib data charts (CDF, multi-panel grids, bar charts) |
**Featured Demos**: Two papers produced entirely by AI agents using the **autoresearch** skill. The [Norm Heterogeneity paper](demos/autoresearch-norm-heterogeneity/) demonstrates autonomous research pivoting — the agent refuted its own hypothesis and discovered a stronger finding. The [RL Brain Scan paper](demos/autoresearch-rl-brain-scan/) demonstrates multi-skill orchestration — the agent trained RL models, analyzed internals with interpretability tools, and synthesized the insight that "DPO is rank-1 alignment." Both papers written end-to-end by the agent.
## Skill Structure
Each skill follows a battle-tested format for maximum usefulness:
```
skill-name/
├── SKILL.md # Quick reference (50-150 lines)
│ ├── Metadata (name, description, version)
│ ├── When to use this skill
│ ├── Quick patterns & examples
│ └── Links to references
│
├── references/ # Deep documentation (300KB+)
│ ├── README.md # From GitHub/official docs
│ ├── api.md # API reference
│ ├── tutorials.md # Step-by-step guides
│ ├── issues.md # Real GitHub issues & solutions
│ ├── releases.md # Version history & breaking changes
│ └── file_structure.md # Codebase navigation
│
├── scripts/ # Helper scripts (optional)
└── assets/ # Templates & examples (optional)
```
**Quality Standards**
- 300KB+ documentation from official sources
- Real GitHub issues & solutions (when available)
- Code examples with language detection
- Version history & breaking changes
- Links to official docs
## Roadmap
The library spans 98 comprehensive skills across the full AI research lifecycle. See our [detailed roadmap](docs/ROADMAP.md) for the complete development plan.
[View Full Roadmap →](docs/ROADMAP.md)
**View Detailed Statistics**
| Metric | Current | Target |
|--------|---------|--------|
| **Skills** | **87** (high-quality, standardized YAML) | 80 ✅ |
| **Avg Lines/Skill** | **420 lines** (focused + progressive disclosure) | 200-600 lines |
| **Documentation** | **~130,000 lines** total (SKILL.md + references) | 100,000+ lines |
| **Gold Standard Skills** | **65** with comprehensive references | 50+ |
| **Contributors** | 1 | 100+ |
| **Coverage** | Architecture, Tokenization, Fine-Tuning, Mechanistic Interpretability, Data Processing, Post-Training, Safety, Distributed, Optimization, Evaluation, Infrastructure, Inference, Agents, RAG, Multimodal, Prompt Engineering, MLOps, Observability, ML Paper Writing, Ideation, Autoresearch | Full Lifecycle ✅ |
**Recent Progress**: npm package `@orchestra-research/ai-research-skills` for one-command installation across all coding agents
**Philosophy**: Quality > Quantity. Following [Anthropic official best practices](anthropic_official_docs/best_practices.md) - each skill provides 200-500 lines of focused, actionable guidance with progressive disclosure.
## Repository Structure
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Use Cases
### For Researchers
"I need to fine-tune Llama 3 with custom data"
→ **03-fine-tuning/axolotl/** - YAML configs, 100+ model support
### For ML Engineers
"How do I optimize inference latency?"
→ **12-inference-serving/vllm/** - PagedAttention, batching
### For Students
"I want to learn how transformers work"
→ **01-model-architecture/litgpt/** - Clean implementations
### For Teams
"We need to scale training to 100 GPUs"
→ **08-distributed-training/deepspeed/** - ZeRO stages, 3D parallelism
## License
MIT License - See [LICENSE](LICENSE) for details.
**Note**: Individual skills may reference libraries with different licenses. Please check each project's license before use.
## Citation
If you use AI Research Skills in your work or find it helpful for a publication, we'd appreciate a citation:
**BibTeX**
```bibtex
@software{ai_research_skills,
title = {AI Research Skills Library},
author = {{Orchestra Research}},
year = {2025},
url = {https://github.com/orchestra-research/AI-research-SKILLs},
note = {Open-source skills library enabling AI agents to autonomously conduct AI research}
}
```
**APA**
> Orchestra Research. (2025). *AI Research Skills Library* [Computer software]. https://github.com/orchestra-research/AI-research-SKILLs
**Chicago**
> Orchestra Research. "AI Research Skills Library." GitHub, 2025. https://github.com/orchestra-research/AI-research-SKILLs.
**IEEE**
> Orchestra Research, "AI Research Skills Library," 2025. [Online]. Available: https://github.com/orchestra-research/AI-research-SKILLs
> **Tip**: You can also click **"Cite this repository"** in the GitHub sidebar for auto-formatted citations.
## Acknowledgments
Built with:
- **[Claude Code](https://www.claude.com/product/claude-code)** - AI pair programming
- **[Skill Seeker](https://github.com/yusufkaraaslan/Skill_Seekers)** - Automated doc scraping
- **Open Source AI Community** - For amazing tools and docs
Special thanks to:
- EleutherAI, HuggingFace, NVIDIA, Lightning AI, Meta AI, Anthropic
- All researchers who maintain excellent documentation
## Contributors
Thanks to all the people who have contributed to the AI Research Skills Library:
We welcome contributions from the AI research community! See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines on:
- Adding new skills
- Improving existing skills
- Quality standards and best practices
- Submission process
## Recent Updates
**June 2026 - v1.7.1 🧹 Inventory Consistency, Drift Guard & Security Hardening**
- 📊 **Repo-wide inventory reconciled to 98 skills / 23 categories** — corrected stale counts that had drifted apart across files: CLAUDE.md (said 90), CONTRIBUTING.md (said 86/22), the README sync line (said 87), WELCOME.md and the npm package README (said 86/22). Also fixed wrong per-category listings (TorchTitan, SwanLab, A-Evolve, ML Training Recipes, Cosmos Policy/OpenPI/OpenVLA-OFT, the paper-writing skills)
- 🛡️ **New CI drift guard** — `scripts/check-inventory.sh` + `check-inventory.yml` fail CI whenever the documented skill/category counts diverge from the actual `SKILL.md` count on disk, so the inventory can't silently drift again
- 📦 **Marketplace sync hardening** — `sync-skills.yml` now prunes build artifacts (`node_modules`/`__pycache__`/`*.pyc`/`.ipynb_checkpoints`) before zipping and fails loudly above 190 files instead of hitting the marketplace's 200-file rejection
- 🔒 **Security** — pinned CLI dependencies (`chalk`/`inquirer`/`ora`) to exact, patched versions with a regenerated lockfile (`inquirer@9.3.8` clears the `tmp` path-traversal advisory; `npm audit` now reports 0 vulnerabilities)
- 🧹 Full open PR/issue triage pass against scope + contribution standards
**April 2026 - v1.6.0 🧬 Agent-Native Research Artifact (ARA) — 23rd Category, 98 Skills**
- 🧬 **NEW CATEGORY**: `22-agent-native-research-artifact/` (the 23rd category) — three skills that turn research outputs into a falsifiable, agent-traversable artifact:
- 🛠️ **[ARA Compiler](22-agent-native-research-artifact/compiler/)** — compiles any input (PDF papers, GitHub repos, experiment logs, raw notes) into a structured ARA with cognitive layer (claims, concepts, heuristics), physical layer (configs, code stubs), exploration graph (research DAG), and grounded evidence
- 📋 **[ARA Research Manager](22-agent-native-research-artifact/research-manager/)** — post-task epilogue that scans conversation history at session end and writes decisions, experiments, dead ends, claims, heuristics, and pivots into the `ara/` directory with `user` / `ai-suggested` / `ai-executed` / `user-revised` provenance tags
- 🔍 **[ARA Rigor Reviewer](22-agent-native-research-artifact/rigor-reviewer/)** — Seal Level 2 semantic epistemic review scoring six dimensions of research rigor (evidence relevance, falsifiability, scope calibration, argument coherence, exploration integrity, methodological rigor) and emitting a severity-ranked report with a Strong Accept-to-Reject recommendation
- 🔗 Sourced from the [Agent-Native-Research-Artifact-Init](https://github.com/Orchestra-Research/Agent-Native-Research-Artifact-Init) reference repo, restructured to AI-research-SKILLs standards (kebab-case names, third-person descriptions, Title-Case tags, one-level-deep references)
- 🧩 Plugin entry `agent-native-research-artifact` added to `.claude-plugin/marketplace.json`; CLI category registered as `22-agent-native-research-artifact` with three individual skill entries in the npm installer
- 🔄 Auto-syncs to Orchestra marketplace via `sync-skills.yml` on push; npm package republished as `@orchestra-research/ai-research-skills@1.6.0` via `publish-npm.yml` on version bump
- 📊 **98 total skills** across **23 categories** — full lifecycle from idea → paper → falsifiable, auditable artifact
**March 2026 - v1.4.0 🔬 Autoresearch & 86 Skills — Full Research Lifecycle**
- 🔬 **NEW SKILL**: **Autoresearch** — autonomous research orchestration using a two-loop architecture (inner optimization loop + outer synthesis loop)
- 🧠 Manages the full research lifecycle: literature survey → ideation → experiments → synthesis → paper writing
- 🔄 Routes to all 86 domain skills automatically — agents don't need to know which skill to use
- ⏰ Mandatory `/loop` (Claude Code) and cron job (OpenClaw) for continuous autonomous operation
- 📊 Generates research presentations (HTML/PDF) with optimization trajectory plots for human review
- 📝 Findings.md as persistent project memory across sessions with "Lessons and Constraints" tracking
- 🗂️ Structured workspace: research-state.yaml, findings.md, research-log.md, literature/, experiments/, src/, data/, to_human/
- 📄 **Two demo papers produced by autoresearch**: [Norm Heterogeneity → LoRA Brittleness](demos/autoresearch-norm-heterogeneity/) and [RL Algorithm Brain Scan](demos/autoresearch-rl-brain-scan/)
- 🚀 WELCOME.md for cold-start agent bootstrap — one URL to go from zero to autonomous research
- 📦 npm v1.4.x with Windows symlink fallback, all 22 categories installable
- 🤖 **Supported agents**: Claude Code, Hermes Agent, OpenCode, OpenClaw, Qoder, Cursor, Codex, Gemini CLI, Qwen Code
- 📊 **87 total skills** across **22 categories** — complete research lifecycle coverage
**February 2026 - v0.15.0 🛡️ Prompt Guard & 83 Skills**
- 🛡️ **NEW SKILL**: Prompt Guard - Meta's 86M prompt injection & jailbreak detector
- ⚡ 99%+ TPR, <1% FPR, <2ms GPU latency, multilingual (8 languages)
- 🔒 3 workflows: user input filtering, third-party data filtering, batch RAG processing
- 📊 **83 total skills** across 20 categories
**January 2026 - v0.14.0 📦 npm Package & 82 Skills**
- 📦 **NEW**: `npx @orchestra-research/ai-research-skills` - One-command installation for all coding agents
- 🤖 **Supported agents**: Claude Code, OpenCode, Cursor, Codex, Gemini CLI, Qwen Code
- ✨ Interactive installer with category/individual skill selection
- 🔄 Update installed skills, selective uninstall
- 📊 **82 total skills** (5 new post-training skills: verl, slime, miles, torchforge + TorchTitan)
- 🏗️ Megatron-Core moved to Distributed Training category
**January 2026 - v0.13.0 📝 ML Paper Writing & Demos Gallery**
- 📝 **NEW CATEGORY**: ML Paper Writing (20th category, 77th skill)
- 🎯 Write publication-ready papers for NeurIPS, ICML, ICLR, ACL, AAAI, COLM
- 📚 Writing philosophy from top researchers (Neel Nanda, Farquhar, Gopen & Swan, Lipton, Perez)
- 🔬 Citation verification workflow - never hallucinate references
- 📄 LaTeX templates for 6 major conferences
- 🎪 **NEW**: Curated demos gallery (`demos/`) showcasing skills in action
- 🔗 Demo repos: NeMo Evaluator benchmark, LoRA Without Regret reproduction
- 📖 936-line comprehensive SKILL.md with 4 workflows
**January 2026 - v0.12.0 📊 NeMo Evaluator SDK**
- 📊 **NEW SKILL**: NeMo Evaluator SDK for enterprise LLM benchmarking
- 🔧 NVIDIA's evaluation platform with 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM)
- ⚡ Multi-backend execution: local Docker, Slurm HPC, Lepton cloud
- 📦 Container-first architecture for reproducible evaluation
- 📝 454 lines SKILL.md + 4 comprehensive reference files (~48KB documentation)
**December 2025 - v0.11.0 🔬 Mechanistic Interpretability**
- 🔬 **NEW CATEGORY**: Mechanistic Interpretability (4 skills)
- 🔍 TransformerLens skill: Neel Nanda's library for mech interp with HookPoints, activation caching, circuit analysis
- 🧠 SAELens skill: Sparse Autoencoder training and analysis for feature discovery, monosemanticity research
- ⚡ pyvene skill: Stanford's causal intervention library with declarative configs, DAS, activation patching
- 🌐 nnsight skill: Remote interpretability via NDIF, run experiments on 70B+ models without local GPUs
- 📝 ~6,500 new lines of documentation across 16 files
- **76 total skills** (filling the missing 04 category slot)
**November 25, 2025 - v0.10.0 🎉 70 Skills Complete!**
- 🎉 **ROADMAP COMPLETE**: Reached 70-skill milestone!
- 🚀 Added 4 skills: Lambda Labs, Segment Anything (SAM), BLIP-2, AudioCraft
- ☁️ Lambda Labs skill: Reserved/on-demand GPU cloud with H100/A100, persistent filesystems, 1-Click Clusters
- 🖼️ SAM skill: Meta's Segment Anything for zero-shot image segmentation with points/boxes/masks
- 👁️ BLIP-2 skill: Vision-language pretraining with Q-Former, image captioning, VQA
- 🎵 AudioCraft skill: Meta's MusicGen/AudioGen for text-to-music and text-to-sound generation
- 📝 ~10,000 new lines of documentation across 12 files
- **70 total skills** (100% roadmap complete!)
**November 25, 2025 - v0.9.0**
- 🚀 Added 2 infrastructure skills: Modal, SkyPilot
- ☁️ Modal skill: Serverless GPU cloud with Python-native API, T4-H200 on-demand, auto-scaling
- 🌐 SkyPilot skill: Multi-cloud orchestration across 20+ providers with spot recovery
- ✨ New Infrastructure category (2 skills - serverless GPU and multi-cloud orchestration)
- 📝 ~2,500 new lines of documentation across 6 files
- **66 total skills** (94% towards 70-skill target)
**November 25, 2025 - v0.8.0**
- 🚀 Added 5 high-priority skills: HQQ, GGUF, Phoenix, AutoGPT, Stable Diffusion
- ⚡ HQQ skill: Half-Quadratic Quantization without calibration data, multi-backend support
- 📦 GGUF skill: llama.cpp quantization format, K-quant methods, CPU/Metal inference
- 👁️ Phoenix skill: Open-source AI observability with OpenTelemetry tracing and LLM evaluation
- 🤖 AutoGPT skill: Autonomous AI agent platform with visual workflow builder
- 🎨 Stable Diffusion skill: Text-to-image generation via Diffusers, SDXL, ControlNet, LoRA
- 📝 ~9,000 new lines of documentation across 15 files
- **64 total skills** (91% towards 70-skill target)
**November 25, 2025 - v0.7.0**
- 🚀 Added 5 high-priority skills: PEFT, CrewAI, Qdrant, AWQ, LangSmith
- ✨ New Observability category with LangSmith for LLM tracing and evaluation
- 🎯 PEFT skill: Parameter-efficient fine-tuning with LoRA, QLoRA, DoRA, 25+ methods
- 🤖 CrewAI skill: Multi-agent orchestration with role-based collaboration
- 🔍 Qdrant skill: High-performance Rust vector search with hybrid filtering
- ⚡ AWQ skill: Activation-aware 4-bit quantization with minimal accuracy loss
- 📝 ~8,000 new lines of documentation across 15 files
- **59 total skills** (84% towards 70-skill target)
**November 15, 2025 - v0.6.0**
- 📊 Added 3 comprehensive MLOps skills: Weights & Biases, MLflow, TensorBoard
- ✨ New MLOps category (3 skills - experiment tracking, model registry, visualization)
- 📝 ~10,000 new lines of documentation across 13 files
- 🔧 Comprehensive coverage: experiment tracking, hyperparameter sweeps, model registry, profiling, embeddings visualization
- **54 total skills** (77% towards 70-skill target)
**November 12, 2025 - v0.5.0**
- 🎯 Added 4 comprehensive prompt engineering skills: DSPy, Instructor, Guidance, Outlines
- ✨ New Prompt Engineering category (4 skills - DSPy, Instructor, Guidance, Outlines)
- 📝 ~10,000 new lines of documentation across 16 files
- 🔧 Comprehensive coverage: declarative programming, structured outputs, constrained generation, FSM-based generation
- **47 total skills** (67% towards 70-skill target)
**November 9, 2025 - v0.4.0**
- 🤖 Added 11 comprehensive skills: LangChain, LlamaIndex, Chroma, FAISS, Sentence Transformers, Pinecone, CLIP, Whisper, LLaVA
- ✨ New Agents category (2 skills - LangChain, LlamaIndex)
- 🔍 New RAG category (4 skills - Chroma, FAISS, Sentence Transformers, Pinecone)
- 🎨 New Multimodal category (3 skills - CLIP, Whisper, LLaVA)
- 📝 ~15,000 new lines of documentation
- **43 total skills** (61% towards 70-skill target)
**November 8, 2025 - v0.3.0**
- 🚀 Added 8 comprehensive skills: TensorRT-LLM, llama.cpp, SGLang, GPTQ, HuggingFace Tokenizers, SentencePiece, Ray Data, NeMo Curator
- ⚡ Completed Inference & Serving category (4/4 skills)
- 🔤 New Tokenization category (2 skills)
- 📊 New Data Processing category (2 skills)
- 📝 9,617 new lines of documentation across 30 files
- **32 total skills** (45% towards 70-skill target)
**November 6, 2025 - v0.2.0**
- Added 10 skills from GitHub (Megatron-Core, Lightning, Ray Train, etc.)
- Improved skill structure with comprehensive references
- Created strategic roadmap to 70 skills
- Added contribution guidelines
**November 3, 2025 - v0.1.0**
- 🎉 Initial release with 5 fine-tuning skills
## Community
Join our community to stay updated, ask questions, and connect with other AI researchers:
- **[SkillEvolve Meta-Skill](https://github.com/Skill-Evolve/meta-skill)** - Connect your agent to the collective intelligence of the community. Captures techniques discovered during sessions and shares them back as curated skills.
- **[Slack Community](https://join.slack.com/t/orchestrarese-efu1990/shared_invite/zt-3iu6gr8io-zJvpkZTPToEviQ9KFZvNSg)** - Chat with the team and other users
- **[Twitter/X](https://x.com/orch_research)** - Follow for updates and announcements
- **[LinkedIn](https://www.linkedin.com/company/orchestra-research/)** - Connect professionally
## Star History
---
## File: 04-mechanistic-interpretability/transformer-lens/references/README.md
# TransformerLens Reference Documentation
This directory contains comprehensive reference materials for TransformerLens.
## Contents
- [api.md](api.md) - Complete API reference for HookedTransformer, ActivationCache, and HookPoints
- [tutorials.md](tutorials.md) - Step-by-step tutorials for common interpretability workflows
- [papers.md](papers.md) - Key research papers and foundational concepts
## Quick Links
- **Official Documentation**: https://transformerlensorg.github.io/TransformerLens/
- **GitHub Repository**: https://github.com/TransformerLensOrg/TransformerLens
- **Model Properties Table**: https://transformerlensorg.github.io/TransformerLens/generated/model_properties_table.html
## Installation
```bash
pip install transformer-lens
```
## Basic Usage
```python
from transformer_lens import HookedTransformer
# Load model
model = HookedTransformer.from_pretrained("gpt2-small")
# Run with activation caching
tokens = model.to_tokens("Hello world")
logits, cache = model.run_with_cache(tokens)
# Access activations
residual = cache["resid_post", 5] # Layer 5 residual stream
attention = cache["pattern", 3] # Layer 3 attention patterns
```
## Key Concepts
### HookPoints
Every activation in the transformer has a HookPoint wrapper, enabling:
- Reading activations via `run_with_cache()`
- Modifying activations via `run_with_hooks()`
### Activation Cache
The `ActivationCache` stores all intermediate activations with helper methods for:
- Residual stream decomposition
- Logit attribution
- Layer-wise analysis
### Supported Models (50+)
GPT-2, LLaMA, Mistral, Pythia, GPT-Neo, OPT, Gemma, Phi, and more.
---
## File: 04-mechanistic-interpretability/saelens/references/README.md
# SAELens Reference Documentation
This directory contains comprehensive reference materials for SAELens.
## Contents
- [api.md](api.md) - Complete API reference for SAE, TrainingSAE, and configuration classes
- [tutorials.md](tutorials.md) - Step-by-step tutorials for training and analyzing SAEs
- [papers.md](papers.md) - Key research papers on sparse autoencoders
## Quick Links
- **GitHub Repository**: https://github.com/jbloomAus/SAELens
- **Neuronpedia**: https://neuronpedia.org (browse pre-trained SAE features)
- **HuggingFace SAEs**: Search for tag `saelens`
## Installation
```bash
pip install sae-lens
```
Requirements: Python 3.10+, transformer-lens>=2.0.0
## Basic Usage
```python
from transformer_lens import HookedTransformer
from sae_lens import SAE
# Load model and SAE
model = HookedTransformer.from_pretrained("gpt2-small", device="cuda")
sae, cfg_dict, sparsity = SAE.from_pretrained(
release="gpt2-small-res-jb",
sae_id="blocks.8.hook_resid_pre",
device="cuda"
)
# Encode activations to sparse features
tokens = model.to_tokens("Hello world")
_, cache = model.run_with_cache(tokens)
activations = cache["resid_pre", 8]
features = sae.encode(activations) # Sparse feature activations
reconstructed = sae.decode(features) # Reconstructed activations
```
## Key Concepts
### Sparse Autoencoders
SAEs decompose dense neural activations into sparse, interpretable features:
- **Encoder**: Maps d_model → d_sae (typically 4-16x expansion)
- **ReLU/TopK**: Enforces sparsity
- **Decoder**: Reconstructs original activations
### Training Loss
`Loss = MSE(original, reconstructed) + L1_coefficient × L1(features)`
### Key Metrics
- **L0**: Average number of active features (target: 50-200)
- **CE Loss Score**: Cross-entropy recovered vs original model (target: 80-95%)
- **Dead Features**: Features that never activate (target: <5%)
## Available Pre-trained SAEs
| Release | Model | Description |
|---------|-------|-------------|
| `gpt2-small-res-jb` | GPT-2 Small | Residual stream SAEs |
| `gemma-2b-res` | Gemma 2B | Residual stream SAEs |
| Various | Search HuggingFace | Community-trained SAEs |
---
## File: 04-mechanistic-interpretability/pyvene/references/README.md
# pyvene Reference Documentation
This directory contains comprehensive reference materials for pyvene.
## Contents
- [api.md](api.md) - Complete API reference for IntervenableModel, intervention types, and configurations
- [tutorials.md](tutorials.md) - Step-by-step tutorials for causal tracing, activation patching, and trainable interventions
## Quick Links
- **Official Documentation**: https://stanfordnlp.github.io/pyvene/
- **GitHub Repository**: https://github.com/stanfordnlp/pyvene
- **Paper**: https://arxiv.org/abs/2403.07809 (NAACL 2024)
## Installation
```bash
pip install pyvene
```
## Basic Usage
```python
import pyvene as pv
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load model
model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")
# Define intervention
config = pv.IntervenableConfig(
representations=[
pv.RepresentationConfig(
layer=5,
component="block_output",
intervention_type=pv.VanillaIntervention,
)
]
)
# Create intervenable model
intervenable = pv.IntervenableModel(config, model)
# Run intervention (swap activations from source to base)
base_inputs = tokenizer("The cat sat on the", return_tensors="pt")
source_inputs = tokenizer("The dog ran through the", return_tensors="pt")
_, outputs = intervenable(
base=base_inputs,
sources=[source_inputs],
)
```
## Key Concepts
### Intervention Types
- **VanillaIntervention**: Swap activations between runs
- **AdditionIntervention**: Add source to base activations
- **ZeroIntervention**: Zero out activations (ablation)
- **CollectIntervention**: Collect activations without modifying
- **RotatedSpaceIntervention**: Trainable intervention for causal discovery
### Components
Target specific parts of the model:
- `block_input`, `block_output`
- `mlp_input`, `mlp_output`, `mlp_activation`
- `attention_input`, `attention_output`
- `query_output`, `key_output`, `value_output`
### HuggingFace Integration
Save and load interventions via HuggingFace Hub for reproducibility.
---
## File: 04-mechanistic-interpretability/nnsight/references/README.md
# nnsight Reference Documentation
This directory contains comprehensive reference materials for nnsight.
## Contents
- [api.md](api.md) - Complete API reference for LanguageModel, tracing, and proxy objects
- [tutorials.md](tutorials.md) - Step-by-step tutorials for local and remote interpretability
## Quick Links
- **Official Documentation**: https://nnsight.net/
- **GitHub Repository**: https://github.com/ndif-team/nnsight
- **NDIF (Remote Execution)**: https://ndif.us/
- **Community Forum**: https://discuss.ndif.us/
- **Paper**: https://arxiv.org/abs/2407.14561 (ICLR 2025)
## Installation
```bash
# Basic installation
pip install nnsight
# For vLLM support
pip install "nnsight[vllm]"
```
## Basic Usage
```python
from nnsight import LanguageModel
# Load model
model = LanguageModel("openai-community/gpt2", device_map="auto")
# Trace and access internals
with model.trace("The Eiffel Tower is in") as tracer:
# Access layer output
hidden = model.transformer.h[5].output[0].save()
# Modify activations
model.transformer.h[8].output[0][:] *= 0.5
# Get final output
logits = model.output.save()
# Access saved values outside context
print(hidden.shape)
```
## Key Concepts
### Tracing
The `trace()` context enables deferred execution - operations are recorded and executed together.
### Proxy Objects
Inside trace, module accesses return Proxies. Call `.save()` to retrieve values after execution.
### Remote Execution (NDIF)
Run the same code on massive models (70B+) without local GPUs:
```python
# Same code, just add remote=True
with model.trace("Hello", remote=True):
hidden = model.model.layers[40].output[0].save()
```
## NDIF Setup
1. Sign up at https://login.ndif.us/
2. Get API key
3. Set environment variable: `export NDIF_API_KEY=your_key`
## Available Remote Models
- Llama-3.1-8B, 70B, 405B
- DeepSeek-R1 models
- More at https://ndif.us/
---
## File: 06-post-training/grpo-rl-training/README.md
# GRPO/RL Training Skill
**Expert-level guidance for Group Relative Policy Optimization with TRL**
## 📁 Skill Structure
```
grpo-rl-training/
├── SKILL.md # Main skill documentation (READ THIS FIRST)
├── README.md # This file
├── templates/
│ └── basic_grpo_training.py # Production-ready training template
└── examples/
└── reward_functions_library.py # 20+ reward function examples
```
## 🚀 Quick Start
1. **Read SKILL.md** - Comprehensive guide with all concepts and patterns
2. **Copy `templates/basic_grpo_training.py`** - Start with working code
3. **Browse `examples/reward_functions_library.py`** - Pick reward functions for your task
4. **Modify for your use case** - Adapt dataset, rewards, and config
## 💡 What's Inside
### SKILL.md (Main Documentation)
- Core GRPO concepts and algorithm fundamentals
- Complete implementation workflow (dataset → rewards → training → deployment)
- 10+ reward function examples with code
- Hyperparameter tuning guide
- Training insights (loss behavior, metrics, debugging)
- Troubleshooting guide
- Production best practices
### Templates
- **basic_grpo_training.py**: Minimal, production-ready training script
- Uses Qwen 2.5 1.5B Instruct
- 3 reward functions (format + correctness)
- LoRA for efficient training
- Fully documented and ready to run
### Examples
- **reward_functions_library.py**: 20+ battle-tested reward functions
- Correctness rewards (exact match, fuzzy match, numeric, code execution)
- Format rewards (XML, JSON, strict/soft)
- Length rewards (ideal length, min/max)
- Style rewards (reasoning quality, citations, repetition penalty)
- Combined rewards (multi-objective optimization)
- Preset collections for common tasks
## 📖 Usage for Agents
When this skill is loaded in your agent's context:
1. **Always read SKILL.md first** before implementing
2. **Start simple** - Use length-based reward to validate setup
3. **Build incrementally** - Add one reward function at a time
4. **Reference examples** - Copy patterns from reward_functions_library.py
5. **Monitor training** - Watch reward metrics (not loss!)
## 🎯 Common Use Cases
| Task Type | Recommended Rewards | Template |
|-----------|---------------------|----------|
| Math reasoning | `MATH_REASONING_REWARDS` preset | basic_grpo_training.py |
| Code generation | `CODE_GENERATION_REWARDS` preset | Modify dataset in template |
| Summarization | `SUMMARIZATION_REWARDS` preset | Adjust prompts + rewards |
| Q&A | `QA_REWARDS` preset | Use fuzzy match + citations |
## ⚠️ Critical Reminders
- **Loss goes UP during training** - This is normal (it's KL divergence)
- **Use 3-5 reward functions** - Single rewards often fail
- **Test rewards before training** - Debug each function independently
- **Monitor reward_std** - Should stay > 0.1 (avoid mode collapse)
- **Start with num_generations=4-8** - Scale up if GPU allows
## 🔗 External Resources
- [TRL Documentation](https://huggingface.co/docs/trl)
- [DeepSeek R1 Paper](https://arxiv.org/abs/2501.12948)
- [Open R1 Implementation](https://github.com/huggingface/open-r1)
- [Unsloth (2-3x faster)](https://docs.unsloth.ai/)
## 📝 Version
**v1.0.0** - Initial release (January 2025)
## 👨💻 Maintained By
Orchestra Research
For questions or improvements, see https://orchestra.com
---
**License:** MIT
**Last Updated:** January 2025
---
## File: 14-agents/a-evolve/references/README.md
# A-Evolve Official Documentation Reference
> This document consolidates key information from the official A-Evolve documentation
> at [github.com/A-EVO-Lab/a-evolve](https://github.com/A-EVO-Lab/a-evolve).
## Table of Contents
- [Project Overview](#project-overview)
- [Installation Guide](#installation-guide)
- [Quick Start Guide](#quick-start-guide)
- [Architecture Overview](#architecture-overview)
- [Agent Protocol](#agent-protocol)
- [Benchmark Adapters](#benchmark-adapters)
- [Evolution Engines](#evolution-engines)
- [Workspace Contract](#workspace-contract)
- [Configuration Reference](#configuration-reference)
- [Built-in Agents](#built-in-agents)
- [Built-in Benchmarks](#built-in-benchmarks)
- [Evolution Algorithms](#evolution-algorithms)
- [Skill System](#skill-system)
- [Memory System](#memory-system)
- [Version Control](#version-control)
- [Observation Pipeline](#observation-pipeline)
- [FAQ](#faq)
---
## Project Overview
A-Evolve is the universal infrastructure for evolving AI agents through self-improvement. It enables automatic, data-driven optimization of agents across any domain using any evolution algorithm.
### Design Principles
1. **File-system as contract**: All evolvable agent state lives as plain files in a workspace directory. No databases, no learned weights, no opaque parameters. Every mutation is an explicit edit to a text file.
2. **Pluggable everything**: Three interfaces — `BaseAgent`, `BenchmarkAdapter`, `EvolutionEngine` — enable any combination of agent, benchmark, and algorithm.
3. **Git for versioning**: Every evolution cycle creates git snapshots. Changes are diffable, rollbackable, and human-readable.
4. **LLM-in-the-loop**: The default evolution engine uses an LLM with bash tools to analyze observations and directly mutate workspace files. The evolver is itself an AI agent improving other AI agents.
5. **Zero manual engineering**: Once configured, evolution runs autonomously. The loop handles solving, evaluation, mutation, gating, and convergence detection.
### Key Results
Using Claude Opus 4.6 as both the solver and evolver model:
| Benchmark | Score | Leaderboard Position |
|-----------|-------|---------------------|
| MCP-Atlas | 79.4% | #1 |
| SWE-bench Verified | 76.8% | ~#5 |
| Terminal-Bench 2.0 | 76.5% | ~#7 |
| SkillsBench | 34.9% | #2 |
These results demonstrate that LLM-driven evolution of prompts, skills, and memory can produce state-of-the-art agent performance across diverse domains.
---
## Installation Guide
### Requirements
- Python >= 3.11
- Git (for workspace versioning)
- An LLM API key (Anthropic, OpenAI, or AWS Bedrock credentials)
### Installation Options
```bash
# Core package (matplotlib, pyyaml)
pip install a-evolve
# With specific LLM provider support
pip install a-evolve[anthropic] # Anthropic Claude API
pip install a-evolve[openai] # OpenAI API
pip install a-evolve[bedrock] # AWS Bedrock (boto3)
pip install a-evolve[litellm] # Multi-provider via LiteLLM
# With domain-specific dependencies
pip install a-evolve[swe] # SWE-bench (strands-agents, datasets, swebench)
pip install a-evolve[mcp] # MCP-Atlas (mcp, strands-agents, litellm)
pip install a-evolve[skillbench] # SkillsBench (strands-agents)
# Everything
pip install a-evolve[all]
# Development
pip install a-evolve[dev] # pytest, ruff, hypothesis
```
### From Source
```bash
git clone https://github.com/A-EVO-Lab/a-evolve.git
cd a-evolve
pip install -e ".[all,dev]"
```
### Verifying Installation
```python
import agent_evolve as ae
print(ae.__version__) # Should print version
print(ae.Evolver) # Should print class reference
```
---
## Quick Start Guide
### 3-Line Evolution
```python
import agent_evolve as ae
evolver = ae.Evolver(agent="swe", benchmark="swe-verified")
results = evolver.run(cycles=10)
print(f"Final score: {results.final_score}")
```
This:
1. Copies the built-in SWE seed workspace to a working directory
2. Instantiates `SweAgent` from the workspace manifest
3. Runs 10 evolution cycles against SWE-bench Verified
4. Returns `EvolutionResult` with scores, convergence status, and details
### With Custom Configuration
```python
import agent_evolve as ae
config = ae.EvolveConfig(
batch_size=15, # 15 tasks per cycle
max_cycles=25, # Up to 25 evolution rounds
evolve_prompts=True, # Mutate system prompt
evolve_skills=True, # Discover and refine skills
evolve_memory=True, # Build episodic memory
holdout_ratio=0.2, # 20% held out for validation
evolver_model="us.anthropic.claude-opus-4-6-v1",
egl_threshold=0.02, # Stop if < 2% improvement
egl_window=5, # Over 5 consecutive cycles
)
evolver = ae.Evolver(
agent="swe",
benchmark="swe-verified",
config=config,
)
results = evolver.run()
# Inspect results
print(f"Cycles: {results.cycles_completed}")
print(f"Score: {results.final_score:.3f}")
print(f"Converged: {results.converged}")
print(f"Score history: {results.score_history}")
```
---
## Architecture Overview
### System Diagram
```
User Code (3 lines)
│
▼
┌──────────────────────────────────────┐
│ Evolver API │
│ - Resolves agent, benchmark, config │
│ - Creates EvolutionLoop │
│ - Returns EvolutionResult │
└──────────────┬───────────────────────┘
│
┌──────────▼──────────┐
│ EvolutionLoop │
│ For each cycle: │
│ 1. Solve │
│ 2. Observe │
│ 3. Snapshot │
│ 4. Evolve │
│ 5. Snapshot │
│ 6. Record │
│ 7. Reload │
│ 8. Converge? │
└──────────┬──────────┘
│
┌──────────┼──────────┐
│ │ │
▼ ▼ ▼
Agent Benchmark Engine
solve() evaluate() step()
│ │ │
└──────────┼──────────┘
│
▼
Agent Workspace
(filesystem + git)
```
### Component Interactions
**Forward flow (solve):**
1. `EvolutionLoop` calls `benchmark.get_tasks()` to get a batch of tasks
2. For each task, calls `agent.solve(task)` to get a `Trajectory`
3. Calls `benchmark.evaluate(task, trajectory)` to get `Feedback`
4. Bundles into `Observation(task, trajectory, feedback)` triples
**Evolution flow (mutate):**
1. `EvolutionLoop` passes observations to `engine.step()`
2. Engine reads workspace files, analyzes observations
3. Engine mutates workspace files (prompts, skills, memory)
4. Returns `StepResult(mutated, summary, metadata)`
**Reload flow (sync):**
1. `EvolutionLoop` calls `agent.reload_from_fs()`
2. Agent re-reads prompts, skills, memory from workspace
3. Next cycle uses evolved state
---
## Agent Protocol
### BaseAgent Abstract Class
All evolvable agents inherit from `BaseAgent`:
```python
from agent_evolve.protocol.base_agent import BaseAgent
from agent_evolve.types import Task, Trajectory
class MyAgent(BaseAgent):
def __init__(self, workspace_dir: str):
super().__init__(workspace_dir)
# Initialize your LLM client, tools, etc.
def solve(self, task: Task) -> Trajectory:
"""Solve a single task and return the trajectory.
This is the only method you MUST override.
"""
# Your solving logic here
return Trajectory(
task_id=task.id,
output="solution",
steps=[{"tool": "llm", "action": "generate"}],
)
```
### Agent Lifecycle
1. **Construction**: `__init__(workspace_dir)` — set up LLM client, load initial state
2. **State loading**: `reload_from_fs()` — read prompts, skills, memory from workspace
3. **Solving**: `solve(task)` — process one task, return trajectory
4. **Memory buffering**: `remember(content, category)` — store lessons during solve
5. **State export**: `export_to_fs()` — flush buffered memories and skill proposals
6. **Hot reload**: `reload_from_fs()` — re-read after evolution mutates files
### Agent Properties
| Property | Type | Description |
|----------|------|-------------|
| `system_prompt` | `str` | Content of `prompts/system.md` |
| `skills` | `list[SkillMeta]` | Available skills from `skills/` directory |
| `memories` | `list[dict]` | Loaded episodic/semantic memories |
### Agent Best Practices
1. **Always use `self.system_prompt`** — don't hardcode prompts
2. **Inject skills into LLM context** — they're the primary evolution mechanism
3. **Call `remember()` for reusable lessons** — not for task-specific notes
4. **Keep `solve()` deterministic** when possible (temperature=0 for reproducibility)
5. **Truncate trajectories** — don't store full conversation if not needed for evolution
---
## Benchmark Adapters
### BenchmarkAdapter Abstract Class
```python
from agent_evolve.benchmarks.base import BenchmarkAdapter
from agent_evolve.types import Task, Trajectory, Feedback
class MyBenchmark(BenchmarkAdapter):
def get_tasks(self, split="train", limit=10):
"""Return tasks from the benchmark dataset.
Args:
split: "train" or "test" (for holdout evaluation)
limit: Maximum number of tasks to return (default 10)
"""
return [Task(id="1", input="task description")]
def evaluate(self, task, trajectory):
"""Evaluate an agent's trajectory on a task.
Returns Feedback with:
- success: bool (binary pass/fail)
- score: float (0.0 to 1.0 continuous)
- detail: str (human-readable explanation)
"""
return Feedback(success=True, score=0.9, detail="Passed 9/10 tests")
```
### Benchmark Best Practices
1. **Rich feedback details** — the evolver reads `feedback.detail` to decide what to mutate
2. **Deterministic evaluation** — same input should produce same score
3. **Diverse task coverage** — include easy, medium, and hard tasks
4. **Strict train/test split** — no overlap between splits
5. **Score granularity** — continuous scores (0.0-1.0) are more useful than binary pass/fail
---
## Evolution Engines
### EvolutionEngine Abstract Class
```python
from agent_evolve.engine.base import EvolutionEngine
from agent_evolve.types import StepResult
class MyEngine(EvolutionEngine):
def step(self, workspace, observations, history, trial):
"""Mutate the workspace based on observations.
Args:
workspace: AgentWorkspace — typed I/O for agent files
observations: list[Observation] — recent (task, trajectory, feedback) triples
history: EvolutionHistory — query past cycles and workspace versions
trial: TrialRunner — optional live evaluation runner
Returns:
StepResult with mutated flag, summary, and metadata
"""
# Analyze observations, mutate workspace
return StepResult(mutated=True, summary="Updated prompts")
def on_cycle_end(self, accepted: bool, score: float):
"""Optional callback after gating decision."""
pass
```
### Engine Selection Guide
| Engine | When to Use | Compute Cost |
|--------|-------------|-------------|
| AEvolveEngine (default) | General-purpose, diverse domains | High (full LLM call) |
| GuidedSynthesisEngine | Skill discovery focus | Medium |
| AdaptiveEvolutionEngine | Noisy evaluation, fine control | Medium |
| AdaptiveSkillEngine | Skill-heavy domains | Medium |
| Custom | Domain-specific mutation logic | Variable |
---
## Workspace Contract
### Directory Structure
```
workspace/
├── manifest.yaml # Required: agent metadata
├── prompts/
│ ├── system.md # Main system prompt
│ └── fragments/ # Modular prompt pieces
│ ├── reasoning.md
│ └── output_format.md
├── skills/
│ ├── _drafts/ # Proposed skills pending review
│ │ └── new-skill.md
│ └── verify-solution/ # Accepted skills
│ └── SKILL.md
├── tools/
│ ├── registry.yaml # Tool manifest
│ └── custom_tool.py # Tool implementations
├── memory/
│ ├── episodic.jsonl # Failure lessons
│ └── semantic.jsonl # Domain knowledge
└── evolution/ # Managed by loop
├── observations/
│ ├── batch_0001.jsonl
│ └── batch_0002.jsonl
├── history.jsonl
└── metrics.json
```
### Manifest Format
```yaml
agent:
type: reference # Must be "reference"
entrypoint: my_package.agents.MyAgent # Dotted Python path
evolvable_layers: # Which directories can be mutated
- prompts # System prompt + fragments
- skills # Skill library
- memory # Episodic/semantic memory
# - tools # Tool implementations (optional)
reload_strategy: hot # "hot" (re-read files) or "cold" (restart)
```
### AgentWorkspace API
The `AgentWorkspace` class provides typed read/write access:
**Prompts:**
- `read_prompt() -> str` — reads `prompts/system.md`
- `write_prompt(content: str)` — writes `prompts/system.md`
- `read_fragment(name: str) -> str` — reads `prompts/fragments/{name}`
- `write_fragment(name: str, content: str)` — writes a fragment
- `list_fragments() -> list[str]` — lists fragment filenames
**Skills:**
- `list_skills() -> list[SkillMeta]` — lists skills with name, description, path
- `read_skill(name: str) -> str` — reads skill content (frontmatter stripped)
- `write_skill(name: str, content: str)` — writes or updates a skill
- `delete_skill(name: str)` — removes a skill directory
**Drafts:**
- `list_drafts() -> list[dict]` — lists pending skill proposals
- `write_draft(name: str, content: str)` — writes a draft proposal
- `clear_drafts()` — removes all pending drafts
**Memory:**
- `add_memory(entry: dict, category: str = "episodic")` — appends to category JSONL
- `read_memories(category: str = "episodic", limit: int = 100) -> list[dict]`
- `read_all_memories(limit: int = 100) -> list[dict]` — all categories combined
**Tools:**
- `read_tool_registry() -> list[dict]` — reads `tools/registry.yaml`
- `write_tool_registry(tools: list[dict])` — writes tool manifest
- `read_tool(name: str) -> str` — reads tool source code
- `write_tool(name: str, content: str)` — writes tool implementation
**Evolution Metadata:**
- `read_evolution_history() -> list[dict]` — reads `evolution/history.jsonl`
- `read_evolution_metrics() -> dict` — reads `evolution/metrics.json`
---
## Configuration Reference
### EvolveConfig Fields
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `batch_size` | `int` | `10` | Tasks per solve round |
| `max_cycles` | `int` | `20` | Maximum evolution iterations |
| `holdout_ratio` | `float` | `0.2` | Fraction held out for validation |
| `evolve_prompts` | `bool` | `True` | Allow prompt mutation |
| `evolve_skills` | `bool` | `True` | Allow skill creation/modification |
| `evolve_memory` | `bool` | `True` | Allow memory writes |
| `evolve_tools` | `bool` | `False` | Allow tool implementation changes |
| `trajectory_only` | `bool` | `False` | Hide scores from evolver |
| `evolver_model` | `str` | `"us.anthropic.claude-opus-4-6-v1"` | LLM for evolution engine |
| `evolver_max_tokens` | `int` | `16384` | Max tokens for evolver calls |
| `egl_threshold` | `float` | `0.05` | Convergence epsilon |
| `egl_window` | `int` | `3` | Cycles for plateau detection |
| `extra` | `dict` | `{}` | Extension point for custom params |
### Loading from YAML
```yaml
# config.yaml
batch_size: 15
max_cycles: 30
evolve_prompts: true
evolve_skills: true
evolve_memory: false
evolver_model: us.anthropic.claude-opus-4-6-v1
egl_threshold: 0.03
egl_window: 5
extra:
solver_proposed: true
merge_threshold: 0.7
```
```python
config = ae.EvolveConfig.from_yaml("config.yaml")
```
### Configuration Strategies
**Conservative (stable improvement):**
```python
config = ae.EvolveConfig(
batch_size=10,
max_cycles=10,
evolve_prompts=True,
evolve_skills=False,
evolve_memory=False,
egl_threshold=0.05,
)
```
**Aggressive (maximum exploration):**
```python
config = ae.EvolveConfig(
batch_size=20,
max_cycles=50,
evolve_prompts=True,
evolve_skills=True,
evolve_memory=True,
evolve_tools=True,
egl_threshold=0.01,
egl_window=7,
)
```
**Skill-focused (procedure discovery):**
```python
config = ae.EvolveConfig(
batch_size=10,
max_cycles=25,
evolve_prompts=False,
evolve_skills=True,
evolve_memory=True,
)
```
---
## Built-in Agents
### SWE Agent (`seed_workspaces/swe/`)
**Domain**: SWE-bench code patching
**Model**: Claude Opus 4.6 via AWS Bedrock
**Framework**: Strands-agents (CodeDojo-compatible)
Key features:
- Verify-fix loop: runs tests before and after each edit
- Hypothesis-first approach: form theory before exploring
- Skill proposal generation: agent reflects on verification process
- Conversation capture with per-turn token tracking
- Dynamic tool loading from workspace `tools/registry.yaml`
**Tools available**: bash, submit, text_editor, python_exec
### Terminal Agent (`seed_workspaces/terminal/`)
**Domain**: Terminal-Bench 2.0 shell challenges
**Model**: Claude Sonnet 4 via AWS Bedrock
**Framework**: Strands-agents
Key features:
- Concurrent timeout enforcement via ThreadPoolExecutor
- Test file copying only during evaluation (prevents cheating)
- Pre-built skills: self-verification, environment-discovery, scientific-computing, debug-and-fix
- Memory injection disabled (time-sensitive tasks)
- Graceful timeout fallback
**Tools available**: bash, python, submit
### MCP Agent (`seed_workspaces/mcp/`)
**Domain**: MCP-Atlas tool calling
**Model**: Claude Opus 4.6 via AWS Bedrock
**Framework**: Strands-agents with MCP integration
Key features:
- MCP server connection management
- Tool discovery and invocation
- Multi-provider support via LiteLLM
---
## Built-in Benchmarks
### SWE-bench Verified
**Module**: `agent_evolve.benchmarks.swe_verified`
**Tasks**: Real GitHub issues from popular Python repositories
**Evaluation**: Runs test suite, checks if agent's patch fixes the issue
**Metric**: Pass rate (0.0 to 1.0)
### MCP-Atlas
**Module**: `agent_evolve.benchmarks.mcp_atlas`
**Tasks**: Tool calling scenarios with MCP servers
**Evaluation**: Checks correct tool selection and parameter passing
**Metric**: Accuracy (0.0 to 1.0)
### Terminal-Bench 2.0
**Module**: `agent_evolve.benchmarks.terminal2`
**Tasks**: Shell command challenges (file manipulation, system admin, scripting)
**Evaluation**: Runs test scripts to verify terminal state
**Metric**: Pass rate (0.0 to 1.0)
### SkillsBench
**Module**: `agent_evolve.benchmarks.skill_bench`
**Tasks**: Multi-step procedural tasks
**Evaluation**: Checks step-by-step correctness
**Metric**: Accuracy (0.0 to 1.0)
### ARC-AGI-3
**Module**: `agent_evolve.benchmarks.arc_agi3`
**Tasks**: Interactive game levels (25 games, 181 levels)
**Evaluation**: RHAE score (ratio of human to agent actions, squared)
**Metric**: Average RHAE across levels (0.0 to 1.0)
---
## Evolution Algorithms
### AEvolveEngine (SkillForge)
**Module**: `agent_evolve.algorithms.skillforge.engine`
**Strategy**: LLM-driven workspace mutation
The default engine gives an LLM full bash tool access to the workspace and asks it to improve the agent based on observations. This is the most flexible engine — it can make arbitrary changes to any workspace file.
**Context provided to the LLM:**
- Recent observations (task inputs, agent outputs, feedback)
- Current system prompt
- Current skill library
- Pending draft proposals
- Score history
**Mutation capabilities:**
- Edit system prompt (refine, consolidate, extend)
- Create new skills from observed patterns
- Merge overlapping skills
- Write episodic memory entries
- Review and curate draft proposals
### GuidedSynthesisEngine
**Module**: `agent_evolve.algorithms.guided_synth`
**Strategy**: Memory-first, curated skills
Emphasizes learning from failures before creating skills. Conservative approach that prevents skill bloat.
**Process:**
1. Extract lessons from failed tasks
2. Write episodic memory entries
3. After accumulating patterns, synthesize skill proposals
4. Curate proposals: ACCEPT, MERGE, or SKIP
### AdaptiveEvolutionEngine
**Module**: `agent_evolve.algorithms.adaptive`
**Strategy**: Reward tracking + observation filtering
Adjusts intervention intensity based on score trends. Makes smaller changes when improving, larger changes when plateaued.
### AdaptiveSkillEngine
**Module**: `agent_evolve.algorithms.adaptive_skill`
**Strategy**: Skill-centric discovery
Focuses exclusively on building the skill library. Identifies task categories where the agent fails and creates targeted skills.
---
## Skill System
### Skill File Format
```markdown
---
name: verify-edge-cases
---
## Pattern
Test all falsy-but-valid values: 0, False, "", [], {}
## Process
1. List all input boundaries
2. Run each against the implementation
3. Check both output AND side effects
```
### Skill Discovery Process
1. **Agent proposes**: During `solve()`, agent writes draft to `skills/_drafts/`
2. **Engine reviews**: During `step()`, engine reads drafts and decides:
- **ACCEPT**: Move to `skills/{name}/SKILL.md`
- **MERGE**: Combine with existing similar skill
- **SKIP**: Discard (too narrow, redundant, or incorrect)
3. **Engine creates**: Engine can also create skills directly from observation analysis
4. **Refinement**: Existing skills are updated based on new observations
### Skill Library Management
Target: 5-10 broad, reusable skills per workspace. Avoid:
- 30+ narrow skills (library bloat)
- Skills that duplicate system prompt content
- Skills with no TRIGGER condition (always-on = should be in prompt)
---
## Memory System
### Episodic Memory
Records specific lessons from task attempts:
```json
{"content": "pytest --no-header flag needed for clean output", "category": "episodic", "task_id": "django-16379"}
{"content": "Off-by-one errors common in range() with len()", "category": "episodic", "task_id": "numpy-8823"}
```
### Semantic Memory
General domain knowledge:
```json
{"content": "Django uses reverse URL resolution via urlpatterns", "category": "semantic"}
{"content": "NumPy broadcasting rules: dimensions must match or be 1", "category": "semantic"}
```
### Memory Limits
- `BaseAgent.reload_from_fs()` loads up to 200 memory entries by default
- `AgentWorkspace.read_memories()` defaults to limit=100
- Old memories should be pruned or consolidated during evolution
---
## Version Control
### Git Tagging Convention
| Tag | When Created | Purpose |
|-----|-------------|---------|
| `pre-evo-1` | Before cycle 1 evolution | Snapshot of solve-only state |
| `evo-1` | After cycle 1 evolution | Snapshot of evolved state |
| `pre-evo-2` | Before cycle 2 evolution | Snapshot before next mutation |
| `evo-2` | After cycle 2 evolution | Snapshot of evolved state |
### Useful Git Commands
```bash
# See all evolution checkpoints
git tag -l "evo-*"
# Compare two evolution stages
git diff evo-1 evo-10
# See what changed in a specific cycle
git diff pre-evo-5 evo-5
# Read a file at a specific point in time
git show evo-3:prompts/system.md
# Revert to a known good state
git checkout evo-5 -- .
```
---
## Observation Pipeline
### JSONL Format
Each observation is stored in `evolution/observations/batch_{label}.jsonl`:
```json
{
"task_id": "django__django-16379",
"task_input": "Fix FileBasedCache has_key method...",
"task_metadata": {},
"agent_output": "--- a/django/core/cache/backends/filebased.py\n+++ ...",
"steps": [
{"tool": "bash", "action": "read_file", "file": "django/core/cache/backends/filebased.py"},
{"tool": "text_editor", "action": "edit", "file": "django/core/cache/backends/filebased.py"}
],
"success": true,
"score": 1.0,
"feedback_detail": "All 24 tests passed"
}
```
### Querying Observations
```python
history = EvolutionHistory("./my-workspace")
# All observations from last 3 cycles
recent = history.get_observations(last_n_cycles=3)
# Only failures
failures = history.get_observations(only_failures=True)
# Score curve
scores = history.get_score_curve() # [(1, 0.62), (2, 0.68), ...]
```
---
## FAQ
### Can I use A-Evolve with any LLM?
Yes. The agent can use any LLM for solving. The evolver model is configurable via `EvolveConfig.evolver_model`. Supported providers: Anthropic (direct API), OpenAI, AWS Bedrock, LiteLLM (multi-provider).
### Does evolution require training data?
No in the traditional ML sense. You need a `BenchmarkAdapter` that provides tasks and evaluation, but there are no training/gradient steps. Evolution is purely file-system mutation guided by LLM reasoning.
### How many cycles should I run?
Start with 10 cycles and check convergence. If score is still improving, run more. Default convergence detection (`egl_threshold=0.05`, `egl_window=3`) stops automatically when improvement plateaus.
### Can I resume evolution after stopping?
Yes. The workspace retains its evolved state. Create a new `Evolver` pointing to the same workspace and call `run()` again.
### Is evolution deterministic?
No. LLM calls are inherently non-deterministic. Running the same config twice may produce different evolved agents with similar final scores.
### Can I evolve multiple agents simultaneously?
Yes, but each must have its own workspace directory. The evolution loop modifies workspace files directly, so concurrent access to the same workspace is not safe.
### What's the cost per evolution cycle?
Each cycle involves: (batch_size) agent solve calls + 1 evolver call. For batch_size=10 with Claude, expect ~$5-20 per cycle depending on task complexity and model used.
### Can I use A-Evolve without a benchmark?
Not directly. The evolution loop requires `BenchmarkAdapter.evaluate()` to produce `Feedback`. However, you can implement a custom benchmark that uses human evaluation, LLM-as-judge, or any other scoring mechanism.
---
## File: 20-ml-paper-writing/ml-paper-writing/templates/acl/README.md
# *ACL Paper Styles
This directory contains the latest LaTeX templates for *ACL conferences.
## Instructions for authors
Paper submissions to *ACL conferences must use the official ACL style
templates.
The LaTeX style files are available
- as an [Overleaf template](https://www.overleaf.com/latex/templates/association-for-computational-linguistics-acl-conference/jvxskxpnznfj)
- in this repository
- as a [.zip file](https://github.com/acl-org/acl-style-files/archive/refs/heads/master.zip)
Please see [`acl_latex.tex`](https://github.com/acl-org/acl-style-files/blob/master/acl_latex.tex) for an example.
Please follow the paper formatting guidelines general to *ACL
conferences:
- [Paper formatting guidelines](https://acl-org.github.io/ACLPUB/formatting.html)
Authors may not modify these style files or use templates designed for
other conferences.
## Instructions for publications chairs
To adapt the style files for your conference, please fork this repository and
make necessary changes. Minimally, you'll need to update the name of
the conference and rename the files.
If you make improvements to the templates that should be propagated to
future conferences, please submit a pull request. Thank you in
advance!
In older versions of the templates, authors were asked to fill in the
START submission ID so that it would be stamped at the top of each
page of the anonymized version. This is no longer needed, because it
is now possible to do this stamping automatically within
START. Currently, the way to do this is for the program chair to email
support@softconf.com and request it.
## Instructions for making changes to style files
- merge pull request in github, or push to github
- git pull from github to a local repository
- then, git push from your local repository to overleaf project
- Overleaf project is https://www.overleaf.com/project/5f64f1fb97c4c50001b60549
- Overleaf git url is https://git.overleaf.com/5f64f1fb97c4c50001b60549
- then, click "Submit" and then "Submit as Template" in overleaf in order to ask overleaf to update the overleaf template from the overleaf project
---
## File: 20-ml-paper-writing/ml-paper-writing/templates/aaai2026/README.md
# AAAI 2026 统一LaTeX模板使用说明 / AAAI 2026 Unified LaTeX Template Guide
> **📝 重要说明 / Important Notice**: 本仓库借助Cursor在AAAI 2026官方模板基础上改进得到。如果遇到不满足或有冲突的情况，请积极提issues。
>
> **📝 Important Notice**: This repository is improved based on the official AAAI 2026 template with the assistance of Cursor. If you encounter any issues or conflicts, please actively submit issues.
[中文](#中文版本) | [English](#english-version)
---
## 🌐 在线查看 / Online Access
**📖 在线阅读和测试模板**: [https://cn.overleaf.com/read/wyhcnvcrtpyt#cd4a07](https://cn.overleaf.com/read/wyhcnvcrtpyt#cd4a07)
**📖 Online View and Test Template**: [https://cn.overleaf.com/read/wyhcnvcrtpyt#cd4a07](https://cn.overleaf.com/read/wyhcnvcrtpyt#cd4a07)
💡 **提示 / Tips**:
- 中文：您可以通过上述链接在Overleaf中直接查看、编辑和编译模板，无需本地安装LaTeX环境
- English: You can view, edit, and compile the template directly in Overleaf using the link above, without needing a local LaTeX installation
---
## 中文版本
### 概述 ✅
我已经将AAAI 2026的两个版本（匿名投稿版本和camera-ready版本）**完整合并**成一个统一的模板文件 `aaai2026-unified-template.tex`。
该模板包含了原始两个模板的**所有完整内容**（共886行，比原始文件更全面），包括：
- 所有格式化说明和要求
- 完整的示例代码和表格
- 图片处理指南
- 参考文献格式要求
- 所有章节和附录内容
- 版本特定的Acknowledgments部分
### 主要差异分析
通过比较原始的两个模板，我发现主要差异在于：
#### 1. 包的加载方式
- **匿名版本**: `\usepackage[submission]{aaai2026}`
- **Camera-ready版本**: `\usepackage{aaai2026}`
#### 2. 标题差异
- **匿名版本**: "AAAI Press Anonymous Submission Instructions for Authors Using LaTeX"
- **Camera-ready版本**: "AAAI Press Formatting Instructions for Authors Using LaTeX --- A Guide"
#### 3. Links环境的处理
- **匿名版本**: Links环境被注释掉，防止泄露作者身份
- **Camera-ready版本**: Links环境正常显示
#### 4. 内容部分差异
- **匿名版本**: 包含"Preparing an Anonymous Submission"部分的特殊说明
- **Camera-ready版本**: 包含完整的格式说明和版权信息
### 依赖文件检查结果
✅ **已验证并复制到主目录的文件**：
- `aaai2026.sty` - AAAI 2026 样式文件（两个版本完全相同）
- `aaai2026.bst` - 参考文献样式文件（两个版本完全相同）
- `aaai2026.bib` - 示例参考文献文件
- `figure1.pdf` 和 `figure2.pdf` - 示例图片文件
所有这些文件在两个版本中都是相同的，因此统一模板可以正常工作。
### 如何使用统一模板
#### 切换到匿名投稿版本
在模板文件第11行，**取消注释**这一行：
```latex
\def\aaaianonymous{true}
```
#### 切换到Camera-ready版本
在模板文件第11行，**注释掉**或**删除**这一行：
```latex
% \def\aaaianonymous{true}
```
### 一键切换的核心机制
统一模板使用了LaTeX的条件编译功能：
```latex
% 条件包加载
\ifdefined\aaaianonymous
\usepackage[submission]{aaai2026} % 匿名版本
\else
\usepackage{aaai2026} % Camera-ready版本
\fi
% 条件标题设置
\ifdefined\aaaianonymous
\title{AAAI Press Anonymous Submission\\Instructions for Authors Using \LaTeX{}}
\else
\title{AAAI Press Formatting Instructions \\for Authors Using \LaTeX{} --- A Guide}
\fi
% 条件内容显示
\ifdefined\aaaianonymous
% 匿名版本特有内容
\else
% Camera-ready版本特有内容
\fi
```
### 文件清单
主目录现在包含以下文件：
- `aaai2026-unified-template.tex` - 统一主论文模板文件
- `aaai2026-unified-supp.tex` - 统一补充材料模板文件
- `aaai2026.sty` - AAAI 2026 LaTeX 样式文件
- `aaai2026.bst` - 参考文献样式文件
- `aaai2026.bib` - 示例参考文献文件
- `figure1.pdf` - 示例图片1
- `figure2.pdf` - 示例图片2
- `README.md` - 本说明文档
### 补充材料模板 (Supplementary Material Template)
#### 概述
`aaai2026-unified-supp.tex` 是专门为AAAI 2026补充材料设计的统一模板，与主论文模板使用相同的版本切换机制。
#### 主要功能
- **版本切换**: 通过修改一行代码在匿名投稿和camera-ready版本间切换
- **补充内容支持**: 支持额外的实验、推导、数据、图表、算法等
- **格式一致性**: 与主论文模板保持完全一致的格式要求
- **代码示例**: 包含算法、代码列表等补充材料的示例
#### 使用方法
与主论文模板相同，只需修改第11行：
```latex
% 匿名投稿版本
\def\aaaianonymous{true}
% Camera-ready版本
% \def\aaaianonymous{true}
```
#### 补充材料内容建议
- 额外的实验结果和消融研究
- 详细的数学推导和证明
- 更多的图表和可视化
- 算法伪代码和实现细节
- 数据集描述和预处理步骤
- 超参数设置和实验配置
- 失败案例分析
- 计算复杂度分析
### 使用检查清单 (Usage Checklist)
#### 📋 投稿前检查清单 (Pre-Submission Checklist)
**版本设置**:
- [ ] 已设置 `\def\aaaianonymous{true}` (匿名投稿)
- [ ] 已注释掉所有可能暴露身份的信息
- [ ] 已匿名化参考文献（移除作者姓名）
**内容完整性**:
- [ ] 标题、摘要、关键词已填写
- [ ] 所有章节内容完整
- [ ] 图表编号连续且正确
- [ ] 参考文献格式正确
- [ ] 补充材料（如有）已准备
**格式检查**:
- [ ] 页面边距符合要求
- [ ] 字体和字号正确
- [ ] 行间距符合标准
- [ ] 图表位置和大小合适
- [ ] 数学公式格式正确
**技术检查**:
- [ ] LaTeX编译无错误
- [ ] 参考文献正确生成
- [ ] PDF输出正常
- [ ] 文件大小在限制范围内
#### 📋 录用后检查清单 (Post-Acceptance Checklist)
**版本切换**:
- [ ] 已注释掉 `\def\aaaianonymous{true}` (camera-ready)
- [ ] 已添加完整的作者信息
- [ ] 已添加所有作者单位信息
- [ ] 已恢复所有被注释的内容
**内容更新**:
- [ ] 已根据审稿意见修改内容
- [ ] 已更新所有图表和实验
- [ ] 已完善补充材料
- [ ] 已检查所有链接和引用
**最终检查**:
- [ ] 最终PDF质量检查
- [ ] 所有文件已备份
- [ ] 符合会议最终提交要求
- [ ] 补充材料已单独提交（如需要）
#### 📋 补充材料检查清单 (Supplementary Material Checklist)
**内容组织**:
- [ ] 补充材料与主论文内容对应
- [ ] 章节结构清晰合理
- [ ] 图表编号与主论文不冲突
- [ ] 参考文献格式一致
**技术细节**:
- [ ] 算法伪代码清晰完整
- [ ] 实验设置详细说明
- [ ] 数据预处理步骤明确
- [ ] 超参数配置完整
**格式要求**:
- [ ] 使用统一的supp模板
- [ ] 页面设置与主论文一致
- [ ] 字体和格式符合要求
- [ ] 文件大小在限制范围内
### 实际使用建议
1. **投稿阶段**:
- 取消注释 `\def\aaaianonymous{true}`
- 确保不包含任何可能暴露身份的信息
- 检查参考文献是否已匿名化
2. **录用后准备final版本**:
- 注释掉或删除 `\def\aaaianonymous{true}` 这一行
- 添加完整的作者信息和affiliations
- 取消注释links环境（如果需要）
3. **编译测试**:
- 分别在两种模式下编译，确保都能正常工作
- 检查输出的PDF是否符合要求
- 验证参考文献格式是否正确
4. **依赖文件确认**:
- 确保所有依赖文件都在同一目录下
- 如果移动模板文件，记得同时移动依赖文件
### 重要注意事项
⚠️ **关于Bibliography Style**:
- `aaai2026.sty`文件已经自动设置了`\bibliographystyle{aaai2026}`
- **不要**在文档中再次添加`\bibliographystyle{aaai2026}`命令
- 否则会出现"`Illegal, another \bibstyle command`"错误
- 只需要使用`\bibliography{aaai2026}`命令即可
### 编译命令示例
```bash
# 编译LaTeX文档
pdflatex aaai2026-unified-template.tex
bibtex aaai2026-unified-template
pdflatex aaai2026-unified-template.tex
pdflatex aaai2026-unified-template.tex
```
### 常见问题解决
#### 1. "Illegal, another \bibstyle command"错误
**原因**: 重复设置了bibliography style
**解决方案**: 删除文档中的`\bibliographystyle{aaai2026}`命令，`aaai2026.sty`会自动处理
#### 2. 参考文献格式不正确
**原因**: 可能缺少natbib包或者BibTeX文件问题
**解决方案**: 确保按照标准的LaTeX编译流程：pdflatex → bibtex → pdflatex → pdflatex
---
## English Version
### Overview ✅
I have **completely merged** the two AAAI 2026 versions (anonymous submission and camera-ready) into a single unified template file `aaai2026-unified-template.tex`.
This template contains **all complete content** from both original templates (886 lines total, more comprehensive than the original files), including:
- All formatting instructions and requirements
- Complete example codes and tables
- Image processing guidelines
- Reference formatting requirements
- All sections and appendix content
- Version-specific Acknowledgments sections
### Key Differences Analysis
By comparing the two original templates, the main differences are:
#### 1. Package Loading Method
- **Anonymous version**: `\usepackage[submission]{aaai2026}`
- **Camera-ready version**: `\usepackage{aaai2026}`
#### 2. Title Differences
- **Anonymous version**: "AAAI Press Anonymous Submission Instructions for Authors Using LaTeX"
- **Camera-ready version**: "AAAI Press Formatting Instructions for Authors Using LaTeX --- A Guide"
#### 3. Links Environment Handling
- **Anonymous version**: Links environment commented out to prevent identity disclosure
- **Camera-ready version**: Links environment displayed normally
#### 4. Content Section Differences
- **Anonymous version**: Contains special instructions in "Preparing an Anonymous Submission" section
- **Camera-ready version**: Contains complete formatting instructions and copyright information
### Dependency Files Verification
✅ **Files verified and copied to main directory**:
- `aaai2026.sty` - AAAI 2026 style file (identical in both versions)
- `aaai2026.bst` - Bibliography style file (identical in both versions)
- `aaai2026.bib` - Sample bibliography file
- `figure1.pdf` and `figure2.pdf` - Sample image files
All these files are identical in both versions, so the unified template works properly.
### How to Use the Unified Template
#### Switch to Anonymous Submission Version
On line 11 of the template file, **uncomment** this line:
```latex
\def\aaaianonymous{true}
```
#### Switch to Camera-ready Version
On line 11 of the template file, **comment out** or **delete** this line:
```latex
% \def\aaaianonymous{true}
```
### Core Mechanism of One-Click Switching
The unified template uses LaTeX conditional compilation:
```latex
% Conditional package loading
\ifdefined\aaaianonymous
\usepackage[submission]{aaai2026} % Anonymous version
\else
\usepackage{aaai2026} % Camera-ready version
\fi
% Conditional title setting
\ifdefined\aaaianonymous
\title{AAAI Press Anonymous Submission\\Instructions for Authors Using \LaTeX{}}
\else
\title{AAAI Press Formatting Instructions \\for Authors Using \LaTeX{} --- A Guide}
\fi
% Conditional content display
\ifdefined\aaaianonymous
% Anonymous version specific content
\else
% Camera-ready version specific content
\fi
```
### File List
The main directory now contains the following files:
- `aaai2026-unified-template.tex` - Unified main paper template file
- `aaai2026-unified-supp.tex` - Unified supplementary material template file
- `aaai2026.sty` - AAAI 2026 LaTeX style file
- `aaai2026.bst` - Bibliography style file
- `aaai2026.bib` - Sample bibliography file
- `figure1.pdf` - Sample image 1
- `figure2.pdf` - Sample image 2
- `README.md` - This documentation
### Supplementary Material Template
#### Overview
`aaai2026-unified-supp.tex` is a unified template specifically designed for AAAI 2026 supplementary materials, using the same version switching mechanism as the main paper template.
#### Key Features
- **Version Switching**: Switch between anonymous submission and camera-ready versions by modifying one line of code
- **Supplementary Content Support**: Supports additional experiments, derivations, data, figures, algorithms, etc.
- **Format Consistency**: Maintains complete format consistency with the main paper template
- **Code Examples**: Includes examples for algorithms, code listings, and other supplementary materials
#### Usage
Same as the main paper template, just modify line 11:
```latex
% Anonymous submission version
\def\aaaianonymous{true}
% Camera-ready version
% \def\aaaianonymous{true}
```
#### Supplementary Material Content Suggestions
- Additional experimental results and ablation studies
- Detailed mathematical derivations and proofs
- More figures and visualizations
- Algorithm pseudocode and implementation details
- Dataset descriptions and preprocessing steps
- Hyperparameter settings and experimental configurations
- Failure case analysis
- Computational complexity analysis
### Usage Checklist
#### 📋 Pre-Submission Checklist
**Version Setup**:
- [ ] Set `\def\aaaianonymous{true}` (anonymous submission)
- [ ] Commented out all information that could reveal identity
- [ ] Anonymized references (removed author names)
**Content Completeness**:
- [ ] Title, abstract, and keywords filled
- [ ] All sections complete
- [ ] Figure and table numbers consecutive and correct
- [ ] Reference format correct
- [ ] Supplementary materials prepared (if any)
**Format Check**:
- [ ] Page margins meet requirements
- [ ] Font and font size correct
- [ ] Line spacing meets standards
- [ ] Figure and table positions and sizes appropriate
- [ ] Mathematical formula format correct
**Technical Check**:
- [ ] LaTeX compilation error-free
- [ ] References generated correctly
- [ ] PDF output normal
- [ ] File size within limits
#### 📋 Post-Acceptance Checklist
**Version Switch**:
- [ ] Commented out `\def\aaaianonymous{true}` (camera-ready)
- [ ] Added complete author information
- [ ] Added all author affiliation information
- [ ] Restored all commented content
**Content Updates**:
- [ ] Modified content according to reviewer comments
- [ ] Updated all figures and experiments
- [ ] Completed supplementary materials
- [ ] Checked all links and citations
**Final Check**:
- [ ] Final PDF quality check
- [ ] All files backed up
- [ ] Meets conference final submission requirements
- [ ] Supplementary materials submitted separately (if needed)
#### 📋 Supplementary Material Checklist
**Content Organization**:
- [ ] Supplementary materials correspond to main paper content
- [ ] Chapter structure clear and reasonable
- [ ] Figure and table numbers don't conflict with main paper
- [ ] Reference format consistent
**Technical Details**:
- [ ] Algorithm pseudocode clear and complete
- [ ] Experimental setup explained in detail
- [ ] Data preprocessing steps clear
- [ ] Hyperparameter configuration complete
**Format Requirements**:
- [ ] Using unified supp template
- [ ] Page settings consistent with main paper
- [ ] Font and format meet requirements
- [ ] File size within limits
### Practical Usage Recommendations
1. **Submission Stage**:
- Uncomment `\def\aaaianonymous{true}`
- Ensure no information that could reveal identity is included
- Check that references are anonymized
2. **Preparing final version after acceptance**:
- Comment out or delete the `\def\aaaianonymous{true}` line
- Add complete author information and affiliations
- Uncomment links environment (if needed)
3. **Compilation Testing**:
- Compile in both modes to ensure proper functionality
- Check if the output PDF meets requirements
- Verify reference formatting is correct
4. **Dependency File Confirmation**:
- Ensure all dependency files are in the same directory
- Remember to move dependency files when moving the template file
### Important Notes
⚠️ **About Bibliography Style**:
- The `aaai2026.sty` file automatically sets `\bibliographystyle{aaai2026}`
- **Do NOT** add `\bibliographystyle{aaai2026}` command again in your document
- Otherwise you'll get "`Illegal, another \bibstyle command`" error
- Just use the `\bibliography{aaai2026}` command
### Compilation Commands Example
```bash
# Compile LaTeX document
pdflatex aaai2026-unified-template.tex
bibtex aaai2026-unified-template
pdflatex aaai2026-unified-template.tex
pdflatex aaai2026-unified-template.tex
```
### Common Issues and Solutions
#### 1. "Illegal, another \bibstyle command" Error
**Cause**: Duplicate bibliography style setting
**Solution**: Remove the `\bibliographystyle{aaai2026}` command from your document, `aaai2026.sty` handles it automatically
#### 2. Incorrect Reference Format
**Cause**: Missing natbib package or BibTeX file issues
**Solution**: Follow the standard LaTeX compilation process: pdflatex → bibtex → pdflatex → pdflatex
---
## 版本信息 / Version Information
- **模板版本 / Template Version**: AAAI 2026 Unified (Main + Supplementary)
- **创建日期 / Created**: 2024年12月
- **支持格式 / Supported Formats**: Anonymous Submission & Camera-Ready
- **模板类型 / Template Types**: Main Paper Template & Supplementary Material Template
- **兼容性 / Compatibility**: LaTeX 2020+ / TeXLive 2024+
---
🎉 **现在您只需要修改一行代码就可以在两个版本之间切换，同时所有必要的依赖文件都已经准备就绪！**
🎉 **Now you only need to modify one line of code to switch between the two versions, with all necessary dependency files ready to use!**
--- METRICS ---
- Files Extracted: 10
- Estimated Token Budget: ~33773 tokens
- Recency Window: Active (< 180 days)
- Canonical Reference: https://codewiki.google/github.com/Orchestra-Research/AI-Research-SKILLs