# Technical Documentation: Orchestra-Research/AI-Research-SKILLs > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/Orchestra-Research/AI-Research-SKILLs) ### Overview (`README.md`) # AI Research `Skills` Library > **The most comprehensive open-source skills library enabling AI agents to autonomously conduct AI research — from idea to paper**

AI Research Skills Demo

License: MIT npm version Blog Post Slack Twitter LinkedIn

### **98 Skills Powering AI Research in 2026**
View All 23 Categories
| | | | |:---:|:---:|:---:| | **Autoresearch** (1) | **Ideation** (2) | **ML Paper Writing** (2) | | **Model Architecture** (5) | **Fine-Tuning** (4) | **Post-Training** (8) | | **Distributed Training** (6) | **Optimization** (6) | **Inference** (4) | | **Tokenization** (2) | **Data Processing** (2) | **Evaluation** (3) | | **Safety & Alignment** (4) | **Agents** (4) | **RAG** (5) | | **Multimodal** (7) | **Prompt Engineering** (4) | **MLOps** (3) | | **Observability** (2) | **Infrastructure** (3) | **Mech Interp** (4) | | **Emerging Techniques** (6) | **Agent-Native Research Artifact** (3) | |
--- ## Table of Contents - [Our Mission](#our-mission) - [Path Towards AI Research Agent](#path-towards-ai-research-agent) - [Available AI Research Engineering Skills](#available-ai-research-engineering-skills) - [Demos](#demos) - [Skill Structure](#skill-structure) - [Roadmap](#roadmap) - [Repository Structure](#repository-structure) - [Use Cases](#use-cases) - [Contributors](#contributors) - [Citation](#citation) - [Community](#community) ## Our Mission We enable AI agents to **autonomously conduct AI research** — from literature survey and idea generation through experiment execution to paper writing. The library provides both the **research orchestration layer** (autoresearch, ideation, paper writing) and the **engineering skills** (training, evaluation, deployment) needed at each stage.

AI Research Agent System
System diagram of an AI research agent

## Path Towards AI Research Agent Modern AI research requires mastering dozens of specialized tools and frameworks. AI Researchers spend more time debugging infrastructure than testing hypotheses — slowing the pace of scientific discovery. We provide a comprehensive skills library that enables AI agents to autonomously conduct the full research lifecycle — from brainstorming ideas to writing the paper. - Autonomous Research - The **autoresearch** skill orchestrates the entire research workflow using a two-loop architecture, routing to domain skills as needed - Specialized Expertise - Each domain skill provides deep, production-ready knowledge of a specific framework (Megatron-LM, vLLM, TRL, etc.) - End-to-End Coverage - 98 skills spanning the full AI research lifecycle, from ideation and literature survey to experiments and paper writing - Research-Grade Quality - Documentation sourced from official repos, real GitHub issues, and battle-tested production workflows ## Available AI Research Engineering Skills **Quality over quantity**: Each skill provides comprehensive, expert-level guidance with real code examples, troubleshooting guides, and production-ready workflows. ### 📦 Quick Install (Recommended) **For humans** — interactive installer with one command: ```bash npx @orchestra-research/ai-research-skills ``` **For AI agents** — point your agent to the welcome doc and it handles the rest: ``` Read https://www.orchestra-research.com/ai-research-skills/welcome.md and follow the instructions to install and use AI Research Skills. ``` This installs all 98 skills, loads the **autoresearch** orchestration layer, and starts autonomous research.
What the installer does - **Auto-detects** your installed coding agents (Claude Code, Hermes Agent, OpenCode, Qoder, Cursor, Gemini CLI, etc.) - **Installs** skills to `~/.orchestra/skills/` with symlinks to each agent (falls back to copy on Windows) - **Offers** everything, quickstart bundle, by category, or individual skills - **Updates** installed skills with latest versions - **Uninstalls** all or selected skills
CLI Commands ```bash # Interactive installer (recommended) npx @orchestra-research/ai-research-skills # Direct commands npx @orchestra-research/ai-research-skills list # View installed skills npx @orchestra-research/ai-research-skills update # Update installed skills ```
Claude Code Marketplace (Alternative) Install skill categories directly using the **Claude Code CLI**: ```bash # Add the marketplace /plugin marketplace add orchestra-research/AI-research-SKILLs # Install by category (23 categories available) /plugin install fine-tuning@ai-research-skills # Axolotl, LLaMA-Factory, PEFT, Unsloth /plugin install post-training@ai-research-skills # TRL, GRPO, OpenRLHF, SimPO, verl, slime, miles, torchforge /plugin install inference-serving@ai-research-skills # vLLM, TensorRT-LLM, llama.cpp, SGLang /plugin install distributed-training@ai-research-skills /plugin install optimization@ai-research-skills ```
### All 23 Categories (98 Skills) | Category | Skills | Included | |----------|--------|----------| | **Autoresearch** | **1** | **Autonomous research orchestration — central layer that manages the full lifecycle and routes to all other skills** | | Ideation | 2 | Research Brainstorming, Creative Thinking | | ML Paper Writing | 2 | ML Paper Writing (LaTeX templates, citation verification), Academic Plotting | | Model Architecture | 5 | LitGPT, Mamba, NanoGPT, RWKV, TorchTitan | | Tokenization | 2 | HuggingFace Tokenizers, SentencePiece | | Fine-Tuning | 4 | Axolotl, LLaMA-Factory, PEFT, Unsloth | | Mech Interp | 4 | TransformerLens, SAELens, pyvene, nnsight | | Data Processing | 2 | NeMo Curator, Ray Data | | Post-Training | 8 | TRL, GRPO, OpenRLHF, SimPO, verl, slime, miles, torchforge | | Safety | 4 | Constitutional AI, LlamaGuard, NeMo Guardrails, Prompt Guard | | Distributed | 6 | DeepSpeed, FSDP, Accelerate, Megatron-Core, Lightning, Ray Train | | Infrastructure | 3 | Modal, Lambda Labs, SkyPilot | | Optimization | 6 | Flash Attention, bitsandbytes, GPTQ, AWQ, HQQ, GGUF | | Evaluation | 3 | lm-eval-harness, BigCode, NeMo Evaluator | | Inference | 4 | vLLM, TensorRT-LLM, llama.cpp, SGLang | | MLOps | 3 | W&B, MLflow, TensorBoard | | Agents | 4 | LangChain, LlamaIndex, CrewAI, AutoGPT | | RAG | 5 | Chroma, FAISS, Pinecone, Qdrant, Sentence Transformers | | Prompt Eng | 4 | DSPy, Instructor, Guidance, Outlines | | Observability | 2 | LangSmith, Phoenix | | Multimodal | 7 | CLIP, Whisper, LLaVA, BLIP-2, SAM, Stable Diffusion, AudioCraft | | Emerging | 6 | MoE, Model Merging, Long Context, Speculative Decoding, Distillation, Pruning | | Agent-Native Research Artifact | 3 | ARA Compiler, Research Manager, Rigor Reviewer |
View All 98 Skills in Details ### 🔬 Autoresearch (1 skill) — Central Orchestration Layer - **[Autoresearch](0-autoresearch-skill/)** - Autonomous research orchestration using a two-loop architecture (inner optimization + outer synthesis). Manages the full lifecycle from literature survey to paper writing, routing to all domain-specific skills. Supports Claude Code /loop and OpenClaw heartbeat for continuous operation (390 lines + 3 refs) ### 🏗️ Model Architecture (5 skills) - **[LitGPT](01-model-architecture/litgpt/)** - Lightning AI's 20+ clean LLM implementations with production training recipes (462 lines + 4 refs) - **[Mamba](01-model-architecture/mamba/)** - State-space models with O(n) complexity, 5× faster than Transformers (253 lines + 3 refs) - **[RWKV](01-model-architecture/rwkv/)** - RNN+Transformer hybrid, infinite context, Linux Foundation project (253 lines + 3 refs) - **[NanoGPT](01-model-architecture/nanogpt/)** - Educational GPT in ~300 lines by Karpathy (283 lines + 3 refs) - **[TorchTitan](01-model-architecture/torchtitan/)** - PyTorch-native distributed training for Llama 3.1 with 4D parallelism ### 🔤 Tokenization (2 skills) - **[HuggingFace Tokenizers](02-tokenization/huggingface-tokenizers/)** - Rust-based, <20s/GB, BPE/WordPiece/Unigram algorithms (486 lines + 4 refs) - **[SentencePiece](02-tokenization/sentencepiece/)** - Language-independent, 50k sentences/sec, used by T5/ALBERT (228 lines + 2 refs) ### 🎯 Fine-Tuning (4 skills) - **[Axolotl](03-fine-tuning/axolotl/)** - YAML-based fine-tuning with 100+ models (156 lines + 4 refs) - **[LLaMA-Factory](03-fine-tuning/llama-factory/)** - WebUI no-code fine-tuning (78 lines + 5 refs) - **[Unsloth](03-fine-tuning/unsloth/)** - 2x faster QLoRA fine-tuning (75 lines + 4 refs) - **[PEFT](03-fine-tuning/peft/)** - Parameter-efficient fine-tuning with LoRA, QLoRA, DoRA, 25+ methods (431 lines + 2 refs) ### 🔬 Mechanistic Interpretability (4 skills) - **[TransformerLens](04-mechanistic-interpretability/transformer-lens/)** - Neel Nanda's library for mech interp with HookPoints, activation caching (346 lines + 3 refs) - **[SAELens](04-mechanistic-interpretability/saelens/)** - Sparse Autoencoder training and analysis for feature discovery (386 lines + 3 refs) - **[pyvene](04-mechanistic-interpretability/pyvene/)** - Stanford's causal intervention library with declarative configs (473 lines + 3 refs) - **[nnsight](04-mechanistic-interpretability/nnsight/)** - Remote interpretability via NDIF, run experiments on 70B+ models (436 lines + 3 refs) ### 📊 Data Processing (2 skills) - **[Ray Data](05-data-processing/ray-data/)** - Distributed ML data processing, streaming execution, GPU support (318 lines + 2 refs) - **[NeMo Curator](05-data-processing/nemo-curator/)** - GPU-accelerated data curation, 16× faster deduplication (375 lines + 2 refs) ### 🎓 Post-Training (8 skills) - **[TRL Fine-Tuning](06-post-training/trl-fine-tuning/)** - Transformer Reinforcement Learning (447 lines + 4 refs) - **[GRPO-RL-Training](06-post-training/grpo-rl-training/)** (TRL) - Group Relative Policy Optimization with TRL (569 lines, **gold standard**) - **[OpenRLHF](06-post-training/openrlhf/)** - Full RLHF pipeline with Ray + vLLM (241 lines + 4 refs) - **[SimPO](06-post-training/simpo/)** - Simple Preference Optimization, no reference model needed (211 lines + 3 refs) - **[verl](06-post-training/verl/)** - ByteDance's HybridFlow RL framework, FSDP/Megatron + vLLM/SGLang backends (389 lines + 2 refs) - **[slime](06-post-training/slime/)** - THUDM's Megatron+SGLang framework powering GLM-4.x models (464 lines + 2 refs) - **[miles](06-post-training/miles/)** - Enterprise fork of slime with FP8, INT4, speculative RL for MoE training (315 lines + 2 refs) - **[torchforge](06-post-training/torchforge/)** - Meta's PyTorch-native RL with Monarch+TorchTitan+vLLM (380 lines + 2 refs) ### 🛡️ Safety & Alignment (4 skills) - **[Constitutional AI](07-safety-alignment/constitutional-ai/)** - AI-driven self-improvement via principles (282 lines) - **[LlamaGuard](07-safety-alignment/llamaguard/)** - Safety classifier for LLM inputs/outputs (329 lines) - **[NeMo Guardrails](07-safety-alignment/nemo-guardrails/)** - Programmable guardrails with Colang (289 lines) - **[Prompt Guard](07-safety-alignment/prompt-guard/)** - Meta's 86M prompt injection & jailbreak detector, 99%+ TPR, <2ms GPU (313 lines) ### ⚡ Distributed Training (6 skills) - **[Megatron-Core](08-distributed-training/megatron-core/)** - NVIDIA's framework for training 2B-462B param models with 47% MFU on H100 (359 lines + 4 refs) - **[DeepSpeed](08-distributed-training/deepspeed/)** - Microsoft's ZeRO optimization (137 lines + 9 refs) - **[PyTorch FSDP2](08-distributed-training/pytorch-fsdp2/)** - Fully Sharded Data Parallel v2 with `fully_shard` and DTensor (231 lines + 12 refs) - **[Accelerate](08-distributed-training/accelerate/)** - HuggingFace's 4-line distributed training API (324 lines + 3 refs) - **[PyTorch Lightning](08-distributed-training/pytorch-lightning/)** - High-level training framework with Trainer class (339 lines + 3 refs) - **[Ray Train](08-distributed-training/ray-train/)** - Multi-node orchestration and hyperparameter tuning (399 lines + 1 ref) ### 🚀 Optimization (6 skills) - **[Flash Attention](10-optimization/flash-attention/)** - 2-4x faster attention with memory efficiency (359 lines + 2 refs) - **[bitsandbytes](10-optimization/bitsandbytes/)** - 8-bit/4-bit quantization for 50-75% memory reduction (403 lines + 3 refs) - **[GPTQ](10-optimization/gptq/)** - 4-bit post-training quantization, 4× memory reduction, <2% accuracy loss (443 lines + 3 refs) - **[AWQ](10-optimization/awq/)** - Activation-aware weight quantization, 4-bit with minimal accuracy loss (310 lines + 2 refs) - **[HQQ](10-optimization/hqq/)** - Half-Quadratic Quantization, no calibration data needed, multi-backend (370 lines + 2 refs) - **[GGUF](10-optimization/gguf/)** - llama.cpp quantization format, K-quant methods, CPU/Metal inference (380 lines + 2 refs) ### 📊 Evaluation (3 skills) - **[lm-evaluation-harness](11-evaluation/lm-evaluation-harness/)** - EleutherAI's standard for benchmarking LLMs across 60+ tasks (482 lines + 4 refs) - **[BigCode Evaluation Harness](11-evaluation/bigcode-evaluation-harness/)** - Code model benchmarking with HumanEval, MBPP, MultiPL-E, pass@k metrics (406 lines + 3 refs) - **[NeMo Evaluator](11-evaluation/nemo-evaluator/)** - NVIDIA's enterprise platform for 100+ benchmarks across 18+ harnesses with multi-backend execution (454 lines + 4 refs) ### ☁️ Infrastructure (3 skills) - **[Modal](09-infrastructure/modal/)** - Serverless GPU cloud with Python-native API, T4-H200 on-demand (342 lines + 2 refs) - **[SkyPilot](09-infrastructure/skypilot/)** - Multi-cloud orchestration across 20+ providers with spot recovery (390 lines + 2 refs) - **[Lambda Labs](09-infrastructure/lambda-labs/)** - Reserved/on-demand GPU cloud with H100/A100, persistent filesystems (390 lines + 2 refs) ### 🔥 Inference & Serving (4 skills) - **[vLLM](12-inference-serving/vllm/)** - High-throughput LLM serving with PagedAttention (356 lines + 4 refs, **production-ready**) - **[TensorRT-LLM](12-inference-serving/tensorrt-llm/)** - NVIDIA's fastest inference, 24k tok/s, FP8/INT4 quantization (180 lines + 3 refs) - **[llama.cpp](12-inference-serving/llama-cpp/)** - CPU/Apple Silicon inference, GGUF quantization (251 lines + 3 refs) - **[SGLang](12-inference-serving/sglang/)** - Structured generation with RadixAttention, 5-10× faster for agents (435 lines + 3 refs) ### 🤖 Agents (4 skills) - **[LangChain](14-agents/langchain/)** - Most popular agent framework, 500+ integrations, ReAct pattern (658 lines + 3 refs, **production-ready**) - **[LlamaIndex](14-agents/llamaindex/)** - Data framework for LLM apps, 300+ connectors, RAG-focused (535 lines + 3 refs) - **[CrewAI](14-agents/crewai/)** - Multi-agent orchestration, role-based collaboration, autonomous workflows (498 lines + 3 refs) - **[AutoGPT](14-agents/autogpt/)** - Autonomous AI agent platform, visual workflow builder, continuous execution (400 lines + 2 refs) ### 🔍 RAG (5 skills) - **[Chroma](15-rag/chroma/)** - Open-source embedding database, local/cloud, 24k stars (385 lines + 1 ref) - **[FAISS](15-rag/faiss/)** - Facebook's similarity search, billion-scale, GPU acceleration (295 lines) - **[Sentence Transformers](15-rag/sentence-transformers/)** - 5000+ embedding models, multilingual, 15k stars (370 lines) - **[Pinecone](15-rag/pinecone/)** - Managed vector database, auto-scaling, <100ms latency (410 lines) - **[Qdrant](15-rag/qdrant/)** - High-performance vector search, Rust-powered, hybrid search with filtering (493 lines + 2 refs) ### 🎨 Multimodal (7 skills) - **[CLIP](18-multimodal/clip/)** - OpenAI's vision-language model, zero-shot classification, 25k stars (320 lines) - **[Whisper](18-multimodal/whisper/)** - Robust speech recognition, 99 languages, 73k stars (395 lines) - **[LLaVA](18-multimodal/llava/)** - Vision-language assistant, image chat, GPT-4V level (360 lines) - **[Stable Diffusion](18-multimodal/stable-diffusion/)** - Text-to-image generation via HuggingFace Diffusers, SDXL, ControlNet (380 lines + 2 refs) - **[Segment Anything](18-multimodal/segment-anything/)** - Meta's SAM for zero-shot image segmentation with points/boxes (500 lines + 2 refs) - **[BLIP-2](18-multimodal/blip-2/)** - Vision-language pretraining with Q-Former, image captioning, VQA (500 lines + 2 refs) - **[AudioCraft](18-multimodal/audiocraft/)** - Meta's MusicGen/AudioGen for text-to-music and text-to-sound (470 lines + 2 refs) ### 🎯 Prompt Engineering (4 skills) - **[DSPy](16-prompt-engineering/dspy/)** - Declarative prompt programming with optimizers, Stanford NLP, 22k stars (438 lines + 3 refs) - **[Instructor](16-prompt-engineering/instructor/)** - Structured LLM outputs with Pydantic validation, 15k stars (726 lines + 3 refs) - **[Guidance](16-prompt-engineering/guidance/)** - Constrained generation with regex/grammars, Microsoft Research, 18k stars (485 lines + 3 refs) - **[Outlines](16-prompt-engineering/outlines/)** - Structured text with FSM, zero-overhead, 8k stars (601 lines + 3 refs) ### 📊 MLOps (3 skills) - **[Weights & Biases](13-mlops/weights-and-biases/)** - Experiment tracking, sweeps, artifacts, model registry (427 lines + 3 refs) - **[MLflow](13-mlops/mlflow/)** - Model registry, tracking, deployment, autologging (514 lines + 3 refs) - **[TensorBoard](13-mlops/tensorboard/)** - Visualization, profiling, embeddings, scalars/images (538 lines + 3 refs) ### 👁️ Observability (2 skills) - **[LangSmith](17-observability/langsmith/)** - LLM observability, tracing, evaluation, monitoring for AI apps (422 lines + 2 refs) - **[Phoenix](17-observability/phoenix/)** - Open-source AI observability with OpenTelemetry tracing and LLM evaluation (380 lines + 2 refs) ### 🔬 Emerging Techniques (6 skills) - **[MoE Training](19-emerging-techniques/moe-training/)** - Mixture of Experts training with DeepSpeed, Mixtral 8x7B, 5× cost reduction (515 lines + 3 refs) - **[Model Merging](19-emerging-techniques/model-merging/)** - Combine models with TIES, DARE, SLERP using mergekit (528 lines + 3 refs) - **[Long Context](19-emerging-techniques/long-context/)** - Extend context windows with RoPE, YaRN, ALiBi, 32k-128k tokens (624 lines + 3 refs) - **[Speculative Decoding](19-emerging-techniques/speculative-decoding/)** - 1.5-3.6× faster inference with Medusa, Lookahead (379 lines) - **[Knowledge Distillation](19-emerging-techniques/knowledge-distillation/)** - Compress models 70B→7B with MiniLLM, temperature scaling (424 lines) - **[Model Pruning](19-emerging-techniques/model-pruning/)** - 50% sparsity with Wanda, SparseGPT, <1% accuracy loss (417 lines) ### 📝 ML Paper Writing (2 skills) - **[ML Paper Writing](20-ml-paper-writing/)** - Write publication-ready papers for NeurIPS, ICML, ICLR, ACL, AAAI, COLM with LaTeX templates, citation verification, and writing best practices (532 lines + 5 refs) - **[Academic Plotting](20-ml-paper-writing/academic-plotting/)** - Generate publication-quality figures for ML papers: architecture diagrams via Gemini AI and data-driven charts via matplotlib/seaborn with venue-specific styling (479 lines + 3 refs) ### 💡 Ideation (2 skills) - **[Research Brainstorming](21-research-ideation/brainstorming-research-ideas/)** - Structured ideation frameworks for discovering high-impact research directions with 10 complementary lenses (384 lines) - **[Creative Thinking](21-research-ideation/creative-thinking-for-research/)** - Cognitive science frameworks (bisociation, structure-mapping, constraint manipulation) for genuinely novel research ideas (366 lines) ### 🧬 Agent-Native Research Artifact (3 skills) - **[ARA Compiler](22-agent-native-research-artifact/compiler/)** - Compiles any research input (PDF papers, repos, experiment logs, raw notes) into a complete Agent-Native Research Artifact with claims, exploration graph, evidence, and code stubs (245 lines + 3 refs) - **[ARA Research Manager](22-agent-native-research-artifact/research-manager/)** - Post-task research recorder that runs at session end to extract decisions, experiments, dead ends, and pivots from conversation history into the `ara/` directory with user-vs-AI provenance tags (324 lines + 3 refs) - **[ARA Rigor Reviewer](22-agent-native-research-artifact/rigor-reviewer/)** - ARA Seal Level 2 semantic epistemic review scoring six dimensions of research rigor (evidence relevance, falsifiability, scope, coherence, exploration integrity, methodology) with severity-ranked findings (322 lines + 1 ref)
## Demos All 98 skills in this repo are automatically synced to [Orchestra Research](https://www.orchestra-research.com/research-skills), where you can add them to your projects with one click and use them with AI research agents. **See skills in action → [demos/](demos/README.md)** We maintain a curated collection of demo repositories showing how to use skills for real AI research tasks: | Demo | Skills Used | What It Does | |------|-------------|--------------| | **[Norm Heterogeneity → LoRA Brittleness](demos/autoresearch-norm-heterogeneity/)** | Autoresearch, ML Paper Writing, Ideation | Agent autonomously discovered norm heterogeneity predicts fine-tuning difficulty (r=-0.99), pivoting from a null result on ETF overlaps | | **[RL Algorithm Brain Scan](demos/autoresearch-rl-brain-scan/)** | Autoresearch, GRPO, TRL, SAELens, TransformerLens, ML Paper Writing | Agent found DPO is a rank-1 perturbation (95.6% recovery from one SVD direction) while online RL is distributed and structure-preserving | | **[NeMo Eval: GPQA Benchmark](https://github.com/zechenzhangAGI/Nemo-Eval-Skill-Demo)** | NeMo Evaluator | Compare Llama 8B/70B/405B on graduate-level science questions | | **[LoRA Without Regret Reproduction](https://www.orchestra-research.com/perspectives/LLM-with-Orchestra)** | GRPO, TRL | Reproduce SFT + GRPO RL experiments via prompting | | **[Layer-Wise Quantization Experiment](https://github.com/AmberLJC/llama-quantization-experiment)** | llama.cpp, GGUF | Investigate optimal layer precision allocation—early layers at Q8 achieve 1.9× compression with 1.3% perplexity loss | | **[Cross-Lingual Alignment Analysis](https://github.com/AmberLJC/faiss-demo)** | FAISS | Quantify how well multilingual embeddings align semantic concepts across 8 languages using FAISS similarity search | | **[Scientific Plotting Demo](demos/scientific-plotting-demo/)** | Academic Plotting | Generate publication-quality figures for the Andes QoE-aware LLM serving paper — Gemini AI architecture diagrams + matplotlib data charts (CDF, multi-panel grids, bar charts) | **Featured Demos**: Two papers produced entirely by AI agents using the **autoresearch** skill. The [Norm Heterogeneity paper](demos/autoresearch-norm-heterogeneity/) demonstrates autonomous research pivoting — the agent refuted its own hypothesis and discovered a stronger finding. The [RL Brain Scan paper](demos/autoresearch-rl-brain-scan/) demonstrates multi-skill orchestration — the agent trained RL models, analyzed internals with interpretability tools, and synthesized the insight that "DPO is rank-1 alignment." Both papers written end-to-end by the agent. ## Skill Structure Each skill follows a battle-tested format for maximum usefulness: ``` skill-name/ ├── SKILL.md # Quick reference (50-150 lines) │ ├── Metadata (name, description, version) │ ├── When to use this skill │ ├── Quick patterns & examples │ └── Links to references │ ├── references/ # Deep documentation (300KB+) │ ├── README.md # From GitHub/official docs │ ├── api.md # API reference │ ├── tutorials.md # Step-by-step guides │ ├── issues.md # Real GitHub issues & solutions │ ├── releases.md # Version history & breaking changes │ └── file_structure.md # Codebase navigation │ ├── scripts/ # Helper scripts (optional) └── assets/ # Templates & examples (optional) ```
Quality Standards - 300KB+ documentation from official sources - Real GitHub issues & solutions (when available) - Code examples with language detection - Version history & breaking changes - Links to official docs
## Roadmap The library spans 98 comprehensive skills across the full AI research lifecycle. See our [detailed roadmap](docs/ROADMAP.md) for the complete development plan. [View Full Roadmap →](docs/ROADMAP.md)
View Detailed Statistics | Metric | Current | Target | |--------|---------|--------| | **Skills** | **87** (high-quality, standardized YAML) | 80 ✅ | | **Avg Lines/Skill** | **420 lines** (focused + progressive disclosure) | 200-600 lines | | **Documentation** | **~130,000 lines** total (SKILL.md + references) | 100,000+ lines | | **Gold Standard Skills** | **65** with comprehensive references | 50+ | | **Contributors** | 1 | 100+ | | **Coverage** | Architecture, Tokenization, Fine-Tuning, Mechanistic Interpretability, Data Processing, Post-Training, Safety, Distributed, Optimization, Evaluation, Infrastructure, Inference, Agents, RAG, Multimodal, Prompt Engineering, MLOps, Observability, ML Paper Writing, Ideation, Autoresearch | Full Lifecycle ✅ | **Recent Progress**: npm package `@orchestra-research/ai-research-skills` for one-command installation across all coding agents **Philosophy**: Quality > Quantity. Following [Anthropic official best practices](anthropic_official_docs/best_practices.md) - each skill provides 200-500 lines of focused, actionable guidance with progressive disclosure.
## Repository Structure ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Use Cases ### For Researchers "I need to fine-tune Llama 3 with custom data" → **03-fine-tuning/axolotl/** - YAML configs, 100+ model support ### For ML Engineers "How do I optimize inference latency?" → **12-inference-serving/vllm/** - PagedAttention, batching ### For Students "I want to learn how transformers work" → **01-model-architecture/litgpt/** - Clean implementations ### For Teams "We need to scale training to 100 GPUs" → **08-distributed-training/deepspeed/** - ZeRO stages, 3D parallelism ## License MIT License - See [LICENSE](LICENSE) for details. **Note**: Individual skills may reference libraries with different licenses. Please check each project's license before use. ## Citation If you use AI Research Skills in your work or find it helpful for a publication, we'd appreciate a citation: **BibTeX** ```bibtex @software{ai_research_skills, title = {AI Research Skills Library}, author = {{Orchestra Research}}, year = {2025}, url = {https://github.com/orchestra-research/AI-research-SKILLs}, note = {Open-source skills library enabling AI agents to autonomously conduct AI research} } ``` **APA** > Orchestra Research. (2025). *AI Research Skills Library* [Computer software]. https://github.com/orchestra-research/AI-research-SKILLs **Chicago** > Orchestra Research. "AI Research Skills Library." GitHub, 2025. https://github.com/orchestra-research/AI-research-SKILLs. **IEEE** > Orchestra Research, "AI Research Skills Library," 2025. [Online]. Available: https://github.com/orchestra-research/AI-research-SKILLs > **Tip**: You can also click **"Cite this repository"** in the GitHub sidebar for auto-formatted citations. ## Acknowledgments Built with: - **[Claude Code](https://www.claude.com/product/claude-code)** - AI pair programming - **[Skill Seeker](https://github.com/yusufkaraaslan/Skill_Seekers)** - Automated doc scraping - **Open Source AI Community** - For amazing tools and docs Special thanks to: - EleutherAI, HuggingFace, NVIDIA, Lightning AI, Meta AI, Anthropic - All researchers who maintain excellent documentation ## Contributors Thanks to all the people who have contributed to the AI Research Skills Library: We welcome contributions from the AI research community! See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines on: - Adding new skills - Improving existing skills - Quality standards and best practices - Submission process ## Recent Updates
June 2026 - v1.7.1 🧹 Inventory Consistency, Drift Guard & Security Hardening - 📊 **Repo-wide inventory reconciled to 98 skills / 23 categories** — corrected stale counts that had drifted apart across files: CLAUDE.md (said 90), CONTRIBUTING.md (said 86/22), the README sync line (said 87), WELCOME.md and the npm package README (said 86/22). Also fixed wrong per-category listings (TorchTitan, SwanLab, A-Evolve, ML Training Recipes, Cosmos Policy/OpenPI/OpenVLA-OFT, the paper-writing skills) - 🛡️ **New CI drift guard** — `scripts/check-inventory.sh` + `check-inventory.yml` fail CI whenever the documented skill/category counts diverge from the actual `SKILL.md` count on disk, so the inventory can't silently drift again - 📦 **Marketplace sync hardening** — `sync-skills.yml` now prunes build artifacts (`node_modules`/`__pycache__`/`*.pyc`/`.ipynb_checkpoints`) before zipping and fails loudly above 190 files instead of hitting the marketplace's 200-file rejection - 🔒 **Security** — pinned CLI dependencies (`chalk`/`inquirer`/`ora`) to exact, patched versions with a regenerated lockfile (`inquirer@9.3.8` clears the `tmp` path-traversal advisory; `npm audit` now reports 0 vulnerabilities) - 🧹 Full open PR/issue triage pass against scope + contribution standards
April 2026 - v1.6.0 🧬 Agent-Native Research Artifact (ARA) — 23rd Category, 98 Skills - 🧬 **NEW CATEGORY**: `22-agent-native-research-artifact/` (the 23rd category) — three skills that turn research outputs into a falsifiable, agent-traversable artifact: - 🛠️ **[ARA Compiler](22-agent-native-research-artifact/compiler/)** — compiles any input (PDF papers, GitHub repos, experiment logs, raw notes) into a structured ARA with cognitive layer (claims, concepts, heuristics), physical layer (configs, code stubs), exploration graph (research DAG), and grounded evidence - 📋 **[ARA Research Manager](22-agent-native-research-artifact/research-manager/)** — post-task epilogue that scans conversation history at session end and writes decisions, experiments, dead ends, claims, heuristics, and pivots into the `ara/` directory with `user` / `ai-suggested` / `ai-executed` / `user-revised` provenance tags - 🔍 **[ARA Rigor Reviewer](22-agent-native-research-artifact/rigor-reviewer/)** — Seal Level 2 semantic epistemic review scoring six dimensions of research rigor (evidence relevance, falsifiability, scope calibration, argument coherence, exploration integrity, methodological rigor) and emitting a severity-ranked report with a Strong Accept-to-Reject recommendation - 🔗 Sourced from the [Agent-Native-Research-Artifact-Init](https://github.com/Orchestra-Research/Agent-Native-Research-Artifact-Init) reference repo, restructured to AI-research-SKILLs standards (kebab-case names, third-person descriptions, Title-Case tags, one-level-deep references) - 🧩 Plugin entry `agent-native-research-artifact` added to `.claude-plugin/marketplace.json`; CLI category registered as `22-agent-native-research-artifact` with three individual skill entries in the npm installer - 🔄 Auto-syncs to Orchestra marketplace via `sync-skills.yml` on push; npm package republished as `@orchestra-research/ai-research-skills@1.6.0` via `publish-npm.yml` on version bump - 📊 **98 total skills** across **23 categories** — full lifecycle from idea → paper → falsifiable, auditable artifact
March 2026 - v1.4.0 🔬 Autoresearch & 86 Skills — Full Research Lifecycle - 🔬 **NEW SKILL**: **Autoresearch** — autonomous research orchestration using a two-loop architecture (inner optimization loop + outer synthesis loop) - 🧠 Manages the full research lifecycle: literature survey → ideation → experiments → synthesis → paper writing - 🔄 Routes to all 86 domain skills automatically — agents don't need to know which skill to use - ⏰ Mandatory `/loop` (Claude Code) and cron job (OpenClaw) for continuous autonomous operation - 📊 Generates research presentations (HTML/PDF) with optimization trajectory plots for human review - 📝 Findings.md as persistent project memory across sessions with "Lessons and Constraints" tracking - 🗂️ Structured workspace: research-state.yaml, findings.md, research-log.md, literature/, experiments/, src/, data/, to_human/ - 📄 **Two demo papers produced by autoresearch**: [Norm Heterogeneity → LoRA Brittleness](demos/autoresearch-norm-heterogeneity/) and [RL Algorithm Brain Scan](demos/autoresearch-rl-brain-scan/) - 🚀 WELCOME.md for cold-start agent bootstrap — one URL to go from zero to autonomous research - 📦 npm v1.4.x with Windows symlink fallback, all 22 categories installable - 🤖 **Supported agents**: Claude Code, Hermes Agent, OpenCode, OpenClaw, Qoder, Cursor, Codex, Gemini CLI, Qwen Code - 📊 **87 total skills** across **22 categories** — complete research lifecycle coverage
February 2026 - v0.15.0 🛡️ Prompt Guard & 83 Skills - 🛡️ **NEW SKILL**: Prompt Guard - Meta's 86M prompt injection & jailbreak detector - ⚡ 99%+ TPR, <1% FPR, <2ms GPU latency, multilingual (8 languages) - 🔒 3 workflows: user input filtering, third-party data filtering, batch RAG processing - 📊 **83 total skills** across 20 categories
January 2026 - v0.14.0 📦 npm Package & 82 Skills - 📦 **NEW**: `npx @orchestra-research/ai-research-skills` - One-command installation for all coding agents - 🤖 **Supported agents**: Claude Code, OpenCode, Cursor, Codex, Gemini CLI, Qwen Code - ✨ Interactive installer with category/individual skill selection - 🔄 Update installed skills, selective uninstall - 📊 **82 total skills** (5 new post-training skills: verl, slime, miles, torchforge + TorchTitan) - 🏗️ Megatron-Core moved to Distributed Training category
January 2026 - v0.13.0 📝 ML Paper Writing & Demos Gallery - 📝 **NEW CATEGORY**: ML Paper Writing (20th category, 77th skill) - 🎯 Write publication-ready papers for NeurIPS, ICML, ICLR, ACL, AAAI, COLM - 📚 Writing philosophy from top researchers (Neel Nanda, Farquhar, Gopen & Swan, Lipton, Perez) - 🔬 Citation verification workflow - never hallucinate references - 📄 LaTeX templates for 6 major conferences - 🎪 **NEW**: Curated demos gallery (`demos/`) showcasing skills in action - 🔗 Demo repos: NeMo Evaluator benchmark, LoRA Without Regret reproduction - 📖 936-line comprehensive SKILL.md with 4 workflows
January 2026 - v0.12.0 📊 NeMo Evaluator SDK - 📊 **NEW SKILL**: NeMo Evaluator SDK for enterprise LLM benchmarking - 🔧 NVIDIA's evaluation platform with 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) - ⚡ Multi-backend execution: local Docker, Slurm HPC, Lepton cloud - 📦 Container-first architecture for reproducible evaluation - 📝 454 lines SKILL.md + 4 comprehensive reference files (~48KB documentation)
December 2025 - v0.11.0 🔬 Mechanistic Interpretability - 🔬 **NEW CATEGORY**: Mechanistic Interpretability (4 skills) - 🔍 TransformerLens skill: Neel Nanda's library for mech interp with HookPoints, activation caching, circuit analysis - 🧠 SAELens skill: Sparse Autoencoder training and analysis for feature discovery, monosemanticity research - ⚡ pyvene skill: Stanford's causal intervention library with declarative configs, DAS, activation patching - 🌐 nnsight skill: Remote interpretability via NDIF, run experiments on 70B+ models without local GPUs - 📝 ~6,500 new lines of documentation across 16 files - **76 total skills** (filling the missing 04 category slot)
November 25, 2025 - v0.10.0 🎉 70 Skills Complete! - 🎉 **ROADMAP COMPLETE**: Reached 70-skill milestone! - 🚀 Added 4 skills: Lambda Labs, Segment Anything (SAM), BLIP-2, AudioCraft - ☁️ Lambda Labs skill: Reserved/on-demand GPU cloud with H100/A100, persistent filesystems, 1-Click Clusters - 🖼️ SAM skill: Meta's Segment Anything for zero-shot image segmentation with points/boxes/masks - 👁️ BLIP-2 skill: Vision-language pretraining with Q-Former, image captioning, VQA - 🎵 AudioCraft skill: Meta's MusicGen/AudioGen for text-to-music and text-to-sound generation - 📝 ~10,000 new lines of documentation across 12 files - **70 total skills** (100% roadmap complete!)
November 25, 2025 - v0.9.0 - 🚀 Added 2 infrastructure skills: Modal, SkyPilot - ☁️ Modal skill: Serverless GPU cloud with Python-native API, T4-H200 on-demand, auto-scaling - 🌐 SkyPilot skill: Multi-cloud orchestration across 20+ providers with spot recovery - ✨ New Infrastructure category (2 skills - serverless GPU and multi-cloud orchestration) - 📝 ~2,500 new lines of documentation across 6 files - **66 total skills** (94% towards 70-skill target)
November 25, 2025 - v0.8.0 - 🚀 Added 5 high-priority skills: HQQ, GGUF, Phoenix, AutoGPT, Stable Diffusion - ⚡ HQQ skill: Half-Quadratic Quantization without calibration data, multi-backend support - 📦 GGUF skill: llama.cpp quantization format, K-quant methods, CPU/Metal inference - 👁️ Phoenix skill: Open-source AI observability with OpenTelemetry tracing and LLM evaluation - 🤖 AutoGPT skill: Autonomous AI agent platform with visual workflow builder - 🎨 Stable Diffusion skill: Text-to-image generation via Diffusers, SDXL, ControlNet, LoRA - 📝 ~9,000 new lines of documentation across 15 files - **64 total skills** (91% towards 70-skill target)
November 25, 2025 - v0.7.0 - 🚀 Added 5 high-priority skills: PEFT, CrewAI, Qdrant, AWQ, LangSmith - ✨ New Observability category with LangSmith for LLM tracing and evaluation - 🎯 PEFT skill: Parameter-efficient fine-tuning with LoRA, QLoRA, DoRA, 25+ methods - 🤖 CrewAI skill: Multi-agent orchestration with role-based collaboration - 🔍 Qdrant skill: High-performance Rust vector search with hybrid filtering - ⚡ AWQ skill: Activation-aware 4-bit quantization with minimal accuracy loss - 📝 ~8,000 new lines of documentation across 15 files - **59 total skills** (84% towards 70-skill target)
November 15, 2025 - v0.6.0 - 📊 Added 3 comprehensive MLOps skills: Weights & Biases, MLflow, TensorBoard - ✨ New MLOps category (3 skills - experiment tracking, model registry, visualization) - 📝 ~10,000 new lines of documentation across 13 files - 🔧 Comprehensive coverage: experiment tracking, hyperparameter sweeps, model registry, profiling, embeddings visualization - **54 total skills** (77% towards 70-skill target)
November 12, 2025 - v0.5.0 - 🎯 Added 4 comprehensive prompt engineering skills: DSPy, Instructor, Guidance, Outlines - ✨ New Prompt Engineering category (4 skills - DSPy, Instructor, Guidance, Outlines) - 📝 ~10,000 new lines of documentation across 16 files - 🔧 Comprehensive coverage: declarative programming, structured outputs, constrained generation, FSM-based generation - **47 total skills** (67% towards 70-skill target)
November 9, 2025 - v0.4.0 - 🤖 Added 11 comprehensive skills: LangChain, LlamaIndex, Chroma, FAISS, Sentence Transformers, Pinecone, CLIP, Whisper, LLaVA - ✨ New Agents category (2 skills - LangChain, LlamaIndex) - 🔍 New RAG category (4 skills - Chroma, FAISS, Sentence Transformers, Pinecone) - 🎨 New Multimodal category (3 skills - CLIP, Whisper, LLaVA) - 📝 ~15,000 new lines of documentation - **43 total skills** (61% towards 70-skill target)
November 8, 2025 - v0.3.0 - 🚀 Added 8 comprehensive skills: TensorRT-LLM, llama.cpp, SGLang, GPTQ, HuggingFace Tokenizers, SentencePiece, Ray Data, NeMo Curator - ⚡ Completed Inference & Serving category (4/4 skills) - 🔤 New Tokenization category (2 skills) - 📊 New Data Processing category (2 skills) - 📝 9,617 new lines of documentation across 30 files - **32 total skills** (45% towards 70-skill target)
November 6, 2025 - v0.2.0 - Added 10 skills from GitHub (Megatron-Core, Lightning, Ray Train, etc.) - Improved skill structure with comprehensive references - Created strategic roadmap to 70 skills - Added contribution guidelines
November 3, 2025 - v0.1.0 - 🎉 Initial release with 5 fine-tuning skills
## Community Join our community to stay updated, ask questions, and connect with other AI researchers: - **[SkillEvolve Meta-Skill](https://github.com/Skill-Evolve/meta-skill)** - Connect your agent to the collective intelligence of the community. Captures techniques discovered during sessions and shares them back as curated skills. - **[Slack Community](https://join.slack.com/t/orchestrarese-efu1990/shared_invite/zt-3iu6gr8io-zJvpkZTPToEviQ9KFZvNSg)** - Chat with the team and other users - **[Twitter/X](https://x.com/orch_research)** - Follow for updates and announcements - **[LinkedIn](https://www.linkedin.com/company/orchestra-research/)** - Connect professionally ## Star History Star History Chart --- ### Overview (`demos/README.md`) # AI Research Skills - Demo Gallery > **Curated collection of demo repositories showcasing skills in action** Each demo is a standalone repository demonstrating how to use specific skills from this library to accomplish real AI research tasks. Demos include complete code, results, analysis, and documentation. --- ## Available Demos ### 1. NeMo Evaluator: GPQA Diamond Benchmark **Repository:** [zechenzhangAGI/Nemo-Eval-Skill-Demo](https://github.com/zechenzhangAGI/Nemo-Eval-Skill-Demo) **Skills Used:** [NeMo Evaluator](../11-evaluation/nemo-evaluator/) **What It Does:** Compares Llama models (8B, 70B, 405B) on the GPQA Diamond benchmark—198 graduate-level science questions. Demonstrates end-to-end evaluation workflow using NVIDIA NeMo Evaluator. **Key Results:** | Model | Accuracy | Notes | |-------|----------|-------| | Llama-3.1-8B-Instruct | 27.3% | 20.7% extraction failures | | Llama-3.3-70B-Instruct | 48.0% | Clean extraction | | Llama-3.1-405B-Instruct | 53.0% | Best performance | **What You'll Learn:** - Setting up NeMo Evaluator with NVIDIA Build API - Writing evaluation configs for different models - Analyzing benchmark results across model scales - Creating visualizations (accuracy plots, Venn diagrams, failure taxonomy) **Repository Contents:** ``` ├── configs/ # YAML configs for each model ├── results/ # Raw evaluation outputs ├── analysis/ # Analysis scripts and visualizations │ ├── model_accuracy.png │ ├── failure_taxonomy_plot.png │ └── venn_diagrams.png └── README.md # Full documentation ``` --- ### 2. Reproducing "LoRA Without Regret" with AI Agents **Repository:** Featured on [Orchestra Research Blog](https://www.orchestra-research.com/perspectives/LLM-with-Orchestra) **Skills Used:** [GRPO RL Training](../06-post-training/grpo-rl-training/), [TRL Fine-Tuning](../06-post-training/trl-fine-tuning/) **What It Does:** Reproduces Thinking Machines Lab's "LoRA Without Regret" paper findings **entirely through prompting an AI agent**. The agent autonomously: - Writes training code for both SFT and GRPO reinforcement learning - Provisions H100 GPUs and runs experiments overnight - Performs LoRA rank ablation studies (rank 1 through 256) - Generates publication-ready analysis and visualizations **Why It's Impressive:** A researcher simply described the paper they wanted to reproduce, and the AI agent handled everything—from understanding the methodology to executing multi-day GPU experiments to analyzing results. No manual coding required. **What You'll Learn:** - How to prompt AI agents for autonomous research reproduction - End-to-end SFT and GRPO training pipelines - LoRA vs full fine-tuning experimental design - Automated analysis and reporting **Resources:** - [Blog Post](https://www.orchestra-research.com/perspectives/LLM-with-Orchestra) - Full walkthrough - [Video Demo](https://www.youtube.com/watch?v=X0DoLYfXl5I) - See the agent in action --- ### 3. Layer-Wise Quantization Experiment **Repository:** [AmberLJC/llama-quantization-experiment](https://github.com/AmberLJC/llama-quantization-experiment) **Skills Used:** [llama.cpp](../12-inference-serving/llama-cpp/), [GGUF](../10-optimization/gguf/) **What It Does:** Investigates optimal layer precision allocation for quantized LLMs. Demonstrates that early layers at Q8 achieve 1.9× compression with only 1.3% perplexity loss—showing not all layers are created equal when it comes to quantization. **What You'll Learn:** - Layer-wise quantization strategies for LLMs - Measuring perplexity impact of different precision levels per layer - Using llama.cpp and GGUF for quantization experiments - Identifying which layers are most sensitive to reduced precision --- ### 4. Cross-Lingual Alignment Analysis **Repository:** [AmberLJC/faiss-demo](https://github.com/AmberLJC/faiss-demo) **Skills Used:** [FAISS](../15-rag/faiss/) **What It Does:** Quantifies how well multilingual embeddings align semantic concepts across 8 languages using FAISS similarity search. Reveals the structure of cross-lingual representations and where alignment breaks down. **What You'll Learn:** - Building and querying FAISS indexes for multilingual embeddings - Measuring cross-lingual semantic alignment quality - Analyzing embedding space structure across languages - Using similarity search to evaluate multilingual models --- ### 5. Autoresearch: Embedding Norm Heterogeneity Drives LoRA Brittleness **Paper:** [autoresearch-norm-heterogeneity/](autoresearch-norm-heterogeneity/) **Skills Used:** [Autoresearch](../0-autoresearch-skill/), [ML Paper Writing](../20-ml-paper-writing/), [Research Ideation](../21-research-ideation/) **What It Does:** An AI agent ran the full autoresearch workflow autonomously. Starting from a hypothesis about ETF crystallization, the agent discovered a null result — ETF overlaps do NOT predict fine-tuning difficulty — then **pivoted** to identify embedding norm heterogeneity as the actual causal predictor (r=-0.99 at 1.4B scale). The agent wrote the paper end-to-end. **Why It's Impressive:** The research pivot was autonomous. The agent refuted its own starting hypothesis, identified a better predictor, validated it causally (equalizing norms improves fine-tunability by 79%), and wrote a paper with a stronger finding than the original plan. --- ### 6. Autoresearch: The RL Algorithm Brain Scan **Paper:** [autoresearch-rl-brain-scan/](autoresearch-rl-brain-scan/) **Skills Used:** [Autoresearch](../0-autoresearch-skill/), [GRPO RL Training](../06-post-training/grpo-rl-training/), [TRL](../06-post-training/trl-fine-tuning/), [SAELens](../04-mechanistic-interpretability/saelens/), [TransformerLens](../04-mechanistic-interpretability/transformer-lens/), [ML Paper Writing](../20-ml-paper-writing/) **What It Does:** An AI agent systematically compared what RLOO, GRPO, and DPO do to model internals using SVD analysis of weight deltas and SAE feature overlap. Key discovery: DPO is a rank-1 perturbation (one SVD direction recovers 95.6% of its behavioral effect), while online RL methods produce distributed, structure-preserving changes. **Why It's Impressive:** The agent orchestrated multiple domain skills (RL training, mechanistic interpretability, paper writing) across the full research lifecycle. The insight that "DPO is rank-1 alignment" is a conceptual contribution that emerged from the outer synthesis loop — not just metric optimization. --- ### 7. Scientific Plotting: Publication-Quality Figures **Demo:** [scientific-plotting-demo/](scientific-plotting-demo/) **Skills Used:** [Academic Plotting](../20-ml-paper-writing/academic-plotting/) **What It Does:** Generates all key figures for the [Andes QoE-aware LLM serving paper](https://arxiv.org/abs/2404.16283) using both workflows from the academic-plotting skill: - **Workflow 1 (Gemini AI):** System architecture diagram using `gemini-3-pro-image-preview` with 6-section prompt structure, Style B "Modern Minimal", and Nord palette — 3 non-deterministic attempts with best-of-3 selection - **Workflow 2 (matplotlib):** Five data-driven figures — QoE definition illustration, 3-panel CDF comparison, 4x3 multi-panel burst intensity grid, summary bar charts — all with publication rcParams, colorblind-safe palette, and PDF+PNG export **Key Results:** | Metric | Result | |--------|--------| | QoE improvement over vLLM | **4.7x** | | GPU resource savings | **61%** | | Gemini text accuracy | **100%** (all labels spelled correctly) | | Figures generated | **6** (1 AI diagram + 5 data charts) | **What You'll Learn:** - Crafting 6-section Gemini prompts for architecture diagrams - Multi-attempt generation with evaluation rubric - Publication-quality matplotlib figures with venue-specific styling - Colorblind-safe palettes, multi-panel layouts, and dual PDF/PNG export **Repository Contents:** ``` scientific-plotting-demo/ ├── README.md # Full demo documentation with all figures └── figures/ ├── gen_fig_andes_architecture_gemini.py # Gemini AI diagram script ├── gen_fig_andes_workflow.py # matplotlib architecture alternative ├── gen_fig_experiment_results.py # Data charts (CDF, grid, bars, QoE) ├── fig_andes_architecture*.png # Gemini outputs (best + 3 attempts) ├── fig_cdf_comparison.{pdf,png} # 3-panel CDF ├── fig_burst_intensity.{pdf,png} # 4x3 multi-panel grid ├── fig_qoe_definition.{pdf,png} # QoE metric illustration └── fig_summary_improvements.{pdf,png} # Summary bar charts ``` --- ## Coming Soon ### ML Paper Writing: From Repo to Publication **Skills Used:** [ML Paper Writing](../20-ml-paper-writing/) **What It Will Do:** Transform a research repository with experimental results into a publication-ready paper for top ML conferences (NeurIPS, ICML, ICLR). *Status: In development* --- ## How Demos Are Organized Each demo repository follows a consistent structure: ``` demo-name/ ├── README.md # Overview, results summary, how to run ├── configs/ # Configuration files ├── results/ # Raw outputs and data ├── analysis/ # Scripts and visualizations ├── .env.example # Required environment variables └── requirements.txt # Python dependencies (if applicable) ``` **Design Principles:** - **Self-contained**: Clone and run without external dependencies (except API keys) - **Reproducible**: Clear instructions to replicate results - **Educational**: Explains the "why" not just the "how" - **Real results**: Actual outputs, not mock data --- ## Contributing a Demo Want to showcase a skill? We welcome demo contributions! **Requirements:** 1. Uses one or more skills from this library 2. Produces meaningful, reproducible results 3. Includes clear documentation 4. Has visual outputs (plots, tables, reports) **To contribute:** 1. Create your demo repository 2. Follow the structure above 3. Open an issue or PR to add it to this index --- ## Quick Links - [Main Skills Library](../README.md) - [All 98 Skills](../README.md#available-ai-research-engineering-skills) - [Contributing Guide](../CONTRIBUTING.md) --- ### Overview (`06-post-training/grpo-rl-training/README.md`) # GRPO/RL Training Skill **Expert-level guidance for Group Relative Policy Optimization with TRL** ## 📁 Skill Structure ``` grpo-rl-training/ ├── SKILL.md # Main skill documentation (READ THIS FIRST) ├── README.md # This file ├── templates/ │ └── basic_grpo_training.py # Production-ready training template └── examples/ └── reward_functions_library.py # 20+ reward function examples ``` ## 🚀 Quick Start 1. **Read SKILL.md** - Comprehensive guide with all concepts and patterns 2. **Copy `templates/basic_grpo_training.py`** - Start with working code 3. **Browse `examples/reward_functions_library.py`** - Pick reward functions for your task 4. **Modify for your use case** - Adapt dataset, rewards, and config ## 💡 What's Inside ### SKILL.md (Main Documentation) - Core GRPO concepts and algorithm fundamentals - Complete implementation workflow (dataset → rewards → training → deployment) - 10+ reward function examples with code - Hyperparameter tuning guide - Training insights (loss behavior, metrics, debugging) - Troubleshooting guide - Production best practices ### Templates - **basic_grpo_training.py**: Minimal, production-ready training script - Uses Qwen 2.5 1.5B Instruct - 3 reward functions (format + correctness) - LoRA for efficient training - Fully documented and ready to run ### Examples - **reward_functions_library.py**: 20+ battle-tested reward functions - Correctness rewards (exact match, fuzzy match, numeric, code execution) - Format rewards (XML, JSON, strict/soft) - Length rewards (ideal length, min/max) - Style rewards (reasoning quality, citations, repetition penalty) - Combined rewards (multi-objective optimization) - Preset collections for common tasks ## 📖 Usage for Agents When this skill is loaded in your agent's context: 1. **Always read SKILL.md first** before implementing 2. **Start simple** - Use length-based reward to validate setup 3. **Build incrementally** - Add one reward function at a time 4. **Reference examples** - Copy patterns from reward_functions_library.py 5. **Monitor training** - Watch reward metrics (not loss!) ## 🎯 Common Use Cases | Task Type | Recommended Rewards | Template | |-----------|---------------------|----------| | Math reasoning | `MATH_REASONING_REWARDS` preset | basic_grpo_training.py | | Code generation | `CODE_GENERATION_REWARDS` preset | Modify dataset in template | | Summarization | `SUMMARIZATION_REWARDS` preset | Adjust prompts + rewards | | Q&A | `QA_REWARDS` preset | Use fuzzy match + citations | ## ⚠️ Critical Reminders - **Loss goes UP during training** - This is normal (it's KL divergence) - **Use 3-5 reward functions** - Single rewards often fail - **Test rewards before training** - Debug each function independently - **Monitor reward_std** - Should stay > 0.1 (avoid mode collapse) - **Start with num_generations=4-8** - Scale up if GPU allows ## 🔗 External Resources - [TRL Documentation](https://huggingface.co/docs/trl) - [DeepSeek R1 Paper](https://arxiv.org/abs/2501.12948) - [Open R1 Implementation](https://github.com/huggingface/open-r1) - [Unsloth (2-3x faster)](https://docs.unsloth.ai/) ## 📝 Version **v1.0.0** - Initial release (January 2025) ## 👨‍💻 Maintained By Orchestra Research For questions or improvements, see https://orchestra.com --- **License:** MIT **Last Updated:** January 2025 --- ### Overview (`demos/autoresearch-norm-heterogeneity/README.md`) # Autoresearch Demo: Embedding Norm Heterogeneity Drives LoRA Fine-Tuning Brittleness **Paper:** [norm-heterogeneity-lora-brittleness.pdf](norm-heterogeneity-lora-brittleness.pdf) **Skills Used:** [Autoresearch](../../0-autoresearch-skill/), [ML Paper Writing](../../20-ml-paper-writing/), [Research Ideation](../../21-research-ideation/) ## What Happened An AI agent ran the full autoresearch workflow autonomously — from literature survey through experiments to paper writing. Starting from the hypothesis that ETF crystallization drives LoRA fine-tuning brittleness in overtrained models, the agent: 1. **Surveyed literature** connecting two recent papers: NeurIPS 2025 Best Paper Runner-Up on superposition/ETF structure and ICML 2025 on catastrophic overtraining 2. **Ran inner loop experiments** across Pythia-410M and Pythia-1.4B checkpoints, computing ETF overlap metrics and norm statistics at each checkpoint, then applying LoRA fine-tuning 3. **Discovered a null result** — ETF overlap geometry does NOT predict fine-tuning difficulty (r=0.14), refuting the starting hypothesis 4. **Pivoted** — identified embedding norm heterogeneity (coefficient of variation) as the actual causal predictor (r=-0.84 at 410M, r=-0.99 at 1.4B) 5. **Deepened** with causal experiments — equalizing norms before LoRA increases fine-tunability by up to 79% 6. **Wrote the paper** using the ml-paper-writing skill ## Key Findings - ETF overlap metrics show no correlation with LoRA fine-tuning difficulty — a clear negative result - Norm CV of LM head rows strongly predicts deconfounded fine-tunability (r=-0.99 at 1.4B) - Equalizing norms before LoRA increases relative fine-tunability by up to 79% - The effect is rank-independent — increasing LoRA rank does not mitigate it - Norms encode semantic specificity, creating an impedance mismatch with LoRA's uniform low-rank updates ## Why This Demo Matters This demonstrates the autoresearch two-loop architecture working as designed: - **Inner loop** ran constrained experiments (checkpoint analysis, LoRA fine-tuning, metric computation) - **Outer loop** synthesized a null result into a pivot, leading to a stronger finding than the original hypothesis - The agent autonomously went from "ETF predicts brittleness" to "actually no, norm heterogeneity does" — a genuine research pivot that produced a more interesting paper --- ### Overview (`demos/autoresearch-rl-brain-scan/README.md`) # Autoresearch Demo: The RL Algorithm Brain Scan **Paper:** [rl_algorithm_brain_scan.pdf](rl_algorithm_brain_scan.pdf) **Skills Used:** [Autoresearch](../../0-autoresearch-skill/), [ML Paper Writing](../../20-ml-paper-writing/), [GRPO RL Training](../../06-post-training/grpo-rl-training/), [TRL](../../06-post-training/trl-fine-tuning/), [SAELens](../../04-mechanistic-interpretability/saelens/), [TransformerLens](../../04-mechanistic-interpretability/transformer-lens/) ## What Happened An AI agent autonomously investigated what RL alignment algorithms actually do to model internals — a question no prior work had systematically addressed. The agent: 1. **Surveyed literature** on RLOO, GRPO, and DPO, identifying the gap: nobody had compared what these algorithms do at the weight and feature level on the same base model 2. **Ran inner loop experiments** training GPT-2 Small with RLOO, GRPO, and DPO on sentiment and toxicity tasks, then analyzing weight deltas via SVD and feature changes via SAELens 3. **Discovered three key findings** through outer loop synthesis: - DPO is a rank-1 perturbation (top-1 SVD direction recovers 95.6% of behavioral effect) - Online RL (RLOO/GRPO) produces distributed, structure-preserving modifications (effective rank 200 vs 119) - DPO creates a "concentrated perturbation cascade" disrupting 2x more SAE features in later layers 4. **Validated causally** with SVD ablation experiments — not just correlation but causal evidence 5. **Wrote the paper** in ICML format using the ml-paper-writing skill ## Key Findings - **DPO is rank-1 alignment**: A single SVD direction per weight matrix recovers 95.6% of DPO's behavioral effect. GRPO needs 50+ directions for equivalent recovery. - **Online RL preserves structure**: RLOO and GRPO maintain higher effective rank (200 vs 119) and better preserve the base model's SAE feature structure (Jaccard 0.83 vs 0.69) - **DPO's concentrated perturbation cascade**: Despite lower-rank changes, DPO disrupts 2x more SAE features in later layers (1619 vs 527-870), amplifying perturbations through the network - Results hold across sentiment and toxicity tasks with statistical significance (n=3 seeds, non-overlapping CIs) ## Why This Demo Matters This demonstrates autoresearch orchestrating multiple domain skills together: - **Post-training skills** (TRL, GRPO) for training the RL models - **Interpretability skills** (SAELens, TransformerLens) for analyzing what changed - **Paper writing skill** for producing the ICML submission - The two-loop architecture enabled the agent to both run experiments AND synthesize them into mechanistic understanding — "DPO is a rank-1 perturbation" is a conceptual insight, not just a metric --- ### Overview (`demos/scientific-plotting-demo/README.md`) # Academic Plotting Skill Demo > Publication-quality figures generated using the **academic-plotting** skill from the [AI Research Skills](https://github.com/Orchestra-Research/AI-Research-SKILLs) library. Demonstrates both **Gemini AI diagram generation** (Workflow 1) and **matplotlib/seaborn data charts** (Workflow 2). --- ## Source Paper **[Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Services](https://arxiv.org/abs/2404.16283)** *Jiachen Liu, Jae-Won Chung, Zhiyu Wu, Fan Lai, Myungjin Lee, Mosharaf Chowdhury* > Andes is a QoE-aware LLM serving system that enhances user experience by ensuring users receive tokens promptly and at a smooth, digestible pace. Its preemptive token-level request scheduler dynamically prioritizes requests based on expected QoE gain and GPU resource usage, achieving up to **4.7x** QoE improvement or **61%** GPU savings compared to existing systems. | Metric | Result | |--------|--------| | QoE improvement over vLLM | **4.7x** | | GPU resource savings | **61%** | | Peak queue length reduction | **85%** | --- ## 1. System Architecture Workflow (Gemini AI) Core contribution diagram showing Andes' co-design of the inference server (Token-Level Request Scheduler + Overhead-Aware Refiner) and client (Token Pacer). Generated using the updated academic-plotting skill: - **Model**: `gemini-3-pro-image-preview` - **Style**: Style B "Modern Minimal" — ultra-clean, spacious, authoritative - **Palette**: "Nord" — desaturated section fills, Aurora Yellow accents for Andes components - **Prompt**: 6-section structure (Framing, Visual Style, Colors, Layout, Connections, Constraints) - **Attempts**: 3 non-deterministic, best selected ### Selected Result (Attempt 1) **Figure 1: Andes QoE-Aware LLM Serving System Architecture** AI-generated diagram showing the full request lifecycle: (1) User submits request, (2) Client enqueues with QoE parameters, (3) Request Tracker feeds state to scheduler, (4) Token-Level Scheduler admits/resumes/preempts at token granularity, (5) Executor streams tokens, (6) Token Pacer delivers smoothly at reading speed. Yellow-accented components are Andes' novel contributions. `gemini-3-pro-image-preview` | `Style B: Modern Minimal` | `Nord Palette` | `Best of 3` ### All 3 Gemini Attempts (for comparison) | Attempt 1 (Selected) | Attempt 2 | Attempt 3 | |:--------------------:|:---------:|:---------:| | | | | | Best spacing, color accents, arrow routing | Good, slightly tighter spacing | Good separation, dashed preempt | All 3 attempts have **100% text accuracy** — every label spelled correctly (Token Pacer, Overhead Refiner, KV Cache, etc.). This is a major improvement over the previous generation which had misspellings in all attempts. --- ## 2. QoE Metric Definition Four foundational cases illustrating how the QoE metric captures different types of user experience degradation in text streaming services. **Figure 2: User Experience Cases and QoE Definition** (a) Perfect experience: actual delivery matches ideal consumption timeline. (b) Long initial delay: head-of-line blocking inflates TTFT. (c) Slow streaming: token generation slower than consumption speed. (d) Pause in middle: preemption causes mid-stream pause. The shaded area represents QoE degradation (S_delay). `matplotlib` | `Line Plot` --- ## 3. CDF Comparison: QoE, TTFT, TDS Three-panel CDF comparison on real-world BurstGPT traces showing Andes' improvements across all key metrics. Follows the multi-panel figure pattern with shared styling and colorblind-safe colors. **Figure 3: CDF of QoE, TTFT, and TDS on BurstGPT Trace** Andes (orange) achieves near-perfect QoE for 97% of requests (QoE >= 0.95), compared to only 75% for vLLM (blue). TTFT is reduced from 10.5s to 1.8s average. TDS remains comparable, showing Andes doesn't sacrifice throughput. `matplotlib` | `CDF Plot` | `PDF + PNG` --- ## 4. Multi-Panel: Varying Burst Intensity 4x3 grid showing average QoE across 4 models and 3 datasets under varying burst intensities. This demonstrates the academic plotting skill's ability to create complex multi-panel figures with shared axes, model labels, and a unified legend. **Figure 4: Average QoE Under Varying Burst Intensity** Across all 12 model-dataset combinations, Andes (orange) consistently maintains higher QoE than all baselines as burst intensity increases. vLLM (blue), LQSF (green), and Sarathi-Serve (red) degrade significantly under heavy bursts due to FCFS scheduling and head-of-line blocking. Andes achieves up to 4.7x improvement at the highest burst intensity. `matplotlib` | `Multi-Panel Grid` | `4 Methods x 4 Models x 3 Datasets` --- ## 5. Summary: Key Improvements Three-panel bar chart summarizing the headline results from the paper. Each panel uses a distinct color to represent different aspects of improvement. **Figure 5: Summary of Key Improvements** (a) Andes achieves 0.99 average QoE vs 0.88 for vLLM on real-world traces. (b) QoE improvement ranges from 3.2x to 4.7x across different model architectures. (c) Andes saves 61% GPU resources, reduces peak queue by 85%, and handles 2.6x more concurrent requests. `matplotlib` | `Grouped Bar` | `PDF + PNG` --- ## 6. How These Figures Were Generated All figures follow the **academic-plotting** skill's two workflows and publication standards. ### Workflow 1: Gemini AI Diagram The system architecture (Figure 1) uses `gemini-3-pro-image-preview` with the skill's **6-section prompt structure** and **Style B: Modern Minimal** visual style. Key elements: 1. **Framing** — Sets the tone: "ultra-clean, modern, authoritative, like Apple docs meets Nature paper" 2. **Visual Style** — Full Modern Minimal style block: floating boxes with shadow, no borders, thin gray arrows 3. **Color Palette** — Nord palette with exact hex codes for every element 4. **Layout** — Every box named, spatially positioned, with nested sub-components 5. **Connections** — Every arrow individually specified: source, target, style, color, label, routing 6. **Constraints** — What NOT to include, adapted for the Modern Minimal style ```python from google import genai client = genai.Client(api_key=API_KEY) # 6-section prompt: Framing + Style + Colors + Layout + Connections + Constraints PROMPT = """ SECTION 1 — FRAMING: Create an ultra-clean, modern technical architecture diagram for an OSDI paper. Think: Apple developer docs meets Nature paper... SECTION 2 — VISUAL STYLE (Modern Minimal): Ultra-clean geometric shapes, floating boxes with shadow, thin gray arrows... SECTION 3 — COLOR PALETTE (Nord): Deep text: #2E3440, Andes accent: Aurora Yellow #EBCB8B, Executor: Frost #5E81AC... SECTION 4 — LAYOUT: Two zones: CLIENT (#EEF1F6) and SERVER (#EDF3ED), each with floating boxes... SECTION 5 — CONNECTIONS: 8 arrows with step numbers, dashed red preempt path, green delivery flow... SECTION 6 — CONSTRAINTS: ZERO decoration, generous whitespace, CRITICAL TEXT ACCURACY... """ # Generate 3 non-deterministic attempts for i in range(1, 4): response = client.models.generate_content( model="gemini-3-pro-image-preview", contents=PROMPT, config=genai.types.GenerateContentConfig( response_modalities=["IMAGE", "TEXT"])) ``` ### Workflow 2: Data-Driven Charts Experiment figures use matplotlib with publication defaults: serif fonts, colorblind-safe palette, 300 DPI export, venue-appropriate sizing. Each figure exports both PDF (vector for LaTeX) and PNG (raster). ```python # Publication defaults plt.rcParams.update({ "font.family": "serif", "font.size": 10, "axes.spines.top": False, "savefig.dpi": 300, }) # Colorblind-safe palette COLORS = { "blue": "#4C72B0", "orange": "#DD8452", "green": "#55A868", "red": "#C44E52", } ``` --- ## 7. Generated Files ``` demo/ ├── README.md # This demo page └── figures/ ├── gen_fig_andes_architecture_gemini.py # Gemini diagram script (Workflow 1) ├── gen_fig_andes_workflow.py # matplotlib diagram (alternative) ├── gen_fig_experiment_results.py # Data charts script (Workflow 2) ├── fig_andes_architecture.png # Gemini best attempt (selected) ├── fig_andes_architecture_attempt1.png # Gemini attempt 1 ├── fig_andes_architecture_attempt2.png # Gemini attempt 2 ├── fig_andes_architecture_attempt3.png # Gemini attempt 3 ├── fig_andes_workflow.pdf # matplotlib vector diagram ├── fig_andes_workflow.png # matplotlib raster diagram ├── fig_cdf_comparison.pdf # CDF panels (vector) ├── fig_cdf_comparison.png # CDF panels (raster) ├── fig_burst_intensity.pdf # Multi-panel grid (vector) ├── fig_burst_intensity.png # Multi-panel grid (raster) ├── fig_qoe_definition.pdf # QoE illustration (vector) ├── fig_qoe_definition.png # QoE illustration (raster) ├── fig_summary_improvements.pdf # Summary bars (vector) └── fig_summary_improvements.png # Summary bars (raster) ``` --- *Generated using the [academic-plotting](../20-ml-paper-writing/academic-plotting/SKILL.md) skill from [AI Research Skills](https://github.com/Orchestra-Research/AI-Research-SKILLs). Paper: [arXiv:2404.16283](https://arxiv.org/abs/2404.16283). Figures use synthetic data matching paper-reported distributions.* --- ### Overview (`packages/ai-research-skills/README.md`) # @orchestra-research/ai-research-skills Install AI research engineering skills to your coding agents (Claude Code, Hermes Agent, OpenCode, Cursor, Gemini CLI, and more). ```bash npx @orchestra-research/ai-research-skills ``` ## Features - **98 skills** across 23 categories for AI research engineering - **Auto-detects** installed coding agents - **Interactive installer** with guided experience - **Global or local install** — install globally with symlinks, or per-project with `--local` for version-controlled, project-specific skill sets - **Works with 9 agents**: Claude Code, Hermes Agent, OpenCode, OpenClaw, Cursor, Codex, Gemini CLI, Qwen Code, and shared `.agents/` ## Quick Start Run the interactive installer: ```bash npx @orchestra-research/ai-research-skills ``` This will: 1. Detect your installed coding agents 2. Let you choose what to install (everything, categories, or quick start bundle) 3. Download skills from GitHub 4. Create symlinks to each agent's skills directory ## Commands ```bash # Interactive mode (recommended) npx @orchestra-research/ai-research-skills # Install everything (global) npx @orchestra-research/ai-research-skills install --all # Install a specific category npx @orchestra-research/ai-research-skills install post-training # List installed skills npx @orchestra-research/ai-research-skills list # Update all skills npx @orchestra-research/ai-research-skills update ``` ### Local Installation (per-project) Install skills directly into your project directory so different projects can have different skill sets: ```bash # Install all skills locally to the current project npx @orchestra-research/ai-research-skills install --all --local # Install a category locally npx @orchestra-research/ai-research-skills install --category post-training --local # List locally installed skills npx @orchestra-research/ai-research-skills list --local # Update local skills npx @orchestra-research/ai-research-skills update --local # Uninstall local skills npx @orchestra-research/ai-research-skills uninstall --local ``` Local installation copies skills (not symlinks) into agent directories within your project: ``` my-project/ ├── .claude/skills/ # Claude Code picks these up │ ├── grpo-rl-training/ │ └── vllm/ ├── .cursor/skills/ # Cursor picks these up │ ├── grpo-rl-training/ │ └── vllm/ ├── .orchestra-skills.json # Tracks installed skills └── ... ``` Benefits: - **Per-project skills**: Each project gets only the skills it needs - **Version control**: Commit skills to your repo so the whole team has them - **Reproducible**: Lock file (`.orchestra-skills.json`) tracks what's installed ## Categories | Category | Skills | Description | |----------|--------|-------------| | **Autoresearch** | **1** | **Central orchestration — manages full research lifecycle, routes to all other skills** | | Model Architecture | 6 | LitGPT, Mamba, TorchTitan, Megatron... | | Post-Training | 8 | GRPO, verl, slime, miles, torchforge... | | Fine-Tuning | 5 | Axolotl, Unsloth, PEFT, Torchtune... | | Distributed Training | 6 | DeepSpeed, FSDP, Megatron... | | Inference Serving | 4 | vLLM, TensorRT-LLM, SGLang... | | Optimization | 6 | Flash Attention, GPTQ, AWQ... | | And 15 more... | | Ideation, Paper Writing, RAG, Agents, Multimodal... | ## How It Works ### Global Install (default) 1. **Canonical Storage**: Skills are stored once at `~/.orchestra/skills/` 2. **Symlinks**: Each agent gets symlinks pointing to the canonical copy 3. **Auto-activation**: Skills activate when you discuss relevant topics ``` ~/.orchestra/skills/ # Single source of truth ├── 06-post-training/ │ ├── verl/ │ └── grpo-rl-training/ └── ... ~/.claude/skills/ # Symlinks for Claude Code ├── verl → ~/.orchestra/skills/.../verl └── grpo-rl-training → ... ~/.cursor/skills/ # Symlinks for Cursor └── (same links) ``` ### Local Install (`--local`) 1. **Direct Copy**: Skills are copied into agent directories within your project 2. **Version Control**: Files can be committed to git for team sharing 3. **Lock File**: `.orchestra-skills.json` tracks what's installed ``` my-project/ ├── .claude/skills/verl/ # Copied for Claude Code ├── .cursor/skills/verl/ # Copied for Cursor ├── .codex/skills/verl/ # Copied for Codex └── .orchestra-skills.json # Lock file ``` ## Supported Agents | Agent | Config Directory | |-------|-----------------| | Claude Code | `~/.claude` | | OpenCode | `~/.config/opencode` | | OpenClaw | `~/.openclaw` | | Cursor | `~/.cursor` | | Codex (OpenAI) | `~/.codex` | | Gemini CLI | `~/.gemini` | | Qwen Code | `~/.qwen` | | Shared Agents | `~/.agents` | | Hermes Agent | `~/.hermes` | ## License MIT - Orchestra Research --- ### Overview (`04-mechanistic-interpretability/nnsight/references/README.md`) # nnsight Reference Documentation This directory contains comprehensive reference materials for nnsight. ## Contents - [api.md](api.md) - Complete API reference for LanguageModel, tracing, and proxy objects - [tutorials.md](tutorials.md) - Step-by-step tutorials for local and remote interpretability ## Quick Links - **Official Documentation**: https://nnsight.net/ - **GitHub Repository**: https://github.com/ndif-team/nnsight - **NDIF (Remote Execution)**: https://ndif.us/ - **Community Forum**: https://discuss.ndif.us/ - **Paper**: https://arxiv.org/abs/2407.14561 (ICLR 2025) ## Installation ```bash # Basic installation pip install nnsight # For vLLM support pip install "nnsight[vllm]" ``` ## Basic Usage ```python from nnsight import LanguageModel # Load model model = LanguageModel("openai-community/gpt2", device_map="auto") # Trace and access internals with model.trace("The Eiffel Tower is in") as tracer: # Access layer output hidden = model.transformer.h[5].output[0].save() # Modify activations model.transformer.h[8].output[0][:] *= 0.5 # Get final output logits = model.output.save() # Access saved values outside context print(hidden.shape) ``` ## Key Concepts ### Tracing The `trace()` context enables deferred execution - operations are recorded and executed together. ### Proxy Objects Inside trace, module accesses return Proxies. Call `.save()` to retrieve values after execution. ### Remote Execution (NDIF) Run the same code on massive models (70B+) without local GPUs: ```python # Same code, just add remote=True with model.trace("Hello", remote=True): hidden = model.model.layers[40].output[0].save() ``` ## NDIF Setup 1. Sign up at https://login.ndif.us/ 2. Get API key 3. Set environment variable: `export NDIF_API_KEY=your_key` ## Available Remote Models - Llama-3.1-8B, 70B, 405B - DeepSeek-R1 models - More at https://ndif.us/ --- ### Overview (`04-mechanistic-interpretability/pyvene/references/README.md`) # pyvene Reference Documentation This directory contains comprehensive reference materials for pyvene. ## Contents - [api.md](api.md) - Complete API reference for IntervenableModel, intervention types, and configurations - [tutorials.md](tutorials.md) - Step-by-step tutorials for causal tracing, activation patching, and trainable interventions ## Quick Links - **Official Documentation**: https://stanfordnlp.github.io/pyvene/ - **GitHub Repository**: https://github.com/stanfordnlp/pyvene - **Paper**: https://arxiv.org/abs/2403.07809 (NAACL 2024) ## Installation ```bash pip install pyvene ``` ## Basic Usage ```python import pyvene as pv from transformers import AutoModelForCausalLM, AutoTokenizer # Load model model = AutoModelForCausalLM.from_pretrained("gpt2") tokenizer = AutoTokenizer.from_pretrained("gpt2") # Define intervention config = pv.IntervenableConfig( representations=[ pv.RepresentationConfig( layer=5, component="block_output", intervention_type=pv.VanillaIntervention, ) ] ) # Create intervenable model intervenable = pv.IntervenableModel(config, model) # Run intervention (swap activations from source to base) base_inputs = tokenizer("The cat sat on the", return_tensors="pt") source_inputs = tokenizer("The dog ran through the", return_tensors="pt") _, outputs = intervenable( base=base_inputs, sources=[source_inputs], ) ``` ## Key Concepts ### Intervention Types - **VanillaIntervention**: Swap activations between runs - **AdditionIntervention**: Add source to base activations - **ZeroIntervention**: Zero out activations (ablation) - **CollectIntervention**: Collect activations without modifying - **RotatedSpaceIntervention**: Trainable intervention for causal discovery ### Components Target specific parts of the model: - `block_input`, `block_output` - `mlp_input`, `mlp_output`, `mlp_activation` - `attention_input`, `attention_output` - `query_output`, `key_output`, `value_output` ### HuggingFace Integration Save and load interventions via HuggingFace Hub for reproducibility. --- ### Overview (`04-mechanistic-interpretability/saelens/references/README.md`) # SAELens Reference Documentation This directory contains comprehensive reference materials for SAELens. ## Contents - [api.md](api.md) - Complete API reference for SAE, TrainingSAE, and configuration classes - [tutorials.md](tutorials.md) - Step-by-step tutorials for training and analyzing SAEs - [papers.md](papers.md) - Key research papers on sparse autoencoders ## Quick Links - **GitHub Repository**: https://github.com/jbloomAus/SAELens - **Neuronpedia**: https://neuronpedia.org (browse pre-trained SAE features) - **HuggingFace SAEs**: Search for tag `saelens` ## Installation ```bash pip install sae-lens ``` Requirements: Python 3.10+, transformer-lens>=2.0.0 ## Basic Usage ```python from transformer_lens import HookedTransformer from sae_lens import SAE # Load model and SAE model = HookedTransformer.from_pretrained("gpt2-small", device="cuda") sae, cfg_dict, sparsity = SAE.from_pretrained( release="gpt2-small-res-jb", sae_id="blocks.8.hook_resid_pre", device="cuda" ) # Encode activations to sparse features tokens = model.to_tokens("Hello world") _, cache = model.run_with_cache(tokens) activations = cache["resid_pre", 8] features = sae.encode(activations) # Sparse feature activations reconstructed = sae.decode(features) # Reconstructed activations ``` ## Key Concepts ### Sparse Autoencoders SAEs decompose dense neural activations into sparse, interpretable features: - **Encoder**: Maps d_model → d_sae (typically 4-16x expansion) - **ReLU/TopK**: Enforces sparsity - **Decoder**: Reconstructs original activations ### Training Loss `Loss = MSE(original, reconstructed) + L1_coefficient × L1(features)` ### Key Metrics - **L0**: Average number of active features (target: 50-200) - **CE Loss Score**: Cross-entropy recovered vs original model (target: 80-95%) - **Dead Features**: Features that never activate (target: <5%) ## Available Pre-trained SAEs | Release | Model | Description | |---------|-------|-------------| | `gpt2-small-res-jb` | GPT-2 Small | Residual stream SAEs | | `gemma-2b-res` | Gemma 2B | Residual stream SAEs | | Various | Search HuggingFace | Community-trained SAEs | --- ### Overview (`04-mechanistic-interpretability/transformer-lens/references/README.md`) # TransformerLens Reference Documentation This directory contains comprehensive reference materials for TransformerLens. ## Contents - [api.md](api.md) - Complete API reference for HookedTransformer, ActivationCache, and HookPoints - [tutorials.md](tutorials.md) - Step-by-step tutorials for common interpretability workflows - [papers.md](papers.md) - Key research papers and foundational concepts ## Quick Links - **Official Documentation**: https://transformerlensorg.github.io/TransformerLens/ - **GitHub Repository**: https://github.com/TransformerLensOrg/TransformerLens - **Model Properties Table**: https://transformerlensorg.github.io/TransformerLens/generated/model_properties_table.html ## Installation ```bash pip install transformer-lens ``` ## Basic Usage ```python from transformer_lens import HookedTransformer # Load model model = HookedTransformer.from_pretrained("gpt2-small") # Run with activation caching tokens = model.to_tokens("Hello world") logits, cache = model.run_with_cache(tokens) # Access activations residual = cache["resid_post", 5] # Layer 5 residual stream attention = cache["pattern", 3] # Layer 3 attention patterns ``` ## Key Concepts ### HookPoints Every activation in the transformer has a HookPoint wrapper, enabling: - Reading activations via `run_with_cache()` - Modifying activations via `run_with_hooks()` ### Activation Cache The `ActivationCache` stores all intermediate activations with helper methods for: - Residual stream decomposition - Logit attribution - Layer-wise analysis ### Supported Models (50+) GPT-2, LLaMA, Mistral, Pythia, GPT-Neo, OPT, Gemma, Phi, and more. --- ### Overview (`14-agents/a-evolve/references/README.md`) # A-Evolve Official Documentation Reference > This document consolidates key information from the official A-Evolve documentation > at [github.com/A-EVO-Lab/a-evolve](https://github.com/A-EVO-Lab/a-evolve). ## Table of Contents - [Project Overview](#project-overview) - [Installation Guide](#installation-guide) - [Quick Start Guide](#quick-start-guide) - [Architecture Overview](#architecture-overview) - [Agent Protocol](#agent-protocol) - [Benchmark Adapters](#benchmark-adapters) - [Evolution Engines](#evolution-engines) - [Workspace Contract](#workspace-contract) - [Configuration Reference](#configuration-reference) - [Built-in Agents](#built-in-agents) - [Built-in Benchmarks](#built-in-benchmarks) - [Evolution Algorithms](#evolution-algorithms) - [Skill System](#skill-system) - [Memory System](#memory-system) - [Version Control](#version-control) - [Observation Pipeline](#observation-pipeline) - [FAQ](#faq) --- ## Project Overview A-Evolve is the universal infrastructure for evolving AI agents through self-improvement. It enables automatic, data-driven optimization of agents across any domain using any evolution algorithm. ### Design Principles 1. **File-system as contract**: All evolvable agent state lives as plain files in a workspace directory. No databases, no learned weights, no opaque parameters. Every mutation is an explicit edit to a text file. 2. **Pluggable everything**: Three interfaces — `BaseAgent`, `BenchmarkAdapter`, `EvolutionEngine` — enable any combination of agent, benchmark, and algorithm. 3. **Git for versioning**: Every evolution cycle creates git snapshots. Changes are diffable, rollbackable, and human-readable. 4. **LLM-in-the-loop**: The default evolution engine uses an LLM with bash tools to analyze observations and directly mutate workspace files. The evolver is itself an AI agent improving other AI agents. 5. **Zero manual engineering**: Once configured, evolution runs autonomously. The loop handles solving, evaluation, mutation, gating, and convergence detection. ### Key Results Using Claude Opus 4.6 as both the solver and evolver model: | Benchmark | Score | Leaderboard Position | |-----------|-------|---------------------| | MCP-Atlas | 79.4% | #1 | | SWE-bench Verified | 76.8% | ~#5 | | Terminal-Bench 2.0 | 76.5% | ~#7 | | SkillsBench | 34.9% | #2 | These results demonstrate that LLM-driven evolution of prompts, skills, and memory can produce state-of-the-art agent performance across diverse domains. --- ## Installation Guide ### Requirements - Python >= 3.11 - Git (for workspace versioning) - An LLM API key (Anthropic, OpenAI, or AWS Bedrock credentials) ### Installation Options ```bash # Core package (matplotlib, pyyaml) pip install a-evolve # With specific LLM provider support pip install a-evolve[anthropic] # Anthropic Claude API pip install a-evolve[openai] # OpenAI API pip install a-evolve[bedrock] # AWS Bedrock (boto3) pip install a-evolve[litellm] # Multi-provider via LiteLLM # With domain-specific dependencies pip install a-evolve[swe] # SWE-bench (strands-agents, datasets, swebench) pip install a-evolve[mcp] # MCP-Atlas (mcp, strands-agents, litellm) pip install a-evolve[skillbench] # SkillsBench (strands-agents) # Everything pip install a-evolve[all] # Development pip install a-evolve[dev] # pytest, ruff, hypothesis ``` ### From Source ```bash git clone https://github.com/A-EVO-Lab/a-evolve.git cd a-evolve pip install -e ".[all,dev]" ``` ### Verifying Installation ```python import agent_evolve as ae print(ae.__version__) # Should print version print(ae.Evolver) # Should print class reference ``` --- ## Quick Start Guide ### 3-Line Evolution ```python import agent_evolve as ae evolver = ae.Evolver(agent="swe", benchmark="swe-verified") results = evolver.run(cycles=10) print(f"Final score: {results.final_score}") ``` This: 1. Copies the built-in SWE seed workspace to a working directory 2. Instantiates `SweAgent` from the workspace manifest 3. Runs 10 evolution cycles against SWE-bench Verified 4. Returns `EvolutionResult` with scores, convergence status, and details ### With Custom Configuration ```python import agent_evolve as ae config = ae.EvolveConfig( batch_size=15, # 15 tasks per cycle max_cycles=25, # Up to 25 evolution rounds evolve_prompts=True, # Mutate system prompt evolve_skills=True, # Discover and refine skills evolve_memory=True, # Build episodic memory holdout_ratio=0.2, # 20% held out for validation evolver_model="us.anthropic.claude-opus-4-6-v1", egl_threshold=0.02, # Stop if < 2% improvement egl_window=5, # Over 5 consecutive cycles ) evolver = ae.Evolver( agent="swe", benchmark="swe-verified", config=config, ) results = evolver.run() # Inspect results print(f"Cycles: {results.cycles_completed}") print(f"Score: {results.final_score:.3f}") print(f"Converged: {results.converged}") print(f"Score history: {results.score_history}") ``` --- ## Architecture Overview ### System Diagram ``` User Code (3 lines) │ ▼ ┌──────────────────────────────────────┐ │ Evolver API │ │ - Resolves agent, benchmark, config │ │ - Creates EvolutionLoop │ │ - Returns EvolutionResult │ └──────────────┬───────────────────────┘ │ ┌──────────▼──────────┐ │ EvolutionLoop │ │ For each cycle: │ │ 1. Solve │ │ 2. Observe │ │ 3. Snapshot │ │ 4. Evolve │ │ 5. Snapshot │ │ 6. Record │ │ 7. Reload │ │ 8. Converge? │ └──────────┬──────────┘ │ ┌──────────┼──────────┐ │ │ │ ▼ ▼ ▼ Agent Benchmark Engine solve() evaluate() step() │ │ │ └──────────┼──────────┘ │ ▼ Agent Workspace (filesystem + git) ``` ### Component Interactions **Forward flow (solve):** 1. `EvolutionLoop` calls `benchmark.get_tasks()` to get a batch of tasks 2. For each task, calls `agent.solve(task)` to get a `Trajectory` 3. Calls `benchmark.evaluate(task, trajectory)` to get `Feedback` 4. Bundles into `Observation(task, trajectory, feedback)` triples **Evolution flow (mutate):** 1. `EvolutionLoop` passes observations to `engine.step()` 2. Engine reads workspace files, analyzes observations 3. Engine mutates workspace files (prompts, skills, memory) 4. Returns `StepResult(mutated, summary, metadata)` **Reload flow (sync):** 1. `EvolutionLoop` calls `agent.reload_from_fs()` 2. Agent re-reads prompts, skills, memory from workspace 3. Next cycle uses evolved state --- ## Agent Protocol ### BaseAgent Abstract Class All evolvable agents inherit from `BaseAgent`: ```python from agent_evolve.protocol.base_agent import BaseAgent from agent_evolve.types import Task, Trajectory class MyAgent(BaseAgent): def __init__(self, workspace_dir: str): super().__init__(workspace_dir) # Initialize your LLM client, tools, etc. def solve(self, task: Task) -> Trajectory: """Solve a single task and return the trajectory. This is the only method you MUST override. """ # Your solving logic here return Trajectory( task_id=task.id, output="solution", steps=[{"tool": "llm", "action": "generate"}], ) ``` ### Agent Lifecycle 1. **Construction**: `__init__(workspace_dir)` — set up LLM client, load initial state 2. **State loading**: `reload_from_fs()` — read prompts, skills, memory from workspace 3. **Solving**: `solve(task)` — process one task, return trajectory 4. **Memory buffering**: `remember(content, category)` — store lessons during solve 5. **State export**: `export_to_fs()` — flush buffered memories and skill proposals 6. **Hot reload**: `reload_from_fs()` — re-read after evolution mutates files ### Agent Properties | Property | Type | Description | |----------|------|-------------| | `system_prompt` | `str` | Content of `prompts/system.md` | | `skills` | `list[SkillMeta]` | Available skills from `skills/` directory | | `memories` | `list[dict]` | Loaded episodic/semantic memories | ### Agent Best Practices 1. **Always use `self.system_prompt`** — don't hardcode prompts 2. **Inject skills into LLM context** — they're the primary evolution mechanism 3. **Call `remember()` for reusable lessons** — not for task-specific notes 4. **Keep `solve()` deterministic** when possible (temperature=0 for reproducibility) 5. **Truncate trajectories** — don't store full conversation if not needed for evolution --- ## Benchmark Adapters ### BenchmarkAdapter Abstract Class ```python from agent_evolve.benchmarks.base import BenchmarkAdapter from agent_evolve.types import Task, Trajectory, Feedback class MyBenchmark(BenchmarkAdapter): def get_tasks(self, split="train", limit=10): """Return tasks from the benchmark dataset. Args: split: "train" or "test" (for holdout evaluation) limit: Maximum number of tasks to return (default 10) """ return [Task(id="1", input="task description")] def evaluate(self, task, trajectory): """Evaluate an agent's trajectory on a task. Returns Feedback with: - success: bool (binary pass/fail) - score: float (0.0 to 1.0 continuous) - detail: str (human-readable explanation) """ return Feedback(success=True, score=0.9, detail="Passed 9/10 tests") ``` ### Benchmark Best Practices 1. **Rich feedback details** — the evolver reads `feedback.detail` to decide what to mutate 2. **Deterministic evaluation** — same input should produce same score 3. **Diverse task coverage** — include easy, medium, and hard tasks 4. **Strict train/test split** — no overlap between splits 5. **Score granularity** — continuous scores (0.0-1.0) are more useful than binary pass/fail --- ## Evolution Engines ### EvolutionEngine Abstract Class ```python from agent_evolve.engine.base import EvolutionEngine from agent_evolve.types import StepResult class MyEngine(EvolutionEngine): def step(self, workspace, observations, history, trial): """Mutate the workspace based on observations. Args: workspace: AgentWorkspace — typed I/O for agent files observations: list[Observation] — recent (task, trajectory, feedback) triples history: EvolutionHistory — query past cycles and workspace versions trial: TrialRunner — optional live evaluation runner Returns: StepResult with mutated flag, summary, and metadata """ # Analyze observations, mutate workspace return StepResult(mutated=True, summary="Updated prompts") def on_cycle_end(self, accepted: bool, score: float): """Optional callback after gating decision.""" pass ``` ### Engine Selection Guide | Engine | When to Use | Compute Cost | |--------|-------------|-------------| | AEvolveEngine (default) | General-purpose, diverse domains | High (full LLM call) | | GuidedSynthesisEngine | Skill discovery focus | Medium | | AdaptiveEvolutionEngine | Noisy evaluation, fine control | Medium | | AdaptiveSkillEngine | Skill-heavy domains | Medium | | Custom | Domain-specific mutation logic | Variable | --- ## Workspace Contract ### Directory Structure ``` workspace/ ├── manifest.yaml # Required: agent metadata ├── prompts/ │ ├── system.md # Main system prompt │ └── fragments/ # Modular prompt pieces │ ├── reasoning.md │ └── output_format.md ├── skills/ │ ├── _drafts/ # Proposed skills pending review │ │ └── new-skill.md │ └── verify-solution/ # Accepted skills │ └── SKILL.md ├── tools/ │ ├── registry.yaml # Tool manifest │ └── custom_tool.py # Tool implementations ├── memory/ │ ├── episodic.jsonl # Failure lessons │ └── semantic.jsonl # Domain knowledge └── evolution/ # Managed by loop ├── observations/ │ ├── batch_0001.jsonl │ └── batch_0002.jsonl ├── history.jsonl └── metrics.json ``` ### Manifest Format ```yaml agent: type: reference # Must be "reference" entrypoint: my_package.agents.MyAgent # Dotted Python path evolvable_layers: # Which directories can be mutated - prompts # System prompt + fragments - skills # Skill library - memory # Episodic/semantic memory # - tools # Tool implementations (optional) reload_strategy: hot # "hot" (re-read files) or "cold" (restart) ``` ### AgentWorkspace API The `AgentWorkspace` class provides typed read/write access: **Prompts:** - `read_prompt() -> str` — reads `prompts/system.md` - `write_prompt(content: str)` — writes `prompts/system.md` - `read_fragment(name: str) -> str` — reads `prompts/fragments/{name}` - `write_fragment(name: str, content: str)` — writes a fragment - `list_fragments() -> list[str]` — lists fragment filenames **Skills:** - `list_skills() -> list[SkillMeta]` — lists skills with name, description, path - `read_skill(name: str) -> str` — reads skill content (frontmatter stripped) - `write_skill(name: str, content: str)` — writes or updates a skill - `delete_skill(name: str)` — removes a skill directory **Drafts:** - `list_drafts() -> list[dict]` — lists pending skill proposals - `write_draft(name: str, content: str)` — writes a draft proposal - `clear_drafts()` — removes all pending drafts **Memory:** - `add_memory(entry: dict, category: str = "episodic")` — appends to category JSONL - `read_memories(category: str = "episodic", limit: int = 100) -> list[dict]` - `read_all_memories(limit: int = 100) -> list[dict]` — all categories combined **Tools:** - `read_tool_registry() -> list[dict]` — reads `tools/registry.yaml` - `write_tool_registry(tools: list[dict])` — writes tool manifest - `read_tool(name: str) -> str` — reads tool source code - `write_tool(name: str, content: str)` — writes tool implementation **Evolution Metadata:** - `read_evolution_history() -> list[dict]` — reads `evolution/history.jsonl` - `read_evolution_metrics() -> dict` — reads `evolution/metrics.json` --- ## Configuration Reference ### EvolveConfig Fields | Field | Type | Default | Description | |-------|------|---------|-------------| | `batch_size` | `int` | `10` | Tasks per solve round | | `max_cycles` | `int` | `20` | Maximum evolution iterations | | `holdout_ratio` | `float` | `0.2` | Fraction held out for validation | | `evolve_prompts` | `bool` | `True` | Allow prompt mutation | | `evolve_skills` | `bool` | `True` | Allow skill creation/modification | | `evolve_memory` | `bool` | `True` | Allow memory writes | | `evolve_tools` | `bool` | `False` | Allow tool implementation changes | | `trajectory_only` | `bool` | `False` | Hide scores from evolver | | `evolver_model` | `str` | `"us.anthropic.claude-opus-4-6-v1"` | LLM for evolution engine | | `evolver_max_tokens` | `int` | `16384` | Max tokens for evolver calls | | `egl_threshold` | `float` | `0.05` | Convergence epsilon | | `egl_window` | `int` | `3` | Cycles for plateau detection | | `extra` | `dict` | `{}` | Extension point for custom params | ### Loading from YAML ```yaml # config.yaml batch_size: 15 max_cycles: 30 evolve_prompts: true evolve_skills: true evolve_memory: false evolver_model: us.anthropic.claude-opus-4-6-v1 egl_threshold: 0.03 egl_window: 5 extra: solver_proposed: true merge_threshold: 0.7 ``` ```python config = ae.EvolveConfig.from_yaml("config.yaml") ``` ### Configuration Strategies **Conservative (stable improvement):** ```python config = ae.EvolveConfig( batch_size=10, max_cycles=10, evolve_prompts=True, evolve_skills=False, evolve_memory=False, egl_threshold=0.05, ) ``` **Aggressive (maximum exploration):** ```python config = ae.EvolveConfig( batch_size=20, max_cycles=50, evolve_prompts=True, evolve_skills=True, evolve_memory=True, evolve_tools=True, egl_threshold=0.01, egl_window=7, ) ``` **Skill-focused (procedure discovery):** ```python config = ae.EvolveConfig( batch_size=10, max_cycles=25, evolve_prompts=False, evolve_skills=True, evolve_memory=True, ) ``` --- ## Built-in Agents ### SWE Agent (`seed_workspaces/swe/`) **Domain**: SWE-bench code patching **Model**: Claude Opus 4.6 via AWS Bedrock **Framework**: Strands-agents (CodeDojo-compatible) Key features: - Verify-fix loop: runs tests before and after each edit - Hypothesis-first approach: form theory before exploring - Skill proposal generation: agent reflects on verification process - Conversation capture with per-turn token tracking - Dynamic tool loading from workspace `tools/registry.yaml` **Tools available**: bash, submit, text_editor, python_exec ### Terminal Agent (`seed_workspaces/terminal/`) **Domain**: Terminal-Bench 2.0 shell challenges **Model**: Claude Sonnet 4 via AWS Bedrock **Framework**: Strands-agents Key features: - Concurrent timeout enforcement via ThreadPoolExecutor - Test file copying only during evaluation (prevents cheating) - Pre-built skills: self-verification, environment-discovery, scientific-computing, debug-and-fix - Memory injection disabled (time-sensitive tasks) - Graceful timeout fallback **Tools available**: bash, python, submit ### MCP Agent (`seed_workspaces/mcp/`) **Domain**: MCP-Atlas tool calling **Model**: Claude Opus 4.6 via AWS Bedrock **Framework**: Strands-agents with MCP integration Key features: - MCP server connection management - Tool discovery and invocation - Multi-provider support via LiteLLM --- ## Built-in Benchmarks ### SWE-bench Verified **Module**: `agent_evolve.benchmarks.swe_verified` **Tasks**: Real GitHub issues from popular Python repositories **Evaluation**: Runs test suite, checks if agent's patch fixes the issue **Metric**: Pass rate (0.0 to 1.0) ### MCP-Atlas **Module**: `agent_evolve.benchmarks.mcp_atlas` **Tasks**: Tool calling scenarios with MCP servers **Evaluation**: Checks correct tool selection and parameter passing **Metric**: Accuracy (0.0 to 1.0) ### Terminal-Bench 2.0 **Module**: `agent_evolve.benchmarks.terminal2` **Tasks**: Shell command challenges (file manipulation, system admin, scripting) **Evaluation**: Runs test scripts to verify terminal state **Metric**: Pass rate (0.0 to 1.0) ### SkillsBench **Module**: `agent_evolve.benchmarks.skill_bench` **Tasks**: Multi-step procedural tasks **Evaluation**: Checks step-by-step correctness **Metric**: Accuracy (0.0 to 1.0) ### ARC-AGI-3 **Module**: `agent_evolve.benchmarks.arc_agi3` **Tasks**: Interactive game levels (25 games, 181 levels) **Evaluation**: RHAE score (ratio of human to agent actions, squared) **Metric**: Average RHAE across levels (0.0 to 1.0) --- ## Evolution Algorithms ### AEvolveEngine (SkillForge) **Module**: `agent_evolve.algorithms.skillforge.engine` **Strategy**: LLM-driven workspace mutation The default engine gives an LLM full bash tool access to the workspace and asks it to improve the agent based on observations. This is the most flexible engine — it can make arbitrary changes to any workspace file. **Context provided to the LLM:** - Recent observations (task inputs, agent outputs, feedback) - Current system prompt - Current skill library - Pending draft proposals - Score history **Mutation capabilities:** - Edit system prompt (refine, consolidate, extend) - Create new skills from observed patterns - Merge overlapping skills - Write episodic memory entries - Review and curate draft proposals ### GuidedSynthesisEngine **Module**: `agent_evolve.algorithms.guided_synth` **Strategy**: Memory-first, curated skills Emphasizes learning from failures before creating skills. Conservative approach that prevents skill bloat. **Process:** 1. Extract lessons from failed tasks 2. Write episodic memory entries 3. After accumulating patterns, synthesize skill proposals 4. Curate proposals: ACCEPT, MERGE, or SKIP ### AdaptiveEvolutionEngine **Module**: `agent_evolve.algorithms.adaptive` **Strategy**: Reward tracking + observation filtering Adjusts intervention intensity based on score trends. Makes smaller changes when improving, larger changes when plateaued. ### AdaptiveSkillEngine **Module**: `agent_evolve.algorithms.adaptive_skill` **Strategy**: Skill-centric discovery Focuses exclusively on building the skill library. Identifies task categories where the agent fails and creates targeted skills. --- ## Skill System ### Skill File Format ```markdown --- name: verify-edge-cases description: "TRIGGER when: checking boundary conditions. DO NOT TRIGGER: for happy-path tests." --- ## Pattern Test all falsy-but-valid values: 0, False, "", [], {} ## Process 1. List all input boundaries 2. Run each against the implementation 3. Check both output AND side effects ``` ### Skill Discovery Process 1. **Agent proposes**: During `solve()`, agent writes draft to `skills/_drafts/` 2. **Engine reviews**: During `step()`, engine reads drafts and decides: - **ACCEPT**: Move to `skills/{name}/SKILL.md` - **MERGE**: Combine with existing similar skill - **SKIP**: Discard (too narrow, redundant, or incorrect) 3. **Engine creates**: Engine can also create skills directly from observation analysis 4. **Refinement**: Existing skills are updated based on new observations ### Skill Library Management Target: 5-10 broad, reusable skills per workspace. Avoid: - 30+ narrow skills (library bloat) - Skills that duplicate system prompt content - Skills with no TRIGGER condition (always-on = should be in prompt) --- ## Memory System ### Episodic Memory Records specific lessons from task attempts: ```json {"content": "pytest --no-header flag needed for clean output", "category": "episodic", "task_id": "django-16379"} {"content": "Off-by-one errors common in range() with len()", "category": "episodic", "task_id": "numpy-8823"} ``` ### Semantic Memory General domain knowledge: ```json {"content": "Django uses reverse URL resolution via urlpatterns", "category": "semantic"} {"content": "NumPy broadcasting rules: dimensions must match or be 1", "category": "semantic"} ``` ### Memory Limits - `BaseAgent.reload_from_fs()` loads up to 200 memory entries by default - `AgentWorkspace.read_memories()` defaults to limit=100 - Old memories should be pruned or consolidated during evolution --- ## Version Control ### Git Tagging Convention | Tag | When Created | Purpose | |-----|-------------|---------| | `pre-evo-1` | Before cycle 1 evolution | Snapshot of solve-only state | | `evo-1` | After cycle 1 evolution | Snapshot of evolved state | | `pre-evo-2` | Before cycle 2 evolution | Snapshot before next mutation | | `evo-2` | After cycle 2 evolution | Snapshot of evolved state | ### Useful Git Commands ```bash # See all evolution checkpoints git tag -l "evo-*" # Compare two evolution stages git diff evo-1 evo-10 # See what changed in a specific cycle git diff pre-evo-5 evo-5 # Read a file at a specific point in time git show evo-3:prompts/system.md # Revert to a known good state git checkout evo-5 -- . ``` --- ## Observation Pipeline ### JSONL Format Each observation is stored in `evolution/observations/batch_{label}.jsonl`: ```json { "task_id": "django__django-16379", "task_input": "Fix FileBasedCache has_key method...", "task_metadata": {}, "agent_output": "--- a/django/core/cache/backends/filebased.py\n+++ ...", "steps": [ {"tool": "bash", "action": "read_file", "file": "django/core/cache/backends/filebased.py"}, {"tool": "text_editor", "action": "edit", "file": "django/core/cache/backends/filebased.py"} ], "success": true, "score": 1.0, "feedback_detail": "All 24 tests passed" } ``` ### Querying Observations ```python history = EvolutionHistory("./my-workspace") # All observations from last 3 cycles recent = history.get_observations(last_n_cycles=3) # Only failures failures = history.get_observations(only_failures=True) # Score curve scores = history.get_score_curve() # [(1, 0.62), (2, 0.68), ...] ``` --- ## FAQ ### Can I use A-Evolve with any LLM? Yes. The agent can use any LLM for solving. The evolver model is configurable via `EvolveConfig.evolver_model`. Supported providers: Anthropic (direct API), OpenAI, AWS Bedrock, LiteLLM (multi-provider). ### Does evolution require training data? No in the traditional ML sense. You need a `BenchmarkAdapter` that provides tasks and evaluation, but there are no training/gradient steps. Evolution is purely file-system mutation guided by LLM reasoning. ### How many cycles should I run? Start with 10 cycles and check convergence. If score is still improving, run more. Default convergence detection (`egl_threshold=0.05`, `egl_window=3`) stops automatically when improvement plateaus. ### Can I resume evolution after stopping? Yes. The workspace retains its evolved state. Create a new `Evolver` pointing to the same workspace and call `run()` again. ### Is evolution deterministic? No. LLM calls are inherently non-deterministic. Running the same config twice may produce different evolved agents with similar final scores. ### Can I evolve multiple agents simultaneously? Yes, but each must have its own workspace directory. The evolution loop modifies workspace files directly, so concurrent access to the same workspace is not safe. ### What's the cost per evolution cycle? Each cycle involves: (batch_size) agent solve calls + 1 evolver call. For batch_size=10 with Claude, expect ~$5-20 per cycle depending on task complexity and model used. ### Can I use A-Evolve without a benchmark? Not directly. The evolution loop requires `BenchmarkAdapter.evaluate()` to produce `Feedback`. However, you can implement a custom benchmark that uses human evaluation, LLM-as-judge, or any other scoring mechanism. --- ### Overview (`20-ml-paper-writing/ml-paper-writing/templates/README.md`) # LaTeX Templates for ML/AI Conferences This directory contains official LaTeX templates for major machine learning and AI conferences. **Systems conference templates** (OSDI, NSDI, ASPLOS, SOSP) have moved to the [systems-paper-writing](../../systems-paper-writing/templates/) skill. ## Compiling LaTeX to PDF ### Option 1: VS Code with LaTeX Workshop (Recommended) **Setup:** 1. Install [TeX Live](https://www.tug.org/texlive/) (full distribution recommended) - macOS: `brew install --cask mactex` - Ubuntu: `sudo apt install texlive-full` - Windows: Download from [tug.org/texlive](https://www.tug.org/texlive/) 2. Install VS Code extension: **LaTeX Workshop** by James Yu - Open VS Code → Extensions (Cmd/Ctrl+Shift+X) → Search "LaTeX Workshop" → Install **Usage:** - Open any `.tex` file in VS Code - Save the file (Cmd/Ctrl+S) → Auto-compiles to PDF - Click the green play button or use `Cmd/Ctrl+Alt+B` to build - View PDF: Click "View LaTeX PDF" icon or `Cmd/Ctrl+Alt+V` - Side-by-side view: `Cmd/Ctrl+Alt+V` then drag tab **Settings** (add to VS Code `settings.json`): ```json { "latex-workshop.latex.autoBuild.run": "onSave", "latex-workshop.view.pdf.viewer": "tab", "latex-workshop.latex.recipes": [ { "name": "pdflatex → bibtex → pdflatex × 2", "tools": ["pdflatex", "bibtex", "pdflatex", "pdflatex"] } ] } ``` ### Option 2: Command Line ```bash # Basic compilation pdflatex main.tex # With bibliography (full workflow) pdflatex main.tex bibtex main pdflatex main.tex pdflatex main.tex # Using latexmk (handles dependencies automatically) latexmk -pdf main.tex # Continuous compilation (watches for changes) latexmk -pdf -pvc main.tex ``` ### Option 3: Overleaf (Online) 1. Go to [overleaf.com](https://www.overleaf.com) 2. New Project → Upload Project → Upload the template folder as ZIP 3. Edit online with real-time PDF preview 4. No local installation needed ### Option 4: Other IDEs | IDE | Extension/Plugin | Notes | |-----|------------------|-------| | **Cursor** | LaTeX Workshop | Same as VS Code | | **Sublime Text** | LaTeXTools | Popular, well-maintained | | **Vim/Neovim** | VimTeX | Powerful, keyboard-driven | | **Emacs** | AUCTeX | Comprehensive LaTeX environment | | **TeXstudio** | Built-in | Dedicated LaTeX IDE | | **Texmaker** | Built-in | Cross-platform LaTeX editor | ### Troubleshooting Compilation **"File not found" errors:** ```bash # Ensure you're in the template directory cd templates/icml2026 pdflatex example_paper.tex ``` **Bibliography not appearing:** ```bash # Run bibtex after first pdflatex pdflatex main.tex bibtex main # Uses main.aux to find citations pdflatex main.tex # Incorporates bibliography pdflatex main.tex # Resolves references ``` **Missing packages:** ```bash # TeX Live package manager tlmgr install # Or install full distribution to avoid this ``` ## Available Templates ### ML/AI Conferences / ML/AI | Conference | Directory | Year | Source | |------------|-----------|------|--------| | ICML | `icml2026/` | 2026 | [Official ICML](https://icml.cc/Conferences/2026/AuthorInstructions) | | ICLR | `iclr2026/` | 2026 | [Official GitHub](https://github.com/ICLR/Master-Template) | | NeurIPS | `neurips2025/` | 2025 | Community template | | ACL | `acl/` | 2025+ | [Official ACL](https://github.com/acl-org/acl-style-files) | | AAAI | `aaai2026/` | 2026 | [AAAI Author Kit](https://aaai.org/authorkit26/) | | COLM | `colm2025/` | 2025 | [Official COLM](https://github.com/COLM-org/Template) | ### Systems Conferences Systems conference templates (OSDI, NSDI, ASPLOS, SOSP) are now in the [systems-paper-writing](../../systems-paper-writing/templates/) skill. ## Usage ### ICML 2026 ```latex \documentclass{article} \usepackage{icml2026} % For submission % \usepackage[accepted]{icml2026} % For camera-ready \begin{document} % Your paper content \end{document} ``` Key files: - `icml2026.sty` - Style file - `icml2026.bst` - Bibliography style - `example_paper.tex` - Example document ### ICLR 2026 ```latex \documentclass{article} \usepackage[submission]{iclr2026_conference} % For submission % \usepackage[final]{iclr2026_conference} % For camera-ready \begin{document} % Your paper content \end{document} ``` Key files: - `iclr2026_conference.sty` - Style file - `iclr2026_conference.bst` - Bibliography style - `iclr2026_conference.tex` - Example document ### ACL Venues (ACL, EMNLP, NAACL) ```latex \documentclass[11pt]{article} \usepackage[review]{acl} % For review % \usepackage{acl} % For camera-ready \begin{document} % Your paper content \end{document} ``` Key files: - `acl.sty` - Style file - `acl_natbib.bst` - Bibliography style - `acl_latex.tex` - Example document ### AAAI 2026 ```latex \documentclass[letterpaper]{article} \usepackage[submission]{aaai2026} % For submission % \usepackage{aaai2026} % For camera-ready \begin{document} % Your paper content \end{document} ``` Key files: - `aaai2026.sty` - Style file - `aaai2026.bst` - Bibliography style ### COLM 2025 ```latex \documentclass{article} \usepackage[submission]{colm2025_conference} % For submission % \usepackage[final]{colm2025_conference} % For camera-ready \begin{document} % Your paper content \end{document} ``` Key files: - `colm2025_conference.sty` - Style file - `colm2025_conference.bst` - Bibliography style ## Page Limits Summary | Conference | Submission | Camera-Ready | Notes | |------------|-----------|--------------|-------| | ICML 2026 | 8 pages | 9 pages | +unlimited refs/appendix | | ICLR 2026 | 9 pages | 10 pages | +unlimited refs/appendix | | NeurIPS 2025 | 9 pages | 9 pages | +checklist outside limit | | ACL 2025 | 8 pages (long) | varies | +unlimited refs/appendix | | AAAI 2026 | 7 pages | 8 pages | +unlimited refs/appendix | | COLM 2025 | 9 pages | 10 pages | +unlimited refs/appendix | **Systems conferences** (OSDI, NSDI, ASPLOS, SOSP): See the [systems-paper-writing](../../systems-paper-writing/templates/) skill for page limits and templates. ## Common Issues ### Compilation Errors 1. **Missing packages**: Install full TeX distribution (TeX Live Full or MikTeX) 2. **Bibliography errors**: Use the provided `.bst` file with `\bibliographystyle{}` 3. **Font warnings**: Install `cm-super` or use `\usepackage{lmodern}` ### Anonymization For submission, ensure: - No author names in `\author{}` - No acknowledgments section - No grant numbers - Use anonymous repositories - Cite own work in third person ### Common LaTeX Packages ```latex % Recommended packages (check compatibility with venue style) \usepackage{amsmath,amsthm,amssymb} % Math \usepackage{graphicx} % Figures \usepackage{booktabs} % Tables \usepackage{hyperref} % Links \usepackage{algorithm,algorithmic} % Algorithms \usepackage{natbib} % Citations ``` ## Updating Templates Templates are updated annually. Check official sources before each submission: **ML/AI:** - ICML: https://icml.cc/ - ICLR: https://iclr.cc/ - NeurIPS: https://neurips.cc/ - ACL: https://github.com/acl-org/acl-style-files - AAAI: https://aaai.org/ - COLM: https://colmweb.org/ **Systems:** See the [systems-paper-writing](../../systems-paper-writing/) skill for OSDI, NSDI, ASPLOS, SOSP template sources --- ### Overview (`20-ml-paper-writing/ml-paper-writing/templates/aaai2026/README.md`) # AAAI 2026 统一LaTeX模板使用说明 / AAAI 2026 Unified LaTeX Template Guide > **📝 重要说明 / Important Notice**: 本仓库借助Cursor在AAAI 2026官方模板基础上改进得到。如果遇到不满足或有冲突的情况，请积极提issues。 > > **📝 Important Notice**: This repository is improved based on the official AAAI 2026 template with the assistance of Cursor. If you encounter any issues or conflicts, please actively submit issues. [中文](#中文版本) | [English](#english-version) --- ## 🌐 在线查看 / Online Access **📖 在线阅读和测试模板**: [https://cn.overleaf.com/read/wyhcnvcrtpyt#cd4a07](https://cn.overleaf.com/read/wyhcnvcrtpyt#cd4a07) **📖 Online View and Test Template**: [https://cn.overleaf.com/read/wyhcnvcrtpyt#cd4a07](https://cn.overleaf.com/read/wyhcnvcrtpyt#cd4a07) 💡 **提示 / Tips**: - 中文：您可以通过上述链接在Overleaf中直接查看、编辑和编译模板，无需本地安装LaTeX环境 - English: You can view, edit, and compile the template directly in Overleaf using the link above, without needing a local LaTeX installation --- ## 中文版本 ### 概述 ✅ 我已经将AAAI 2026的两个版本（匿名投稿版本和camera-ready版本）**完整合并**成一个统一的模板文件 `aaai2026-unified-template.tex`。 该模板包含了原始两个模板的**所有完整内容**（共886行，比原始文件更全面），包括： - 所有格式化说明和要求 - 完整的示例代码和表格 - 图片处理指南 - 参考文献格式要求 - 所有章节和附录内容 - 版本特定的Acknowledgments部分 ### 主要差异分析 通过比较原始的两个模板，我发现主要差异在于： #### 1. 包的加载方式 - **匿名版本**: `\usepackage[submission]{aaai2026}` - **Camera-ready版本**: `\usepackage{aaai2026}` #### 2. 标题差异 - **匿名版本**: "AAAI Press Anonymous Submission Instructions for Authors Using LaTeX" - **Camera-ready版本**: "AAAI Press Formatting Instructions for Authors Using LaTeX --- A Guide" #### 3. Links环境的处理 - **匿名版本**: Links环境被注释掉，防止泄露作者身份 - **Camera-ready版本**: Links环境正常显示 #### 4. 内容部分差异 - **匿名版本**: 包含"Preparing an Anonymous Submission"部分的特殊说明 - **Camera-ready版本**: 包含完整的格式说明和版权信息 ### 依赖文件检查结果 ✅ **已验证并复制到主目录的文件**： - `aaai2026.sty` - AAAI 2026 样式文件（两个版本完全相同） - `aaai2026.bst` - 参考文献样式文件（两个版本完全相同） - `aaai2026.bib` - 示例参考文献文件 - `figure1.pdf` 和 `figure2.pdf` - 示例图片文件 所有这些文件在两个版本中都是相同的，因此统一模板可以正常工作。 ### 如何使用统一模板 #### 切换到匿名投稿版本 在模板文件第11行，**取消注释**这一行： ```latex \def\aaaianonymous{true} ``` #### 切换到Camera-ready版本 在模板文件第11行，**注释掉**或**删除**这一行： ```latex % \def\aaaianonymous{true} ``` ### 一键切换的核心机制 统一模板使用了LaTeX的条件编译功能： ```latex % 条件包加载 \ifdefined\aaaianonymous \usepackage[submission]{aaai2026} % 匿名版本 \else \usepackage{aaai2026} % Camera-ready版本 \fi % 条件标题设置 \ifdefined\aaaianonymous \title{AAAI Press Anonymous Submission\\Instructions for Authors Using \LaTeX{}} \else \title{AAAI Press Formatting Instructions \\for Authors Using \LaTeX{} --- A Guide} \fi % 条件内容显示 \ifdefined\aaaianonymous % 匿名版本特有内容 \else % Camera-ready版本特有内容 \fi ``` ### 文件清单 主目录现在包含以下文件： - `aaai2026-unified-template.tex` - 统一主论文模板文件 - `aaai2026-unified-supp.tex` - 统一补充材料模板文件 - `aaai2026.sty` - AAAI 2026 LaTeX 样式文件 - `aaai2026.bst` - 参考文献样式文件 - `aaai2026.bib` - 示例参考文献文件 - `figure1.pdf` - 示例图片1 - `figure2.pdf` - 示例图片2 - `README.md` - 本说明文档 ### 补充材料模板 (Supplementary Material Template) #### 概述 `aaai2026-unified-supp.tex` 是专门为AAAI 2026补充材料设计的统一模板，与主论文模板使用相同的版本切换机制。 #### 主要功能 - **版本切换**: 通过修改一行代码在匿名投稿和camera-ready版本间切换 - **补充内容支持**: 支持额外的实验、推导、数据、图表、算法等 - **格式一致性**: 与主论文模板保持完全一致的格式要求 - **代码示例**: 包含算法、代码列表等补充材料的示例 #### 使用方法 与主论文模板相同，只需修改第11行： ```latex % 匿名投稿版本 \def\aaaianonymous{true} % Camera-ready版本 % \def\aaaianonymous{true} ``` #### 补充材料内容建议 - 额外的实验结果和消融研究 - 详细的数学推导和证明 - 更多的图表和可视化 - 算法伪代码和实现细节 - 数据集描述和预处理步骤 - 超参数设置和实验配置 - 失败案例分析 - 计算复杂度分析 ### 使用检查清单 (Usage Checklist) #### 📋 投稿前检查清单 (Pre-Submission Checklist) **版本设置**: - [ ] 已设置 `\def\aaaianonymous{true}` (匿名投稿) - [ ] 已注释掉所有可能暴露身份的信息 - [ ] 已匿名化参考文献（移除作者姓名） **内容完整性**: - [ ] 标题、摘要、关键词已填写 - [ ] 所有章节内容完整 - [ ] 图表编号连续且正确 - [ ] 参考文献格式正确 - [ ] 补充材料（如有）已准备 **格式检查**: - [ ] 页面边距符合要求 - [ ] 字体和字号正确 - [ ] 行间距符合标准 - [ ] 图表位置和大小合适 - [ ] 数学公式格式正确 **技术检查**: - [ ] LaTeX编译无错误 - [ ] 参考文献正确生成 - [ ] PDF输出正常 - [ ] 文件大小在限制范围内 #### 📋 录用后检查清单 (Post-Acceptance Checklist) **版本切换**: - [ ] 已注释掉 `\def\aaaianonymous{true}` (camera-ready) - [ ] 已添加完整的作者信息 - [ ] 已添加所有作者单位信息 - [ ] 已恢复所有被注释的内容 **内容更新**: - [ ] 已根据审稿意见修改内容 - [ ] 已更新所有图表和实验 - [ ] 已完善补充材料 - [ ] 已检查所有链接和引用 **最终检查**: - [ ] 最终PDF质量检查 - [ ] 所有文件已备份 - [ ] 符合会议最终提交要求 - [ ] 补充材料已单独提交（如需要） #### 📋 补充材料检查清单 (Supplementary Material Checklist) **内容组织**: - [ ] 补充材料与主论文内容对应 - [ ] 章节结构清晰合理 - [ ] 图表编号与主论文不冲突 - [ ] 参考文献格式一致 **技术细节**: - [ ] 算法伪代码清晰完整 - [ ] 实验设置详细说明 - [ ] 数据预处理步骤明确 - [ ] 超参数配置完整 **格式要求**: - [ ] 使用统一的supp模板 - [ ] 页面设置与主论文一致 - [ ] 字体和格式符合要求 - [ ] 文件大小在限制范围内 ### 实际使用建议 1. **投稿阶段**: - 取消注释 `\def\aaaianonymous{true}` - 确保不包含任何可能暴露身份的信息 - 检查参考文献是否已匿名化 2. **录用后准备final版本**: - 注释掉或删除 `\def\aaaianonymous{true}` 这一行 - 添加完整的作者信息和affiliations - 取消注释links环境（如果需要） 3. **编译测试**: - 分别在两种模式下编译，确保都能正常工作 - 检查输出的PDF是否符合要求 - 验证参考文献格式是否正确 4. **依赖文件确认**: - 确保所有依赖文件都在同一目录下 - 如果移动模板文件，记得同时移动依赖文件 ### 重要注意事项 ⚠️ **关于Bibliography Style**: - `aaai2026.sty`文件已经自动设置了`\bibliographystyle{aaai2026}` - **不要**在文档中再次添加`\bibliographystyle{aaai2026}`命令 - 否则会出现"`Illegal, another \bibstyle command`"错误 - 只需要使用`\bibliography{aaai2026}`命令即可 ### 编译命令示例 ```bash # 编译LaTeX文档 pdflatex aaai2026-unified-template.tex bibtex aaai2026-unified-template pdflatex aaai2026-unified-template.tex pdflatex aaai2026-unified-template.tex ``` ### 常见问题解决 #### 1. "Illegal, another \bibstyle command"错误 **原因**: 重复设置了bibliography style **解决方案**: 删除文档中的`\bibliographystyle{aaai2026}`命令，`aaai2026.sty`会自动处理 #### 2. 参考文献格式不正确 **原因**: 可能缺少natbib包或者BibTeX文件问题 **解决方案**: 确保按照标准的LaTeX编译流程：pdflatex → bibtex → pdflatex → pdflatex --- ## English Version ### Overview ✅ I have **completely merged** the two AAAI 2026 versions (anonymous submission and camera-ready) into a single unified template file `aaai2026-unified-template.tex`. This template contains **all complete content** from both original templates (886 lines total, more comprehensive than the original files), including: - All formatting instructions and requirements - Complete example codes and tables - Image processing guidelines - Reference formatting requirements - All sections and appendix content - Version-specific Acknowledgments sections ### Key Differences Analysis By comparing the two original templates, the main differences are: #### 1. Package Loading Method - **Anonymous version**: `\usepackage[submission]{aaai2026}` - **Camera-ready version**: `\usepackage{aaai2026}` #### 2. Title Differences - **Anonymous version**: "AAAI Press Anonymous Submission Instructions for Authors Using LaTeX" - **Camera-ready version**: "AAAI Press Formatting Instructions for Authors Using LaTeX --- A Guide" #### 3. Links Environment Handling - **Anonymous version**: Links environment commented out to prevent identity disclosure - **Camera-ready version**: Links environment displayed normally #### 4. Content Section Differences - **Anonymous version**: Contains special instructions in "Preparing an Anonymous Submission" section - **Camera-ready version**: Contains complete formatting instructions and copyright information ### Dependency Files Verification ✅ **Files verified and copied to main directory**: - `aaai2026.sty` - AAAI 2026 style file (identical in both versions) - `aaai2026.bst` - Bibliography style file (identical in both versions) - `aaai2026.bib` - Sample bibliography file - `figure1.pdf` and `figure2.pdf` - Sample image files All these files are identical in both versions, so the unified template works properly. ### How to Use the Unified Template #### Switch to Anonymous Submission Version On line 11 of the template file, **uncomment** this line: ```latex \def\aaaianonymous{true} ``` #### Switch to Camera-ready Version On line 11 of the template file, **comment out** or **delete** this line: ```latex % \def\aaaianonymous{true} ``` ### Core Mechanism of One-Click Switching The unified template uses LaTeX conditional compilation: ```latex % Conditional package loading \ifdefined\aaaianonymous \usepackage[submission]{aaai2026} % Anonymous version \else \usepackage{aaai2026} % Camera-ready version \fi % Conditional title setting \ifdefined\aaaianonymous \title{AAAI Press Anonymous Submission\\Instructions for Authors Using \LaTeX{}} \else \title{AAAI Press Formatting Instructions \\for Authors Using \LaTeX{} --- A Guide} \fi % Conditional content display \ifdefined\aaaianonymous % Anonymous version specific content \else % Camera-ready version specific content \fi ``` ### File List The main directory now contains the following files: - `aaai2026-unified-template.tex` - Unified main paper template file - `aaai2026-unified-supp.tex` - Unified supplementary material template file - `aaai2026.sty` - AAAI 2026 LaTeX style file - `aaai2026.bst` - Bibliography style file - `aaai2026.bib` - Sample bibliography file - `figure1.pdf` - Sample image 1 - `figure2.pdf` - Sample image 2 - `README.md` - This documentation ### Supplementary Material Template #### Overview `aaai2026-unified-supp.tex` is a unified template specifically designed for AAAI 2026 supplementary materials, using the same version switching mechanism as the main paper template. #### Key Features - **Version Switching**: Switch between anonymous submission and camera-ready versions by modifying one line of code - **Supplementary Content Support**: Supports additional experiments, derivations, data, figures, algorithms, etc. - **Format Consistency**: Maintains complete format consistency with the main paper template - **Code Examples**: Includes examples for algorithms, code listings, and other supplementary materials #### Usage Same as the main paper template, just modify line 11: ```latex % Anonymous submission version \def\aaaianonymous{true} % Camera-ready version % \def\aaaianonymous{true} ``` #### Supplementary Material Content Suggestions - Additional experimental results and ablation studies - Detailed mathematical derivations and proofs - More figures and visualizations - Algorithm pseudocode and implementation details - Dataset descriptions and preprocessing steps - Hyperparameter settings and experimental configurations - Failure case analysis - Computational complexity analysis ### Usage Checklist #### 📋 Pre-Submission Checklist **Version Setup**: - [ ] Set `\def\aaaianonymous{true}` (anonymous submission) - [ ] Commented out all information that could reveal identity - [ ] Anonymized references (removed author names) **Content Completeness**: - [ ] Title, abstract, and keywords filled - [ ] All sections complete - [ ] Figure and table numbers consecutive and correct - [ ] Reference format correct - [ ] Supplementary materials prepared (if any) **Format Check**: - [ ] Page margins meet requirements - [ ] Font and font size correct - [ ] Line spacing meets standards - [ ] Figure and table positions and sizes appropriate - [ ] Mathematical formula format correct **Technical Check**: - [ ] LaTeX compilation error-free - [ ] References generated correctly - [ ] PDF output normal - [ ] File size within limits #### 📋 Post-Acceptance Checklist **Version Switch**: - [ ] Commented out `\def\aaaianonymous{true}` (camera-ready) - [ ] Added complete author information - [ ] Added all author affiliation information - [ ] Restored all commented content **Content Updates**: - [ ] Modified content according to reviewer comments - [ ] Updated all figures and experiments - [ ] Completed supplementary materials - [ ] Checked all links and citations **Final Check**: - [ ] Final PDF quality check - [ ] All files backed up - [ ] Meets conference final submission requirements - [ ] Supplementary materials submitted separately (if needed) #### 📋 Supplementary Material Checklist **Content Organization**: - [ ] Supplementary materials correspond to main paper content - [ ] Chapter structure clear and reasonable - [ ] Figure and table numbers don't conflict with main paper - [ ] Reference format consistent **Technical Details**: - [ ] Algorithm pseudocode clear and complete - [ ] Experimental setup explained in detail - [ ] Data preprocessing steps clear - [ ] Hyperparameter configuration complete **Format Requirements**: - [ ] Using unified supp template - [ ] Page settings consistent with main paper - [ ] Font and format meet requirements - [ ] File size within limits ### Practical Usage Recommendations 1. **Submission Stage**: - Uncomment `\def\aaaianonymous{true}` - Ensure no information that could reveal identity is included - Check that references are anonymized 2. **Preparing final version after acceptance**: - Comment out or delete the `\def\aaaianonymous{true}` line - Add complete author information and affiliations - Uncomment links environment (if needed) 3. **Compilation Testing**: - Compile in both modes to ensure proper functionality - Check if the output PDF meets requirements - Verify reference formatting is correct 4. **Dependency File Confirmation**: - Ensure all dependency files are in the same directory - Remember to move dependency files when moving the template file ### Important Notes ⚠️ **About Bibliography Style**: - The `aaai2026.sty` file automatically sets `\bibliographystyle{aaai2026}` - **Do NOT** add `\bibliographystyle{aaai2026}` command again in your document - Otherwise you'll get "`Illegal, another \bibstyle command`" error - Just use the `\bibliography{aaai2026}` command ### Compilation Commands Example ```bash # Compile LaTeX document pdflatex aaai2026-unified-template.tex bibtex aaai2026-unified-template pdflatex aaai2026-unified-template.tex pdflatex aaai2026-unified-template.tex ``` ### Common Issues and Solutions #### 1. "Illegal, another \bibstyle command" Error **Cause**: Duplicate bibliography style setting **Solution**: Remove the `\bibliographystyle{aaai2026}` command from your document, `aaai2026.sty` handles it automatically #### 2. Incorrect Reference Format **Cause**: Missing natbib package or BibTeX file issues **Solution**: Follow the standard LaTeX compilation process: pdflatex → bibtex → pdflatex → pdflatex --- ## 版本信息 / Version Information - **模板版本 / Template Version**: AAAI 2026 Unified (Main + Supplementary) - **创建日期 / Created**: 2024年12月 - **支持格式 / Supported Formats**: Anonymous Submission & Camera-Ready - **模板类型 / Template Types**: Main Paper Template & Supplementary Material Template - **兼容性 / Compatibility**: LaTeX 2020+ / TeXLive 2024+ --- 🎉 **现在您只需要修改一行代码就可以在两个版本之间切换，同时所有必要的依赖文件都已经准备就绪！** 🎉 **Now you only need to modify one line of code to switch between the two versions, with all necessary dependency files ready to use!** --- ### Overview (`20-ml-paper-writing/ml-paper-writing/templates/acl/README.md`) # *ACL Paper Styles This directory contains the latest LaTeX templates for *ACL conferences. ## Instructions for authors Paper submissions to *ACL conferences must use the official ACL style templates. The LaTeX style files are available - as an [Overleaf template](https://www.overleaf.com/latex/templates/association-for-computational-linguistics-acl-conference/jvxskxpnznfj) - in this repository - as a [.zip file](https://github.com/acl-org/acl-style-files/archive/refs/heads/master.zip) Please see [`acl_latex.tex`](https://github.com/acl-org/acl-style-files/blob/master/acl_latex.tex) for an example. Please follow the paper formatting guidelines general to *ACL conferences: - [Paper formatting guidelines](https://acl-org.github.io/ACLPUB/formatting.html) Authors may not modify these style files or use templates designed for other conferences. ## Instructions for publications chairs To adapt the style files for your conference, please fork this repository and make necessary changes. Minimally, you'll need to update the name of the conference and rename the files. If you make improvements to the templates that should be propagated to future conferences, please submit a pull request. Thank you in advance! In older versions of the templates, authors were asked to fill in the START submission ID so that it would be stamped at the top of each page of the anonymized version. This is no longer needed, because it is now possible to do this stamping automatically within START. Currently, the way to do this is for the program chair to email support@softconf.com and request it. ## Instructions for making changes to style files - merge pull request in github, or push to github - git pull from github to a local repository - then, git push from your local repository to overleaf project - Overleaf project is https://www.overleaf.com/project/5f64f1fb97c4c50001b60549 - Overleaf git url is https://git.overleaf.com/5f64f1fb97c4c50001b60549 - then, click "Submit" and then "Submit as Template" in overleaf in order to ask overleaf to update the overleaf template from the overleaf project --- ### Overview (`20-ml-paper-writing/ml-paper-writing/templates/colm2025/README.md`) # Template Template and style files for CoLM 2025 --- ### ROADMAP (`docs/ROADMAP.md`) # 🗺️ Roadmap ## Vision Build the most comprehensive open-source library of AI research skills, enabling AI agents to autonomously conduct experiments from hypothesis to deployment. **Target**: 86 comprehensive skills — achieved ✅ ## Progress Overview | Metric | Current | Target | |--------|---------|--------| | **Skills** | **86** (high-quality, standardized YAML) | 86 ✅ | | **Avg Lines/Skill** | **420 lines** (focused + progressive disclosure) | 200-500 lines | | **Documentation** | **~130,000 lines** total (SKILL.md + references) | 100,000+ lines | | **Gold Standard Skills** | **65** with comprehensive references | 50+ ✅ | | **Coverage** | Autoresearch, Ideation, Paper Writing, Architecture, Tokenization, Fine-Tuning, Data Processing, Post-Training, Safety, Distributed, Infrastructure, Optimization, Evaluation, Inference, Agents, RAG, Multimodal, MLOps, Observability, Prompt Engineering, Emerging Techniques | Full Lifecycle ✅ | ## Development Phases ### ✅ Phase 1: Model Architecture (COMPLETE - 5 skills) **Status**: Core model architectures covered **Completed Skills**: - ✅ **Megatron-Core** - NVIDIA's framework for training 2B-462B param models - ✅ **LitGPT** - Lightning AI's 20+ clean LLM implementations - ✅ **Mamba** - State-space models with O(n) complexity - ✅ **RWKV** - RNN+Transformer hybrid, infinite context - ✅ **NanoGPT** - Educational GPT in ~300 lines by Karpathy ### ✅ Phase 2: Tokenization (COMPLETE - 2 skills) **Status**: Essential tokenization frameworks covered **Completed Skills**: - ✅ **HuggingFace Tokenizers** - Rust-based, BPE/WordPiece/Unigram - ✅ **SentencePiece** - Language-independent tokenization ### ✅ Phase 3: Fine-Tuning (COMPLETE - 4 skills) **Status**: Core fine-tuning frameworks covered **Completed Skills**: - ✅ **Axolotl** - YAML-based fine-tuning with 100+ models - ✅ **LLaMA-Factory** - WebUI no-code fine-tuning - ✅ **Unsloth** - 2x faster QLoRA fine-tuning - ✅ **PEFT** - Parameter-efficient fine-tuning with LoRA, QLoRA, DoRA, 25+ methods ### ✅ Phase 4: Data Processing (COMPLETE - 2 skills) **Status**: Distributed data processing covered **Completed Skills**: - ✅ **Ray Data** - Distributed ML data processing - ✅ **NeMo Curator** - GPU-accelerated data curation ### ✅ Phase 5: Post-Training (COMPLETE - 4 skills) **Status**: RLHF and alignment techniques covered **Completed Skills**: - ✅ **TRL Fine-Tuning** - Transformer Reinforcement Learning - ✅ **GRPO-RL-Training** - Group Relative Policy Optimization (gold standard) - ✅ **OpenRLHF** - Full RLHF pipeline with Ray + vLLM - ✅ **SimPO** - Simple Preference Optimization ### ✅ Phase 6: Safety & Alignment (COMPLETE - 4 skills) **Status**: Core safety frameworks covered **Completed Skills**: - ✅ **Constitutional AI** - AI-driven self-improvement via principles - ✅ **LlamaGuard** - Safety classifier for LLM inputs/outputs - ✅ **NeMo Guardrails** - Programmable guardrails with Colang - ✅ **Prompt Guard** - Meta's 86M prompt injection & jailbreak detector ### ✅ Phase 7: Distributed Training (COMPLETE - 5 skills) **Status**: Major distributed training frameworks covered **Completed Skills**: - ✅ **DeepSpeed** - Microsoft's ZeRO optimization - ✅ **PyTorch FSDP** - Fully Sharded Data Parallel - ✅ **Accelerate** - HuggingFace's distributed training API - ✅ **PyTorch Lightning** - High-level training framework - ✅ **Ray Train** - Multi-node orchestration ### ✅ Phase 8: Optimization (COMPLETE - 6 skills) **Status**: Core optimization techniques covered **Completed Skills**: - ✅ **Flash Attention** - 2-4x faster attention with memory efficiency - ✅ **bitsandbytes** - 8-bit/4-bit quantization - ✅ **GPTQ** - 4-bit post-training quantization - ✅ **AWQ** - Activation-aware weight quantization - ✅ **HQQ** - Half-Quadratic Quantization without calibration data - ✅ **GGUF** - llama.cpp quantization format for CPU/Metal inference ### ✅ Phase 9: Evaluation (COMPLETE - 1 skill) **Status**: Standard benchmarking framework available **Completed Skills**: - ✅ **lm-evaluation-harness** - EleutherAI's standard for benchmarking LLMs ### ✅ Phase 10: Inference & Serving (COMPLETE - 4 skills) **Status**: Production inference frameworks covered **Completed Skills**: - ✅ **vLLM** - High-throughput LLM serving with PagedAttention - ✅ **TensorRT-LLM** - NVIDIA's fastest inference - ✅ **llama.cpp** - CPU/Apple Silicon inference - ✅ **SGLang** - Structured generation with RadixAttention ### ✅ Phase 10.5: Infrastructure (COMPLETE - 3 skills) **Status**: Cloud infrastructure and orchestration covered **Completed Skills**: - ✅ **Modal** - Serverless GPU cloud with Python-native API, T4-H200 on-demand - ✅ **SkyPilot** - Multi-cloud orchestration across 20+ providers with spot recovery - ✅ **Lambda Labs** - Reserved/on-demand GPU cloud with H100/A100, persistent filesystems ### ✅ Phase 11: Agents (COMPLETE - 4 skills) **Status**: Major agent frameworks covered **Completed Skills**: - ✅ **LangChain** - Most popular agent framework, 500+ integrations - ✅ **LlamaIndex** - Data framework for LLM apps, 300+ connectors - ✅ **CrewAI** - Multi-agent orchestration with role-based collaboration - ✅ **AutoGPT** - Autonomous AI agent platform with visual workflow builder ### ✅ Phase 12: RAG (COMPLETE - 5 skills) **Status**: Core RAG and vector database skills covered **Completed Skills**: - ✅ **Chroma** - Open-source embedding database - ✅ **FAISS** - Facebook's similarity search, billion-scale - ✅ **Sentence Transformers** - 5000+ embedding models - ✅ **Pinecone** - Managed vector database - ✅ **Qdrant** - High-performance Rust vector search with hybrid filtering ### ✅ Phase 13: Multimodal (COMPLETE - 7 skills) **Status**: Comprehensive multimodal frameworks covered **Completed Skills**: - ✅ **CLIP** - OpenAI's vision-language model - ✅ **Whisper** - Robust speech recognition, 99 languages - ✅ **LLaVA** - Vision-language assistant, GPT-4V level - ✅ **Stable Diffusion** - Text-to-image generation via HuggingFace Diffusers - ✅ **Segment Anything (SAM)** - Meta's zero-shot image segmentation with points/boxes/masks - ✅ **BLIP-2** - Vision-language pretraining with Q-Former, image captioning, VQA - ✅ **AudioCraft** - Meta's MusicGen/AudioGen for text-to-music and text-to-sound ### ✅ Phase 14: Advanced Optimization (COMPLETE) **Status**: Advanced optimization techniques covered (merged into Phase 8) **Note**: HQQ and GGUF skills have been completed and merged into Phase 8: Optimization. ### ✅ Phase 15: MLOps & Observability (COMPLETE - 5 skills) **Status**: Core MLOps and LLM observability covered **Completed Skills**: - ✅ **MLflow** - Open-source MLOps platform for tracking experiments - ✅ **TensorBoard** - Visualization and experiment tracking - ✅ **Weights & Biases** - Experiment tracking and collaboration - ✅ **LangSmith** - LLM observability, tracing, evaluation - ✅ **Phoenix** - Open-source AI observability with OpenTelemetry tracing ### ✅ Phase 16: Prompt Engineering & Advanced Applications (COMPLETE - 6 skills) **Status**: Core prompt engineering and multi-agent tools covered **Completed Skills**: - ✅ **DSPy** - Declarative prompt optimization and LM programming - ✅ **Guidance** - Constrained generation and structured prompting - ✅ **Instructor** - Structured output with Pydantic models - ✅ **Outlines** - Structured text generation with regex and grammars - ✅ **CrewAI** - Multi-agent orchestration (completed in Phase 11) - ✅ **AutoGPT** - Autonomous agents (completed in Phase 11) ### ✅ Phase 17: Extended Multimodal (COMPLETE) **Status**: All extended multimodal skills complete, merged into Phase 13 **Note**: BLIP-2, SAM, and AudioCraft have been completed and merged into Phase 13: Multimodal. ### ✅ Phase 18: Emerging Techniques (COMPLETE - 6 skills) **Status**: Core emerging techniques covered **Completed Skills**: - ✅ **MoE Training** - Mixture of Experts with DeepSpeed/HuggingFace - ✅ **Model Merging** - mergekit, SLERP, and model composition - ✅ **Long Context** - RoPE extensions, ALiBi, and context scaling - ✅ **Speculative Decoding** - Medusa, Lookahead, and draft models for faster inference - ✅ **Knowledge Distillation** - MiniLLM, reverse KLD, teacher-student training - ✅ **Model Pruning** - Wanda, SparseGPT, and structured pruning ## Contributing to the Roadmap Want to help us achieve these goals? 1. **Pick a skill from the roadmap** - Comment on [GitHub Discussions](https://github.com/orchestra-research/AI-research-SKILLs/discussions) to claim it 2. **Follow the [contribution guide](CONTRIBUTING.md)** - Use our template and quality standards 3. **Submit your PR** - We review within 48 hours ## 🎉 Roadmap Complete! All 70 skills have been completed! The library now covers the full AI research lifecycle: 1. ✅ **Phase 1-10**: Core ML infrastructure (Architecture, Tokenization, Fine-Tuning, Data Processing, Post-Training, Safety, Distributed Training, Optimization, Evaluation, Inference) 2. ✅ **Phase 10.5**: Infrastructure (Modal, SkyPilot, Lambda Labs) 3. ✅ **Phase 11-12**: Applications (Agents, RAG) 4. ✅ **Phase 13**: Multimodal (CLIP, Whisper, LLaVA, Stable Diffusion, SAM, BLIP-2, AudioCraft) 5. ✅ **Phase 14-16**: Advanced (Optimization, MLOps & Observability, Prompt Engineering) 6. ✅ **Phase 17-18**: Extended (Extended Multimodal, Emerging Techniques) ## Future Directions While the 70-skill roadmap is complete, the library will continue to evolve with: - **Updates**: Keeping existing skills current with latest versions - **Community contributions**: Additional skills from contributors - **Emerging tools**: New frameworks and techniques as they mature ## Philosophy **Quality over Quantity**: Each skill must provide real value with comprehensive guidance, not just links to docs. We aim for 300+ lines of expert-level content per skill, with real code examples, troubleshooting guides, and production-ready workflows. --- ### SKILL CREATION GUIDE (`docs/SKILL_CREATION_GUIDE.md`) # Skill Creation Guide **Based on**: [Anthropic Official Best Practices](anthropic_official_docs/best_practices.md) **Last Updated**: November 6, 2025 --- ## Core Principles (from Anthropic) ### 1. Concise is Key **The context window is a public good.** Your skill shares it with system prompts, conversation history, and other skills. **Default assumption: Claude is already smart** Only add context Claude doesn't already have. Challenge each piece of information: - "Does Claude really need this explanation?" - "Can I assume Claude knows this?" - "Does this paragraph justify its token cost?" **Good** (50 tokens): ```markdown ## Extract PDF text Use pdfplumber for text extraction: ```python import pdfplumber with pdfplumber.open("file.pdf") as pdf: text = pdf.pages[0].extract_text() ``` ``` **Bad** (150 tokens): ```markdown ## Extract PDF text PDF (Portable Document Format) files are a common file format that contains text, images, and other content. To extract text from a PDF, you'll need to use a library. There are many libraries available for PDF processing, but we recommend pdfplumber because it's easy to use and handles most cases well. First, you'll need to install it using pip. Then you can use the code below... ``` ### 2. Progressive Disclosure **SKILL.md serves as an overview** that points Claude to detailed materials as needed. - Keep SKILL.md body **under 500 lines** for optimal performance - Aim for **200-300 lines** in practice - Split content into separate reference files - Keep references **ONE LEVEL DEEP** from SKILL.md (no nested references) **Structure**: ``` skill-name/ ├── SKILL.md # Main overview (200-300 lines) ├── server-deployment.md # Specific topic (loaded as needed) ├── offline-inference.md # Another topic (loaded as needed) ├── optimization.md # Advanced topic (loaded as needed) └── scripts/ ├── validate.py # Utility script (executed, not loaded) └── helper.py # Another script ``` ### 3. Use Workflows with Checklists For multi-step tasks, provide copy-paste checklists: ```markdown ## Deployment workflow Copy this checklist and track progress: ``` Task Progress: - [ ] Step 1: Configure server settings - [ ] Step 2: Validate configuration - [ ] Step 3: Deploy to production - [ ] Step 4: Verify deployment ``` **Step 1: Configure server settings** Edit `config.yaml` with production values. **Step 2: Validate configuration** Run validator and fix errors: ```bash python validate.py config.yaml # If errors: fix → validate again → continue ``` **Step 3: Deploy to production** [Specific deployment command] **Step 4: Verify deployment** [Verification steps] ``` ### 4. Feedback Loops for Quality **Common pattern**: Run validator → fix errors → repeat ```markdown ## Document editing process 1. Make your edits to `document.xml` 2. **Validate immediately**: `python validate.py document.xml` 3. If validation fails: - Review the error message carefully - Fix the issues - Run validation again 4. **Only proceed when validation passes** 5. Export final document ``` --- ## YAML Frontmatter Requirements All SKILL.md files **must** include properly formatted YAML frontmatter with the following fields: ```yaml --- name: skill-name-here description: Third-person description of what this does and when to use it. Include key terms and triggers. Maximum 1024 characters. version: 1.0.0 author: Orchestra Research license: MIT tags: [Tag One, Tag Two, Tag Three] dependencies: [package1>=1.0.0, package2>=2.0.0] --- ``` ### Field Requirements | Field | Required | Format | Notes | |-------|----------|--------|-------| | `name` | ✅ Yes | kebab-case | No quotes, lowercase with hyphens | | `description` | ✅ Yes | Plain text | No quotes, concise explanation | | `version` | ✅ Yes | Semantic version | Format: `MAJOR.MINOR.PATCH` | | `author` | ✅ Yes | Plain text | Use "Orchestra Research" | | `license` | ✅ Yes | License identifier | Typically `MIT` | | `tags` | ✅ Yes | Array | Capitalized words, no quotes | | `dependencies` | ⚠️ Optional | Array | Include version constraints | **name** field: - Maximum 64 characters - Lowercase letters, numbers, hyphens only - No XML tags - No reserved words: "anthropic", "claude" - **Recommended**: Use gerund form (e.g., `serving-llms`, `processing-pdfs`, `analyzing-data`) **description** field: - Maximum 1024 characters - Non-empty - No XML tags - No quotes around the text - **MUST be third person**: "Processes files..." not "I can help you..." - Include **what** it does AND **when** to use it - Include key terms for discovery **tags** field: - Use **Title Case** for all tags (capitalize first letter of each word) - Keep acronyms **UPPERCASE** (e.g., `GRPO`, `TRL`, `RLHF`, `DPO`, `MLOps`, `RAG`) - Use descriptive, searchable terms - Include 5-10 relevant tags - No quotes around tags **dependencies** field: - Only include **direct dependencies** needed to use the skill - Include **minimum version constraints** using `>=` - No quotes around package names - List core packages first, optional packages last **Examples**: ✅ **Good**: ```yaml --- name: serving-llms description: Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency, or serving models with limited GPU memory. version: 1.0.0 author: Orchestra Research license: MIT tags: [Inference, Serving, vLLM, PagedAttention, Production Deployment, High Throughput] dependencies: [vllm>=0.2.0, torch>=2.0.0, transformers>=4.35.0] --- ``` ✅ **Good**: ```yaml --- name: processing-pdfs description: Extracts text and tables from PDF files, fills forms, merges documents. Use when working with PDF files or when the user mentions PDFs, forms, or document extraction. version: 1.0.0 author: Orchestra Research license: MIT tags: [PDF Processing, Text Extraction, Document Processing, Forms] dependencies: [pdfplumber>=0.9.0, PyPDF2>=3.0.0] --- ``` ❌ **Bad** (quotes and missing fields): ```yaml --- name: "skill-name" description: "I can help you process PDF files" --- ``` ❌ **Bad** (first person, vague): ```yaml --- name: docs-helper description: Helps with documents version: 1.0.0 author: Orchestra Research license: MIT tags: [documents] --- ``` --- ## Skill Structure Best Practices ### File Organization **Simple skill** (just SKILL.md): ``` skill-name/ └── SKILL.md ``` **Complex skill** (with references): ``` skill-name/ ├── SKILL.md # Overview, points to references ├── server-deployment.md # Topic-specific guide ├── offline-inference.md # Another topic ├── optimization.md # Advanced features ├── troubleshooting.md # Common issues └── scripts/ ├── validate.py # Utility script └── setup.sh # Setup script ``` **Domain-specific organization** (for Skills with multiple domains): ``` bigquery-skill/ ├── SKILL.md # Overview and navigation └── reference/ ├── finance.md # Revenue, billing metrics ├── sales.md # Opportunities, pipeline ├── product.md # API usage, features └── marketing.md # Campaigns, attribution ``` ### Reference Files **One level deep**: All reference files should link directly from SKILL.md ✅ **Good**: ```markdown # SKILL.md **Server deployment**: See [server-deployment.md](server-deployment.md) **Offline inference**: See [offline-inference.md](offline-inference.md) **API reference**: See [api-reference.md](api-reference.md) ``` ❌ **Bad** (nested references): ```markdown # SKILL.md See [advanced.md](advanced.md)... # advanced.md See [details.md](details.md)... # details.md Here's the actual information... ``` **Table of contents**: For reference files >100 lines, include table of contents at top ```markdown # API Reference ## Contents - Authentication and setup - Core methods (create, read, update, delete) - Advanced features (batch operations, webhooks) - Error handling patterns - Code examples ## Authentication and setup ... ``` --- ## Content Guidelines ### Assume Claude is Smart Don't explain basics. Assume Claude knows: - What PDFs are - How libraries work - What APIs are - Common programming concepts - Standard ML/AI terminology Only explain: - Domain-specific concepts unique to this tool - Non-obvious gotchas - Best practices from community experience ### Consistent Terminology Choose one term and use it throughout: ✅ **Good**: - Always "API endpoint" - Always "field" - Always "extract" ❌ **Bad**: - Mix "API endpoint", "URL", "API route", "path" - Mix "field", "box", "element", "control" - Mix "extract", "pull", "get", "retrieve" ### Avoid Time-Sensitive Information ❌ **Bad**: ```markdown If you're doing this before August 2025, use the old API. After August 2025, use the new API. ``` ✅ **Good**: ```markdown ## Current method Use the v2 API endpoint: `api.example.com/v2/messages` ## Old patterns
Legacy v1 API (deprecated 2025-08) The v1 API used: `api.example.com/v1/messages` This endpoint is no longer supported.
``` ### Provide Examples (Input/Output Pairs) For skills where output quality depends on seeing examples: ```markdown ## Commit message format Generate commit messages following these examples: **Example 1:** Input: Added user authentication with JWT tokens Output: ``` feat(auth): implement JWT-based authentication Add login endpoint and token validation middleware ``` **Example 2:** Input: Fixed bug where dates displayed incorrectly in reports Output: ``` fix(reports): correct date formatting in timezone conversion Use UTC timestamps consistently across report generation ``` Follow this style: type(scope): brief description, then detailed explanation. ``` --- ## Common Patterns ### Template Pattern Provide templates for output format. Match strictness to needs. **For strict requirements**: ````markdown ## Report structure ALWAYS use this exact template structure: ```markdown # [Analysis Title] ## Executive summary [One-paragraph overview of key findings] ## Key findings - Finding 1 with supporting data - Finding 2 with supporting data - Finding 3 with supporting data ## Recommendations 1. Specific actionable recommendation 2. Specific actionable recommendation ``` ```` **For flexible guidance**: ````markdown ## Report structure Here is a sensible default format, but use your best judgment: ```markdown # [Analysis Title] ## Executive summary [Overview] ## Key findings [Adapt sections based on what you discover] ## Recommendations [Tailor to the specific context] ``` Adjust sections as needed for the specific analysis type. ```` ### Conditional Workflow Pattern Guide Claude through decision points: ```markdown ## Document modification workflow 1. Determine the modification type: **Creating new content?** → Follow "Creation workflow" below **Editing existing content?** → Follow "Editing workflow" below 2. Creation workflow: - Use docx-js library - Build document from scratch - Export to .docx format 3. Editing workflow: - Unpack existing document - Modify XML directly - Validate after each change - Repack when complete ``` --- ## Anti-Patterns to Avoid ### ❌ Windows-Style Paths Always use forward slashes: ✅ **Good**: `scripts/helper.py`, `reference/guide.md` ❌ **Bad**: `scripts\helper.py`, `reference\guide.md` ### ❌ Too Many Options Don't present multiple approaches unless necessary: ❌ **Bad**: "You can use pypdf, or pdfplumber, or PyMuPDF, or pdf2image, or..." ✅ **Good**: "Use pdfplumber for text extraction: ```python import pdfplumber ``` For scanned PDFs requiring OCR, use pdf2image with pytesseract instead." ### ❌ Nested References ❌ **Bad**: SKILL.md → advanced.md → details.md → actual info ✅ **Good**: SKILL.md → [topic].md (all references one level deep) ### ❌ Over-Explaining Basics ❌ **Bad** (150 tokens): "PDF files are a common format. They contain text and images. To process them, you need a library. Python has many PDF libraries. We recommend pdfplumber because..." ✅ **Good** (30 tokens): "Use pdfplumber for PDF text extraction: ```python import pdfplumber with pdfplumber.open("file.pdf") as pdf: text = pdf.pages[0].extract_text() ```" --- ## Quality Checklist Before submitting a skill: ### Core Quality - [ ] Description is specific and includes key terms - [ ] Description includes both what it does and when to use it - [ ] SKILL.md body is under 500 lines (aim for 200-300) - [ ] Additional details in separate files (if needed) - [ ] No time-sensitive information (or in "old patterns" section) - [ ] Consistent terminology throughout - [ ] Examples are concrete, not abstract - [ ] File references are one level deep - [ ] Progressive disclosure used appropriately - [ ] Workflows have clear steps with checklists ### Code and Scripts - [ ] Scripts solve problems rather than punt to Claude - [ ] Error handling is explicit and helpful - [ ] No "magic numbers" (all values justified) - [ ] Required packages listed in instructions - [ ] No Windows-style paths (all forward slashes) - [ ] Validation/verification steps for critical operations - [ ] Feedback loops included for quality-critical tasks ### Content Quality - [ ] Assumes Claude is smart (no over-explaining basics) - [ ] Third person description - [ ] Gerund naming (e.g., "serving-llms" not "llm-server") - [ ] Clear when to use vs alternatives - [ ] Concrete examples with input/output pairs - [ ] Troubleshooting section with common issues --- ## Recommended Process ### 1. Research Phase - Read official documentation thoroughly - Analyze real-world usage (blog posts, Stack Overflow, GitHub issues) - Identify key concepts and common gotchas - Find production code examples ### 2. Outline Phase Create structure outline: 1. Quick start (20-30 lines) 2. Common workflows with checklists (80-120 lines) 3. When to use vs alternatives (20-30 lines) 4. Common issues (30-50 lines) 5. Advanced topics with links to reference files (10-20 lines) **Target**: 200-300 lines for SKILL.md ### 3. Writing Phase Use SKILL_TEMPLATE.md as starting point: - Fill in YAML frontmatter (name, description) - Write concise quick start - Create 2-3 workflows with copy-paste checklists - Add common issues section - Link to reference files for advanced topics ### 4. Reference Files Phase Create separate markdown files for: - Detailed API documentation - Advanced features - Troubleshooting guides - Configuration references - Domain-specific content Each file: - Has clear purpose - Links directly from SKILL.md - Includes table of contents if >100 lines - Focuses on one topic ### 5. Testing Phase Test with Claude: - Activate the skill - Try common workflows - Verify checklist format works - Test progressive disclosure (does Claude load right files?) - Check cross-references work ### 6. Iteration Phase Based on testing: - Simplify over-explained sections - Add missing common issues - Improve workflow clarity - Reorganize reference files if needed --- ## Examples of Good Skills **For structure reference**, see official Anthropic examples in `anthropic_official_docs/best_practices.md`: - PDF Processing skill (lines 286-307) - BigQuery skill (lines 316-344) - Git Commit Helper (lines 229-233) **From this project**: - Reference GRPO-RL-Training skill for comprehensive workflows - But make it MORE CONCISE following Anthropic guidelines --- ## Common Mistakes to Avoid 1. **Making SKILL.md too long** (>500 lines is RED FLAG) 2. **Over-explaining basics** (assume Claude knows ML/programming) 3. **No workflows with checklists** (makes complex tasks hard) 4. **Nested references** (keep one level deep) 5. **First-person descriptions** (use third person!) 6. **Vague skill names** (use gerund form with specific terms) 7. **No "when to use vs alternatives"** (critical for skill selection) 8. **Missing validation steps** (add feedback loops) 9. **Too many options** (provide default with escape hatch) 10. **Time-sensitive info** (use "old patterns" section instead) --- ## Resources - **Anthropic Official Best Practices**: [anthropic_official_docs/best_practices.md](anthropic_official_docs/best_practices.md) - **Skill Template**: [SKILL_TEMPLATE.md](SKILL_TEMPLATE.md) - **Contributing Guide**: [CONTRIBUTING.md](CONTRIBUTING.md) --- ### SKILL TEMPLATE (`docs/SKILL_TEMPLATE.md`) --- name: example-skill-name description: Brief third-person description of what this skill does and when to use it. Include key terms and triggers for discovery. Maximum 1024 characters. version: 1.0.0 author: Orchestra Research license: MIT tags: [Tag One, Tag Two, Tag Three, Key Concept, Use Case] dependencies: [package1>=1.0.0, package2>=2.0.0] --- # [Skill Title] ## Quick start [One paragraph overview of what this skill provides] **Basic usage**: ```[language] # Minimal working example (5-10 lines) import library result = library.function(input) print(result) ``` ## Common workflows ### Workflow 1: [Primary Use Case] Copy this checklist and track progress: ``` Task Progress: - [ ] Step 1: [First action] - [ ] Step 2: [Second action] - [ ] Step 3: [Validation step] - [ ] Step 4: [Completion step] ``` **Step 1: [First action]** [Brief instruction - assume Claude knows basics] ```[language] # Code example [concise code] ``` **Step 2: [Second action]** [Brief instruction] ```[language] # Code example [concise code] ``` **Step 3: [Validation step]** Run validator and fix errors if found: ```bash validate_script.py input.json # If errors: fix → validate again → continue ``` **Step 4: [Completion step]** [Final action] ### Workflow 2: [Secondary Use Case] [Similar structure with checklist] ## When to use vs alternatives **Use this when:** - [Specific scenario 1] - [Specific scenario 2] **Use [Alternative] instead when:** - [Different scenario] ## Common issues **Issue: [Error message or problem]** Fix by adjusting [parameter]: ```[language] # Solution code [concise fix] ``` **Issue: [Another common problem]** Check [specific requirement], then [action]. ## Advanced topics **[Advanced feature 1]**: See [references/advanced-features.md](references/advanced-features.md) **[Advanced feature 2]**: See [references/optimization.md](references/optimization.md) **[API reference]**: See [references/api-reference.md](references/api-reference.md) ## Resources - Official docs: [URL] - GitHub: [URL] --- ### Npm Package Plan (`docs/npm-package-plan.md`) # NPM Package Plan: @orchestra-research/skills ## Overview Create an npm/npx package that allows users to easily install AI research skills to their preferred coding agents (Claude Code, Cursor, Codex, Windsurf, etc.). ## Package Name Options - `@orchestra-research/skills` (recommended - scoped, professional) - `ai-research-skills` (simple, may conflict) - `orchestra-skills` (short, brandable) ## Architecture ### Inspired By Based on research of existing solutions: - **vercel-labs/skills**: Canonical storage + symlinks, 27 agent support, lock file - **openskills**: Universal loader, AGENTS.md generation - **add-skill**: Zero-dependency, auto-detection ### Core Components ``` @orchestra-research/skills/ ├── bin/ │ └── cli.js # CLI entry point ├── src/ │ ├── agents/ # Agent-specific handlers │ │ ├── claude.js # .claude/skills/ │ │ ├── cursor.js # .cursor/skills/ │ │ ├── codex.js # .codex/skills/ │ │ ├── windsurf.js # .windsurf/skills/ │ │ ├── copilot.js # .github/copilot-instructions.md │ │ └── index.js # Agent registry │ ├── commands/ │ │ ├── install.js # Install skills │ │ ├── list.js # List available/installed skills │ │ ├── update.js # Update skills │ │ ├── remove.js # Remove skills │ │ └── detect.js # Detect installed agents │ ├── storage/ │ │ ├── canonical.js # ~/.orchestra-skills/ management │ │ ├── lock.js # Lock file management │ │ └── symlink.js # Symlink utilities │ ├── registry/ │ │ └── skills.json # Skill manifest (or fetch from GitHub) │ └── utils/ │ ├── fetch.js # Download skills from GitHub │ └── format.js # Format for different agents ├── package.json └── README.md ``` ## Supported Agents (7 Verified) All agents below have been verified to support SKILL.md files with the same format. | Agent | Config Location | Skills Location | Source | |-------|-----------------|-----------------|--------| | Claude Code | `~/.claude/` | `.claude/skills/` | Verified locally | | Cursor | `~/.cursor/` | `.cursor/skills/` | [DeepWiki](https://deepwiki.com/getcursor/cursor) | | Codex (OpenAI) | `~/.codex/` | `.codex/skills/` | [DeepWiki](https://deepwiki.com/openai/codex) | | Windsurf | `~/.windsurf/` | `.windsurf/skills/` | [Windsurf Docs](https://docs.windsurf.com/windsurf/cascade/skills) | | Gemini CLI | `~/.gemini/` | `.gemini/skills/` | [DeepWiki](https://deepwiki.com/google-gemini/gemini-cli) | | Kilo Code | `~/.kilocode/` | `.kilocode/skills/` | [Kilo Docs](https://kilo.ai/docs/agent-behavior/skills) | | Qwen Code | `~/.qwen/` | `.qwen/skills/` | [Qwen Docs](https://qwenlm.github.io/qwen-code-docs/) ## CLI Commands ### Installation ```bash # Install globally npm install @orchestra-research/skills # Or use npx (recommended) npx @orchestra-research/skills ``` ### Commands ```bash # Detect installed coding agents npx @orchestra-research/skills detect # List all available skills npx @orchestra-research/skills list # List skills by category npx @orchestra-research/skills list --category post-training # List available categories npx @orchestra-research/skills categories # Install all skills for detected agents npx @orchestra-research/skills install --all # Install specific category (user selects from list) npx @orchestra-research/skills install --category post-training # Install multiple categories npx @orchestra-research/skills install --category post-training,fine-tuning,inference # Install specific skill npx @orchestra-research/skills install verl # Install for specific agent only npx @orchestra-research/skills install verl --agent claude # Install to project scope (current directory) npx @orchestra-research/skills install verl --scope project # Install to global scope (home directory) npx @orchestra-research/skills install verl --scope global # Interactive mode - prompts user to select categories/skills npx @orchestra-research/skills install --interactive # Update all skills npx @orchestra-research/skills update # Remove a skill npx @orchestra-research/skills remove verl # Show skill info npx @orchestra-research/skills info verl ``` ### Interactive Installation Flow When running `npx @orchestra-research/skills install --interactive`: ``` ? What would you like to install? ○ All skills (86 skills) ○ Select by category ○ Select individual skills ? Select categories to install: (Space to select, Enter to confirm) ◉ 01-model-architecture (6 skills) ◯ 02-tokenization (2 skills) ◯ 03-fine-tuning (5 skills) ◉ 06-post-training (8 skills) ◯ 20-ml-paper-writing (1 skill) ... ? Confirm installation of 14 skills to Claude Code, Cursor, Gemini CLI? (Y/n) ``` ## Storage Strategy ### Canonical Storage (Recommended) Single source of truth with symlinks: ``` ~/.orchestra-skills/ # Canonical storage ├── .lock.json # Lock file for versioning ├── 01-model-architecture/ │ ├── megatron-core/ │ │ └── SKILL.md │ └── litgpt/ │ └── SKILL.md ├── 06-post-training/ │ ├── verl/ │ │ ├── SKILL.md │ │ └── references/ │ ├── slime/ │ └── ... └── ... ~/.claude/skills/ # Symlinks to canonical ├── verl -> ~/.orchestra-skills/06-post-training/verl ├── slime -> ~/.orchestra-skills/06-post-training/slime └── ... ~/.cursor/skills/ # Same symlinks ├── verl -> ~/.orchestra-skills/06-post-training/verl └── ... ``` ### Lock File Format ```json { "version": "1.0.0", "lastUpdated": "2025-01-28T00:00:00Z", "skills": { "verl": { "version": "1.0.0", "category": "06-post-training", "installedAt": "2025-01-28T00:00:00Z", "agents": ["claude", "cursor"] } }, "agents": { "claude": { "detected": true, "scope": "global", "path": "~/.claude/skills" } } } ``` ## Skill Structure Patterns The repository has two skill organization patterns: ### Pattern 1: Nested Skills (Most Categories) ``` XX-category/ ├── skill-name-1/ │ ├── SKILL.md │ └── references/ ├── skill-name-2/ │ └── SKILL.md └── ... ``` Example: `06-post-training/verl/SKILL.md` ### Pattern 2: Standalone Skills (Single Skill = Category) ``` XX-category-name/ ├── SKILL.md ├── references/ └── templates/ ``` Example: `20-ml-paper-writing/SKILL.md` (the category IS the skill) The npm package must handle both patterns when fetching skills. --- ## Skill Registry ### Option A: Embedded (Simpler) Include skill manifest in npm package, update with releases: ```json // src/registry/skills.json { "version": "1.0.0", "categories": { "01-model-architecture": { "name": "Model Architecture", "skills": ["megatron-core", "litgpt", "mamba", "rwkv", "nanogpt"] }, "06-post-training": { "name": "Post-Training (RLHF/DPO/GRPO)", "skills": ["trl", "grpo", "openrlhf", "simpo", "verl", "slime", "miles", "torchforge"] } }, "skills": { "verl": { "name": "verl", "category": "06-post-training", "description": "Volcano Engine RL for LLM post-training", "tags": ["Reinforcement Learning", "RLHF", "GRPO", "PPO"] } } } ``` ### Option B: Remote Fetch (More Flexible) Fetch skill manifest from GitHub API on each run: ```javascript const REPO = 'orchestra-research/AI-research-SKILLs'; const MANIFEST_URL = `https://api.github.com/repos/${REPO}/contents/skill-manifest.json`; async function fetchSkillManifest() { const response = await fetch(MANIFEST_URL); return JSON.parse(atob(response.content)); } ``` **Recommendation**: Start with embedded, add remote fetch as update mechanism. ## Installation Flow ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Agent-Specific Handling ### All 7 Verified Agents All agents use the same SKILL.md format and symlink pattern: ```javascript // src/agents/index.js export const agents = { claude: { name: 'Claude Code', configDir: '~/.claude', skillsDir: '~/.claude/skills', projectSkillsDir: '.claude/skills', }, cursor: { name: 'Cursor', configDir: '~/.cursor', skillsDir: '~/.cursor/skills', projectSkillsDir: '.cursor/skills', }, codex: { name: 'Codex (OpenAI)', configDir: '~/.codex', skillsDir: '~/.codex/skills', projectSkillsDir: '.codex/skills', }, windsurf: { name: 'Windsurf', configDir: '~/.windsurf', skillsDir: '~/.windsurf/skills', projectSkillsDir: '.windsurf/skills', }, gemini: { name: 'Gemini CLI', configDir: '~/.gemini', skillsDir: '~/.gemini/skills', projectSkillsDir: '.gemini/skills', }, kilo: { name: 'Kilo Code', configDir: '~/.kilocode', skillsDir: '~/.kilocode/skills', projectSkillsDir: '.kilocode/skills', }, qwen: { name: 'Qwen Code', configDir: '~/.qwen', skillsDir: '~/.qwen/skills', projectSkillsDir: '.qwen/skills', }, }; // Common install function for all agents function installSkill(agent, skillName, canonicalPath, scope) { const targetDir = scope === 'project' ? agent.projectSkillsDir : expandHome(agent.skillsDir); fs.ensureDirSync(targetDir); fs.symlinkSync(canonicalPath, path.join(targetDir, skillName)); } ``` ## User Experience ### First Run ```bash $ npx @orchestra-research/skills detect 🔍 Detecting installed coding agents... ✓ Claude Code ~/.claude ✓ Cursor ~/.cursor ✗ Codex not found ✗ Windsurf not found ✓ GitHub Copilot available for projects Found 2 global agents, 1 project-only agent. Run 'npx @orchestra-research/skills install --all' to install all skills. ``` ### Installing Skills ```bash $ npx @orchestra-research/skills install post-training 📦 Installing post-training skills... Downloading skills from GitHub... ✓ trl (1.2 KB) ✓ grpo (15.3 KB) ✓ openrlhf (8.7 KB) ✓ simpo (4.2 KB) ✓ verl (12.1 KB) ✓ slime (18.4 KB) ✓ miles (9.8 KB) ✓ torchforge (11.2 KB) Creating symlinks... ✓ Claude Code: 8 skills installed ✓ Cursor: 8 skills installed ✨ Done! Installed 8 skills for 2 agents. Skills are stored in: ~/.orchestra-skills/06-post-training/ Symlinks created in: ~/.claude/skills/, ~/.cursor/skills/ ``` ### Listing Skills ```bash $ npx @orchestra-research/skills list 📚 AI Research Skills (81 total) Model Architecture (5) ○ megatron-core Megatron-Core for large-scale model training ○ litgpt LitGPT for efficient LLM development ○ mamba Mamba state space models ○ rwkv RWKV linear attention models ○ nanogpt NanoGPT for learning/prototyping Post-Training (8) ● verl Volcano Engine RL for LLM post-training ● slime Megatron-SGLang RL training framework ● miles Enterprise-grade RL for large MoE models ● torchforge PyTorch-native agentic RL library ○ trl Transformer Reinforcement Learning ○ grpo Group Relative Policy Optimization ○ openrlhf OpenRLHF training framework ○ simpo Simple Preference Optimization ● = installed, ○ = available ``` ## Implementation Phases ### Phase 1: MVP (Week 1) - [ ] Basic CLI structure with commander.js - [ ] Agent detection (Claude, Cursor, Codex) - [ ] Download skills from GitHub - [ ] Symlink installation to detected agents - [ ] Basic list and install commands ### Phase 2: Full Features (Week 2) - [ ] Canonical storage with lock file - [ ] Update and remove commands - [ ] Category filtering - [ ] Project vs global scope - [ ] Copilot special handling ### Phase 3: Polish (Week 3) - [ ] Interactive mode (inquirer.js prompts) - [ ] Progress bars and better UX - [ ] Error handling and recovery - [ ] Documentation and README - [ ] npm publish and GitHub Actions for releases ## Dependencies ```json { "dependencies": { "commander": "^12.0.0", // CLI framework "chalk": "^5.3.0", // Colored output "ora": "^8.0.0", // Spinners "fs-extra": "^11.2.0", // File utilities "node-fetch": "^3.3.0", // HTTP requests "inquirer": "^9.2.0" // Interactive prompts (optional) } } ``` ## Publishing ```bash # Login to npm npm login # Publish scoped package (public) npm publish --access public ``` ## Alternatives Considered ### 1. Shell Script (Rejected) PR #6 approach - too limited, no cross-platform support, poor UX. ### 2. Python Package (Possible Alternative) Could work with `pipx install orchestra-skills`, but npm/npx is more common for dev tools. ### 3. Homebrew Formula (Future) Could add `brew install orchestra-skills` later for Mac users. ## Next Steps 1. Create new repository or directory for npm package 2. Implement Phase 1 MVP 3. Test with Claude Code and Cursor 4. Publish to npm 5. Update main README with installation instructions 6. Close PR #6 with reference to new approach ## References - [vercel-labs/skills](https://github.com/vercel-labs/skills) - Multi-agent skill installer - [openskills](https://github.com/OpenAgentsInc/openskills) - Universal skill loader - [add-skill](https://github.com/iamnbutler/add-skill) - Zero-dependency installer --- ### Writing Assets ML Paper Guide (`docs/writing-assets/ML_paper_guide.md`) # The Complete Guide to Writing Top-Quality ML Academic Papers Writing successful ML papers for venues like NeurIPS, ICML, and ICLR demands mastery of a specific craft: translating rigorous technical work into a compelling narrative that busy reviewers can quickly evaluate. **The single most critical insight across all expert sources: your paper is not a collection of experiments—it's a story with one clear contribution supported by evidence.** This guide synthesizes advice from prominent researchers including Neel Nanda, Andrej Karpathy, Jacob Steinhardt, and Sebastian Farquhar, alongside official conference guidelines and practical tools for citation management. The stakes are high: top ML conferences maintain **~25% acceptance rates**, and reviewers spend limited time per paper. Seminal work like Adam and Knowledge Distillation faced initial rejections. Success requires not just strong research but strategic communication—front-loading value, maintaining precision, and providing reproducibility details that build reviewer confidence. --- ## The narrative principle that separates accepted papers Every successful ML paper centers on what Neel Nanda calls "the narrative": a short, rigorous, evidence-based technical story with a takeaway readers care about. This narrative rests on three pillars that must be crystal clear by the end of your introduction. **The "What"** consists of one to three specific novel claims fitting within a cohesive theme. Vague contributions like "we study X" fail immediately—reviewers need precise, falsifiable claims. **The "Why"** provides rigorous empirical evidence that convincingly supports those claims, including strong baselines honestly tuned and experiments that distinguish between competing hypotheses rather than merely showing "decent results." **The "So What"** answers why readers should care, connecting your contribution to problems the community recognizes as important. Andrej Karpathy reinforces this: "A paper is not a random collection of experiments you report on. The paper sells a single thing that was not obvious or present before. The entire paper is organized around this core contribution with surgical precision." This applies whether you're presenting a new architecture, a theoretical result, or improved understanding of existing methods—NeurIPS explicitly notes that "originality does not necessarily require an entirely new method." The practical implication is severe: if you cannot state your contribution in one sentence, you don't yet have a paper. Everything else—experiments, related work, discussion—exists only to support that core claim. --- ## Front-load value: the title-to-methods pipeline Readers encounter your paper in a predictable pattern: title → abstract → introduction → figures → maybe the rest. Nanda advises spending "about the same amount of time on each of: the abstract, the intro, the figures, and everything else." This isn't hyperbole—most reviewers form preliminary judgments before reaching your methods section. **The abstract** follows a tight five-sentence structure perfected by Sebastian Farquhar: (1) What you achieved ("We introduce...", "We prove...", "We demonstrate..."), (2) Why this is hard and important, (3) How you do it with specialist keywords, (4-5) What evidence you have, including your most remarkable number. Generic openings like "Large language models have achieved remarkable success" waste precious space—Zachary Lipton's rule: "If the first sentence can be pre-pended to any ML paper, delete it." **The introduction** should not exceed 1-1.5 pages and must include a bullet-point contribution list of 2-4 items (max 1-2 lines each in two-column format). Farquhar emphasizes: "Methods should start by page 2-3 maximum"—if your introduction runs longer, you're burying the actual contribution. **Figure 1** deserves special attention because many readers skip directly to it. It should convey your core idea, approach, or most compelling result. Use vector graphics, ensure readability in black-and-white, and write captions that stand alone without requiring the main text. --- ## Section-by-section execution matters **Related Work** should be organized methodologically, not paper-by-paper. Good: "One line of work uses Floogledoodle's assumption [refs] whereas we use Doobersnoddle's assumption because..." Bad: "Snap et al. introduced X while Crackle et al. introduced Y." Cite generously—reviewers likely authored relevant papers—and distribute citations throughout your paper rather than confining them to one section. **Methods** must enable reimplementation. ICML's checklist requires: conceptual outline or pseudocode, clearly stated algorithms, all hyperparameters listed, and architectural details sufficient for reproduction. Present your final design decisions here; comparative ablations belong in experiments or appendix. **Experiments** require explicit structure. For each experiment, state: what claim it supports, how it connects to your main contribution, the experimental setting (with details in appendix), and explicit guidance on what to observe in figures ("the blue line shows X, which demonstrates Y"). The ICML checklist mandates: error bars with methodology specified (standard deviation vs. standard error), hyperparameter search ranges, compute infrastructure (GPU type, total hours), and seed-setting methods. **Limitations** deserve their own section—NeurIPS and ICML require this. Counter-intuitively, honesty helps: reviewers are explicitly instructed not to penalize papers for acknowledging limitations. Pre-empt criticisms by identifying weaknesses before reviewers do and explaining why they don't undermine your core claims. --- ## Writing style that signals quality Gopen and Swan's "Science of Scientific Writing" establishes principles that ML papers routinely violate. **Place emphasis at sentence ends** (the "stress position"): readers naturally weight final words more heavily. **Put context first**: establish familiar information before introducing new concepts. **Keep subject and verb close together**: anything intervening reads as interruption. **One unit, one function**: each paragraph should make exactly one point. Specific style rules from multiple sources converge on these practices. Minimize pronouns—if you must use "this" or "those," use them as adjectives ("this result") to provide clarity. Place verbs early in sentences for easier parsing. Use minimal-syllable words. Eliminate hedging unless genuine uncertainty exists—"may" and "can" should almost always be dropped. Lipton notes that "provides *very* tight approximation" drips with insecurity compared to "provides tight approximation." Jacob Steinhardt emphasizes precision over brevity: replace "performance" with "accuracy" or "speed" depending on meaning. Use consistent phrasing—referring to the same concept with different terms creates confusion. Avoid vocabulary that signals incremental work: never "combine," "modify," or "expand"; instead "develop" or "propose." For mathematical writing, state all assumptions formally, provide intuitive explanations alongside proofs, and use consistent notation. Ethan Perez's practical tip: unfold apostrophes ("X's Y" → "The Y of X") for clarity. --- ## Tables, figures, and visual communication **Tables** should use the booktabs LaTeX package for professional appearance—avoid vertical lines, use horizontal rules sparingly. Bold the best value per metric and include symbols indicating direction (↑ higher is better, ↓ lower is better). Right-align numerical columns and maintain consistent decimal precision across all values. **Figures** must be vector graphics (PDF, EPS) for plots and diagrams; raster formats (PNG at 600 DPI) only for photographs or dense visualizations. Critical accessibility requirement: **8% of men have color vision deficiency**. Use the Okabe-Ito or Paul Tol palettes, avoid red-green combinations, and verify your figures work in grayscale. The SciencePlots Python package provides publication-ready styles with a single line: `plt.style.use(['science', 'ieee'])`. **Architecture diagrams** benefit from TikZ via PlotNeuralNet (GitHub: HarisIqbal88/PlotNeuralNet), which generates LaTeX code from Python. For training visualizations, include shaded regions showing variance across runs and use log scale when values span multiple orders of magnitude. **Captions** should be self-contained—readers must understand figures without consulting main text. ICML explicitly states: "Do not include a title inside the figure; the caption should serve this function." --- ## Conference requirements every submission must meet **NeurIPS 2025**: 9 content pages plus unlimited references; mandatory paper checklist covering reproducibility, ethics, and societal impact (desk rejection if missing); 6-point scoring system; lay summaries required for accepted papers. Reviews of accepted papers become public. **ICML 2025**: 8 content pages plus one additional page allowed for camera-ready; Broader Impact Statement required at end before references (doesn't count toward limit); reciprocal reviewing requirement—all submissions need a designated reviewer from authors. **ICLR 2025-2026**: 10 pages plus unlimited appendices; double-blind via OpenReview; new LLM policy requiring disclosure of AI use in writing (violations result in desk rejection). All three conferences evaluate papers on four core dimensions: **quality** (technical soundness, well-supported claims), **clarity** (clear writing, reproducible by experts), **significance** (community impact, advances understanding), and **originality** (new insights, clear differentiation from prior work). Reviewers separate concerns into major issues (essential for publication) and minor issues (not essential), and strong reviews follow Daniel Dennett's rules: first re-express the position fairly, then list agreements and what you learned, only then critique. --- ## Citation APIs that prevent hallucination AI-generated citations have a documented **~40% error rate**, including fabricated papers with real author names and fake titles. A reliable workflow requires programmatic verification through multiple APIs. **Semantic Scholar** (api.semanticscholar.org) covers 214M papers with 2.49B citations. Rate limit: 1 RPS with free API key. Python library: `pip install semanticscholar`. Search, retrieve metadata, and access citation graphs—ideal for ML papers specifically. **CrossRef** (api.crossref.org) is the primary source for DOI metadata and offers direct BibTeX retrieval via content negotiation: ```python import requests def doi_to_bibtex(doi): return requests.get(f"https://doi.org/{doi}", headers={"Accept": "application/x-bibtex"}).text ``` **arXiv API** (export.arxiv.org/api) provides metadata for preprints. Python library: `pip install arxiv`. No authentication required, but maintain 3-second delays between requests. **OpenAlex** (api.openalex.org) offers 240M+ works under CC0 license—the open successor to Microsoft Academic Graph. 100K requests/day, 10 RPS with email in query string. **Google Scholar has no official API**—scraping violates ToS. Use SerpApi ($75-275/month) only if Semantic Scholar coverage is insufficient. --- ## The verified citation workflow for AI assistants For any AI-assisted paper writing, implement this verification pipeline to eliminate hallucinated citations: 1. **Search** using Semantic Scholar or OpenAlex APIs with specific queries 2. **Verify existence** by confirming the paper appears in at least two sources (Semantic Scholar + CrossRef, or DOI resolution + arXiv) 3. **Retrieve BibTeX** via DOI content negotiation for guaranteed accuracy 4. **Verify claims** by accessing the actual paper (via DOI link or Semantic Scholar PDF) and confirming the attributed claim appears in the source 5. **Maintain clean .bib files** using Zotero with Better BibTeX plugin for auto-export and consistent citation keys Tools like Citely (citely.ai) and CiteSure provide batch verification of reference lists. ReciteWorks checks that in-text citations match your reference list. For LaTeX, prefer **BibLaTeX with Biber backend** over legacy BibTeX—it provides full Unicode support, extended entry types (@online, @dataset), and flexible customization. Use `\citep{}` for parenthetical citations and `\citet{}` for textual citations. --- ## Conclusion Writing top-quality ML papers is fundamentally about **reducing cognitive load for reviewers** while **maximizing evidence density for your claims**. The hierarchy of importance is clear: narrative clarity beats methodological complexity, front-loaded value beats comprehensive coverage, and verified reproducibility beats impressive numbers. The most actionable insight: treat writing as iterative design. Nanda recommends paper swaps for mutual feedback; Karpathy suggests submitting a 5-page draft with all experiments two weeks before deadline to reveal critical gaps. Sebastian Farquhar captures the modern reality: "If you are a good writer, you are better than LLMs. If you are a bad writer, you need the practice"—but LLMs excel at identifying unclear passages through misinterpretation and simulating harsh reviewer feedback. For citation workflows specifically, the combination of Semantic Scholar search → DOI content negotiation → BibLaTeX management provides a reliable, hallucination-resistant pipeline suitable for integration into AI writing assistants. Every citation must be verified to exist before inclusion—the alternative is contributing to the documented problem of fabricated references that has affected even NeurIPS accepted papers. --- ### Writing Assets Ml Paper Writing Sources (`docs/writing-assets/ml_paper_writing_sources.md`) # Comprehensive Source List: Building an ML Paper Writing Skill This document compiles authoritative sources for creating a Claude skill that writes high-quality ML/AI papers for venues like ICLR, NeurIPS, and ICML. --- ## Part 1: Writing Philosophy & Guides from ML Researchers ### Primary Sources (Must-Read) | Source | Author | URL | Key Value | |--------|--------|-----|-----------| | **Highly Opinionated Advice on How to Write ML Papers** | Neel Nanda | https://www.alignmentforum.org/posts/eJGptPbbFPZGLpjsp/highly-opinionated-advice-on-how-to-write-ml-papers | Core narrative philosophy, "what/why/so what" framework, figure-first approach | | **How to Write ML Papers** | Sebastian Farquhar (DeepMind) | https://sebastianfarquhar.com/on-research/2024/11/04/how_to_write_ml_papers/ | 5-sentence abstract formula, structure templates, reader expectations | | **A Survival Guide to a PhD** | Andrej Karpathy | http://karpathy.github.io/2016/09/07/phd/ | Paper structure recipe, importance of reviewing bad papers, contribution framing | | **Heuristics for Scientific Writing (ML Perspective)** | Zachary Lipton (CMU) | https://www.approximatelycorrect.com/2018/01/29/heuristics-technical-scientific-writing-machine-learning-perspective/ | Snappy maxims for clear prose, vacuous intensifier warnings, section balance | | **Advice for Authors** | Jacob Steinhardt (UC Berkeley) | https://jsteinhardt.stat.berkeley.edu/blog/advice-for-authors | Precision over brevity, consistent terminology, reader-centric writing | | **Easy Paper Writing Tips** | Ethan Perez (Anthropic) | https://ethanperez.net/easy-paper-writing-tips/ | Practical micro-level tips, apostrophe unfolding, clarity tricks | ### Foundational Scientific Writing | Source | Author | URL | Key Value | |--------|--------|-----|-----------| | **The Science of Scientific Writing** | Gopen & Swan | https://cseweb.ucsd.edu/~swanson/papers/science-of-writing.pdf | Topic/stress positions, old-before-new principle, sentence-level clarity | | **Summary of Science of Scientific Writing** | Lawrence Crowl | https://www.crowl.org/Lawrence/writing/GopenSwan90.html | Condensed version of Gopen & Swan principles | ### Additional Researcher Perspectives | Source | URL | Key Value | |--------|-----|-----------| | How To Write A Research Paper In Machine Learning | https://grigorisg9gr.github.io/machine%20learning/research%20paper/how-to-write-a-research-paper-in-machine-learning/ | Practical walkthrough | | A Recipe for Training Neural Networks (Karpathy) | http://karpathy.github.io/2019/04/25/recipe/ | Debugging methodology that translates to paper structure | --- ## Part 2: Official Conference Guidelines ### NeurIPS | Document | URL | Purpose | |----------|-----|---------| | **Paper Checklist Guidelines** | https://neurips.cc/public/guides/PaperChecklist | Mandatory checklist items, reproducibility requirements | | **2025 Reviewer Guidelines** | https://neurips.cc/Conferences/2025/ReviewerGuidelines | What reviewers look for, scoring criteria | | **Formatting Instructions** | https://arxiv.org/html/2505.10292v1 | LaTeX template, page limits, style requirements | ### ICML | Document | URL | Purpose | |----------|-----|---------| | **Paper Guidelines** | https://icml.cc/Conferences/2024/PaperGuidelines | Submission requirements, ethics policy | | **Style & Author Instructions** | https://icml.cc/Conferences/2022/StyleAuthorInstructions | Formatting specifications | | **Reviewer Tutorial** | https://icml.cc/Conferences/2022/ReviewerTutorial | Evaluation criteria from reviewer perspective | | **Reviewer Guidelines (2020)** | https://icml.cc/Conferences/2020/ReviewerGuidelines | Detailed review criteria | | **ICML 2025 LaTeX Template** | https://www.overleaf.com/latex/templates/icml2025-template/dhxrkcgkvnkt | Official Overleaf template | ### ICLR | Document | URL | Purpose | |----------|-----|---------| | **Author Guide 2026** | https://iclr.cc/Conferences/2026/AuthorGuide | Submission requirements, LLM disclosure policy | | **LLM Disclosure Policy** | https://eu.36kr.com/en/p/3443306502428032 | NEW: Mandatory AI use disclosure (desk rejection if missing) | --- ## Part 3: Citation APIs & Tools (Hallucination Prevention) ### Primary APIs for Paper Search & Metadata | API | Documentation URL | Key Features | Rate Limits | |-----|-------------------|--------------|-------------| | **Semantic Scholar API** | https://api.semanticscholar.org/api-docs/ | 214M papers, citation graphs, AI-trained search | 1 RPS with API key | | **Semantic Scholar Tutorial** | https://www.semanticscholar.org/product/api/tutorial | Step-by-step usage guide | - | | **CrossRef REST API** | https://www.crossref.org/documentation/retrieve-metadata/rest-api/ | DOI metadata, direct BibTeX via content negotiation | Polite pool with mailto | | **arXiv API** | https://info.arxiv.org/help/api/basics.html | Preprint metadata, full-text access | 3-second delays | | **OpenAlex API** | https://docs.openalex.org/api-entities/works | 240M+ works, CC0 license, open successor to MAG | 100K/day, 10 RPS | ### Python Libraries | Library | Install | Documentation | Purpose | |---------|---------|---------------|---------| | `semanticscholar` | `pip install semanticscholar` | https://semanticscholar.readthedocs.io/ | Official-ish Python wrapper | | `arxiv` | `pip install arxiv` | https://pypi.org/project/arxiv/ | arXiv search and download | | `habanero` | `pip install habanero` | https://github.com/sckott/habanero | CrossRef Python client | ### BibTeX Retrieval Code Pattern ```python import requests def doi_to_bibtex(doi: str) -> str: """Get BibTeX directly from DOI via CrossRef content negotiation.""" response = requests.get( f"https://doi.org/{doi}", headers={"Accept": "application/x-bibtex"} ) response.raise_for_status() return response.text # Example: Get verified BibTeX for "Attention Is All You Need" bibtex = doi_to_bibtex("10.48550/arXiv.1706.03762") ``` ### Citation Verification Tools | Tool | URL | Purpose | |------|-----|---------| | **Citely** | https://citely.ai/citation-checker | Batch verification of AI-generated citations | | **ReciteWorks** | https://reciteworks.com/ | Check in-text citations match reference list | ### LaTeX Citation Management | Resource | URL | Key Info | |----------|-----|----------| | BibTeX vs BibLaTeX Guide | https://electricalvoice.com/latex-vs-bibtex-vs-biblatex/ | When to use which system | | BibLaTeX Comprehensive Guide | https://latextutorial.net/latex-vs-bibtex-vs-biblatex/ | Modern citation management | --- ## Part 4: The Verified Citation Workflow ### Recommended Pipeline for AI-Assisted Writing ``` 1. SEARCH: User specifies topic → Query Semantic Scholar API └─ Use paper/search endpoint with specific keywords 2. VERIFY EXISTENCE: For each candidate paper: └─ Confirm paper exists in 2+ sources (Semantic Scholar + CrossRef/arXiv) └─ Verify DOI resolves correctly 3. GET BIBTEX: Use DOI content negotiation └─ requests.get(f"https://doi.org/{doi}", headers={"Accept": "application/x-bibtex"}) └─ NEVER generate BibTeX from memory - always fetch 4. VERIFY CLAIMS: Before citing paper for specific claim: └─ Retrieve paper abstract/full-text via Semantic Scholar └─ Confirm the attributed claim actually appears in source 5. BUILD BIBLIOGRAPHY: └─ Maintain .bib file with only verified entries └─ Use consistent citation keys (e.g., author_year_firstword) ``` ### Why This Matters From research on AI citation hallucination: - ~40% of AI-generated citations contain errors (Enago Academy research) - NeurIPS 2025 found 100+ hallucinated citations slipped through review - Common errors: fabricated titles, wrong authors, non-existent papers with plausible metadata --- ## Part 5: Visualization & Formatting Resources ### Figure Creation | Tool | URL | Purpose | |------|-----|---------| | **PlotNeuralNet** | https://github.com/HarisIqbal88/PlotNeuralNet | TikZ neural network diagrams | | **SciencePlots** | https://github.com/garrettj403/SciencePlots | Publication-ready matplotlib styles | | **Okabe-Ito Palette** | https://jfly.uni-koeln.de/color/ | Colorblind-safe color scheme | ### LaTeX Templates | Venue | Template URL | |-------|--------------| | NeurIPS | https://neurips.cc/Conferences/2025/PaperInformation/StyleFiles | | ICML | https://www.overleaf.com/latex/templates/icml2025-template/dhxrkcgkvnkt | | ICLR | https://iclr.cc/Conferences/2026/AuthorGuide (links to template) | --- ## Part 6: Key Principles Summary (For Skill Encoding) ### From Neel Nanda 1. Paper = short, rigorous, evidence-based technical story with a takeaway readers care about 2. Spend equal time on: abstract, intro, figures, everything else 3. Every experiment must support a specific claim connected to contribution 4. "If you can't state your contribution in one sentence, you don't have a paper yet" ### From Karpathy 1. "The paper sells a single thing that was not obvious before" 2. Default structure: Intro → Related Work → Model → Experiments → Conclusions 3. Review bad papers to learn what NOT to do (binary classifier training) ### From Zachary Lipton 1. "If the first sentence can be pre-pended to any ML paper, delete it" 2. Figures should tell coherent story even if reader skips text 3. Sections should be balanced like bullets on slides 4. "provides *very* tight approximation" drips with insecurity → "provides tight approximation" ### From Sebastian Farquhar 1. Methods should start by page 2-3 maximum 2. Abstract formula: (1) What achieved, (2) Why hard/important, (3) How with keywords, (4-5) Evidence + best number 3. Introduction must have 2-4 bullet contribution list (max 1-2 lines each) ### From Gopen & Swan 1. Place emphasis at sentence ends (stress position) 2. Put context (old info) before new information 3. Keep subject and verb close together 4. One unit = one function (each paragraph = one point) --- ## Part 7: Additional Resources ### Hallucination & AI Writing Concerns | Source | URL | |--------|-----| | AI Hallucinations in Research Citations | https://www.enago.com/academy/ai-hallucinations-research-citations/ | | Hallucination in AI-Generated Writing (PMC) | https://pmc.ncbi.nlm.nih.gov/articles/PMC10726751/ | | NeurIPS 2025 AI Hallucination Report | https://byteiota.com/neurips-2025-100-ai-hallucinations-slip-through-review/ | ### ML Conference Review System Analysis | Source | URL | |--------|-----| | Position: ML Conferences Should Have Refutations Track | https://arxiv.org/html/2506.19882v1 | --- ## Usage Notes for Skill Development 1. **For paper structure**: Start with Nanda + Farquhar for high-level philosophy, use conference guidelines for specifics 2. **For writing style**: Combine Lipton's heuristics + Gopen & Swan's principles + Ethan Perez's micro-tips 3. **For citation workflow**: Implement Semantic Scholar → DOI verification → CrossRef BibTeX pipeline; NEVER generate citations from model memory 4. **For figures/tables**: Reference booktabs for tables, SciencePlots for figures, always use colorblind-safe palettes 5. **For reviewer simulation**: Study reviewer guidelines from all three venues to anticipate criticisms --- --- METRICS --- - Files Extracted: 22 - Estimated Token Budget: ~50822 tokens - Canonical Reference: https://codewiki.google/github.com/Orchestra-Research/AI-Research-SKILLs