{"owner":"Orchestra-Research","repo":"AI-Research-SKILLs","hasSkills":true,"hasMcp":false,"mcpConfig":null,"found":["CLAUDE.md","06-post-training/miles/SKILL.md"],"skills":{"CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\n**AI Research Skills Library** - A comprehensive open-source library of 98 AI research skills enabling AI agents to autonomously conduct AI research — from idea to paper. Each skill provides expert-level guidance (200-500 lines) with real code examples, troubleshooting guides, and production-ready workflows.\n\n**Mission**: Enable AI agents to autonomously conduct AI research from hypothesis to experimental verification, covering the full lifecycle: literature survey, ideation, dataset preparation, training pipelines, model deployment, evaluation, and paper writing.\n\n## Repository Architecture\n\n### Directory Structure (98 Skills Across 23 Categories)\n\nSkills are organized into numbered categories representing the AI research lifecycle:\n\n- `0-autoresearch-skill/` - **Autonomous research orchestration** (1 skill: Autoresearch — central layer that manages the full lifecycle and routes to all other skills)\n- `01-model-architecture/` - Model architectures (5 skills: TorchTitan, LitGPT, Mamba, RWKV, NanoGPT)\n- `02-tokenization/` - Tokenizers (2 skills: HuggingFace Tokenizers, SentencePiece)\n- `03-fine-tuning/` - Fine-tuning frameworks (4 skills: Axolotl, LLaMA-Factory, Unsloth, PEFT)\n- `04-mechanistic-interpretability/` - Interpretability tools (4 skills: TransformerLens, SAELens, NNsight, Pyvene)\n- `05-data-processing/` - Data curation (2 skills: Ray Data, NeMo Curator)\n- `06-post-training/` - RLHF/DPO/GRPO (8 skills: TRL, GRPO, OpenRLHF, SimPO, verl, slime, miles, torchforge)\n- `07-safety-alignment/` - Safety and guardrails (4 skills: Constitutional AI, LlamaGuard, NeMo Guardrails, Prompt Guard)\n- `08-distributed-training/` - Distributed systems (6 skills: Megatron-Core, DeepSpeed, FSDP, Accelerate, PyTorch Lightning, Ray Train)\n- `09-infrastructure/` - Cloud compute (3 skills: Modal, SkyPilot, Lambda Labs)\n- `10-optimization/` - Optimization techniques (7 skills: Flash Attention, bitsandbytes, GPTQ, AWQ, HQQ, GGUF, ML Training Recipes)\n- `11-evaluation/` - Benchmarking (3 skills: lm-evaluation-harness, BigCode, NeMo Evaluator)\n- `12-inference-serving/` - Inference engines (4 skills: vLLM, TensorRT-LLM, llama.cpp, SGLang)\n- `13-mlops/` - Experiment tracking (4 skills: Weights & Biases, MLflow, TensorBoard, SwanLab)\n- `14-agents/` - Agent frameworks (5 skills: LangChain, LlamaIndex, CrewAI, AutoGPT, A-Evolve)\n- `15-rag/` - Retrieval-augmented generation (5 skills: Chroma, FAISS, Sentence Transformers, Pinecone, Qdrant)\n- `16-prompt-engineering/` - Structured output (4 skills: DSPy, Instructor, Guidance, Outlines)\n- `17-observability/` - LLM observability (2 skills: LangSmith, Phoenix)\n- `18-multimodal/` - Vision and speech (10 skills: CLIP, Whisper, LLaVA, Stable Diffusion, SAM, BLIP-2, AudioCraft, Cosmos Policy, OpenPI, OpenVLA-OFT)\n- `19-emerging-techniques/` - Advanced methods (6 skills: MoE Training, Model Merging, Long Context, Speculative Decoding, Knowledge Distillation, Model Pruning)\n- `20-ml-paper-writing/` - Paper writing (4 skills: ML Paper Writing with LaTeX templates for NeurIPS, ICML, ICLR, ACL, AAAI, COLM; Systems Paper Writing for OSDI, NSDI, ASPLOS, SOSP; Academic Plotting; Presenting Conference Talks)\n- `21-research-ideation/` - Ideation (2 skills: Research Brainstorming, Creative Thinking)\n- `22-agent-native-research-artifact/` - Agent-Native Research Artifact tooling (3 skills: ARA Compiler, ARA Research Manager, ARA Rigor Reviewer — ingestion, post-task provenance recording, and Seal Level 2 epistemic review)\n\n### Skill File Structure\n\nEach skill follows a standardized format:\n```\nskill-name/\n├── SKILL.md                    # Main guidance (200-600 lines with YAML frontmatter)\n├── references/                 # Deep documentation (300KB+ target)\n│   ├── README.md              # From official docs\n│   ├── api.md                 # API reference\n│   ├── tutorials.md           # Step-by-step guides\n│   ├── issues.md              # Real GitHub issues & solutions\n│   └── releases.md            # Version history\n├── scripts/                    # Helper scripts (optional)\n├── templates/                  # Code templates (optional)\n└── examples/                   # Example implementations (optional)\n```\n\n## Skill Quality Standards\n\n### YAML Frontmatter Requirements (CRITICAL)\n\nAll `SKILL.md` files MUST include YAML frontmatter with these exact fields:\n\n```yaml\n---\nname: skill-name-here              # kebab-case, no quotes, gerund form preferred\ndescription: Third-person description of what AND when to use this skill  # No quotes, max 1024 chars\nversion: 1.0.0                     # Semantic versioning\nauthor: Orchestra Research         # Standard author\nlicense: MIT                       # Standard license\ntags: [Tag One, Tag Two]          # Title Case (except UPPERCASE acronyms like GRPO, TRL, RLHF)\ndependencies: [pkg>=1.0.0]         # Optional, with version constraints\n---\n```\n\n**Critical Rules**:\n- `name`: Use gerund form (e.g., `serving-llms`, `processing-data`, `grpo-rl-training`)\n- `description`: Third person (\"Provides guidance for...\"), include WHAT it does AND WHEN to use it\n- `tags`: Title Case for regular words, UPPERCASE for acronyms (GRPO, TRL, RLHF, DPO, PPO)\n- No quotes around any field values (except in arrays)\n- Dependencies should include version constraints: `transformers>=4.47.0`\n\n### Content Quality Standards\n\n**Core Requirements** (based on Anthropic official best practices):\n- ✅ SKILL.md body: **200-500 lines** (under 500 lines is critical for performance)\n- ✅ Progressive disclosure: SKILL.md as overview, details in separate reference files\n- ✅ Workflows with copy-paste checklists for complex tasks\n- ✅ \"When to use vs alternatives\" guidance section\n- ✅ Common issues section with solutions\n- ✅ Concise content: assume Claude is smart, no over-explaining basics\n- ✅ Code examples with language detection (```python, ```bash, etc.)\n- ✅ References ONE level deep from SKILL.md (no nested references)\n\n**Gold Standard** (aim for this - see `06-post-training/grpo-rl-training/`):\n- ✅ 2-3 complete workflows with step-by-step checklists\n- ✅ Reference files for advanced topics (one level deep)\n- ✅ Feedback loops (validate → fix → repeat) for quality-critical operations\n- ✅ Consistent terminology throughout\n- ✅ Concrete input/output examples\n- ✅ Real GitHub issues with solutions (when available)\n\n**NOT Acceptable**:\n- ❌ SKILL.md over 500 lines (split into reference files instead)\n- ❌ Over-explaining basics that Claude already knows\n- ❌ First-person descriptions (\"I can help you...\")\n- ❌ Vague skill names (\"helper\", \"utils\", \"tools\")\n- ❌ Nested references (SKILL.md → ref1.md → ref2.md)\n- ❌ Missing workflows with checklists for complex tasks\n\n## Development Workflow\n\n### Adding a New Skill\n\n1. **Choose skill from roadmap** (see CONTRIBUTING.md or README.md)\n2. **Create directory structure** in appropriate category (01-19)\n3. **Write SKILL.md** with YAML frontmatter following standards above\n4. **Add reference documentation** (target 300KB+ from official sources)\n5. **Validate quality**:\n   - Check SKILL.md has YAML frontmatter\n   - Verify SKILL.md is 200-500 lines\n   - Ensure code blocks have language tags\n   - Confirm references are one level deep from SKILL.md\n   - Check documentation size: `du -sh skill-name/references/`\n6. **Test the skill** with real use cases before submitting\n\n### Improving Existing Skills\n\nWhen updating skills:\n1. **Maintain YAML frontmatter** format and fields\n2. **Keep SKILL.md under 500 lines** - split into reference files if needed\n3. **Add workflows** with checklists for complex operations\n4. **Update version number** in YAML frontmatter\n5. **Test changes** with representative tasks\n\n### Quality Validation Commands\n\n```bash\n# Check YAML frontmatter exists\nhead -20 skill-name/SKILL.md\n\n# Verify SKILL.md line count (target 200-500 lines)\nwc -l skill-name/SKILL.md\n\n# Check documentation size (target 300KB+)\ndu -sh skill-name/references/\n\n# Verify code blocks have language tags\ngrep -A 1 '```' skill-name/SKILL.md | head -20\n\n# Validate YAML frontmatter syntax\npython -c \"import yaml; yaml.safe_load(open('skill-name/SKILL.md').read().split('---')[1])\"\n```\n\n## Key Files\n\n- **README.md** - Project overview, all 98 skills listed with descriptions and stats\n- **CONTRIBUTING.md** - Complete contribution guidelines and quality standards\n- **SKILL_TEMPLATE.md** - Copy-paste scaffold for new skills\n- **ROADMAP.md** - Development roadmap (98 skills achieved)\n- **anthropic_official_docs/** - Anthropic's official best practices for skills\n\n## Git Workflow\n\nStandard Git workflow:\n```bash\n# Create feature branch\ngit checkout -b add-skill-name\n\n# Add and commit changes\ngit add category/skill-name/\ngit commit -m \"Add [Skill Name] skill\n\n- X lines of documentation\n- Y GitHub issues with solutions\n- API reference and examples included\"\n\n# Push to fork and create PR\ngit push origin add-skill-name\n```\n\n## Automation: Orchestra Skill Marketplace Sync\n\n### How Auto-Sync Works\n\nWhen skills are committed to the `main` branch, GitHub Actions automatically syncs them to the Orchestra skill marketplace:\n\n1. **GitHub Actions detects** changed skill folders on push to `main`\n2. **For each changed skill**:\n   - Extracts metadata from SKILL.md frontmatter (`name`, `author`, etc.)\n   - Creates ZIP file containing entire skill directory (SKILL.md, references/, scripts/, etc.)\n   - Uploads to Orchestra API endpoint\n3. **Orchestra stores** ZIP in Supabase Storage and creates database record\n4. **Skill appears** in marketplace at `https://orchestra.com/research-skills`\n\n### Workflow File Location\n\n- **File**: `.github/workflows/sync-skills.yml`\n- **Triggers**: Push to `main` branch, manual workflow dispatch\n- **What syncs**: Only skill directories that changed in the commit\n\n### Author Detection (Orchestra vs Community)\n\nThe workflow reads the `author:` field from SKILL.md frontmatter to determine badge:\n\n**Official Orchestra Skill**:\n```yaml\n---\nauthor: Orchestra Research  # Contains \"Orchestra\"\n---\n```\n- Result: Source = `orchestra` (Official badge)\n- Storage: `research-skills/orchestra/skill-name.zip`\n\n**Community Skill**:\n```yaml\n---\nauthor: Jane Doe  # Does NOT contain \"Orchestra\"\n---\n```\n- Result: Source = `community` (Community badge)\n- Storage: `research-skills/community/skill-name.zip`\n\n### What Gets Synced\n\nThe workflow zips **ALL contents** of skill directory:\n- ✅ SKILL.md\n- ✅ references/ (all subdirectories)\n- ✅ scripts/ (if exists)\n- ✅ assets/ (if exists)\n- ✅ examples/ (if exists)\n- ✅ templates/ (if exists)\n- ❌ Hidden files (`.gitkeep`, `.DS_Store`)\n\n### Testing the Sync\n\n**Manual trigger**:\n1. Go to GitHub Actions tab\n2. Select \"Sync Skills to Orchestra\" workflow\n3. Click \"Run workflow\"\n\n**Test with commit**:\n```bash\n# Make a small change to any skill\necho \"\\n<!-- Updated $(date) -->\" >> 01-model-architecture/litgpt/SKILL.md\n\n# Commit and push to main\ngit add .\ngit commit -m \"test: trigger auto-sync\"\ngit push origin main\n```\n\n**Verify sync worked**:\n1. Check GitHub Actions tab for workflow run status\n2. Check Orchestra marketplace for updated skill\n3. Check Supabase Storage for ZIP file\n\n### Important Notes\n\n- **GitHub Secrets required**: `ORCHESTRA_API_URL`, `ORCHESTRA_SYNC_API_KEY` (already configured)\n- **Only syncs changed skills**: Workflow detects which skill directories changed in commit\n- **SKILL.md required**: Skills without SKILL.md are skipped with warning\n- **See detailed setup**: `dev_data/GITHUB_SKILLS_SYNC_SETUP.md`\n\n## npm Package Publishing\n\n### How It Works\n\nThe `publish-npm.yml` workflow auto-publishes to npm when the version in `packages/ai-research-skills/package.json` changes on `main`.\n\n- **Auth**: Uses OIDC trusted publishing (no npm tokens). Configured on npmjs.com under the package's Trusted Publishers settings.\n- **Provenance**: `--provenance` flag signs packages with Sigstore for supply chain security.\n- **Workflow**: `.github/workflows/publish-npm.yml`\n\n### Bumping Versions\n\n**Always use `npm version`** (not manual edits) to keep `package-lock.json` in sync:\n\n```bash\ncd packages/ai-research-skills\nnpm version patch   # 1.3.6 → 1.3.7\nnpm version minor   # 1.3.7 → 1.4.0\nnpm version major   # 1.4.0 → 2.0.0\n```\n\nUse `--no-git-tag-version` if you want to commit manually.\n\n### Common Issues\n\n- **`npm ci` fails in CI**: `package-lock.json` is out of sync. Run `npm install` locally and commit the lockfile.\n- **OIDC auth fails**: The trusted publisher config on npmjs.com must match the repo exactly (case-sensitive: `Orchestra-Research/AI-Research-SKILLs`, workflow: `publish-npm.yml`).\n- **`NODE_AUTH_TOKEN` blocks OIDC**: `actions/setup-node` with `registry-url` auto-sets this token. The workflow unsets it before publish so OIDC takes over.\n- **Version unchanged skip**: The workflow compares `HEAD` vs `HEAD~1`. If only the lockfile changed (not `package.json` version), publish is skipped. Bump the version to trigger.\n\n## Important Conventions\n\n### Naming Conventions\n\n- **Skill names**: Use gerund form (verb + -ing) in kebab-case: `processing-pdfs`, `serving-llms`, `grpo-rl-training`\n- **Tags**: Title Case for words, UPPERCASE for acronyms (GRPO, TRL, RLHF, DPO, PPO, FSDP, MoE)\n- **Descriptions**: Third person, include what AND when to use\n\n### Code Examples\n\nAlways use language detection in code blocks:\n```python\n# Good - has language tag\nfrom transformers import AutoModel\n```\n\nNOT:\n```\n# Bad - no language tag\nfrom transformers import AutoModel\n```\n\n### Progressive Disclosure Pattern\n\nSKILL.md should link directly to reference files (one level deep):\n\n```markdown\n## Advanced Features\n\n**API Reference**: See [references/api.md](references/api.md)\n**Troubleshooting**: See [references/issues.md](references/issues.md)\n```\n\n## Philosophy\n\n**Quality over Quantity**: This library maintains high standards by:\n- Requiring 200-500 line SKILL.md files (focused, actionable guidance)\n- Including 300KB+ documentation from official sources\n- Providing real GitHub issues with solutions\n- Following Anthropic's official best practices for skills\n- Testing skills with real use cases before inclusion\n\nEach skill represents expert-level knowledge distilled into a format optimized for AI agent consumption.\n","06-post-training/miles/SKILL.md":"---\nname: miles-rl-training\ndescription: Provides guidance for enterprise-grade RL training using miles, a production-ready fork of slime. Use when training large MoE models with FP8/INT4, needing train-inference alignment, or requiring speculative RL for maximum throughput.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Reinforcement Learning, MoE, FP8, INT4, Enterprise, SGLang, Megatron-LM]\ndependencies: [sglang-router>=0.2.3, ray, torch>=2.0.0, transformers>=4.40.0]\n---\n\n# miles: Enterprise-Grade RL for Large-Scale Model Training\n\nmiles is a high-performance, enterprise-ready RL framework optimized for large-scale model post-training. Built as a production fork of slime, it addresses critical challenges in MoE training stability, low-precision training, and train-inference alignment.\n\n## When to Use miles\n\n**Choose miles when you need:**\n- Training 1TB+ MoE models (DeepSeek V3, Qwen3-MoE)\n- FP8 or INT4 quantization-aware training\n- Bit-wise identical train-inference alignment\n- Speculative RL for maximum throughput\n- Production stability with enterprise support\n\n**Consider alternatives when:**\n- You want the research-grade original → use **slime**\n- You need flexible backend swapping → use **verl**\n- You want PyTorch-native abstractions → use **torchforge**\n\n## Key Features\n\n### Low-Precision Training\n- **Unified FP8**: End-to-end FP8 for both inference and training\n- **INT4 QAT**: 1TB models on single-machine VRAM (H200)\n- **Rollout Routing Replay (R3)**: Bit-wise expert alignment for MoE\n\n### Performance Optimizations\n- **Speculative RL**: 25%+ rollout speedup with online SFT draft models\n- **Zero-Copy Weight Sync**: CUDA IPC zero-copy mapping\n- **Partial Rollout**: Recycle half-finished trajectories\n\n### Train-Inference Alignment\n- **TIS/MIS**: Truncated/Masked Importance Sampling for off-policy correction\n- **Kernel-level optimization**: FlashAttention-3, DeepGEMM integration\n\n## Installation\n\n```bash\n# Recommended: Docker\ndocker pull radixark/miles:latest\ndocker run --rm --gpus all --ipc=host --shm-size=16g \\\n  -it radixark/miles:latest /bin/bash\n\n# From source\ngit clone https://github.com/radixark/miles.git\ncd miles\npip install -r requirements.txt\npip install -e .\n```\n\n## Quick Start\n\nmiles inherits slime's configuration system. Basic training:\n\n```bash\npython train.py \\\n    --advantage-estimator grpo \\\n    --model-name qwen3-30b-a3b \\\n    --hf-checkpoint /path/to/qwen3-30b-a3b-hf \\\n    --rollout-batch-size 512 \\\n    --n-samples-per-prompt 8\n```\n\n---\n\n## Workflow 1: Large MoE Training\n\nUse this workflow for training large MoE models like DeepSeek V3 or Qwen3-MoE.\n\n### Prerequisites Checklist\n- [ ] H100/H200 GPUs with FP8 support\n- [ ] MoE model (DeepSeek V3, Qwen3-MoE)\n- [ ] Docker environment with miles\n\n### Step 1: Environment Setup\n\n```bash\n# FP8 block scaling (recommended for stability)\nexport NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1\nexport CUDA_DEVICE_MAX_CONNECTIONS=1\n```\n\n### Step 2: Configure Training\n\n```bash\npython train.py \\\n    --actor-num-gpus-per-node 8 \\\n    --rollout-num-gpus 8 \\\n    --hf-checkpoint /path/to/deepseek-v3 \\\n    --advantage-estimator grpo \\\n    --tensor-model-parallel-size 8 \\\n    --expert-model-parallel-size 4 \\\n    --prompt-data /path/to/data.jsonl \\\n    --num-rollout 3000\n```\n\n### Verification Checklist\n- [ ] Model loads without errors\n- [ ] Routing decisions are consistent\n- [ ] No NaN/Inf in loss values\n\n---\n\n## Workflow 2: Speculative RL Training\n\nUse this workflow for maximum rollout throughput with EAGLE speculative decoding.\n\n### How Speculative RL Works\n\n1. Small draft model generates candidate tokens\n2. Target model verifies in parallel\n3. Draft model updated via online SFT to track policy\n\n### Step 1: Enable Speculative Decoding\n\nmiles supports EAGLE speculative decoding via SGLang:\n\n```bash\npython train.py \\\n    --actor-num-gpus-per-node 8 \\\n    --hf-checkpoint /path/to/target-model \\\n    --sglang-speculative-algorithm EAGLE \\\n    --sglang-speculative-num-steps 3 \\\n    --sglang-speculative-eagle-topk 1 \\\n    --sglang-speculative-num-draft-tokens 4 \\\n    --sglang-speculative-draft-model-path /path/to/draft-model \\\n    --advantage-estimator grpo \\\n    --prompt-data /path/to/data.jsonl\n```\n\n### Step 2: Enable Online MTP Training (Optional)\n\nFor online SFT of draft model during training:\n\n```bash\n--mtp-num-layers 1 \\\n--enable-mtp-training \\\n--mtp-loss-scaling-factor 0.2\n```\n\n**Note**: Online MTP training requires a torch dist checkpoint with MTP weights. Add `--mtp-num-layers 1` during checkpoint conversion from HuggingFace.\n\n### Expected Speedup\n\n- **Standard rollout**: Baseline\n- **Speculative RL**: 25-40% faster rollout\n- **With partial rollout**: Additional 10-15% throughput\n\n---\n\n## Configuration Reference\n\nmiles inherits all slime arguments. See [slime API Reference](../slime/references/api-reference.md) for the complete list.\n\n### Cluster Resources (from slime)\n\n```bash\n--actor-num-nodes 1\n--actor-num-gpus-per-node 8\n--rollout-num-gpus 8\n--rollout-num-gpus-per-engine 2\n--colocate\n```\n\n### Megatron Parallelism (from slime)\n\n```bash\n--tensor-model-parallel-size 8\n--pipeline-model-parallel-size 2\n--expert-model-parallel-size 4    # MoE expert parallelism\n```\n\n### Speculative Decoding (miles-specific)\n\n```bash\n--sglang-speculative-algorithm EAGLE\n--sglang-speculative-num-steps 3\n--sglang-speculative-eagle-topk 1\n--sglang-speculative-num-draft-tokens 4\n--sglang-enable-draft-weights-cpu-backup\n--sglang-speculative-draft-model-path /your/draft/model/path\n```\n\n### Online MTP Training (miles-specific)\n\n```bash\n--mtp-num-layers 1\n--enable-mtp-training\n--mtp-loss-scaling-factor 0.2\n```\n\n---\n\n## Key Features (Conceptual)\n\nThe following features are documented in miles but specific CLI flags may vary. Consult the miles repository for latest configuration.\n\n### Unified FP8 Pipeline\n\nEnd-to-end FP8 sampling and training that eliminates quantization-induced discrepancy causing RL collapse in MoE models.\n\n### Rollout Routing Replay (R3)\n\nRecords expert routing decisions during SGLang inference and replays them during Megatron training for bit-wise expert alignment.\n\n**How R3 Works**:\n1. During SGLang inference, expert routing decisions are recorded\n2. Routing decisions stored in `sample.rollout_routed_experts`\n3. During Megatron training, routing is replayed instead of recomputed\n4. Ensures identical expert selection between train and inference\n\n### INT4 Quantization-Aware Training\n\nEnables single-machine deployment of 1TB+ models (e.g., on H200).\n\n**Memory Savings with INT4**:\n\n| Model Size | BF16 VRAM | INT4 VRAM | Reduction |\n|------------|-----------|-----------|-----------|\n| 70B | 140GB | 45GB | 3.1x |\n| 235B | 470GB | 150GB | 3.1x |\n| 671B | 1.3TB | 420GB | 3.1x |\n\n### Train-Inference Alignment\n\nmiles achieves \"exactly 0 KL divergence\" between training and inference through:\n- Flash Attention 3\n- DeepGEMM\n- Batch-invariant kernels from Thinking Machines Lab\n- `torch.compile` integration\n\n---\n\n## Sample Data Structure\n\nmiles uses the same `Sample` dataclass as slime with the `rollout_routed_experts` field for MoE routing replay:\n\n```python\n@dataclass\nclass Sample:\n    prompt: str | list[dict]\n    tokens: list[int]\n    response: str\n    reward: float | dict\n    loss_mask: list[int]\n    status: Status\n    metadata: dict\n    rollout_log_probs: list[float]\n    rollout_routed_experts: list[list[int]]  # MoE routing for R3\n```\n\nSee [slime API Reference](../slime/references/api-reference.md) for the complete Sample definition.\n\n---\n\n## Common Issues and Solutions\n\n### Issue: FP8 Training Collapse\n\n**Symptoms**: Loss explodes, NaN values\n\n**Solutions**:\n- Use block scaling: `export NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1`\n- Reduce learning rate: `--lr 5e-7`\n- Ensure MoE routing is consistent between train/inference\n\n### Issue: Speculative Draft Drift\n\n**Symptoms**: Low acceptance rate over time\n\n**Solutions**:\n- Enable online MTP training to keep draft model aligned\n- Reduce speculative steps: `--sglang-speculative-num-steps 2`\n- Use CPU backup: `--sglang-enable-draft-weights-cpu-backup`\n\n### Issue: Train-Inference Mismatch\n\n**Symptoms**: Policy divergence, reward collapse\n\n**Solutions**:\n- Use TIS for off-policy correction: `--use-tis --tis-threshold 0.9`\n- Verify log probs match between SGLang and Megatron\n- Enable R3 for MoE models\n\n---\n\n## Supported Models\n\n| Family | Models | MoE Support |\n|--------|--------|-------------|\n| DeepSeek | R1, V3, V3.2 | Full |\n| Qwen | 2, 2.5, 3 (including MoE) | Full |\n| Llama | 3, 3.1, 3.3, 4 | Dense only |\n| Gemma | 2, 3, 3N | Dense only |\n| GLM | 4.5, 4.6, 4.7 | Dense only |\n| MiniMax | M2, M2.1 | Full |\n\n---\n\n## Resources\n\n- **GitHub**: https://github.com/radixark/miles\n- **Introduction Blog**: https://lmsys.org/blog/2025-11-19-miles/\n- **Slime (upstream)**: https://github.com/THUDM/slime\n- **SGLang**: https://github.com/sgl-project/sglang\n\n"},"files":{"CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\n**AI Research Skills Library** - A comprehensive open-source library of 98 AI research skills enabling AI agents to autonomously conduct AI research — from idea to paper. Each skill provides expert-level guidance (200-500 lines) with real code examples, troubleshooting guides, and production-ready workflows.\n\n**Mission**: Enable AI agents to autonomously conduct AI research from hypothesis to experimental verification, covering the full lifecycle: literature survey, ideation, dataset preparation, training pipelines, model deployment, evaluation, and paper writing.\n\n## Repository Architecture\n\n### Directory Structure (98 Skills Across 23 Categories)\n\nSkills are organized into numbered categories representing the AI research lifecycle:\n\n- `0-autoresearch-skill/` - **Autonomous research orchestration** (1 skill: Autoresearch — central layer that manages the full lifecycle and routes to all other skills)\n- `01-model-architecture/` - Model architectures (5 skills: TorchTitan, LitGPT, Mamba, RWKV, NanoGPT)\n- `02-tokenization/` - Tokenizers (2 skills: HuggingFace Tokenizers, SentencePiece)\n- `03-fine-tuning/` - Fine-tuning frameworks (4 skills: Axolotl, LLaMA-Factory, Unsloth, PEFT)\n- `04-mechanistic-interpretability/` - Interpretability tools (4 skills: TransformerLens, SAELens, NNsight, Pyvene)\n- `05-data-processing/` - Data curation (2 skills: Ray Data, NeMo Curator)\n- `06-post-training/` - RLHF/DPO/GRPO (8 skills: TRL, GRPO, OpenRLHF, SimPO, verl, slime, miles, torchforge)\n- `07-safety-alignment/` - Safety and guardrails (4 skills: Constitutional AI, LlamaGuard, NeMo Guardrails, Prompt Guard)\n- `08-distributed-training/` - Distributed systems (6 skills: Megatron-Core, DeepSpeed, FSDP, Accelerate, PyTorch Lightning, Ray Train)\n- `09-infrastructure/` - Cloud compute (3 skills: Modal, SkyPilot, Lambda Labs)\n- `10-optimization/` - Optimization techniques (7 skills: Flash Attention, bitsandbytes, GPTQ, AWQ, HQQ, GGUF, ML Training Recipes)\n- `11-evaluation/` - Benchmarking (3 skills: lm-evaluation-harness, BigCode, NeMo Evaluator)\n- `12-inference-serving/` - Inference engines (4 skills: vLLM, TensorRT-LLM, llama.cpp, SGLang)\n- `13-mlops/` - Experiment tracking (4 skills: Weights & Biases, MLflow, TensorBoard, SwanLab)\n- `14-agents/` - Agent frameworks (5 skills: LangChain, LlamaIndex, CrewAI, AutoGPT, A-Evolve)\n- `15-rag/` - Retrieval-augmented generation (5 skills: Chroma, FAISS, Sentence Transformers, Pinecone, Qdrant)\n- `16-prompt-engineering/` - Structured output (4 skills: DSPy, Instructor, Guidance, Outlines)\n- `17-observability/` - LLM observability (2 skills: LangSmith, Phoenix)\n- `18-multimodal/` - Vision and speech (10 skills: CLIP, Whisper, LLaVA, Stable Diffusion, SAM, BLIP-2, AudioCraft, Cosmos Policy, OpenPI, OpenVLA-OFT)\n- `19-emerging-techniques/` - Advanced methods (6 skills: MoE Training, Model Merging, Long Context, Speculative Decoding, Knowledge Distillation, Model Pruning)\n- `20-ml-paper-writing/` - Paper writing (4 skills: ML Paper Writing with LaTeX templates for NeurIPS, ICML, ICLR, ACL, AAAI, COLM; Systems Paper Writing for OSDI, NSDI, ASPLOS, SOSP; Academic Plotting; Presenting Conference Talks)\n- `21-research-ideation/` - Ideation (2 skills: Research Brainstorming, Creative Thinking)\n- `22-agent-native-research-artifact/` - Agent-Native Research Artifact tooling (3 skills: ARA Compiler, ARA Research Manager, ARA Rigor Reviewer — ingestion, post-task provenance recording, and Seal Level 2 epistemic review)\n\n### Skill File Structure\n\nEach skill follows a standardized format:\n```\nskill-name/\n├── SKILL.md                    # Main guidance (200-600 lines with YAML frontmatter)\n├── references/                 # Deep documentation (300KB+ target)\n│   ├── README.md              # From official docs\n│   ├── api.md                 # API reference\n│   ├── tutorials.md           # Step-by-step guides\n│   ├── issues.md              # Real GitHub issues & solutions\n│   └── releases.md            # Version history\n├── scripts/                    # Helper scripts (optional)\n├── templates/                  # Code templates (optional)\n└── examples/                   # Example implementations (optional)\n```\n\n## Skill Quality Standards\n\n### YAML Frontmatter Requirements (CRITICAL)\n\nAll `SKILL.md` files MUST include YAML frontmatter with these exact fields:\n\n```yaml\n---\nname: skill-name-here              # kebab-case, no quotes, gerund form preferred\ndescription: Third-person description of what AND when to use this skill  # No quotes, max 1024 chars\nversion: 1.0.0                     # Semantic versioning\nauthor: Orchestra Research         # Standard author\nlicense: MIT                       # Standard license\ntags: [Tag One, Tag Two]          # Title Case (except UPPERCASE acronyms like GRPO, TRL, RLHF)\ndependencies: [pkg>=1.0.0]         # Optional, with version constraints\n---\n```\n\n**Critical Rules**:\n- `name`: Use gerund form (e.g., `serving-llms`, `processing-data`, `grpo-rl-training`)\n- `description`: Third person (\"Provides guidance for...\"), include WHAT it does AND WHEN to use it\n- `tags`: Title Case for regular words, UPPERCASE for acronyms (GRPO, TRL, RLHF, DPO, PPO)\n- No quotes around any field values (except in arrays)\n- Dependencies should include version constraints: `transformers>=4.47.0`\n\n### Content Quality Standards\n\n**Core Requirements** (based on Anthropic official best practices):\n- ✅ SKILL.md body: **200-500 lines** (under 500 lines is critical for performance)\n- ✅ Progressive disclosure: SKILL.md as overview, details in separate reference files\n- ✅ Workflows with copy-paste checklists for complex tasks\n- ✅ \"When to use vs alternatives\" guidance section\n- ✅ Common issues section with solutions\n- ✅ Concise content: assume Claude is smart, no over-explaining basics\n- ✅ Code examples with language detection (```python, ```bash, etc.)\n- ✅ References ONE level deep from SKILL.md (no nested references)\n\n**Gold Standard** (aim for this - see `06-post-training/grpo-rl-training/`):\n- ✅ 2-3 complete workflows with step-by-step checklists\n- ✅ Reference files for advanced topics (one level deep)\n- ✅ Feedback loops (validate → fix → repeat) for quality-critical operations\n- ✅ Consistent terminology throughout\n- ✅ Concrete input/output examples\n- ✅ Real GitHub issues with solutions (when available)\n\n**NOT Acceptable**:\n- ❌ SKILL.md over 500 lines (split into reference files instead)\n- ❌ Over-explaining basics that Claude already knows\n- ❌ First-person descriptions (\"I can help you...\")\n- ❌ Vague skill names (\"helper\", \"utils\", \"tools\")\n- ❌ Nested references (SKILL.md → ref1.md → ref2.md)\n- ❌ Missing workflows with checklists for complex tasks\n\n## Development Workflow\n\n### Adding a New Skill\n\n1. **Choose skill from roadmap** (see CONTRIBUTING.md or README.md)\n2. **Create directory structure** in appropriate category (01-19)\n3. **Write SKILL.md** with YAML frontmatter following standards above\n4. **Add reference documentation** (target 300KB+ from official sources)\n5. **Validate quality**:\n   - Check SKILL.md has YAML frontmatter\n   - Verify SKILL.md is 200-500 lines\n   - Ensure code blocks have language tags\n   - Confirm references are one level deep from SKILL.md\n   - Check documentation size: `du -sh skill-name/references/`\n6. **Test the skill** with real use cases before submitting\n\n### Improving Existing Skills\n\nWhen updating skills:\n1. **Maintain YAML frontmatter** format and fields\n2. **Keep SKILL.md under 500 lines** - split into reference files if needed\n3. **Add workflows** with checklists for complex operations\n4. **Update version number** in YAML frontmatter\n5. **Test changes** with representative tasks\n\n### Quality Validation Commands\n\n```bash\n# Check YAML frontmatter exists\nhead -20 skill-name/SKILL.md\n\n# Verify SKILL.md line count (target 200-500 lines)\nwc -l skill-name/SKILL.md\n\n# Check documentation size (target 300KB+)\ndu -sh skill-name/references/\n\n# Verify code blocks have language tags\ngrep -A 1 '```' skill-name/SKILL.md | head -20\n\n# Validate YAML frontmatter syntax\npython -c \"import yaml; yaml.safe_load(open('skill-name/SKILL.md').read().split('---')[1])\"\n```\n\n## Key Files\n\n- **README.md** - Project overview, all 98 skills listed with descriptions and stats\n- **CONTRIBUTING.md** - Complete contribution guidelines and quality standards\n- **SKILL_TEMPLATE.md** - Copy-paste scaffold for new skills\n- **ROADMAP.md** - Development roadmap (98 skills achieved)\n- **anthropic_official_docs/** - Anthropic's official best practices for skills\n\n## Git Workflow\n\nStandard Git workflow:\n```bash\n# Create feature branch\ngit checkout -b add-skill-name\n\n# Add and commit changes\ngit add category/skill-name/\ngit commit -m \"Add [Skill Name] skill\n\n- X lines of documentation\n- Y GitHub issues with solutions\n- API reference and examples included\"\n\n# Push to fork and create PR\ngit push origin add-skill-name\n```\n\n## Automation: Orchestra Skill Marketplace Sync\n\n### How Auto-Sync Works\n\nWhen skills are committed to the `main` branch, GitHub Actions automatically syncs them to the Orchestra skill marketplace:\n\n1. **GitHub Actions detects** changed skill folders on push to `main`\n2. **For each changed skill**:\n   - Extracts metadata from SKILL.md frontmatter (`name`, `author`, etc.)\n   - Creates ZIP file containing entire skill directory (SKILL.md, references/, scripts/, etc.)\n   - Uploads to Orchestra API endpoint\n3. **Orchestra stores** ZIP in Supabase Storage and creates database record\n4. **Skill appears** in marketplace at `https://orchestra.com/research-skills`\n\n### Workflow File Location\n\n- **File**: `.github/workflows/sync-skills.yml`\n- **Triggers**: Push to `main` branch, manual workflow dispatch\n- **What syncs**: Only skill directories that changed in the commit\n\n### Author Detection (Orchestra vs Community)\n\nThe workflow reads the `author:` field from SKILL.md frontmatter to determine badge:\n\n**Official Orchestra Skill**:\n```yaml\n---\nauthor: Orchestra Research  # Contains \"Orchestra\"\n---\n```\n- Result: Source = `orchestra` (Official badge)\n- Storage: `research-skills/orchestra/skill-name.zip`\n\n**Community Skill**:\n```yaml\n---\nauthor: Jane Doe  # Does NOT contain \"Orchestra\"\n---\n```\n- Result: Source = `community` (Community badge)\n- Storage: `research-skills/community/skill-name.zip`\n\n### What Gets Synced\n\nThe workflow zips **ALL contents** of skill directory:\n- ✅ SKILL.md\n- ✅ references/ (all subdirectories)\n- ✅ scripts/ (if exists)\n- ✅ assets/ (if exists)\n- ✅ examples/ (if exists)\n- ✅ templates/ (if exists)\n- ❌ Hidden files (`.gitkeep`, `.DS_Store`)\n\n### Testing the Sync\n\n**Manual trigger**:\n1. Go to GitHub Actions tab\n2. Select \"Sync Skills to Orchestra\" workflow\n3. Click \"Run workflow\"\n\n**Test with commit**:\n```bash\n# Make a small change to any skill\necho \"\\n<!-- Updated $(date) -->\" >> 01-model-architecture/litgpt/SKILL.md\n\n# Commit and push to main\ngit add .\ngit commit -m \"test: trigger auto-sync\"\ngit push origin main\n```\n\n**Verify sync worked**:\n1. Check GitHub Actions tab for workflow run status\n2. Check Orchestra marketplace for updated skill\n3. Check Supabase Storage for ZIP file\n\n### Important Notes\n\n- **GitHub Secrets required**: `ORCHESTRA_API_URL`, `ORCHESTRA_SYNC_API_KEY` (already configured)\n- **Only syncs changed skills**: Workflow detects which skill directories changed in commit\n- **SKILL.md required**: Skills without SKILL.md are skipped with warning\n- **See detailed setup**: `dev_data/GITHUB_SKILLS_SYNC_SETUP.md`\n\n## npm Package Publishing\n\n### How It Works\n\nThe `publish-npm.yml` workflow auto-publishes to npm when the version in `packages/ai-research-skills/package.json` changes on `main`.\n\n- **Auth**: Uses OIDC trusted publishing (no npm tokens). Configured on npmjs.com under the package's Trusted Publishers settings.\n- **Provenance**: `--provenance` flag signs packages with Sigstore for supply chain security.\n- **Workflow**: `.github/workflows/publish-npm.yml`\n\n### Bumping Versions\n\n**Always use `npm version`** (not manual edits) to keep `package-lock.json` in sync:\n\n```bash\ncd packages/ai-research-skills\nnpm version patch   # 1.3.6 → 1.3.7\nnpm version minor   # 1.3.7 → 1.4.0\nnpm version major   # 1.4.0 → 2.0.0\n```\n\nUse `--no-git-tag-version` if you want to commit manually.\n\n### Common Issues\n\n- **`npm ci` fails in CI**: `package-lock.json` is out of sync. Run `npm install` locally and commit the lockfile.\n- **OIDC auth fails**: The trusted publisher config on npmjs.com must match the repo exactly (case-sensitive: `Orchestra-Research/AI-Research-SKILLs`, workflow: `publish-npm.yml`).\n- **`NODE_AUTH_TOKEN` blocks OIDC**: `actions/setup-node` with `registry-url` auto-sets this token. The workflow unsets it before publish so OIDC takes over.\n- **Version unchanged skip**: The workflow compares `HEAD` vs `HEAD~1`. If only the lockfile changed (not `package.json` version), publish is skipped. Bump the version to trigger.\n\n## Important Conventions\n\n### Naming Conventions\n\n- **Skill names**: Use gerund form (verb + -ing) in kebab-case: `processing-pdfs`, `serving-llms`, `grpo-rl-training`\n- **Tags**: Title Case for words, UPPERCASE for acronyms (GRPO, TRL, RLHF, DPO, PPO, FSDP, MoE)\n- **Descriptions**: Third person, include what AND when to use\n\n### Code Examples\n\nAlways use language detection in code blocks:\n```python\n# Good - has language tag\nfrom transformers import AutoModel\n```\n\nNOT:\n```\n# Bad - no language tag\nfrom transformers import AutoModel\n```\n\n### Progressive Disclosure Pattern\n\nSKILL.md should link directly to reference files (one level deep):\n\n```markdown\n## Advanced Features\n\n**API Reference**: See [references/api.md](references/api.md)\n**Troubleshooting**: See [references/issues.md](references/issues.md)\n```\n\n## Philosophy\n\n**Quality over Quantity**: This library maintains high standards by:\n- Requiring 200-500 line SKILL.md files (focused, actionable guidance)\n- Including 300KB+ documentation from official sources\n- Providing real GitHub issues with solutions\n- Following Anthropic's official best practices for skills\n- Testing skills with real use cases before inclusion\n\nEach skill represents expert-level knowledge distilled into a format optimized for AI agent consumption.\n","06-post-training/miles/SKILL.md":"---\nname: miles-rl-training\ndescription: Provides guidance for enterprise-grade RL training using miles, a production-ready fork of slime. Use when training large MoE models with FP8/INT4, needing train-inference alignment, or requiring speculative RL for maximum throughput.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Reinforcement Learning, MoE, FP8, INT4, Enterprise, SGLang, Megatron-LM]\ndependencies: [sglang-router>=0.2.3, ray, torch>=2.0.0, transformers>=4.40.0]\n---\n\n# miles: Enterprise-Grade RL for Large-Scale Model Training\n\nmiles is a high-performance, enterprise-ready RL framework optimized for large-scale model post-training. Built as a production fork of slime, it addresses critical challenges in MoE training stability, low-precision training, and train-inference alignment.\n\n## When to Use miles\n\n**Choose miles when you need:**\n- Training 1TB+ MoE models (DeepSeek V3, Qwen3-MoE)\n- FP8 or INT4 quantization-aware training\n- Bit-wise identical train-inference alignment\n- Speculative RL for maximum throughput\n- Production stability with enterprise support\n\n**Consider alternatives when:**\n- You want the research-grade original → use **slime**\n- You need flexible backend swapping → use **verl**\n- You want PyTorch-native abstractions → use **torchforge**\n\n## Key Features\n\n### Low-Precision Training\n- **Unified FP8**: End-to-end FP8 for both inference and training\n- **INT4 QAT**: 1TB models on single-machine VRAM (H200)\n- **Rollout Routing Replay (R3)**: Bit-wise expert alignment for MoE\n\n### Performance Optimizations\n- **Speculative RL**: 25%+ rollout speedup with online SFT draft models\n- **Zero-Copy Weight Sync**: CUDA IPC zero-copy mapping\n- **Partial Rollout**: Recycle half-finished trajectories\n\n### Train-Inference Alignment\n- **TIS/MIS**: Truncated/Masked Importance Sampling for off-policy correction\n- **Kernel-level optimization**: FlashAttention-3, DeepGEMM integration\n\n## Installation\n\n```bash\n# Recommended: Docker\ndocker pull radixark/miles:latest\ndocker run --rm --gpus all --ipc=host --shm-size=16g \\\n  -it radixark/miles:latest /bin/bash\n\n# From source\ngit clone https://github.com/radixark/miles.git\ncd miles\npip install -r requirements.txt\npip install -e .\n```\n\n## Quick Start\n\nmiles inherits slime's configuration system. Basic training:\n\n```bash\npython train.py \\\n    --advantage-estimator grpo \\\n    --model-name qwen3-30b-a3b \\\n    --hf-checkpoint /path/to/qwen3-30b-a3b-hf \\\n    --rollout-batch-size 512 \\\n    --n-samples-per-prompt 8\n```\n\n---\n\n## Workflow 1: Large MoE Training\n\nUse this workflow for training large MoE models like DeepSeek V3 or Qwen3-MoE.\n\n### Prerequisites Checklist\n- [ ] H100/H200 GPUs with FP8 support\n- [ ] MoE model (DeepSeek V3, Qwen3-MoE)\n- [ ] Docker environment with miles\n\n### Step 1: Environment Setup\n\n```bash\n# FP8 block scaling (recommended for stability)\nexport NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1\nexport CUDA_DEVICE_MAX_CONNECTIONS=1\n```\n\n### Step 2: Configure Training\n\n```bash\npython train.py \\\n    --actor-num-gpus-per-node 8 \\\n    --rollout-num-gpus 8 \\\n    --hf-checkpoint /path/to/deepseek-v3 \\\n    --advantage-estimator grpo \\\n    --tensor-model-parallel-size 8 \\\n    --expert-model-parallel-size 4 \\\n    --prompt-data /path/to/data.jsonl \\\n    --num-rollout 3000\n```\n\n### Verification Checklist\n- [ ] Model loads without errors\n- [ ] Routing decisions are consistent\n- [ ] No NaN/Inf in loss values\n\n---\n\n## Workflow 2: Speculative RL Training\n\nUse this workflow for maximum rollout throughput with EAGLE speculative decoding.\n\n### How Speculative RL Works\n\n1. Small draft model generates candidate tokens\n2. Target model verifies in parallel\n3. Draft model updated via online SFT to track policy\n\n### Step 1: Enable Speculative Decoding\n\nmiles supports EAGLE speculative decoding via SGLang:\n\n```bash\npython train.py \\\n    --actor-num-gpus-per-node 8 \\\n    --hf-checkpoint /path/to/target-model \\\n    --sglang-speculative-algorithm EAGLE \\\n    --sglang-speculative-num-steps 3 \\\n    --sglang-speculative-eagle-topk 1 \\\n    --sglang-speculative-num-draft-tokens 4 \\\n    --sglang-speculative-draft-model-path /path/to/draft-model \\\n    --advantage-estimator grpo \\\n    --prompt-data /path/to/data.jsonl\n```\n\n### Step 2: Enable Online MTP Training (Optional)\n\nFor online SFT of draft model during training:\n\n```bash\n--mtp-num-layers 1 \\\n--enable-mtp-training \\\n--mtp-loss-scaling-factor 0.2\n```\n\n**Note**: Online MTP training requires a torch dist checkpoint with MTP weights. Add `--mtp-num-layers 1` during checkpoint conversion from HuggingFace.\n\n### Expected Speedup\n\n- **Standard rollout**: Baseline\n- **Speculative RL**: 25-40% faster rollout\n- **With partial rollout**: Additional 10-15% throughput\n\n---\n\n## Configuration Reference\n\nmiles inherits all slime arguments. See [slime API Reference](../slime/references/api-reference.md) for the complete list.\n\n### Cluster Resources (from slime)\n\n```bash\n--actor-num-nodes 1\n--actor-num-gpus-per-node 8\n--rollout-num-gpus 8\n--rollout-num-gpus-per-engine 2\n--colocate\n```\n\n### Megatron Parallelism (from slime)\n\n```bash\n--tensor-model-parallel-size 8\n--pipeline-model-parallel-size 2\n--expert-model-parallel-size 4    # MoE expert parallelism\n```\n\n### Speculative Decoding (miles-specific)\n\n```bash\n--sglang-speculative-algorithm EAGLE\n--sglang-speculative-num-steps 3\n--sglang-speculative-eagle-topk 1\n--sglang-speculative-num-draft-tokens 4\n--sglang-enable-draft-weights-cpu-backup\n--sglang-speculative-draft-model-path /your/draft/model/path\n```\n\n### Online MTP Training (miles-specific)\n\n```bash\n--mtp-num-layers 1\n--enable-mtp-training\n--mtp-loss-scaling-factor 0.2\n```\n\n---\n\n## Key Features (Conceptual)\n\nThe following features are documented in miles but specific CLI flags may vary. Consult the miles repository for latest configuration.\n\n### Unified FP8 Pipeline\n\nEnd-to-end FP8 sampling and training that eliminates quantization-induced discrepancy causing RL collapse in MoE models.\n\n### Rollout Routing Replay (R3)\n\nRecords expert routing decisions during SGLang inference and replays them during Megatron training for bit-wise expert alignment.\n\n**How R3 Works**:\n1. During SGLang inference, expert routing decisions are recorded\n2. Routing decisions stored in `sample.rollout_routed_experts`\n3. During Megatron training, routing is replayed instead of recomputed\n4. Ensures identical expert selection between train and inference\n\n### INT4 Quantization-Aware Training\n\nEnables single-machine deployment of 1TB+ models (e.g., on H200).\n\n**Memory Savings with INT4**:\n\n| Model Size | BF16 VRAM | INT4 VRAM | Reduction |\n|------------|-----------|-----------|-----------|\n| 70B | 140GB | 45GB | 3.1x |\n| 235B | 470GB | 150GB | 3.1x |\n| 671B | 1.3TB | 420GB | 3.1x |\n\n### Train-Inference Alignment\n\nmiles achieves \"exactly 0 KL divergence\" between training and inference through:\n- Flash Attention 3\n- DeepGEMM\n- Batch-invariant kernels from Thinking Machines Lab\n- `torch.compile` integration\n\n---\n\n## Sample Data Structure\n\nmiles uses the same `Sample` dataclass as slime with the `rollout_routed_experts` field for MoE routing replay:\n\n```python\n@dataclass\nclass Sample:\n    prompt: str | list[dict]\n    tokens: list[int]\n    response: str\n    reward: float | dict\n    loss_mask: list[int]\n    status: Status\n    metadata: dict\n    rollout_log_probs: list[float]\n    rollout_routed_experts: list[list[int]]  # MoE routing for R3\n```\n\nSee [slime API Reference](../slime/references/api-reference.md) for the complete Sample definition.\n\n---\n\n## Common Issues and Solutions\n\n### Issue: FP8 Training Collapse\n\n**Symptoms**: Loss explodes, NaN values\n\n**Solutions**:\n- Use block scaling: `export NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1`\n- Reduce learning rate: `--lr 5e-7`\n- Ensure MoE routing is consistent between train/inference\n\n### Issue: Speculative Draft Drift\n\n**Symptoms**: Low acceptance rate over time\n\n**Solutions**:\n- Enable online MTP training to keep draft model aligned\n- Reduce speculative steps: `--sglang-speculative-num-steps 2`\n- Use CPU backup: `--sglang-enable-draft-weights-cpu-backup`\n\n### Issue: Train-Inference Mismatch\n\n**Symptoms**: Policy divergence, reward collapse\n\n**Solutions**:\n- Use TIS for off-policy correction: `--use-tis --tis-threshold 0.9`\n- Verify log probs match between SGLang and Megatron\n- Enable R3 for MoE models\n\n---\n\n## Supported Models\n\n| Family | Models | MoE Support |\n|--------|--------|-------------|\n| DeepSeek | R1, V3, V3.2 | Full |\n| Qwen | 2, 2.5, 3 (including MoE) | Full |\n| Llama | 3, 3.1, 3.3, 4 | Dense only |\n| Gemma | 2, 3, 3N | Dense only |\n| GLM | 4.5, 4.6, 4.7 | Dense only |\n| MiniMax | M2, M2.1 | Full |\n\n---\n\n## Resources\n\n- **GitHub**: https://github.com/radixark/miles\n- **Introduction Blog**: https://lmsys.org/blog/2025-11-19-miles/\n- **Slime (upstream)**: https://github.com/THUDM/slime\n- **SGLang**: https://github.com/sgl-project/sglang\n\n"},"items":[{"name":"CLAUDE.md","path":"CLAUDE.md","title":"CLAUDE.md","content":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\n**AI Research Skills Library** - A comprehensive open-source library of 98 AI research skills enabling AI agents to autonomously conduct AI research — from idea to paper. Each skill provides expert-level guidance (200-500 lines) with real code examples, troubleshooting guides, and production-ready workflows.\n\n**Mission**: Enable AI agents to autonomously conduct AI research from hypothesis to experimental verification, covering the full lifecycle: literature survey, ideation, dataset preparation, training pipelines, model deployment, evaluation, and paper writing.\n\n## Repository Architecture\n\n### Directory Structure (98 Skills Across 23 Categories)\n\nSkills are organized into numbered categories representing the AI research lifecycle:\n\n- `0-autoresearch-skill/` - **Autonomous research orchestration** (1 skill: Autoresearch — central layer that manages the full lifecycle and routes to all other skills)\n- `01-model-architecture/` - Model architectures (5 skills: TorchTitan, LitGPT, Mamba, RWKV, NanoGPT)\n- `02-tokenization/` - Tokenizers (2 skills: HuggingFace Tokenizers, SentencePiece)\n- `03-fine-tuning/` - Fine-tuning frameworks (4 skills: Axolotl, LLaMA-Factory, Unsloth, PEFT)\n- `04-mechanistic-interpretability/` - Interpretability tools (4 skills: TransformerLens, SAELens, NNsight, Pyvene)\n- `05-data-processing/` - Data curation (2 skills: Ray Data, NeMo Curator)\n- `06-post-training/` - RLHF/DPO/GRPO (8 skills: TRL, GRPO, OpenRLHF, SimPO, verl, slime, miles, torchforge)\n- `07-safety-alignment/` - Safety and guardrails (4 skills: Constitutional AI, LlamaGuard, NeMo Guardrails, Prompt Guard)\n- `08-distributed-training/` - Distributed systems (6 skills: Megatron-Core, DeepSpeed, FSDP, Accelerate, PyTorch Lightning, Ray Train)\n- `09-infrastructure/` - Cloud compute (3 skills: Modal, SkyPilot, Lambda Labs)\n- `10-optimization/` - Optimization techniques (7 skills: Flash Attention, bitsandbytes, GPTQ, AWQ, HQQ, GGUF, ML Training Recipes)\n- `11-evaluation/` - Benchmarking (3 skills: lm-evaluation-harness, BigCode, NeMo Evaluator)\n- `12-inference-serving/` - Inference engines (4 skills: vLLM, TensorRT-LLM, llama.cpp, SGLang)\n- `13-mlops/` - Experiment tracking (4 skills: Weights & Biases, MLflow, TensorBoard, SwanLab)\n- `14-agents/` - Agent frameworks (5 skills: LangChain, LlamaIndex, CrewAI, AutoGPT, A-Evolve)\n- `15-rag/` - Retrieval-augmented generation (5 skills: Chroma, FAISS, Sentence Transformers, Pinecone, Qdrant)\n- `16-prompt-engineering/` - Structured output (4 skills: DSPy, Instructor, Guidance, Outlines)\n- `17-observability/` - LLM observability (2 skills: LangSmith, Phoenix)\n- `18-multimodal/` - Vision and speech (10 skills: CLIP, Whisper, LLaVA, Stable Diffusion, SAM, BLIP-2, AudioCraft, Cosmos Policy, OpenPI, OpenVLA-OFT)\n- `19-emerging-techniques/` - Advanced methods (6 skills: MoE Training, Model Merging, Long Context, Speculative Decoding, Knowledge Distillation, Model Pruning)\n- `20-ml-paper-writing/` - Paper writing (4 skills: ML Paper Writing with LaTeX templates for NeurIPS, ICML, ICLR, ACL, AAAI, COLM; Systems Paper Writing for OSDI, NSDI, ASPLOS, SOSP; Academic Plotting; Presenting Conference Talks)\n- `21-research-ideation/` - Ideation (2 skills: Research Brainstorming, Creative Thinking)\n- `22-agent-native-research-artifact/` - Agent-Native Research Artifact tooling (3 skills: ARA Compiler, ARA Research Manager, ARA Rigor Reviewer — ingestion, post-task provenance recording, and Seal Level 2 epistemic review)\n\n### Skill File Structure\n\nEach skill follows a standardized format:\n```\nskill-name/\n├── SKILL.md                    # Main guidance (200-600 lines with YAML frontmatter)\n├── references/                 # Deep documentation (300KB+ target)\n│   ├── README.md              # From official docs\n│   ├── api.md                 # API reference\n│   ├── tutorials.md           # Step-by-step guides\n│   ├── issues.md              # Real GitHub issues & solutions\n│   └── releases.md            # Version history\n├── scripts/                    # Helper scripts (optional)\n├── templates/                  # Code templates (optional)\n└── examples/                   # Example implementations (optional)\n```\n\n## Skill Quality Standards\n\n### YAML Frontmatter Requirements (CRITICAL)\n\nAll `SKILL.md` files MUST include YAML frontmatter with these exact fields:\n\n```yaml\n---\nname: skill-name-here              # kebab-case, no quotes, gerund form preferred\ndescription: Third-person description of what AND when to use this skill  # No quotes, max 1024 chars\nversion: 1.0.0                     # Semantic versioning\nauthor: Orchestra Research         # Standard author\nlicense: MIT                       # Standard license\ntags: [Tag One, Tag Two]          # Title Case (except UPPERCASE acronyms like GRPO, TRL, RLHF)\ndependencies: [pkg>=1.0.0]         # Optional, with version constraints\n---\n```\n\n**Critical Rules**:\n- `name`: Use gerund form (e.g., `serving-llms`, `processing-data`, `grpo-rl-training`)\n- `description`: Third person (\"Provides guidance for...\"), include WHAT it does AND WHEN to use it\n- `tags`: Title Case for regular words, UPPERCASE for acronyms (GRPO, TRL, RLHF, DPO, PPO)\n- No quotes around any field values (except in arrays)\n- Dependencies should include version constraints: `transformers>=4.47.0`\n\n### Content Quality Standards\n\n**Core Requirements** (based on Anthropic official best practices):\n- ✅ SKILL.md body: **200-500 lines** (under 500 lines is critical for performance)\n- ✅ Progressive disclosure: SKILL.md as overview, details in separate reference files\n- ✅ Workflows with copy-paste checklists for complex tasks\n- ✅ \"When to use vs alternatives\" guidance section\n- ✅ Common issues section with solutions\n- ✅ Concise content: assume Claude is smart, no over-explaining basics\n- ✅ Code examples with language detection (```python, ```bash, etc.)\n- ✅ References ONE level deep from SKILL.md (no nested references)\n\n**Gold Standard** (aim for this - see `06-post-training/grpo-rl-training/`):\n- ✅ 2-3 complete workflows with step-by-step checklists\n- ✅ Reference files for advanced topics (one level deep)\n- ✅ Feedback loops (validate → fix → repeat) for quality-critical operations\n- ✅ Consistent terminology throughout\n- ✅ Concrete input/output examples\n- ✅ Real GitHub issues with solutions (when available)\n\n**NOT Acceptable**:\n- ❌ SKILL.md over 500 lines (split into reference files instead)\n- ❌ Over-explaining basics that Claude already knows\n- ❌ First-person descriptions (\"I can help you...\")\n- ❌ Vague skill names (\"helper\", \"utils\", \"tools\")\n- ❌ Nested references (SKILL.md → ref1.md → ref2.md)\n- ❌ Missing workflows with checklists for complex tasks\n\n## Development Workflow\n\n### Adding a New Skill\n\n1. **Choose skill from roadmap** (see CONTRIBUTING.md or README.md)\n2. **Create directory structure** in appropriate category (01-19)\n3. **Write SKILL.md** with YAML frontmatter following standards above\n4. **Add reference documentation** (target 300KB+ from official sources)\n5. **Validate quality**:\n   - Check SKILL.md has YAML frontmatter\n   - Verify SKILL.md is 200-500 lines\n   - Ensure code blocks have language tags\n   - Confirm references are one level deep from SKILL.md\n   - Check documentation size: `du -sh skill-name/references/`\n6. **Test the skill** with real use cases before submitting\n\n### Improving Existing Skills\n\nWhen updating skills:\n1. **Maintain YAML frontmatter** format and fields\n2. **Keep SKILL.md under 500 lines** - split into reference files if needed\n3. **Add workflows** with checklists for complex operations\n4. **Update version number** in YAML frontmatter\n5. **Test changes** with representative tasks\n\n### Quality Validation Commands\n\n```bash\n# Check YAML frontmatter exists\nhead -20 skill-name/SKILL.md\n\n# Verify SKILL.md line count (target 200-500 lines)\nwc -l skill-name/SKILL.md\n\n# Check documentation size (target 300KB+)\ndu -sh skill-name/references/\n\n# Verify code blocks have language tags\ngrep -A 1 '```' skill-name/SKILL.md | head -20\n\n# Validate YAML frontmatter syntax\npython -c \"import yaml; yaml.safe_load(open('skill-name/SKILL.md').read().split('---')[1])\"\n```\n\n## Key Files\n\n- **README.md** - Project overview, all 98 skills listed with descriptions and stats\n- **CONTRIBUTING.md** - Complete contribution guidelines and quality standards\n- **SKILL_TEMPLATE.md** - Copy-paste scaffold for new skills\n- **ROADMAP.md** - Development roadmap (98 skills achieved)\n- **anthropic_official_docs/** - Anthropic's official best practices for skills\n\n## Git Workflow\n\nStandard Git workflow:\n```bash\n# Create feature branch\ngit checkout -b add-skill-name\n\n# Add and commit changes\ngit add category/skill-name/\ngit commit -m \"Add [Skill Name] skill\n\n- X lines of documentation\n- Y GitHub issues with solutions\n- API reference and examples included\"\n\n# Push to fork and create PR\ngit push origin add-skill-name\n```\n\n## Automation: Orchestra Skill Marketplace Sync\n\n### How Auto-Sync Works\n\nWhen skills are committed to the `main` branch, GitHub Actions automatically syncs them to the Orchestra skill marketplace:\n\n1. **GitHub Actions detects** changed skill folders on push to `main`\n2. **For each changed skill**:\n   - Extracts metadata from SKILL.md frontmatter (`name`, `author`, etc.)\n   - Creates ZIP file containing entire skill directory (SKILL.md, references/, scripts/, etc.)\n   - Uploads to Orchestra API endpoint\n3. **Orchestra stores** ZIP in Supabase Storage and creates database record\n4. **Skill appears** in marketplace at `https://orchestra.com/research-skills`\n\n### Workflow File Location\n\n- **File**: `.github/workflows/sync-skills.yml`\n- **Triggers**: Push to `main` branch, manual workflow dispatch\n- **What syncs**: Only skill directories that changed in the commit\n\n### Author Detection (Orchestra vs Community)\n\nThe workflow reads the `author:` field from SKILL.md frontmatter to determine badge:\n\n**Official Orchestra Skill**:\n```yaml\n---\nauthor: Orchestra Research  # Contains \"Orchestra\"\n---\n```\n- Result: Source = `orchestra` (Official badge)\n- Storage: `research-skills/orchestra/skill-name.zip`\n\n**Community Skill**:\n```yaml\n---\nauthor: Jane Doe  # Does NOT contain \"Orchestra\"\n---\n```\n- Result: Source = `community` (Community badge)\n- Storage: `research-skills/community/skill-name.zip`\n\n### What Gets Synced\n\nThe workflow zips **ALL contents** of skill directory:\n- ✅ SKILL.md\n- ✅ references/ (all subdirectories)\n- ✅ scripts/ (if exists)\n- ✅ assets/ (if exists)\n- ✅ examples/ (if exists)\n- ✅ templates/ (if exists)\n- ❌ Hidden files (`.gitkeep`, `.DS_Store`)\n\n### Testing the Sync\n\n**Manual trigger**:\n1. Go to GitHub Actions tab\n2. Select \"Sync Skills to Orchestra\" workflow\n3. Click \"Run workflow\"\n\n**Test with commit**:\n```bash\n# Make a small change to any skill\necho \"\\n<!-- Updated $(date) -->\" >> 01-model-architecture/litgpt/SKILL.md\n\n# Commit and push to main\ngit add .\ngit commit -m \"test: trigger auto-sync\"\ngit push origin main\n```\n\n**Verify sync worked**:\n1. Check GitHub Actions tab for workflow run status\n2. Check Orchestra marketplace for updated skill\n3. Check Supabase Storage for ZIP file\n\n### Important Notes\n\n- **GitHub Secrets required**: `ORCHESTRA_API_URL`, `ORCHESTRA_SYNC_API_KEY` (already configured)\n- **Only syncs changed skills**: Workflow detects which skill directories changed in commit\n- **SKILL.md required**: Skills without SKILL.md are skipped with warning\n- **See detailed setup**: `dev_data/GITHUB_SKILLS_SYNC_SETUP.md`\n\n## npm Package Publishing\n\n### How It Works\n\nThe `publish-npm.yml` workflow auto-publishes to npm when the version in `packages/ai-research-skills/package.json` changes on `main`.\n\n- **Auth**: Uses OIDC trusted publishing (no npm tokens). Configured on npmjs.com under the package's Trusted Publishers settings.\n- **Provenance**: `--provenance` flag signs packages with Sigstore for supply chain security.\n- **Workflow**: `.github/workflows/publish-npm.yml`\n\n### Bumping Versions\n\n**Always use `npm version`** (not manual edits) to keep `package-lock.json` in sync:\n\n```bash\ncd packages/ai-research-skills\nnpm version patch   # 1.3.6 → 1.3.7\nnpm version minor   # 1.3.7 → 1.4.0\nnpm version major   # 1.4.0 → 2.0.0\n```\n\nUse `--no-git-tag-version` if you want to commit manually.\n\n### Common Issues\n\n- **`npm ci` fails in CI**: `package-lock.json` is out of sync. Run `npm install` locally and commit the lockfile.\n- **OIDC auth fails**: The trusted publisher config on npmjs.com must match the repo exactly (case-sensitive: `Orchestra-Research/AI-Research-SKILLs`, workflow: `publish-npm.yml`).\n- **`NODE_AUTH_TOKEN` blocks OIDC**: `actions/setup-node` with `registry-url` auto-sets this token. The workflow unsets it before publish so OIDC takes over.\n- **Version unchanged skip**: The workflow compares `HEAD` vs `HEAD~1`. If only the lockfile changed (not `package.json` version), publish is skipped. Bump the version to trigger.\n\n## Important Conventions\n\n### Naming Conventions\n\n- **Skill names**: Use gerund form (verb + -ing) in kebab-case: `processing-pdfs`, `serving-llms`, `grpo-rl-training`\n- **Tags**: Title Case for words, UPPERCASE for acronyms (GRPO, TRL, RLHF, DPO, PPO, FSDP, MoE)\n- **Descriptions**: Third person, include what AND when to use\n\n### Code Examples\n\nAlways use language detection in code blocks:\n```python\n# Good - has language tag\nfrom transformers import AutoModel\n```\n\nNOT:\n```\n# Bad - no language tag\nfrom transformers import AutoModel\n```\n\n### Progressive Disclosure Pattern\n\nSKILL.md should link directly to reference files (one level deep):\n\n```markdown\n## Advanced Features\n\n**API Reference**: See [references/api.md](references/api.md)\n**Troubleshooting**: See [references/issues.md](references/issues.md)\n```\n\n## Philosophy\n\n**Quality over Quantity**: This library maintains high standards by:\n- Requiring 200-500 line SKILL.md files (focused, actionable guidance)\n- Including 300KB+ documentation from official sources\n- Providing real GitHub issues with solutions\n- Following Anthropic's official best practices for skills\n- Testing skills with real use cases before inclusion\n\nEach skill represents expert-level knowledge distilled into a format optimized for AI agent consumption.\n","category":"root","tokens":3582},{"name":"SKILL.md","path":"06-post-training/miles/SKILL.md","title":"miles Skill","content":"---\nname: miles-rl-training\ndescription: Provides guidance for enterprise-grade RL training using miles, a production-ready fork of slime. Use when training large MoE models with FP8/INT4, needing train-inference alignment, or requiring speculative RL for maximum throughput.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Reinforcement Learning, MoE, FP8, INT4, Enterprise, SGLang, Megatron-LM]\ndependencies: [sglang-router>=0.2.3, ray, torch>=2.0.0, transformers>=4.40.0]\n---\n\n# miles: Enterprise-Grade RL for Large-Scale Model Training\n\nmiles is a high-performance, enterprise-ready RL framework optimized for large-scale model post-training. Built as a production fork of slime, it addresses critical challenges in MoE training stability, low-precision training, and train-inference alignment.\n\n## When to Use miles\n\n**Choose miles when you need:**\n- Training 1TB+ MoE models (DeepSeek V3, Qwen3-MoE)\n- FP8 or INT4 quantization-aware training\n- Bit-wise identical train-inference alignment\n- Speculative RL for maximum throughput\n- Production stability with enterprise support\n\n**Consider alternatives when:**\n- You want the research-grade original → use **slime**\n- You need flexible backend swapping → use **verl**\n- You want PyTorch-native abstractions → use **torchforge**\n\n## Key Features\n\n### Low-Precision Training\n- **Unified FP8**: End-to-end FP8 for both inference and training\n- **INT4 QAT**: 1TB models on single-machine VRAM (H200)\n- **Rollout Routing Replay (R3)**: Bit-wise expert alignment for MoE\n\n### Performance Optimizations\n- **Speculative RL**: 25%+ rollout speedup with online SFT draft models\n- **Zero-Copy Weight Sync**: CUDA IPC zero-copy mapping\n- **Partial Rollout**: Recycle half-finished trajectories\n\n### Train-Inference Alignment\n- **TIS/MIS**: Truncated/Masked Importance Sampling for off-policy correction\n- **Kernel-level optimization**: FlashAttention-3, DeepGEMM integration\n\n## Installation\n\n```bash\n# Recommended: Docker\ndocker pull radixark/miles:latest\ndocker run --rm --gpus all --ipc=host --shm-size=16g \\\n  -it radixark/miles:latest /bin/bash\n\n# From source\ngit clone https://github.com/radixark/miles.git\ncd miles\npip install -r requirements.txt\npip install -e .\n```\n\n## Quick Start\n\nmiles inherits slime's configuration system. Basic training:\n\n```bash\npython train.py \\\n    --advantage-estimator grpo \\\n    --model-name qwen3-30b-a3b \\\n    --hf-checkpoint /path/to/qwen3-30b-a3b-hf \\\n    --rollout-batch-size 512 \\\n    --n-samples-per-prompt 8\n```\n\n---\n\n## Workflow 1: Large MoE Training\n\nUse this workflow for training large MoE models like DeepSeek V3 or Qwen3-MoE.\n\n### Prerequisites Checklist\n- [ ] H100/H200 GPUs with FP8 support\n- [ ] MoE model (DeepSeek V3, Qwen3-MoE)\n- [ ] Docker environment with miles\n\n### Step 1: Environment Setup\n\n```bash\n# FP8 block scaling (recommended for stability)\nexport NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1\nexport CUDA_DEVICE_MAX_CONNECTIONS=1\n```\n\n### Step 2: Configure Training\n\n```bash\npython train.py \\\n    --actor-num-gpus-per-node 8 \\\n    --rollout-num-gpus 8 \\\n    --hf-checkpoint /path/to/deepseek-v3 \\\n    --advantage-estimator grpo \\\n    --tensor-model-parallel-size 8 \\\n    --expert-model-parallel-size 4 \\\n    --prompt-data /path/to/data.jsonl \\\n    --num-rollout 3000\n```\n\n### Verification Checklist\n- [ ] Model loads without errors\n- [ ] Routing decisions are consistent\n- [ ] No NaN/Inf in loss values\n\n---\n\n## Workflow 2: Speculative RL Training\n\nUse this workflow for maximum rollout throughput with EAGLE speculative decoding.\n\n### How Speculative RL Works\n\n1. Small draft model generates candidate tokens\n2. Target model verifies in parallel\n3. Draft model updated via online SFT to track policy\n\n### Step 1: Enable Speculative Decoding\n\nmiles supports EAGLE speculative decoding via SGLang:\n\n```bash\npython train.py \\\n    --actor-num-gpus-per-node 8 \\\n    --hf-checkpoint /path/to/target-model \\\n    --sglang-speculative-algorithm EAGLE \\\n    --sglang-speculative-num-steps 3 \\\n    --sglang-speculative-eagle-topk 1 \\\n    --sglang-speculative-num-draft-tokens 4 \\\n    --sglang-speculative-draft-model-path /path/to/draft-model \\\n    --advantage-estimator grpo \\\n    --prompt-data /path/to/data.jsonl\n```\n\n### Step 2: Enable Online MTP Training (Optional)\n\nFor online SFT of draft model during training:\n\n```bash\n--mtp-num-layers 1 \\\n--enable-mtp-training \\\n--mtp-loss-scaling-factor 0.2\n```\n\n**Note**: Online MTP training requires a torch dist checkpoint with MTP weights. Add `--mtp-num-layers 1` during checkpoint conversion from HuggingFace.\n\n### Expected Speedup\n\n- **Standard rollout**: Baseline\n- **Speculative RL**: 25-40% faster rollout\n- **With partial rollout**: Additional 10-15% throughput\n\n---\n\n## Configuration Reference\n\nmiles inherits all slime arguments. See [slime API Reference](../slime/references/api-reference.md) for the complete list.\n\n### Cluster Resources (from slime)\n\n```bash\n--actor-num-nodes 1\n--actor-num-gpus-per-node 8\n--rollout-num-gpus 8\n--rollout-num-gpus-per-engine 2\n--colocate\n```\n\n### Megatron Parallelism (from slime)\n\n```bash\n--tensor-model-parallel-size 8\n--pipeline-model-parallel-size 2\n--expert-model-parallel-size 4    # MoE expert parallelism\n```\n\n### Speculative Decoding (miles-specific)\n\n```bash\n--sglang-speculative-algorithm EAGLE\n--sglang-speculative-num-steps 3\n--sglang-speculative-eagle-topk 1\n--sglang-speculative-num-draft-tokens 4\n--sglang-enable-draft-weights-cpu-backup\n--sglang-speculative-draft-model-path /your/draft/model/path\n```\n\n### Online MTP Training (miles-specific)\n\n```bash\n--mtp-num-layers 1\n--enable-mtp-training\n--mtp-loss-scaling-factor 0.2\n```\n\n---\n\n## Key Features (Conceptual)\n\nThe following features are documented in miles but specific CLI flags may vary. Consult the miles repository for latest configuration.\n\n### Unified FP8 Pipeline\n\nEnd-to-end FP8 sampling and training that eliminates quantization-induced discrepancy causing RL collapse in MoE models.\n\n### Rollout Routing Replay (R3)\n\nRecords expert routing decisions during SGLang inference and replays them during Megatron training for bit-wise expert alignment.\n\n**How R3 Works**:\n1. During SGLang inference, expert routing decisions are recorded\n2. Routing decisions stored in `sample.rollout_routed_experts`\n3. During Megatron training, routing is replayed instead of recomputed\n4. Ensures identical expert selection between train and inference\n\n### INT4 Quantization-Aware Training\n\nEnables single-machine deployment of 1TB+ models (e.g., on H200).\n\n**Memory Savings with INT4**:\n\n| Model Size | BF16 VRAM | INT4 VRAM | Reduction |\n|------------|-----------|-----------|-----------|\n| 70B | 140GB | 45GB | 3.1x |\n| 235B | 470GB | 150GB | 3.1x |\n| 671B | 1.3TB | 420GB | 3.1x |\n\n### Train-Inference Alignment\n\nmiles achieves \"exactly 0 KL divergence\" between training and inference through:\n- Flash Attention 3\n- DeepGEMM\n- Batch-invariant kernels from Thinking Machines Lab\n- `torch.compile` integration\n\n---\n\n## Sample Data Structure\n\nmiles uses the same `Sample` dataclass as slime with the `rollout_routed_experts` field for MoE routing replay:\n\n```python\n@dataclass\nclass Sample:\n    prompt: str | list[dict]\n    tokens: list[int]\n    response: str\n    reward: float | dict\n    loss_mask: list[int]\n    status: Status\n    metadata: dict\n    rollout_log_probs: list[float]\n    rollout_routed_experts: list[list[int]]  # MoE routing for R3\n```\n\nSee [slime API Reference](../slime/references/api-reference.md) for the complete Sample definition.\n\n---\n\n## Common Issues and Solutions\n\n### Issue: FP8 Training Collapse\n\n**Symptoms**: Loss explodes, NaN values\n\n**Solutions**:\n- Use block scaling: `export NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1`\n- Reduce learning rate: `--lr 5e-7`\n- Ensure MoE routing is consistent between train/inference\n\n### Issue: Speculative Draft Drift\n\n**Symptoms**: Low acceptance rate over time\n\n**Solutions**:\n- Enable online MTP training to keep draft model aligned\n- Reduce speculative steps: `--sglang-speculative-num-steps 2`\n- Use CPU backup: `--sglang-enable-draft-weights-cpu-backup`\n\n### Issue: Train-Inference Mismatch\n\n**Symptoms**: Policy divergence, reward collapse\n\n**Solutions**:\n- Use TIS for off-policy correction: `--use-tis --tis-threshold 0.9`\n- Verify log probs match between SGLang and Megatron\n- Enable R3 for MoE models\n\n---\n\n## Supported Models\n\n| Family | Models | MoE Support |\n|--------|--------|-------------|\n| DeepSeek | R1, V3, V3.2 | Full |\n| Qwen | 2, 2.5, 3 (including MoE) | Full |\n| Llama | 3, 3.1, 3.3, 4 | Dense only |\n| Gemma | 2, 3, 3N | Dense only |\n| GLM | 4.5, 4.6, 4.7 | Dense only |\n| MiniMax | M2, M2.1 | Full |\n\n---\n\n## Resources\n\n- **GitHub**: https://github.com/radixark/miles\n- **Introduction Blog**: https://lmsys.org/blog/2025-11-19-miles/\n- **Slime (upstream)**: https://github.com/THUDM/slime\n- **SGLang**: https://github.com/sgl-project/sglang\n\n","category":"06-post-training","tokens":2224}]}