{"owner":"InternLM","repo":"xtuner","hasSkills":true,"totalSkillsCount":18,"totalTokensCount":37419,"categories":["claude-rule","anthropic-skill","root-instruction","plugin-manifest"],"hasMcp":false,"mcpConfig":null,"found":[".claude/CLAUDE.md",".claude/rules/datasets_architecture.md",".claude/rules/rl_review.md",".claude/skills/add-chat-template/SKILL.md",".claude/skills/add-chat-template/agents/openai.yaml",".claude/skills/add_hf_model/SKILL.md",".claude/skills/check-hf-config-save/SKILL.md",".claude/skills/check-hf-config-save/agents/openai.yaml",".claude/skills/model_normalize/SKILL.md",".claude/skills/model_patch/SKILL.md",".claude/skills/sphinx-debug/SKILL.md",".claude/skills/xtuner-custom-judger/SKILL.md",".claude/skills/xtuner-sync-supported-models/SKILL.md",".claude/skills/xtuner-trace/SKILL.md",".claude/skills/xtuner-trace/references/blackbox-agent-trace.md",".claude/skills/xtuner-trace/references/trace-patterns.md",".codex/AGENTS.md","xtuner/tools/llm_calc/README.md"],"skills":{".claude/CLAUDE.md":"# XTuner Programming Standards\n\n## Python\n\n### Development Standards\n\n1. When adding methods or functions to a module or class, declare **public interfaces** as early as possible and place **private interfaces** toward the end.\n2. Python **>= 3.10** is required. Use modern syntax accordingly (e.g., `X | Y` union types).\n3. Use `{}` instead of `dict()` for creating dictionaries. Use `[]` instead of `list()` for creating lists.\n4. Use **double quotes** (`\"`) for strings instead of single quotes (`'`).\n5. Use `TypedDict` with `{}` syntax when a parameter requires a typed dictionary.\n6. Prefer `pathlib.Path` over `os.path` for filesystem operations.\n7. All new code must include **type hints** for function signatures (parameters and return types).\n8. Maximum line length is **119** characters (configured in `pyproject.toml`).\n\n### Docstring Standards\n\n1. Use **Google Style** docstrings.\n2. For classes, include `__init__` documentation directly under the **class docstring**; do not repeat it under the `__init__` method.\n3. Parameter types **must** be declared in docstrings — include the type hint in parentheses immediately following the parameter name.\n4. The **return type** must be explicitly declared in the docstring.\n5. Do **not** declare raised exceptions in the docstring.\n6. Do **not** declare Attributes in the class docstring.\n7. If the return value is `None`, omit the `Returns` section.\n8. Only provide docstrings for **public methods**. Private methods do not require docstrings.\n\n### Linting, Formatting & Type Checking\n\n- **Ruff** for linting and formatting, **mypy** for type checking — see `pyproject.toml` for configuration.\n- `mypy` is strict for `xtuner/v1/*` (`ignore_missing_imports = false`).\n- Pre-commit hooks apply **only** to `xtuner/v1/` and `autotest/` — run `pre-commit run --all-files` before submitting a PR.\n- Docstrings are wrapped at **119** characters by `docformatter`.\n\n### Error Handling\n\n1. Never use bare `except:` — always catch specific exception types.\n2. Avoid silencing exceptions with `pass` unless there is a documented reason.\n\n### Testing\n\n1. Every bug fix PR **must** include a regression test that reproduces the original bug.\n2. Test functions must be named `test_` and grouped in classes named `Test`.\n3. Use `pytest` as the test framework. Prefer `pytest.raises` for exception assertions.\n4. Tests should be self-contained — avoid reliance on external services or GPUs unless explicitly marked with `@pytest.mark.gpu`.\n\n---\n\n## Git\n\n### Branch Naming\n\n- Bug fixes: `fix/`\n- Features: `feat/`\n- Refactors: `refactor/`\n\n### Commit Message Standards\n\n1. Follow the style of previous commit messages to maintain a **consistent** project history.\n2. Use the format: `[Tag] Short description` (e.g., `[Fix] Fix OOM in MoE expert routing`).\n3. Common tags: `[Fix]`, `[Feature]`, `[Refactor]`, `[CI]`, `[Docs]`, `[Test]`.\n4. **Never** include metadata or watermarks unrelated to the code itself, such as \"generated by AI\" or \"Co-Authored-By: Claude.\"\n\n---\n\n## Pull Request Standards\n\n### Bug Fix PRs\n\n1. **Title**: `[Fix] `\n2. **Body** must include:\n - **Root Cause**: A clear explanation of why the bug occurred.\n - **Fix**: What was changed and why this is the correct fix.\n - **Test Plan**: How the fix was verified (unit test, manual reproduction, etc.).\n3. Keep the diff **minimal** — fix only the bug. Do not refactor surrounding code, add unrelated improvements, or change formatting of untouched lines.\n4. If the fix touches performance-critical paths (data loading, forward/backward pass, communication), include **before/after** benchmarks or justify why benchmarks are unnecessary.\n\n### General PR Rules\n\n1. One logical change per PR. Do not mix bug fixes with features or refactors.\n2. All CI checks must pass before requesting review.\n3. Resolve all review comments before merging — do not dismiss without explanation.\n\n---\n\n## Code Review Standards\n\n### What to Check\n\n1. **Correctness**: Does the code do what it claims? Are edge cases handled?\n2. **Concurrency & Distributed**: Check for race conditions, deadlocks, incorrect `dist.barrier()` placement, and mismatched collective operations across ranks.\n3. **Memory**: Watch for GPU memory leaks — tensors not freed, caches growing unbounded, missing `torch.no_grad()` in inference paths.\n4. **Numerical Stability**: Flag potential issues like `log(0)`, division by zero, `float32`/`float16` overflow in loss computation or gradient scaling.\n5. **Performance**: Unnecessary CPU-GPU synchronization (`item()`, `.cpu()` in hot paths), redundant data copies, inefficient collective communication patterns.\n6. **API Contracts**: Do public interface changes maintain backward compatibility? Are deprecation warnings added for breaking changes?\n7. **Resource Cleanup**: Are file handles, NCCL communicators, and CUDA streams properly cleaned up?\n\n### Area-Specific Review Rules\n\nIf a PR changes code under `xtuner/v1/rl`, read `.claude/rules/rl_review.md` before commenting or\nposting the final review summary.\n\n### Review Output Format\n\nWhen performing code reviews, use the following structure:\n\n```\n## Summary\n<1-2 sentence overview of the change>\n\n## Issues\n### Critical\n- [file:line] Description of the issue and suggested fix\n\n### Warning\n- [file:line] Description of the concern\n\n### Nit\n- [file:line] Minor style or readability suggestion\n\n## Verdict\nAPPROVE / REQUEST_CHANGES / COMMENT\n```\n\n### Review Principles\n\n1. **Be specific** — always reference the exact file and line number.\n2. **Explain why** — don't just say \"this is wrong\"; explain the failure mode.\n3. **Suggest a fix** — provide a concrete code suggestion when possible.\n4. **Distinguish severity** — separate critical bugs from style nits.\n5. Do not flag issues in code that is **outside the scope** of the PR diff.\n\n\n\n## Rules\n\nPlease refer to the docs in `.claude/rules` for the development guidelines.\n",".claude/rules/datasets_architecture.md":"---\npaths:\n - \"xtuner/v1/datasets/**/*\"\n---\n\n# XTuner v1 Datasets Architecture Rules\n\n## Core Data Contract (DataItem Protocol)\n\nEvery dataset layer must produce/consume `DataItem`:\n```python\n{\n \"input_ids\": list[int], # tokenized token IDs\n \"labels\": list[int], # same length; first token masked to -100\n \"num_tokens\": int, # len(input_ids); used for sampler and packing\n}\n```\n\n- `labels[0] = -100` always (pretrain: first token not predicted)\n- `HardPackDataset.__getitem__` returns `list[DataItem]` (one per source sample in pack)\n- All other datasets return a single `DataItem`\n\n## Tokenization Functions\n\n**Standard pretrain**: `PretrainTokenizeFunction`\n- Accepts: `{\"content\": \"text\"}` or `{\"messages\": [{\"role\": \"pretrain\", \"content\": \"text\"}]}`\n- Hash: `\"{tokenizer_hash}_{source_hash}_{add_bos_token}_{add_eos_token}\"`\n\n**Long text pretrain**: `LongTextPretrainTokenizeFunction`\n- Two-phase: cache phase returns chunk metadata, runtime phase tokenizes specific char ranges\n- Parameters: `chunk_size`, `tokenizer_chunk_chars`, `overlap_chars`, `min_chunk_tokens`\n- BOS only at `char_start==0`, EOS only at document end\n- Hash includes all chunking parameters\n\n## JsonlDataset Architecture\n\n- **Shared memory**: entire file loaded once at init\n- **Offset table**: `np.ndarray` for O(1) line access\n- **Chunked mode**: activated when `chunks.npy` exists in cache\n - Expands `sampled` to per-chunk entries\n - `__getitem__` calls `tokenize_fn(data, char_start=cs, char_end=ce)`\n - `max_length` filter skipped (would incorrectly discard valid long-text samples)\n - `__len__` returns chunk count (not line count)\n\n## Cache Directory Layout\n\n```\ncache_dir/\n .xpuyu-cache-meta.json\n {file_xxhash}/\n offsets.npy\n {tokenize_fn_hash}/\n num_tokens.npy # always present\n chunks.npy # only for LongText mode: (total_chunks, 4)\n # [line_idx, char_start, char_end, num_tokens]\n```\n\nCache key: `(file xxhash, tokenizer hash, tokenize_fn source hash)`\n- Tag-based lookup via `cache_tag` on `DatasetConfig`\n- Source hash auto-invalidates on code change\n\n## Packing Strategies\n\n**HardPackDataset**:\n- Static pack_infos computed at init (deterministic from seed)\n- Returns `list[DataItem]` per pack\n- `global_pack=False`: per-dataset packing (default)\n- `global_pack=True`: concat all datasets then pack\n\n**PresetPackDataset**:\n- User-provided pack config (NPY directory + `paths.json`)\n- Validates structure and token ranges via vectorized numpy\n- Strategies for length mismatch: error/padding (short), error/truncate (long)\n- No state (deterministic from config file)\n\n## Samplers\n\n**LengthGroupedSampler**:\n- Megabatch logic: group similar lengths, sort DESC, shuffle chunks\n- State: `{epoch, step, world_size, num_samples, total_size}`\n- `step` = `consumed_samples % total_size`\n\n**PresetSampler**:\n- User-provided order via `sampler_config_path` (`.npy` path only, mmap read)\n- Round-down to `global_batch_size * world_size` multiple (truncate tail; error if length too short)\n- Per-rank slice: `effective[local_rank::world_size]` on the truncated mmap-backed view (`self.global_order`)\n- `pack_level=\"preset\"` in `DataloaderConfig`; requires `pack_config_path` and `sampler_config_path`\n- State dict compatible with existing resume logic\n\n## Resume Contract\n\n```python\n# Save\nstate = dataloader.get_state_dict(consumed_samples)\n# → {\"sampler\": {...}, \"dataset\": {}}\n\n# Restore\ndataloader.load_state_dict(state)\n```\n\n- `consumed_samples` = total globally-seen samples (across all ranks)\n- Sampler converts to per-epoch `step` by modulo\n- Packing datasets have no state (deterministic)\n\n## Multi-Dataset Support\n\n- `DatasetConfigList` = list of `{dataset, tokenize_fn}` pairs\n- `build_datasets()` returns `list[JsonlDataset]` (1 file → 1 dataset)\n- `sample_ratio` applied per-DatasetConfig via `sampled[]` repetition\n\n## Key Implementation Files\n\n| File | Purpose |\n|------|---------|\n| `config.py` | Pipeline orchestration via `DataloaderConfig.build()` |\n| `pt_tokenize_fn/text.py` | `PretrainTokenizeFunction` |\n| `pt_tokenize_fn/long_text.py` | `LongTextPretrainTokenizeFunction` |\n| `jsonl.py` | `JsonlDataset` with shared memory + offset table |\n| `packing.py` | `HardPackDataset`, `ExpandSoftPackDataset` |\n| `sampler.py` | `LengthGroupedSampler`, `ParallelSampler` |\n| `preset_pack.py` | `PresetPackDataset` |\n| `preset_sampler.py` | `PresetSampler` |\n| `dataloader.py` | `Dataloader` wrapper |\n| `data_item.py` | `DataItem` schema |\n\n## Design Principles\n\n1. **Protocol-based**: All layers communicate via `DataItem` contract\n2. **Cache-aware**: Tokenization cached with automatic invalidation\n3. **Deterministic**: Packing and sampling reproducible from seed/config\n4. **Resumable**: State dict pattern for checkpoint/resume\n5. **Composable**: Tokenize → Dataset → Pack → Sample → Collate pipeline\n",".claude/rules/rl_review.md":"# RL Review Guidelines\n\nUse these rules whenever a PR changes code under `xtuner/v1/rl`. They extend the general review\nstandards in `.claude/CLAUDE.md`; they do not replace them.\n\n## Required Review Output\n\nEvery review for an RL change must state the ProduceBatchResult impact near the top-level summary:\n\n```text\nProduceBatchResult impact: \n```\n\nAlso include routed-experts impact when the PR touches routed-experts logic, rollout response\nhandling, `RolloutState.extra_fields`, object references, or memory ownership around rollout outputs:\n\n```text\nRoutedExperts impact: \n```\n\nAlso include Ray concurrency impact when the PR touches Ray actor methods, actor construction,\nmethod decorators or wrappers, concurrency-group definitions, or rollout control paths:\n\n```text\nRay concurrency impact: \n```\n\nDo not leave these impacts implicit in the finding text. If a finding is about rollout status,\npause/abort/timeout behavior, response handling, producer aggregation, routed-experts ownership, or\nRay actor concurrency, repeat the relevant impact line inside that finding so the downstream effect\nis visible at the point of review.\n\n## ProduceBatchResult Checklist\n\nReview whether the PR can change trainer-visible batch accounting, timing, or reward semantics in\n`ProduceBatchResult`.\n\nCheck this area when the PR changes any of these paths:\n\n- `RolloutState.status`, `finish_reason`, or status conversion.\n- Abort, filter, expire, retry, timeout, cancellation, or failure handling.\n- Producer, agent loop, judger, replay buffer, rollout worker, rollout controller, or backend pause\n cleanup logic.\n- Writers, readers, tests, or fake implementations that construct or consume `ProduceBatchResult`,\n including trainer-facing code outside `xtuner/v1/rl`.\n\nName the concrete field-level impact when any of these fields can change:\n\n- Batch status: `status`.\n- Returned groups: `rollout_states`.\n- Generation timing: `group_gen_count`, `group_gen_mean_s`, `group_gen_p50_s`, `group_gen_p99_s`,\n `group_gen_p99_p50_ratio`, `group_gen_pause_time_s`.\n- Replay-buffer leftovers: `leftover_init`, `leftover_completed`, `leftover_aborted`,\n `leftover_expired`, `leftover_failed`, `leftover_filtered`.\n- Reward accounting: `raw_rewards_sum`, `raw_rewards_count`.\n- Produced work counters: `produced_samples`, `produced_tokens`, `produce_time_s`.\n- Multi-task aggregation: `task_batch_sizes`, `task_results`.\n\nCommon impacts to call out explicitly:\n\n- A sample moving between `ABORTED`, `FAILED`, `FILTERED`, `EXPIRED`, and `COMPLETED` can change the\n corresponding `leftover_*` counts.\n- Pause, abort, timeout, and cancellation changes can inflate or deflate `group_gen_*` timing,\n especially `group_gen_pause_time_s`.\n- Reward or filter-path changes can change `raw_rewards_sum`, `raw_rewards_count`,\n `produced_samples`, and `produced_tokens`.\n\n## Ray Actor Concurrency Checklist\n\nReview whether high-volume rollout calls can starve pause, abort, health-check, restart, or shutdown\nRPCs.\n\nCheck this area when the PR changes any of these paths:\n\n- Ray actor methods decorated with `@ray.method(...)` or `@ray_method(...)`.\n- Decorators or wrappers stacked on Ray actor methods, especially wrappers using `functools.wraps`\n or exposing `__wrapped__`.\n- Actor construction through `ray.remote(...)`, including `concurrency_groups` and\n `max_concurrency`.\n- Generate, pause, continue, abort, health-check, restart, or shutdown methods.\n\nWhen a Ray method is combined with another decorator, keep the Ray method decorator closest to the\nfunction definition:\n\n```python\n@trace_rollout_endpoint(\"rollout.worker.generate\")\n@ray.method(concurrency_group=ROLLOUT_CONCURRENCY_GROUP_GENERATE)\nasync def generate(...):\n ...\n```\n\nDo not use the reverse order:\n\n```python\n@ray.method(concurrency_group=ROLLOUT_CONCURRENCY_GROUP_GENERATE)\n@trace_rollout_endpoint(\"rollout.worker.generate\")\nasync def generate(...):\n ...\n```\n\nThe current Ray actor-class processing unwraps decorated methods before reading Ray method metadata.\nIf `@ray.method(...)` is outside a wrapper, metadata such as `__ray_concurrency_group__` can remain\non the discarded wrapper, causing the method to fall back to the default concurrency group.\n\nReview the effective dispatch behavior, not only the visible decorator order:\n\n- After following `__wrapped__`, the callable inspected by Ray must retain the expected Ray method\n metadata.\n- Every named method concurrency group must be declared by the corresponding `ray.remote(...)`\n actor construction.\n- High-volume `generate` methods must not accidentally fall back to the default group.\n- Pause, abort, and other control RPCs must remain schedulable while generation is saturated.\n- Slow health or lifecycle methods must not occupy every slot needed by critical control RPCs.\n- Blocking work inside async actor methods must not block the actor event loop or concurrency lane.\n\n## RoutedExperts Checklist\n\nReview whether routed-experts ownership and cleanup remain correct.\n\nCheck this area when the PR changes any of these paths:\n\n- LMDeploy rollout response handling.\n- `return_routed_experts`, `routed_experts`, `RolloutState.extra_fields`, or object-ref plumbing.\n- Abort, cancellation, timeout, filter, retry, or failure paths before response handling completes.\n- Background tasks, replay-buffer storage, trainer batches, metrics, tests, or fake rollout\n responses that can retain routed-experts object refs.\n\nFor LMDeploy rollout, `rollout_worker` obtains routed-experts object refs from the LMDeploy shared\nstore. At that point, ownership moves from LMDeploy to XTuner. A review finding should call out both\nsides of the ownership boundary when relevant:\n\n- Leak before transfer: requests whose routed experts remain in LMDeploy because XTuner never\n obtains the object refs.\n- Leak after transfer: object refs that XTuner keeps alive too long through `RolloutState`, replay\n buffer, trainer batches, metrics, fake tests, or background tasks.\n",".claude/skills/add-chat-template/SKILL.md":"---\nname: add-chat-template\ndescription: Add or audit an XTuner chat-template implementation and its loss mask from an official Hugging Face model repository or local model directory. Use when supporting a new model under xtuner/v1/data_proto/messages, checking an existing implementation against tokenizer or processor apply_chat_template, deciding which assistant output and stop/EOS tokens receive loss, adding regression tests, or validating that exported training behavior matches vLLM and SGLang inference.\n---\n\n# Add Chat Template\n\n## Goal\n\nTreat the official Hugging Face tokenizer, or processor for a multimodal model,\nas the rendering oracle. Produce the same conversation token sequence in\nXTuner, supervise model-generated output by default, and supervise the\ncontext-appropriate token that ends each assistant generation. Prove the\nbehavior through XTuner's public tokenize API and, when available, the target\nvLLM and SGLang versions.\n\nKeep the implementation small and model-local. Do not infer a template or stop\ncontract from another model in the family.\n\n## 1. Establish the reference\n\n1. Record the official HF repo or local directory, immutable revision, tokenizer\n class, Transformers version, and whether `trust_remote_code=True` is required.\n2. Inspect the actual artifacts used by inference:\n `tokenizer_config.json`, `chat_template.jinja` or `chat_template`,\n `special_tokens_map.json`, `generation_config.json`, and `config.json`.\n Inspect `processor_config.json` and remote processor/template code when the\n official multimodal path uses `AutoProcessor.apply_chat_template`.\n3. Run the bundled inventory against the same revision:\n\n ```bash\n python .claude/skills/add-chat-template/scripts/audit_hf_chat_template.py \\\n --trust-remote-code\n ```\n\n Omit `--trust-remote-code` unless the official model requires it. For\n model-specific branches such as tools, thinking, or multimodal inputs, pass a\n JSON case file with `--cases `; see the script's `--help` output.\n The script inventories the tokenizer side; for a processor-owned multimodal\n template, run the corresponding `AutoProcessor` renderer separately.\n4. Render every supported branch with the same `tokenizer.apply_chat_template`\n or `processor.apply_chat_template` entry point used by the official example.\n For multimodal models, do not assume tokenizer-only rendering reproduces\n media placeholders or processor expansion. Record all required template\n kwargs and their defaults, including\n `add_generation_prompt`, `continue_final_message`, tool definitions,\n thinking/reasoning flags, named templates, and multimodal options. Do not\n copy rendered strings from a model card when the executable tokenizer is\n available.\n5. When the selected official template contains Jinja `{% generation %}`\n regions, call its `apply_chat_template` with `tokenize=True`,\n `return_dict=True`, and\n `return_assistant_tokens_mask=True`. Use its assistant mask as an additional\n oracle, then separately audit stop boundaries: a next-role stop token can sit\n outside the official generation region while still belonging to the previous\n assistant for SFT.\n6. Pin the reference revision in tests or CI configuration. A moving HF branch\n is not a stable oracle.\n\n## 2. Audit before implementing\n\nSearch these integration points first:\n\n- `xtuner/v1/data_proto/templates/__init__.py`\n- `xtuner/v1/data_proto/messages/`\n- `xtuner/v1/data_proto/messages/__init__.py`\n- `xtuner/v1/datasets/sft_tokenize_fn/openai.py`\n- multimodal tokenize functions when applicable\n- `tests/chat_template/` and `tests/datasets/`\n\nIf the model already exists, run its public tokenize path before changing it:\n\n```python\ntokenize_fn = OpenaiTokenizeFunctionConfig(chat_template=\"\").build(tokenizer)\nresult = tokenize_fn({\"messages\": messages, \"tools\": tools})\n```\n\nCompare the result with the official renderer and inspect existing tests. Find a\nconcrete failing message sequence before fixing a mismatch. Do not rewrite a\nworking implementation because its structure looks unusual.\n\nUse `HybridChatTemplate` plus `ChatMessages` only when fixed role wrappers fully\nexpress the official template. Add a dedicated\n`xtuner/v1/data_proto/messages/_chat.py` when rendering depends on\nmessage history, tools, reasoning, content types, or the next role. Follow\n`glm52_chat.py` and `qwen35_chat.py` as integration examples, not as rendering\nspecifications for another model.\n\n## 3. Write the stop contract first\n\nInventory three distinct sources; do not collapse them into one `eos_token`:\n\n1. **Configured generation stops**: `tokenizer.eos_token_id`, `config.json`,\n `generation_config.json`, including integer lists and stop strings.\n2. **Template boundaries**: the exact token or token sequence following an\n assistant when the next item is user, tool/observation, system/developer, or\n the end of the sample.\n3. **Engine stops**: the IDs and strings actually installed by the target vLLM\n and SGLang versions after loading the exported checkpoint.\n\nCreate an explicit table before writing the mask:\n\n| Transition | Official serialized boundary | Token IDs | Engine mechanism | Loss when assistant loss is true |\n|---|---|---|---|---|\n| assistant → user | exact role/EOT sequence | exact IDs | EOS ID, stop ID/string, or parser | yes for the generated stop target |\n| assistant tool call → tool | exact observation boundary | exact IDs | EOS ID, stop ID/string, or parser | yes for the generated stop target |\n| final assistant → end | exact EOT/EOS, or documented training-only EOS | exact IDs | EOS ID or stop ID/string | yes |\n\nVerify each token by both encoding the exact boundary and converting its ID back\nto a token. Reject unknown-token conversions. Handle a multi-token stop as a\nsequence rather than assuming every stop is one special token.\n\n### Role tokens can belong to the preceding assistant\n\nSome official templates have no dedicated per-turn EOT. The next role token is\nthen the token the model must generate to stop the previous assistant. In that\ncase, keep the official serialized order and assign loss only on that boundary\ntoken or sequence to the preceding assistant:\n\n```text\nassistant answer user text\n ^ assistant loss ^ masked\n```\n\nFor a GLM-like protocol this can mean supervising the user boundary after a\nnormal answer, the observation boundary after a tool call, and the configured\nterminal EOS only when no following role supplies a boundary.\n\nDo not append a generic EOS after every assistant when the official inference\ntemplate places a role/observation boundary there. That creates training history\nthat vLLM and SGLang do not render. Conversely, when the official full-message\nrenderer omits the final token that the model must generate to terminate,\nappend exactly that one terminal token for SFT and document this single\ntraining-only suffix.\n\nConfigured EOS IDs are alternatives, not a sequence to append together.\nSupervise the actual context-specific stop token present in each training path.\n\n## 4. Implement rendering and loss together\n\nPrefer one cohesive renderer that returns rendered text plus a character- or\ntoken-level loss mask. Keep context-dependent boundary ownership beside the\nrendering branch that emits the boundary; avoid a collection of shallow helper\nfunctions.\n\nApply these defaults:\n\n- Treat `assistant` messages with no `loss` field as `loss=True`.\n- Mask system/developer/user/tool inputs and role scaffolding unless the stop\n contract proves that a boundary is generated by the preceding assistant.\n- Supervise all model-generated assistant components: visible content,\n reasoning that remains in the rendered sample, reasoning closing syntax,\n tool calls, and the appropriate stop target.\n- Treat `loss=False` as masking the entire assistant output and its stop target.\n- Mask `add_generation_prompt` completely; it is prompt scaffolding, not a\n demonstrated model output.\n- Preserve explicit model behavior such as clearing historical reasoning only\n when the official template does so for the chosen kwargs.\n\nIf character offsets are used, verify special-token and Unicode boundaries with\nthe real tokenizer. A token overlapping a loss span needs a deliberate,\ntested rule. Keep a slow, independent token-level oracle for regression tests;\ndo not call the production renderer from both sides of the same parity test.\n\nMaintain exact rendering parity before any documented training suffix:\n\n```mermaid\nflowchart LR\n A[\"Official HF messages\"] --> B[\"tokenizer.apply_chat_template\"]\n A --> C[\"XTuner public tokenize API\"]\n B --> D[\"Exact text and token IDs\"]\n C --> D\n D --> E[\"Optional one terminal SFT stop\"]\n E --> F[\"Loss-mask contract\"]\n```\n\nDo not use decoded-text equality as the sole check: decoding can hide a token-ID\nmismatch.\n\n## 5. Register the smallest integration\n\nFor a dedicated message implementation, update only the required seams:\n\n1. Export the message class from `xtuner/v1/data_proto/messages/__init__.py`.\n2. Add the public name and serving stop metadata to `CHAT_TEMPLATE_MAP`.\n3. Dispatch that name in `OpenaiTokenizeFunction`; update multimodal dispatch\n only if the model uses it.\n4. Update typed CLI/config literals only where they restrict the new name.\n\nKeep `CHAT_TEMPLATE_MAP.stop_words` consistent with the discovered contract,\nbut do not treat it as proof of engine behavior. vLLM and SGLang load their\nruntime stops from the exported HF artifacts and request/engine configuration.\n\n## 6. Add good regression tests\n\nUse the real official tokenizer/processor and XTuner's public tokenize\nfunction. Mock only unavailable external services. Cover every supported\ntemplate branch with the smallest useful matrix:\n\n- user → assistant, with `loss` omitted;\n- assistant → user in a multi-turn conversation;\n- assistant tool call → tool/observation → assistant;\n- final assistant;\n- `loss=False` followed by another role and at sample end;\n- system/developer input;\n- `add_generation_prompt=True`;\n- reasoning enabled/disabled and historical reasoning behavior;\n- tools and multimodal content when supported;\n- repeated assistant text, Unicode, and adjacent special tokens to expose\n faulty substring or offset alignment.\n\nAssert public behavior:\n\n1. Official tokenizer/processor rendered text equals XTuner rendered text,\n except for the one explicitly documented final SFT suffix if required.\n2. Official token IDs equal XTuner `input_ids` before that suffix.\n3. `len(input_ids) == len(labels)` and every label is either the matching input\n ID or `IGNORE_INDEX`.\n4. Assistant output receives loss by default; `loss=False` removes all of it.\n5. Each transition's exact stop token IDs receive or do not receive loss as the\n stop table specifies.\n6. User/tool content and generation prompts remain masked.\n7. The fast implementation matches an independent slow oracle.\n\nTest token positions and IDs directly. A substring assertion can supplement but\nmust not replace the loss assertion. Avoid snapshots that pass after both the\nrenderer and expected string are changed to the same incorrect value.\n\n## 7. Validate vLLM and SGLang\n\nResolve exact target versions. Prefer the user's production versions; otherwise\ninspect the current installed versions and latest stable official source. For\neach engine:\n\n1. Load the same exported tokenizer, `config.json`, and\n `generation_config.json`.\n2. Compare the engine's chat-render/tokenize output with the official\n tokenizer/processor renderer for every stop-table transition.\n3. Inspect or log the final stop token IDs/strings installed in the generation\n request. Distinguish tokenizer EOS, model generation-config EOS, request stop\n strings, and model-specific parsers.\n4. Run generation smoke tests for a normal final answer and a tool call. Use a\n sufficiently high output limit and assert the engine reports a stop rather\n than a length limit, with no role leakage.\n\nObtain the repository GPU lock before a local GPU server or generation test. If\nthe exact runtime cannot be executed, audit the exact tagged engine source and\nreport the result as a static source audit, not runtime compatibility.\n\nSwitching from Chat Completions to Responses does not remove this requirement:\nfor ordinary HF models both interfaces ultimately render model messages through\nthe chat template.\n\n## 8. Completion checklist\n\nFinish only when the report includes:\n\n- official HF repo, revision, and Transformers version;\n- implementation path and whether it is generic or dedicated;\n- rendering-parity matrix;\n- stop-contract table with token strings, IDs, and loss ownership;\n- tests proving default assistant loss and `loss=False`;\n- vLLM/SGLang versions and runtime or exact-tag source validation;\n- every intentional difference from the official renderer, normally at most a\n final training stop suffix.\n\nIf auditing an existing implementation, state which requirements already had\ntests, which were missing, and the concrete mismatches found. Do not claim the\nimplementation is correct merely because a test file exists.\n",".claude/skills/add-chat-template/agents/openai.yaml":"interface:\n display_name: \"Add Chat Template\"\n short_description: \"Add and verify XTuner chat templates\"\n default_prompt: \"Use $add-chat-template to add or audit XTuner chat-template rendering, loss masks, and stop-token behavior for this official Hugging Face model.\"\n",".claude/skills/add_hf_model/SKILL.md":"---\nname: add_hf_model\ndescription: >\n Add support for a HuggingFace model in XTuner so it can be trained with the\n full set of parallelism strategies and optimization switches. The user can\n provide a HuggingFace Hub repo id, a local model directory, or a model family\n already supported by `transformers`. This skill walks through the full path:\n locating the reference implementation, classifying the model, splitting new\n code across the model / module / ops layers, implementing the XTuner model\n class and config, registering the entry point, validating bitwise numerical\n parity with HuggingFace, writing regression tests, and enabling the training\n optimizations (EP/SP, micro-batch, torch.compile, fp8, activation offload).\n---\n\n# Add a HuggingFace model to XTuner\n\nA complete handbook for porting any HuggingFace causal LM into XTuner with\ncorrect training behavior. Covers four buckets: **Dense LLM**, **MoE LLM**,\n**VLM / compose model**, and **`trust_remote_code` model**. Every step below is\ngrounded in concrete `file:line` references in this repository — read the cited\ncode before writing yours.\n\n> **Anti-spaghetti reminder** — XTuner separates *what the model is* (`*Config`),\n> *what it computes* (`Dense` / `MoE` / `BaseComposeModel` subclass), and *how its\n> weights map to HF safetensors* (`to_hf_key_list`, optional `safetensors_to_params`\n> / `param_to_safetensor`). Keep new code inside those three seams. **Never** add\n> `if model_name == \"...\"` branches to the base classes.\n\n---\n\n## 0. Locate the source code\n\nThe only goal of this step is to find the **HuggingFace reference implementation\nyou will port from**, and to note the `model_type` you will register under. Do\nnot pick attention/router/layer here — that happens naturally in §1 as you read\nthe code.\n\n1. If the user gives a **remote-code** model, the modeling code ships in the repo\n as `modeling_*.py` — read it directly.\n2. If the user gives **weights only**, confirm the target Python environment with\n the user, then find the matching implementation inside that env's\n `transformers` install. If no code exists for that `model_type`, stop and ask\n the user.\n3. Confirm the weights are **training-suitable** — at least bf16 — so training\n precision can be aligned later. Flag fp8/int4-only checkpoints to the user.\n4. Note two strings from the HF config — you will need both downstream:\n ```python\n from transformers import AutoConfig\n cfg = AutoConfig.from_pretrained(path, trust_remote_code=True)\n cfg.model_type # registration key (§4)\n cfg.architectures[0] # the modeling class you are mirroring\n ```\n\n---\n\n## 1. How to do the migration\n\nXTuner splits a model into three layers. You port by reusing what already exists\nat each layer and only adding what is genuinely new — so the work is \"read the HF\nmodeling code, then match each of its pieces to the XTuner component below\". The\nlocations are listed plainly; you do not need extra guidance to map your model's\nattention / router / layer onto the matching one.\n\n- **model layer** (`xtuner/v1/model`) — the model body.\n - Dense body: `xtuner/v1/model/dense/dense.py`\n - MoE body: `xtuner/v1/model/moe/moe.py`\n - Multi-modal (compose) bodies: `xtuner/v1/model/compose/` — reuse the language\n model and add only the ViT / projector pieces.\n- **module layer** (`xtuner/v1/module`):\n - Decoder layers: `DenseDecoderLayer`, `MoEDecoderLayer` —\n `xtuner/v1/module/decoder_layer`. Small changes add a switch on the existing\n layer; a genuinely novel architecture may add `{model_name}_decoder_layer`.\n - Attention configs: `MHAConfig` (`xtuner/v1/module/attention/mha.py`),\n `MLAConfig` (`.../mla.py`, DeepSeek-style latent),\n `GatedDeltaNetConfig` (`.../gated_deltanet.py`, linear).\n - Router configs (MoE): `GreedyRouterConfig`\n (`xtuner/v1/module/router/greedy.py`, softmax top-k — Qwen3 MoE, GptOss),\n `NoAuxRouterConfig` (`xtuner/v1/module/router/noaux_router.py`, grouped sigmoid\n aux-loss-free — DeepSeek V3).\n - Also: rope, rms_norm, etc.\n- **ops layer** (`xtuner/v1/ops`) — kernels such as attention and rms_norm.\n\n### Existing models to copy from\n\nPick the one whose attention + (router) match yours; the closer it is, the\nsmaller your diff.\n\n| Your model is… | Copy from |\n|--------------------------------------|--------------------------------------|\n| Dense LLM | `xtuner/v1/model/dense/qwen3.py` |\n| MoE LLM | `xtuner/v1/model/moe/qwen3.py` |\n| MoE with MLA + aux-loss-free router | `xtuner/v1/model/moe/deepseek_v3.py` |\n| MoE needing weight-layout transform | `xtuner/v1/model/moe/gpt_oss.py` |\n| Compose / VLM | `xtuner/v1/model/compose/qwen3_vl/` |\n\n\nAfter you have a plan, **write an integration design doc into `docs/design/model`**\nand confirm with the user that the decomposition is reasonable before\nimplementing.\n\n### File layout for the new model\n\n| Bucket | Model file | Public re-export | Test file |\n|----------------------|-------------------------------------------------------------|-----------------------------------------------|--------------------------------------------|\n| Dense LLM | `xtuner/v1/model/dense/.py` | `xtuner/v1/model/__init__.py` | `tests/model/test__dense.py` |\n| MoE LLM | `xtuner/v1/model/moe/.py` | `xtuner/v1/model/__init__.py` | `tests/model/test__moe.py` |\n| Compose / VLM | `xtuner/v1/model/compose//...` | `xtuner/v1/model/__init__.py` | `tests/model/test_.py` |\n\n---\n\n## 2. Implement the model class\n\nSubclass `Dense`, `MoE`, or `BaseComposeModel`. You typically only need to\noverride one method.\n\n### 2.1 `to_hf_key_list(self, key: str) -> list[str]` — **mandatory, abstract on `BaseModel`**\n\nDeclared at `xtuner/v1/model/base.py`. Translates an **XTuner-side** parameter\nname into one or more **HF-side** safetensors keys. Return-list cardinality:\n\n- **1 → 1** for plain weights (most params).\n- **1 → N** for fused MoE experts (one fused param explodes into N per-expert\n HF keys; see `xtuner/v1/model/moe/qwen3.py`).\n\nPatterns you almost always need (study `xtuner/v1/model/dense/qwen3.py`\nand `xtuner/v1/model/moe/qwen3.py`):\n\n```python\n# 1. Tied embeddings: redirect lm_head → embed_tokens before adding the model. prefix.\nif self.config.tie_word_embeddings and \"lm_head\" in key:\n key = key.replace(\"lm_head\", \"embed_tokens\")\n\n# 2. Add the \"model.\" prefix that HF wraps everything in.\nif \"layers\" in key or \"embed_tokens\" in key:\n key = \"model.\" + key\n\n# 3. HF MoE nests experts/gate under .mlp.; XTuner does not.\nif \"layers\" in key:\n key = re.sub(r\"layers\\.(\\d+)\\.(experts|gate)\", r\"layers.\\1.mlp.\\2\", key)\n\n# 4. Top-level norm.\nif key.startswith(\"norm.\"):\n return [key.replace(\"norm.\", \"model.norm.\")]\n\n# 5. MoE expert fusion: one fused param → N HF keys.\nif \"fused_w1w3.weight\" in key:\n out = []\n for i in range(self.config.n_routed_experts):\n out.append(key.replace(\"fused_w1w3.weight\", f\"{i}.gate_proj.weight\"))\n out.append(key.replace(\"fused_w1w3.weight\", f\"{i}.up_proj.weight\"))\n return out\n\n# 6. Model-specific buffers (e.g. FoPE: rotary_emb.sin_coef / cos_coef are persistent buffers).\nif key.startswith(\"rotary_emb.\"):\n return [key.replace(\"rotary_emb.\", \"model.rotary_emb.\")]\n```\n\nA prefix-region remap that differs by *deployment* (e.g. a text tower nested under a\nVLM as `model.language_model.`) does **not** belong here — put it in the config's\n`hf_key_mapping`. See §5.1.\n\n### 2.2 `safetensors_to_params` / `param_to_safetensor` — **optional, only for layout mismatches**\n\nIf the HF storage layout cannot be expressed as “same tensor, different name”,\nyou must transform the bytes. Canonical example: GptOss stores expert weights as\n`(num_experts, hidden_size, expert_dim * 2)` but XTuner fuses them as\n`(num_experts * 2 * expert_dim, hidden_size)` — see overrides at\n`xtuner/v1/model/moe/gpt_oss.py`.\n\nUse this hook only when transposing/reshaping is unavoidable. Renames go in\n`to_hf_key_list`; numerics go here.\n\n### 2.3 Do **not** override\n\n- `build_layers`, `build_embeddings`, `_init_weights` — the base handles them\n via the config. Override only if the new model adds a layer type the base\n cannot express (rare).\n- `forward` — same reasoning. If you find yourself touching it, re-evaluate the\n bucket choice; the problem is probably in the config (`attention`, `router`,\n `first_k_dense_replace`).\n\n---\n\n## 3. Implement the config\n\nTwo layers: a **base config** that reads HF and a **size-specific subclass** per\nreleased checkpoint.\n\n### 3.1 Base config — three required members\n\nMirror `Qwen3MoEConfig` (`xtuner/v1/model/moe/qwen3.py`) or\n`Qwen3DenseConfig` (`xtuner/v1/model/dense/qwen3.py`).\n\n1. **`build(self) -> `** — return `(self)`.\n\n2. **`@classmethod from_hf(cls, hf_path) -> Self`** — read the HF config and map\n its fields one-to-one onto your config. The field set is whatever the live\n config class declares (`TransformerConfig` in `xtuner/v1/model/base.py`,\n `MoEConfig` in `xtuner/v1/model/moe/moe.py`); read it, don't work from a\n hard-coded list. A few non-obvious traps:\n - Use `RopeParametersConfig.from_hf_config(hf_config)` from\n `xtuner/v1/module/rope/rope.py`. It already handles both the\n `rope_parameters` dict (HF ≥ 5.2.0) and the legacy `rope_scaling` dict\n (HF 4.57.x), plus YARN / FoPE / mrope special fields. **Do not** parse\n rope yourself.\n - For optional HF fields, use `getattr(hf_config, \"\", )`\n rather than `hf_config.` — older checkpoints drop fields.\n\n3. **`@property hf_config(self) -> | None`** — the inverse of\n `from_hf`: every field `from_hf` reads must be re-emitted so `save_hf`\n round-trips.\n\n**Built-in vs. `trust_remote_code` — the only axis that changes `from_hf` /\n`hf_config`:** a `trust_remote_code` model is structurally still Dense / MoE /\nVLM; the *only* difference is whether `transformers` ships a built-in\n`Config` class. Decide which case you are in and follow the matching column\n— this is the single place that contract is defined.\n\n| | Built-in config (e.g. Qwen3) | `trust_remote_code` (no built-in config) |\n|---|---|---|\n| `from_hf` reads via | `.from_pretrained(hf_path)` | `AutoConfig.from_pretrained(hf_path, trust_remote_code=True)`, then `assert hf_config.model_type == \"\"` |\n| `hf_config` returns | a populated `` re-emitting every field | `None` |\n| `save_hf` weight-map | written from `hf_config` | same |\n| `save_hf` side files | derived from `hf_config` | falls back to copying `config.json` / tokenizer / `*.py` from `self._hf_path` so the dir stays loadable with `trust_remote_code=True`; module-cache test in the matrix below |\n\nCompare `Qwen3MoEConfig.hf_config` (built-in) and `Qwen3MoEFoPEConfig.hf_config`\n(returns `None`), both in `xtuner/v1/model/moe/qwen3.py`. **Do not** invent a\nbuilt-in HF config class for a remote-code model — return `None`.\n\n**Invariants:**\n\n- `from_hf` ↔ `hf_config` must round-trip. The `test_save_hf` test (§7) is the\n enforcement mechanism — every key in the original index must appear in the\n saved index with byte-equal tensors.\n- The config must be **pickleable and side-effect free**. No env vars, no file\n I/O, no `torch.cuda.*` at construction time.\n\n### 3.2 Size-specific subclasses\n\nAdd one subclass per released checkpoint that hard-codes the published\ndimensions. Mirror `Qwen3MoE30BA3Config` / `Qwen3MoE235BA22Config` in\n`xtuner/v1/model/moe/qwen3.py`. Even if only one size ships, define at least one\n— it is the entry point for `get_model_config` and for tests.\n\n---\n\n## 4. Register the entry point\n\nOpen `xtuner/v1/model/__init__.py` — all four edits are short:\n\n1. **Import** — bring in the new config classes.\n2. **Alias** — add `\"\": ()` to `model_mapping`. Lookup\n normalizes case and `-`/`_`, so `\"my-model-7b\"` and `\"my_model_7b\"` are the\n same key.\n3. **Dispatch** — add `elif cfg.model_type == \"\":` to\n `get_model_config_from_hf` returning `.from_hf(model_path)`.\n The catch-all `raise ValueError(...)` must stay last.\n4. **`__all__`** — re-export the new public config names.\n\n**Never** key dispatch on parameter names, file paths, or architecture strings.\nOnly `model_type`.\n\n---\n\n## 5. Compose / VLM specifics\n\nCompose models live under `xtuner/v1/model/compose//` and follow a\nthree-tower pattern. See `xtuner/v1/model/compose/base.py` and\n`xtuner/v1/model/compose/qwen3_vl/qwen3_vl_config.py`.\n\n### 5.1 Config shape\n\n`BaseComposeConfig` (`compose/base.py`) carries three sub-configs:\n\n```python\nvision_config: XTunerBaseModelConfig # e.g. Qwen3VLVisionConfig\nprojector_config: XTunerBaseModelConfig # e.g. Qwen3VLProjectorConfig\ntext_config: XTunerBaseModelConfig # any Dense/MoEConfig subclass\nfreeze_vision: bool = False\nfreeze_projector: bool = False\nfreeze_language: bool = False\n```\n\nReuse existing text-tower configs where possible. `Qwen3VLDense*` delegates to\nthe Dense path via `xtuner/v1/model/dense/qwen3vl_text.py`. **Do not** copy-paste\nthe entire dense/MoE config — compose over them.\n\n**Reused text-tower key prefixes — use `hf_key_mapping`, not `to_hf_key_list`.** A\ntext tower reused inside a VLM keeps its *standalone* `to_hf_key_list` (it emits\n`model.<...>` as if it were a plain LLM). The compose checkpoint nests it one level\ndeeper (`model.language_model.<...>`). Do **not** teach the text tower's\n`to_hf_key_list` about that prefix — that leaks the compose context into a tower that\nmust also work standalone. Set the remap on the **text config** instead:\n`hf_key_mapping = {r\"^model\\.\": \"model.language_model.\"}` (applied in\n`_init_load_spec`). Keep *structural* renames in `to_hf_key_list`; the\ndeployment-dependent prefix region belongs in `hf_key_mapping`.\n\n### 5.2 Model shape\n\n`BaseComposeModel` (`compose/base.py`) builds three sub-models:\n`vision_tower`, `multi_modal_projector`, `language_model`.\n\n- **`from_hf(hf_path)`** — delegates to each sub-model with\n `strict=False` (vision keys are missing from the language-model state dict\n and vice versa) and unions the missing-keys sets.\n- **`save_hf(hf_dir)`** — saves each tower with a distinct prefix\n (`\"model-language\"`, `\"model-vision\"`, `\"model-projector\"`) and merges the\n three `weight_map`s into one `model.safetensors.index.json`.\n\n### 5.3 `hf_config` for VLMs\n\nMany VLMs ship without a stable HF top-level config class; `hf_config` returns\n`None` (`compose/qwen3_vl/qwen3_vl_config.py`). In that case, `save_hf`\nfalls back to copying source files from `self._hf_path` (see §3.1).\n\n### 5.4 Vision-side quirks to watch for\n\n- `attn_impl` selectable per checkpoint: `\"flash_attention\" | \"flex_attention\" |\n \"eager_attention\"`.\n- `deepstack_visual_indexes` (e.g. `[8, 16, 24]`) — auxiliary supervision depths\n on the vision tower; must round-trip.\n- Specialized rope: `rope_type=\"qwen3_vl\"`, `mrope_section=[24, 20, 20]` —\n delegated to `RopeParametersConfig.from_hf_config`.\n- The projector’s `torch.compile` path is enabled on torch ≥ 2.9.1 (see commit\n `f6d74efb`). Tests must validate both compiled and eager outputs match.\n\n---\n\n## 6. Baseline parity — no complex parallelism yet\n\nFirst make the model correct on the **simplest execution path**: single rank, no\nEP/SP, no `torch.compile`, no fp8, no activation offload. Align precision from\nthe inside out — ops → decoder layer → full model — reusing XTuner's own modules\nat every level. Only once this baseline holds do you add the parallel and\noptimization features (§8).\n\n1. **Reuse, don't reimplement.** Use the existing XTuner modules — MHA, MLA,\n GatedDeltaNet, GroupLinear, etc. Do not re-implement them with naive torch.\n\n2. **Route op differences through `XTUNER_HF_IMPL`.** This env var already exists\n (`xtuner/v1/utils/misc.py`) and selects HF-exact ops via per-op selectors:\n `xtuner/v1/ops/attn_imp.py::get_attn_impl_fn` (eager attention),\n `xtuner/v1/ops/rms_norm/__init__.py::get_rms_norm_fn` (native torch rms_norm),\n and `xtuner/v1/ops/gated_deltanet/__init__.py::{get_chunk_gated_delta_rule_fn,\n get_causal_conv1d_fn}` (return the canonical fla / causal_conv1d wrappers HF\n calls instead of XTuner's compile-friendly `torch.library.custom_op` wraps —\n same forward output, but the backward op graph now matches HF). Selectors\n read the env var **live** so a test can toggle it per model instance. If your\n model introduces a new op whose fast path is not bitwise against HF (typical\n for fla-backed linear-attention variants), add its HF-exact branch in the\n corresponding op selector the same way. **You may only patch at the ops\n level** — never take the shortcut of patching the entire\n `XTunerDecoderLayer.forward`.\n\n Pair `XTUNER_HF_IMPL` with `XTUNER_DETERMINISTIC=true`. The latter gates an\n autotune-pin block in `xtuner/v1/__init__.py` that monkey-patches\n `triton.autotune` to lock every kernel to its first config and disable the\n result cache. Without the pin, fla's `@triton.autotune` picks tiling /\n reduction order per kernel and the choice drifts between runs, producing\n 1 ULP per linear-attention layer that the chain rule amplifies to ~1.0 at\n the model boundary (§12). `tests/conftest.py` sets the env var and runs\n `import xtuner.v1` so the patch installs before fla is imported. The patch\n lives on the `xtuner.v1.*` import chain (not in the conftest) because\n `multiprocessing.spawn` children of `MultiProcessTestCase` don't go through\n pytest's conftest — they re-import the test class top-level (which pulls\n `xtuner.v1.model.*`) and the patch must run in those processes too.\n Production training doesn't set the env var, so the block is inert there.\n\n3. **Decoder-layer bitwise parity — forward AND backward, both required.** Build\n one XTuner `DecoderLayer` (one per distinct layer type — e.g. a linear and a\n full layer for a hybrid model) and the matching HF layer, set `XTUNER_HF_IMPL`\n and `XTUNER_DETERMINISTIC`, feed identical inputs, and align **bitwise** — not a\n tolerance. **Backward is not optional**: a `single layer → final norm → lm_head\n → CE loss → loss.backward()` test asserts the **layer output, the loss, and the\n input gradient `dL/dx`** all bitwise-equal to HF — test one full and one linear\n layer separately (reference: `test_decoder_layer_bitwise_parity`). A bitwise\n *forward* does not imply a bitwise backward — the forward values can round\n identically while the backward graph differs (see the fp32-softmax note in §12).\n This is the **primary** path, required at all scales, and the only feasible one\n for large models (it never loads the full model — see \"load only what you use\"\n below). When you hand-build a single layer's\n inputs, the two sides' `causal mask` / `seq_ctx` must match exactly — see §12.\n **Load only what you use** so the test fits any model size: build the XTuner\n tower on `meta` and materialize (`to_empty` + selective load) *only* the tested\n layer + `norm` + `lm_head` via `HFCheckpointLoader.load(key)` (it reads just the\n needed safetensors shard); build a *standalone* HF `DecoderLayer` + norm +\n lm_head and load the same keys. Never `from_pretrained` the whole model for a\n single-layer test. (`build_rotary_embedding` builds the rotary on CPU with real\n buffers even under meta, so `model.rotary_emb` is usable without materializing\n the layers.) For < 40B you may also read per-layer hidden states from one\n full-model forward (HF `output_hidden_states` + XTuner `return_hidden_states`);\n mind the off-by-one in §12.\n\n4. **Model config matches HF hyperparameters** — see the §3.1 built-in vs.\n remote-code contract for how `from_hf` / `hf_config` differ between the two.\n\n5. **Full-model forward parity for models < 40B**, bitwise against HF, on the\n single-rank path (FSDP on or off; no EP / compile / fp8 yet).\n\n6. **`save_hf` / `from_hf` correctness** (round-trip; see §7).\n\n7. **Loss-convergence trace + engine test.** Once model forward and decoder-layer\n forward/backward are aligned, record a loss-convergence trajectory and add an\n engine test case.\n\n**Commit the baseline — do not stop to ask.** Committing is part of this\nworkflow: invoking this skill authorizes the in-workflow commits. Once parity\nholds on the simple path and lint/tests pass, commit immediately so this correct\nbaseline is preserved before any parallel/optimization work begins (the §8\nsub-agents branch off this commit). Commit each logical step as you finish it\n(§10) rather than batching everything to the end or waiting for a separate\ngo-ahead.\n\nReport back to the user: whether bitwise parity was achieved, and the residual\nerror (decoder-layer level and model level) once XTuner's internal components are\nused.\n\n---\n\n## 7. Baseline tests — add these first\n\nBefore any parallel feature, add the most basic regression tests so the baseline\n(§6) is locked and the §8 work has a fixed reference to measure against. Add\n`tests/model/test__.py`, mirroring\n`tests/model/test_qwen3_moe.py` (MoE) or `tests/model/test_qwen3_dense.py`\n(Dense). All tests extend `xtuner._testing.DeterministicDDPTestCase` for\ndeterministic distributed setup.\n\nReference test cases:\n- decoder-layer bitwise parity + save_hf round-trip: `tests/model/test_qwen3_5_dense.py`\n (hybrid linear/full dense VLM; toggles `XTUNER_HF_IMPL` per case)\n- model forward & save & load parity: `tests/model/test_qwen3_moe.py`\n- engine training test: `tests/engine/test_moe_train_engine.py`\n\n### 7.1 Baseline test matrix\n\nAll single execution path — no EP / dispatcher / compile / fp8 here; those are\n§8 tests, written against this baseline.\n\n| Case | Dense | MoE | Compose | Notes |\n|-------------------------------------------------|:-----:|:---:|:-------:|------------------------------------------------------------------------------------|\n| Decoder-layer parity — output + loss + `dL/dx` | ✅ | ✅ | ✅ | **Required at all scales, bitwise.** Test one full and one linear layer *separately*. Standalone & memory-light: build the XTuner tower on `meta` and `materialize_xtuner_submodule` only {tested layer, `norm`, `lm_head`}; build a standalone HF `DecoderLayer`; load via `HFCheckpointLoader`. Under `XTUNER_HF_IMPL` assert the **layer output, loss, and `dL/dx` all bitwise** (forward AND backward at the layer level — a bitwise forward ≠ bitwise backward, see §12). Reference: `test_decoder_layer_bitwise_parity`. |\n| Whole-model forward + backward parity (by scale) | ✅(4B)| scale | ✅(4B) | Run the **top-level model** end-to-end vs HF, eager both sides (`XTUNER_HF_IMPL` + `XTUNER_DETERMINISTIC` + HF `eager`). For a **VLM** that's the compose model on an **image** prompt (vision + projector + text — subsumes \"VLM forward parity\"); for a plain LLM, the text/LM model. With both env switches on plus the autotune pin loaded (§6.2), **forward AND backward are bitwise**: assert logits + loss + `dL/d(pixel_values)` / `dL/d(inputs_embeds)` all `== 0.0`. **Scale**: small = fwd+bwd; medium = forward only; large (can't e2e-forward) = skip → integration. Reference: `test_vl_forward_parity`. |\n| Vision-tower bitwise parity | — | — | ✅ | **Compose/VLM only.** Compare HF's `visual` output (`pooler_output`, which is **post-merger**) bitwise (0.0), both eager. XTuner splits that boundary into `vision_tower` (patches → merged hidden) + `projector` (the merger MLP → text dim), so load **both** standalone to match it — don't build the whole VLM just for the vision path. Build them on the **real device, not `meta`**: the vision rotary is computed into a buffer at forward time, so `to_empty` would leave garbage (unlike the text rotary, which `build_rotary_embedding` builds real even under meta — §6.3). Vision attention is *non-causal* → `eager_attention(causal=False)`, which `XTUNER_HF_IMPL` selects. Reference: `test_vision_tower_bitwise_parity`. |\n| FSDP forward parity | ✅ | ✅ | ✅ | Looser tolerance (≈ 3e-2 is the established budget). |\n| `save_hf` round-trip (byte-equal tensors) | ✅ | ✅ | ✅ | See `tests/model/test_qwen3_moe.py`. |\n\n### 7.2 Parity bar by model scale\n\nTwo independent axes:\n\n**Per-layer (op) parity — required at _all_ scales, bitwise.** Test one full and\none linear decoder layer separately; for each assert layer **output + loss + `dL/dx`**\nbitwise vs HF under `XTUNER_HF_IMPL`. Keep it memory-light (XTuner `meta`-build +\nmaterialize only the tested layer + `norm` + `lm_head`; standalone HF layer; load\nvia `HFCheckpointLoader`) so it runs even for models too large to forward\nend-to-end. This is the bitwise guarantee for both forward and backward ops.\n\n**Whole-model (integration) parity — on the _top-level_ model, graded by what fits\none GPU.** Run the real model end-to-end: for a VLM that's the **compose model on an\nimage prompt** (one test covers vision + projector + text — don't add a separate\ntext-tower model test); for a plain LLM, the text/LM model.\n- **Small** (e.g. dense 4B): both forward (logits/loss) AND backward\n (`dL/d(pixel_values)` for a VLM, `dL/d(inputs_embeds)` for text) **bitwise**.\n Prerequisites: `XTUNER_HF_IMPL=true` so every op selector returns the\n HF-canonical callable (§6.2) AND `XTUNER_DETERMINISTIC=true` so Triton\n autotune is pinned (§6.2). With both on, even hybrid linear/full models match\n HF byte-for-byte through the whole compose chain. Don't check weight\n gradients — they accumulate across positions + tied `lm_head` and aren't the\n right granularity for parity.\n- **Medium**: forward only (logits/loss bitwise).\n- **Large** (cannot forward end-to-end on the test GPUs): skip the whole-model\n test; leave it to integration tests. The per-layer test still gives op-level\n forward+backward bitwise.\n\n### 7.3 Required test idioms\n\n`tests/model/test_qwen3_5_dense.py` is the reference template; extend\n`DeterministicDDPTestCase` (`xtuner._testing`), which provides the shared\nparity scaffolding so the test file only holds model-specific logic:\n\n- `with self.hf_impl():` — sets/restores `XTUNER_HF_IMPL=true`; build the XTuner\n model **inside** the block so its attention picks the eager op at construction.\n- `self.materialize_xtuner_submodule(model, submodule, prefix, loader)` — for a\n model built on `meta`, materialize *only* that submodule on CUDA and load its\n weights (so a single-layer test fits any model size).\n- `self.load_params_from_hf(module, key_for, loader)` /\n `self.xtuner_ckpt_key(model, name)` — selective per-param load by checkpoint key\n (`HFCheckpointLoader` reads only the needed shard); `xtuner_ckpt_key` applies\n `to_hf_key_list` + the config's `hf_key_mapping`.\n\n```python\n# Construct on meta, materialize on device — single-rank baseline, eager, no EP.\nwith torch.device(\"meta\"):\n cfg = get_model_config_from_hf(hf_model_path)\n cfg.compile_cfg = False # baseline tests run eager\n model = cfg.build()._to_device_dtype(dtype=torch.bfloat16, skip_buffers_dtype=True)\n\n# Optionally shard with FSDP (data-parallel only), then load.\nmodel.fully_shard(fsdp_config=FSDPConfig(cpu_offload=False))\nmodel.from_hf(hf_model_path)\n```\n\nOlder `patch_hf_rms_norm` / `patch_hf_rope` helpers also live in\n`xtuner._testing`. **Check they actually fit your model before using them** —\nthey were written for earlier models and match HF modules\nby class-name substring and attribute name (e.g. `variance_epsilon`). On a newer\nmodel they can silently mis-patch (wrong eps attribute, or replacing a\n`zero_centered` RMSNorm with a default one). Often the native XTuner ops already\nmatch HF bitwise (see §12), so no patching is needed; prefer `XTUNER_HF_IMPL`\nover these helpers for parity. The §8 parallel-feature tests reuse this same\nidiom with `cfg.dispatcher` / `cfg.ep_size` (and `FSDPConfig(ep_size=...)`) set.\n\n### 7.4 Checkpoint paths\n\nRead from env vars; **never** hard-code. Example:\n`QWEN3_MOE_PATH = os.environ[\"QWEN3_MOE_PATH\"]` at\n`tests/model/test_qwen3_moe.py`. Document new env vars in the PR body.\n\n### 7.5 Drop-in training config\n\nShip a runnable training config at `ci/config/.py` so a developer (and CI) can verify\nthe port end-to-end with one command: load the HF checkpoint, run a few real training steps,\nwatch the loss curve descend as expected. This is the smoke test that catches anything the\nunit tests miss — the full `from_hf` → `model.forward` → loss → backward → optimizer step →\nFSDP shard/reduce chain — and the file doubles as the example users copy when wiring the\nmodel into their own training pipeline. Mirror `ci/config/qwen3_moe_30BA3.py` (MoE) or\n`ci/config/qwen3_dense.py` (dense), keeping its structure: one `()` (from §3.2)\nfed into a `TrainerConfig` alongside `optim_cfg` / `lr_cfg` / `fsdp_cfg` / `dataset_cfg` /\n`dataloader_cfg` / `loss_cfg`. `load_from` and `tokenizer_path` read from an env var (§7.4 —\ntypically the same one as the parity test). Verify by running ~50 steps and confirming the\nloss drops monotonically into a plausible range for that model size; record the trajectory\nin the PR body alongside the §6 convergence trace.\n\n---\n\n## 8. Training optimizations\n\nOnly after the §6 baseline is committed and the §7 tests are green: verify each\nswitch runs correctly and keeps precision within budget **against that baseline**.\n\n1. EP and SP can be enabled normally.\n2. With `ep_size > 1`, `intra_layer_micro_batch` enables correctly and stays\n precise.\n3. `torch.compile` runs correctly.\n4. fp8 can be enabled.\n5. `XTUNER_ACTIVATION_OFFLOAD` can be enabled.\n\nEach switch needs a test that compares its output to the §6 baseline — e.g. the\ndispatcher × ep_size parity matrix (at minimum `{(None,1), (\"all2all\",4),\n(\"all2all\",8)}`, add `\"deepep\"` if supported), compiled-vs-eager, or fp8-vs-bf16\nwithin tolerance.\n\nThese items are well-suited to the multi-agent workflow in §11: the main agent\nanalyzes and creates tasks (with test cases), sub-agents develop in parallel, and\nthe main agent merges. Report the precision error introduced by each switch.\n\n---\n\n## 9. Reference cheat-sheet\n\n| Concept | Concrete location |\n|-------------------------|--------------------------------------------------------------------------------|\n| Dense base config | `xtuner/v1/model/base.py` (`TransformerConfig`) |\n| MoE base config | `xtuner/v1/model/moe/moe.py` (`MoEConfig`) |\n| Compose base config | `xtuner/v1/model/compose/base.py` (`BaseComposeConfig`) |\n| `from_hf` (model) | `xtuner/v1/model/base.py` (`from_hf`) |\n| `save_hf` (model) | `xtuner/v1/model/base.py` (`save_hf`) |\n| `to_hf_key_list` proto | `xtuner/v1/model/base.py` |\n| MoE layer split | `xtuner/v1/model/moe/moe.py` (`first_k_dense_replace`) |\n| RoPE auto-parse | `xtuner/v1/module/rope/rope.py` (`RopeParametersConfig.from_hf_config`)|\n| Dense reference | `xtuner/v1/model/dense/qwen3.py` |\n| MoE reference | `xtuner/v1/model/moe/qwen3.py` |\n| MoE w/ MLA + NoAux | `xtuner/v1/model/moe/deepseek_v3.py` |\n| MoE w/ layout transform | `xtuner/v1/model/moe/gpt_oss.py` |\n| Remote-code reference | `xtuner/v1/model/moe/qwen3.py` (`Qwen3MoEFoPEConfig`) |\n| Compose reference (VLM) | `xtuner/v1/model/compose/qwen3_vl/` |\n| Dense test reference | `tests/model/test_qwen3_dense.py` |\n| MoE test reference | `tests/model/test_qwen3_moe.py` |\n| Engine test reference | `tests/engine/test_moe_train_engine.py` |\n\n---\n\n## 10. Commit discipline\n\n0. **Commit as you go — this is authorized, do not pause for a separate\n go-ahead.** The user invoking this skill authorizes the commits this workflow\n produces. Finish a logical step → lint/test → commit, then move on. Do not\n leave the whole port uncommitted and end with \"should I commit?\"; that strands\n the work. (Hard-to-reverse or outward actions like `git push` / opening a PR\n still need explicit confirmation — committing locally does not.)\n1. **Follow the stacked-PR convention.** Plan upfront how many commits the model\n port needs — one per logical step (e.g. shared-path fixes; model class +\n config + registration; the `XTUNER_HF_IMPL` switch; baseline tests; then one\n per parallel/optimization feature). Order them by dependency (a shared-path fix\n the baseline relies on goes first). Land every later change as\n `git commit --fixup=` + `git rebase --autosquash` into the commit it\n belongs to. **Never grow the history with endless patch commits just to keep\n fixing things up.**\n2. **Every commit must pass lint:**\n ```bash\n pre-commit run --files $(find xtuner/v1)\n ```\n\n---\n\n## 11. Multi-agent workflow\n\nThis applies **only to the §8 parallel/optimization features** — and only once\nthe §6 baseline and §7 baseline tests are done and committed (§10). The baseline\nitself is built sequentially, not multi-agent; the §8 features are independent of\neach other, which is what makes them safe to develop in parallel.\n\n1. The **main agent** splits the §8 features into tasks and, per §8, writes each\n feature's comparison test against the committed baseline and commits the tests.\n2. **Sub-agents** branch off the baseline/test commit and develop one feature\n each (e.g. one switch / one tower) in parallel, making its test pass.\n3. **The two sides adjust each other** — a shared-interface change is made once by\n the main agent (it updates the test/baseline; sub-agents rebase), and a failing\n test pushes the sub-agent to fix its feature, not to weaken the test.\n4. The **main agent** merges, resolves conflicts, and reports the precision error\n each feature introduces relative to the baseline.\n\n---\n\n## 12. Parity debugging pitfalls\n\nHard-won notes from aligning hybrid (GatedDeltaNet + gated-MHA) models bitwise.\nThe meta-rule: **measure on real GPU and bisect to localize the divergence before\nconcluding anything** — most \"bugs\" here are measurement artifacts, not model\nerrors. A clean way to bisect is to feed one side's intermediate tensors into\nboth modules and compare step by step (norm → q/k/v → qk_norm → rope → attention\ncore → o_proj).\n\n- **Single-layer parity: the two sides' inputs must match exactly — especially\n the causal mask.** When you hand-build one layer's inputs, passing\n `attention_mask=None` to an HF attention/layer makes HF apply **no** causal\n mask while XTuner stays causal, producing a large *false* diff. Build a causal\n mask equivalent to XTuner's `seq_ctx`, or drive the layer through the model so\n the mask is constructed internally.\n\n- **`flash` ≠ `eager`, and HF eager upcasts softmax to fp32.** XTuner defaults to\n flash attention; HF reference eager does softmax in fp32. They are numerically\n close but not bitwise equal, so bitwise parity requires forcing eager on the\n XTuner side (that is exactly what `XTUNER_HF_IMPL` does). XTuner's\n `eager_attention` matches HF's `eager_attention_forward` bitwise.\n\n- **A bitwise forward does not imply a bitwise backward.** The eager softmax must\n run in **fp32** (`softmax(..., dtype=torch.float32).to(x.dtype)`, like HF), not\n bf16. With a bf16 softmax the forward logits can still round identically to HF\n (bitwise), but the *backward* differs (~1e-5 on `dL/dx`) because the gradient\n accumulates at bf16 precision. So always check backward bitwise separately\n (the `loss.backward()` test), not just forward — and align op dtypes (softmax,\n norms) in the backward-sensitive direction, fp32 where HF uses fp32.\n\n- **Whole-model backward is bitwise — but only with the full env stack.**\n Per-layer bitwise backward does not on its own imply whole-model bitwise\n backward; the missing pieces are `XTUNER_HF_IMPL=true` AND\n `XTUNER_DETERMINISTIC=true` AND the conftest must `import xtuner.v1` so the\n autotune-pin block in `xtuner/v1/__init__.py` runs before any test reaches\n fla. Without those, two distinct ULP-level sources compound through the LM\n chain rule and look indistinguishable from \"fundamental bf16 noise\":\n\n 1. **Triton autotune drift in fla.** fla's `@triton.autotune` picks\n tiling / num_warps / reduction-order per kernel. Across runs the choice\n can change, producing a ~1-ULP backward diff per linear-attention layer\n that the chain rule amplifies — a 32-layer LM stack turned a per-layer\n ~2e-3 abs diff at grad magnitude ~30 into a ~1.0 diff on the\n massive-activation channels of `dL/d(pixel_values)`. The pin (configs\n held to the first one, cache disabled) is in `xtuner/v1/__init__.py`\n under an `XTUNER_DETERMINISTIC` gate. It must run **before any module\n imports fla** — fla's `@triton.autotune` decorators evaluate at import\n time, so patching after the fact has no effect on kernels already\n constructed. That's why the patch is on the `xtuner.v1.*` import chain\n rather than in the conftest: `multiprocessing.spawn` children of\n `MultiProcessTestCase` re-import the test class top-level (which pulls\n `xtuner.v1.model.*`) without running conftest, so the conftest alone\n can't install the patch in those processes.\n 2. **`torch.library.custom_op` wraps for the linear-attention path**\n (`GatedDeltaNet`, `causal_conv1d`) produce a different backward autograd\n graph from the canonical fla / causal_conv1d call HF uses, even though\n the underlying kernel and forward output are identical. `XTUNER_HF_IMPL`\n routes through `get_chunk_gated_delta_rule_fn` / `get_causal_conv1d_fn`\n (§6.2) so the parity path matches HF byte-for-byte.\n\n Production train doesn't set either env var; the custom_op wraps + autotune\n still operate. These switches are **parity-test only**. If the whole-model\n backward is not bitwise but the per-layer backward is, suspect one of the\n two before suspecting depth-accumulated bf16 noise.\n\n- **`flash`(XTuner varlen) vs `flash_attention_2`(HF) is bitwise only for short,\n single-tile sequences.** They are the same kernel, so at a short seqlen (e.g.\n 32) the logits are bitwise (0.0); at longer sequences the tiling /\n accumulation order differs and they drift ~1e-2. So a short-seq text test can\n assert flash-vs-FA2 bitwise, but anything realistic-length (or a VLM whose\n image tokens make the sequence long) needs the **eager** path for\n seqlen-independent bitwise. Don't conclude \"flash matches FA2\" from one short\n input.\n\n- **Shared kernels are already bitwise; focus on what isn't.** GatedDeltaNet /\n linear attention go through the same `fla` / `causal_conv1d` kernels as HF, and\n native rms_norm / rope match HF bitwise — these come out at 0.0 with no work.\n Spend your effort on the path that actually differs (usually full attention).\n\n- **`output_hidden_states` off-by-one (full-forward route, < 40B only).** HF's\n tuple is `[emb, L0_out, …, L{n-2}_out, post_norm]`: index `i+1` is layer `i`'s\n pre-norm output only for `i ≤ n-2`; the **last** entry is post-norm, not the\n last layer's pre-norm output. Compare the last layer via `model.norm(...)` or\n you will see the final layer \"explode\".\n\n- **A big max-diff is not automatically a bug.** Late layers have *massive\n activations* (a few channels in the tens/hundreds); a tiny relative error there\n shows up as a large absolute max-diff. Look at the mean and the trend, not just\n the max. Conversely, feeding `randn` inputs to a single attention layer is\n ill-conditioned and misleads — use real (or model-produced) hidden states.\n\n- **A single-model failure is often a general-path bug.** E.g. `Dense.build_layers`\n not forwarding `rms_norm_type` to `DenseDecoderLayer` broke any `zero_centered`\n dense model, not just this one. Fix the shared invariant, not the symptom.\n\n- **Loading a standalone sub-tower uses `strict=False`.** A text tower loaded\n directly from a full VLM checkpoint will see vision (and deferred `mtp.*`) keys\n as unexpected; load with `strict=False` so they are ignored.\n\n- **Test the whole VLM, not just the text tower.** Text-tower parity leaves the\n vision tower and projector completely untested. A compose port needs a\n full-model forward-parity test with real image inputs (§7.1). The text path of\n the compose forward still runs a *dummy* vision forward to keep the tower in the\n autograd graph, so vision bugs surface even on text prompts.\n\n- **Vision attention is non-causal — give `eager_attention` a `causal=False`\n path.** XTuner's `eager_attention` defaulted to a causal block-diagonal mask;\n that is wrong for a vision tower (bidirectional within each image), which is why\n forcing vision to the causal eager op produced large diffs. The op now takes\n `causal=True|False` (False → non-causal block-diagonal mask), the vision module\n already calls it with `causal=False`, and the vision tower routes through\n `get_attn_impl_fn` so `XTUNER_HF_IMPL` selects eager. With that, the *whole* VLM\n is bitwise in one run, everything eager: `XTUNER_HF_IMPL` on the XTuner side\n (text causal-eager + vision non-causal-eager) and HF loaded with a single\n `attn_implementation=\"eager\"`. Prefer eager for parity — it is\n seqlen-independent, unlike flash/FA2 (see the flash seqlen bullet above).\n\n- **Vision pos-embed dtype when reusing the Qwen3-VL tower.** HF's\n `fast_pos_embed_interpolate` returns fp32 (bilinear weights are fp32) and HF's\n vision forward casts it (`pos_embeds.to(hidden_states.dtype)`); XTuner's vision\n forward adds it without a cast. If you patch in the HF method for parity, cast\n the result back to the param dtype, or you get an fp32/bf16 `LayerNorm` mismatch\n (`expected scalar type Float but found BFloat16`).\n\n- **Hybrid-model `torch.compile`: choose `fullgraph` per layer — verify, then fall\n back; don't gate *whether* to compile.** When a decoder layer's forward is wrapped\n into a higher-order op (HOP) — e.g. by an activation-checkpoint wrapper, which\n `torch.compile` lowers to a HOP whose body must be functional so it can be\n recomputed in backward — *and* that layer is a linear-attention layer\n (GatedDeltaNet), the layer mutates an outer-scope variable inside the HOP\n (GatedDeltaNet writes back into `seq_ctx`). Under `fullgraph=True` that raises\n `HigherOrderOperator ... Mutating a variable not in the current scope (SideEffects)`.\n This is **not** a blanket \"linear attention ⇒ `fullgraph=False`\" rule — whether it\n triggers depends on whether the layer is actually HOP-wrapped and whether that\n linear impl really writes outer state. So for such a HOP + linear combination,\n **first test whether `fullgraph=True` runs**; keep it if it does (it compiles more\n of the layer), and only **fall back to `fullgraph=False`** if it doesn't —\n `fullgraph=False` survives by graph-breaking at the mutation and running that\n side-effecting write in eager. Make this a **per-layer** choice in\n `Dense.fully_shard` (decide each layer's `fullgraph`), not a decision about whether\n a layer is compiled at all.\n\n",".claude/skills/check-hf-config-save/SKILL.md":"---\nname: check-hf-config-save\ndescription: Implement a missing XTuner Hugging Face config export from the official HF model when possible, then validate it against the installed Transformers round-trip and versioned inference-engine field contracts. Use when adding a model, when hf_config is missing or returns None, when changing from_hf/hf_config/save_hf, when reviewing config.json differences, or when debugging vLLM/SGLang checkpoint-load or inference failures. Also use for requests named check_hf_config_save.\n---\n\n# Check HF Config Save\n\n## Overview\n\nFirst ensure that a model with an official built-in Transformers config has an\ninverse `from_hf <-> hf_config` mapping. If that export is missing, implement it\nfrom the official HF model before using `xtuner._testing.check_hf_config_save`\nto test the real public `from_hf -> save_hf` path. Separate changes forced by\nthe installed Transformers version from fields dropped by XTuner, then protect\nfields that exact inference engine versions use for model construction or\nweight loading.\n\n## Workflow\n\n### 1. Fix the scope and version matrix\n\n1. Identify the source HF model directory, XTuner config class, export path, and\n any engine/version named by the user or failure log.\n2. Record the executable environment versions for Python, Transformers, and any\n installed engines. Use the requested environment; otherwise follow the repo's\n default environment instructions.\n3. A user-specified or production-log version takes precedence. If a component\n version is unspecified, look up the latest stable release from its official\n PyPI project/API or official release page and inspect the matching source tag.\n By default audit both vLLM and SGLang.\n4. List every checked version in the result. Distinguish an installed runtime\n test from a static audit of an exact source tag.\n\nUse a table with at least these columns:\n\n| Component | Version | How selected | Validation |\n|---|---:|---|---|\n| Transformers | exact version | active environment | executable round-trip |\n| vLLM | exact version | user/log/latest official | runtime or exact-tag source audit |\n| SGLang | exact version | user/log/latest official | runtime or exact-tag source audit |\n\nDo not write `latest` without resolving it to an exact version and source.\n\n### 2. Implement a missing HF config export\n\nExercise the supported model's public conversion path first:\n\n```python\nconfig = get_model_config_from_hf(SOURCE_HF_DIR)\nexported_config = config.hf_config\n```\n\n`exported_config is None`, or the corresponding public `config.save_hf(...)`\nraising the base missing-`hf_config` `NotImplementedError`, means the config-only\nexport is not implemented. Classify the official model before changing code:\n\n```mermaid\nflowchart TD\n A[\"Official HF model and exact revision\"] --> B{\"Built-in official Transformers config?\"}\n B -->|Yes| C{\"XTuner hf_config implemented?\"}\n C -->|No| D[\"Implement inverse from_hf and hf_config mapping\"]\n C -->|Yes| E[\"Continue round-trip validation\"]\n D --> E\n B -->|No; trust_remote_code only| F[\"Keep hf_config as None and validate source-config copying\"]\n```\n\nFor a built-in official Transformers config:\n\n1. Read the official checkpoint's raw `config.json` and resolve its exact\n `model_type`, `architectures`, repository revision, and official config\n implementation. Load it with the selected Transformers version and record\n the concrete `PretrainedConfig` subclass. Do not use a third-party model\n implementation as the reference.\n2. Implement or complete `from_hf` with that official class. Map every\n architecture value needed by XTuner, use `getattr` for genuinely optional\n versioned fields, and use `RopeParametersConfig.from_hf_config` for RoPE.\n3. Implement `hf_config` by constructing the same official config class from\n the XTuner config's current values. It must be the inverse of `from_hf`, not\n a cached copy of the source object. Re-emit every field consumed by\n `from_hf`, plus raw compatibility fields required by vLLM or SGLang even when\n Transformers treats them as optional or legacy.\n4. Preserve the official `model_type` and architecture name. Do not substitute\n a generic `PretrainedConfig`, duplicate the official config class in XTuner,\n or return a hand-written dictionary.\n5. Verify through public behavior that `config.hf_config` has the expected\n official type and that `config.save_hf(...)` succeeds, then continue with\n the helper below.\n\nIf the official repository only works through `trust_remote_code` and the\nselected Transformers versions have no built-in config class, do not invent an\n`hf_config` implementation. Keep `hf_config=None`, retain the source `_hf_path`,\nand test the model's public HF save path that copies the original config,\ntokenizer, and required remote-code files. If the entire model or dispatch is\nnot yet supported by XTuner, follow `$add_hf_model`; this skill only fills a\nmissing exporter for an otherwise supported model.\n\n### 3. Establish the Transformers reference\n\nCompare three raw JSON states:\n\n```mermaid\nflowchart LR\n A[\"Source config.json\"] --> B[\"Current Transformers load + save\"]\n A --> C[\"XTuner from_hf + save_hf\"]\n B --> D[\"Expected serialized reference\"]\n C --> E[\"XTuner export\"]\n D --> F[\"Public helper comparison\"]\n E --> F\n```\n\n- Read the source `config.json` before `AutoConfig` normalization.\n- Load and save it directly with the active Transformers version. This is the\n serialized reference and captures forced defaults or `__post_init__` changes.\n- Build through XTuner's public `get_model_config_from_hf`/`from_hf` API and call\n the public `save_hf` API.\n- Compare the Transformers reference with the XTuner export. Do not use direct\n source-versus-export equality as the primary assertion: it misclassifies\n Transformers normalization as an XTuner bug.\n\nInspect the exact Transformers config implementation, including generated or\nmodular source and `__post_init__`, for every source-to-reference difference.\nTypical examples are derived `head_dim`, generated `layer_types`, new defaults,\nand serializer metadata, but never assume these examples cover a new model.\n\n### 4. Audit inference-engine dependencies\n\nFor every changed, missing, newly defaulted, or architecture-selecting field:\n\n1. Search the exact engine tag, not an arbitrary installed or main-branch copy.\n2. Follow model registration, config access, module construction, and the weight\n loader. Search aliases, `getattr` defaults, direct attribute access, and tensor\n name/shape conditions.\n3. Classify the use as one of:\n - module/parameter registration;\n - checkpoint key or tensor-shape selection;\n - layer topology or MoE routing;\n - attention/RoPE behavior;\n - ignored or default-compatible.\n4. Encode every value-sensitive dependency as `HFConfigFieldDependency`, with\n exact engine version, JSON pointer, expected value, reason, and an official\n source permalink.\n\nA field can be HF-equivalent yet engine-critical. For example, an engine may\nregister a checkpoint parameter only when a legacy routing field has one exact\nvalue; omitting that field then becomes a real weight-loader failure.\n\nStatic source inspection proves the dependency but not end-to-end engine\ncompatibility. Run an engine checkpoint-load smoke test when the exact runtime\nand required hardware are available. Otherwise report `exact-tag source audit`\nand do not claim runtime success. Follow the repo GPU-lock instructions before\nany local GPU run.\n\n### 5. Add the model regression test\n\nDelete narrow hand-written assertions that duplicate this contract, then add a\nmodel test on the real public conversion path:\n\n```python\nimport transformers\n\nfrom xtuner._testing import HFConfigFieldDependency, check_hf_config_save\nfrom xtuner.v1.model import get_model_config_from_hf\n\n\ndef test_save_hf_matches_transformers_and_engine_contracts():\n config = get_model_config_from_hf(SOURCE_HF_DIR)\n assert isinstance(config.hf_config, OFFICIAL_HF_CONFIG_CLASS)\n report = check_hf_config_save(\n config,\n SOURCE_HF_DIR,\n engine_dependencies=(\n HFConfigFieldDependency(\n engine=\"vllm\",\n version=\"\",\n path=\"/\",\n expected=\"\",\n reason=\"\",\n source=\"\",\n ),\n ),\n )\n\n assert report.transformers_version == transformers.__version__\n assert report.checked_engine_versions == (\"vllm==\",)\n```\n\nThe helper performs two independent checks:\n\n- XTuner export matches the active Transformers direct round-trip.\n- Exported values satisfy the declared inference-engine contracts, even when\n Transformers itself does not consume those fields.\n\nUse `allowed_export_differences={\"/json/pointer\": \"specific reason\"}` only for\nan intentional XTuner difference that is not already an engine dependency.\nEvery exception needs a non-empty model-level reason.\n\n### 6. Prove the regression and run the matrix\n\n1. Run the helper's own behavior tests.\n2. Run the generated model test in the project's pinned Transformers version.\n3. Run it in each user-requested/current upgraded Transformers environment.\n4. If the old broken export is available, pass its `config.json` through the\n same helper and show that the expected missing/extra paths fail. This is the\n minimal proof that the test catches the original bug.\n5. Run formatting and the smallest relevant model test suite.\n\nReport:\n\n- whether `hf_config` already existed, was implemented from which official\n config class/revision, or correctly remained `None` for remote code;\n- source-to-Transformers normalization paths;\n- Transformers-reference-to-XTuner paths (normally none, apart from documented\n allowed contracts);\n- engine dependency, expected value, exact version, and effect;\n- which checks were executable and which were source-only.\n\n## Guardrails\n\n- Compare raw JSON, not only `AutoConfig` attributes; unknown compatibility\n fields may disappear from a newer config class while engines still read them.\n- Exercise public APIs and real conversion behavior. Do not mock XTuner model\n internals.\n- Do not replace the helper with a blanket list of expected keys.\n- Do not silently refresh an engine version in a test. Re-audit its exact source\n before updating the version and permalink.\n- HF semantic equivalence is not evidence of vLLM/SGLang compatibility.\n- Do not set `hf_config=None` for a model with a built-in official config merely\n to bypass config reconstruction or round-trip failures.\n- Preserve unrelated worktree changes and keep the implementation model-agnostic.\n\n## Completion criteria\n\nThe work is complete only when the official config export path exists where it\nshould, the public round-trip matches Transformers, every audited engine\ncontract is preserved, and the report identifies the exact official model\nrevision and component versions used.\n",".claude/skills/check-hf-config-save/agents/openai.yaml":"interface:\n display_name: \"Check HF Config Save\"\n short_description: \"Implement and validate exported HF configs\"\n default_prompt: \"Use $check-hf-config-save to implement any missing official Hugging Face config export and validate it across Transformers and inference engines.\"\n",".claude/skills/model_normalize/SKILL.md":"---\nname: model_normalize\ndescription: Normalize an HF model checkpoint — quantize bf16/fp16 weights to per-block FP8, and/or repack engine-named safetensors into HF-standard ~4GB shards.\n---\n\n# model_normalize\n\nTwo complementary capabilities for getting a training-engine checkpoint into a\nformat that HF / vLLM / SGLang can consume:\n\n1. **FP8 quantization** of large linears, in two modes:\n - **Reference-guided** — point at an existing FP8 checkpoint of the same\n architecture; the tool reads its index and reproduces the exact same\n quantization decisions on the source.\n - **Heuristic** — apply a built-in rule set that quantizes only the large\n matrix multiplications (attention, MLP, MoE experts) and keeps norms,\n routers, biases, embeddings, and vision tower in their original dtype.\n2. **Standard repack** — re-shard arbitrary safetensors directories into the\n HF-standard ``model-{i:05d}-of-{n:05d}.safetensors`` layout with ~4GB\n shards and a fresh ``model.safetensors.index.json``.\n\nWhen ``to-fp8`` detects non-standard source shard names, it automatically\nruns the repack step on the quantized output, so the user-visible result is\nalways a clean HF-conformant directory.\n\n## Layout\n\n```\n.claude/skills/model_normalize/\n├── SKILL.md # this file\n├── model_normalize.py # top-level CLI (to-fp8 / repack)\n├── hf_to_fp8.py # FP8 conversion library + standalone CLI\n├── heuristics.py # codified default quantization rule set\n├── repack_hf.py # 4GB-shard repacker library + CLI\n└── convert_interns2_to_fp8.sh # example invocation for InternS2\n```\n\n## Usage\n\n### Mode 1 — Reference-guided FP8 conversion\n\n```bash\npython model_normalize.py to-fp8 \\\n --source /path/to/bf16-model \\\n --output /path/to/fp8-model \\\n --reference /path/to/fp8-reference\n```\n\nBehavior:\n- Loads the reference's ``model.safetensors.index.json``.\n- For each ``_scale_inv`` entry in the reference index, marks ````\n as an FP8-quantized tensor.\n- Applies that exact set to the source as a literal name match (no regex\n surprise; works even when names don't follow any pattern).\n\n### Mode 2 — Heuristic FP8 conversion (no reference)\n\n```bash\npython model_normalize.py to-fp8 \\\n --source /path/to/bf16-model \\\n --output /path/to/fp8-model\n```\n\nThe rule set is defined in ``heuristics.py``. Concretely, **quantize iff** the\ntensor name matches one of:\n\n| Group | Pattern (anchored) |\n| -------------------------------- | ------------------------------------------------------------------------------------------------------------------- |\n| Self-attention projections | ``[…]layers..self_attn.{q,k,v,o}_proj.weight`` |\n| Dense MLP | ``[…]layers..mlp.{gate,up,down}_proj.weight`` |\n| MoE per-expert (unfused) | ``[…]layers..mlp.experts..{gate,up,down}_proj.weight`` |\n| MoE fused-expert (3D tensors) | ``[…]layers..mlp.experts.{gate_up_proj, down_proj}`` |\n| MoE shared expert | ``[…]layers..mlp.shared_expert.{gate,up,down}_proj.weight`` |\n| Linear-attn wide projections | ``[…]layers..linear_attn.{in_proj_qkv, in_proj_z, out_proj}.weight`` |\n\nThe ``[…]`` prefix allows ``model.`` and ``model.language_model.``; the same\npatterns repeat under ``mtp.layers..`` for multi-token-prediction blocks.\n\n**Everything else is kept in the original dtype**, in particular:\n- all norms (``*norm*``, ``*_layernorm.*``)\n- MoE routers (``mlp.gate.weight``, ``mlp.shared_expert_gate.weight``)\n- embeddings (``embed_tokens``, ``lm_head``, ``pos_embed``, ``patch_embed``, ``mtp.fc``)\n- vision tower (``model.visual.*``)\n- all ``*.bias``\n- ``linear_attn`` control-flow tensors (``A_log``, ``conv1d.weight``, ``dt_bias``, ``in_proj_a/b.weight``, ``norm.weight``)\n\nThis rule set was validated against the InternS2 / Qwen3-MoE FP8 references:\nit reproduces their quantization decisions on every tensor (no false\npositives or false negatives).\n\n### Mode 3 — Standalone repack (no quantization)\n\n```bash\npython model_normalize.py repack \\\n --source /path/to/messy-shards \\\n --output /path/to/standard-shards \\\n --shard-size-gb 4\n```\n\nGreedy bin-packs tensors into ``model-{i:05d}-of-{n:05d}.safetensors`` shards\nof at most ``--shard-size-gb`` GiB, writes a fresh\n``model.safetensors.index.json`` (including ``metadata.total_size``), and\ncopies all non-safetensors files (config, tokenizer, etc.) through.\n\n## When to use which mode\n\n| Situation | Mode |\n| ---------------------------------------------------------------------------- | ------------------ |\n| You already have an FP8 reference for this architecture. | Reference-guided |\n| New architecture / first-time conversion; standard naming. | Heuristic |\n| Quantized model has engine-specific shard names; need HF-conformant output. | (auto, see below) |\n| You only need to standardize shard naming; weights are already FP8 / bf16. | ``repack`` |\n\nThe ``to-fp8`` subcommand auto-detects non-standard source shard names (anything\nthat isn't ``model.safetensors`` or ``model-NNNNN-of-NNNNN.safetensors``) and\nchains the repack step. Pass ``--no-repack`` to keep the source naming.\n\n## Quantization details\n\n- Block size **128 × 128**, dtype ``float8_e4m3fn`` with saturating cast.\n- Per-block scale stored as ``_scale_inv`` (``float32``) in the same\n shard as the parent weight. Layout matches what vLLM / SGLang load for\n per-block FP8.\n- 3D MoE-expert tensors (shape ``(E, D0, D1)``) are quantized per-expert; the\n per-expert scales are stacked along a leading dim.\n- ``config.json`` receives a ``quantization_config`` block with\n ``quant_method=fp8``, ``weight_block_size=[128, 128]``,\n ``scale_fmt=ue8m0``, and an explicit ``modules_to_not_convert`` list\n derived from the actual quantization decisions.\n\n## Post-conversion handoff (mandatory for the agent)\n\nAfter **any** successful run of `to-fp8` or `repack`, the agent must ask the\nuser — via `AskUserQuestion` — where to **place** the produced\n`*.safetensors` and `model.safetensors.index.json` files. The conversion\noutput is treated as a staging area; the user decides the final home.\n\nOffer at least these options:\n1. **Keep in place** — leave the files at the `--output` path as-is.\n2. **Copy to a new directory** — let the user specify a target; copy the\n shards + index there. Always **copy** (`cp`), never move — the\n `--output` staging directory must remain intact unless the user\n explicitly asks to delete it as a separate step.\n3. **Copy into an existing model directory** — copy shards/index into a\n user-specified directory while preserving its other files (tokenizer,\n config, etc.).\n\nIf the user picks (2) or (3), execute the copy with `cp` (confirming the\ndestination path back to the user before any overwrite). Do not infer the\ndestination from earlier turns — the user may have a fresh target in mind\neach time. Cleaning up the staging directory afterwards is a separate,\nuser-initiated step — never delete it as part of the handoff.\n\n## Example: InternS2 Preview\n\n```bash\npython model_normalize.py to-fp8 \\\n --source /mnt/shared-storage-user/puyudelivery/user/yehaochen/models/InternS2PreviewCandidate5 \\\n --output /tmp/InternS2-FP8 \\\n --reference /mnt/shared-storage-user/puyudelivery/user/yehaochen/models/InternS2PreviewCandidate5-FP8\n```\n\nThe source uses engine-specific names like\n``model-language-0001-fused-save_rank0.safetensors``; the output will be\nrepacked into the standard ``model-00001-of-NNNNN.safetensors`` layout.\n",".claude/skills/model_patch/SKILL.md":"---\nname: model_patch\ndescription: Merge a base HF model dir with an extra HF model dir by index diff — take tensors only present in extra, append them to a new output dir in HF-standard shard layout. Base wins on overlap.\n---\n\n# model_patch\n\nProduce ``out = base ∪ (extra \\ base)`` at the **tensor level**, by diffing the\ntwo ``model.safetensors.index.json`` files. Useful when ``extra`` is a sibling\ncheckpoint (different training run, fine-tune, or fp8 conversion) that holds a\nfew tensors the base is missing — e.g. an updated head, a newly-added MTP\nblock, or auxiliary projections — and you want a self-contained merged dir\nwithout re-running the whole conversion pipeline.\n\nOverlap rule: **base wins**. Any tensor present in both indexes is taken from\n``base``; the corresponding entry in ``extra`` is ignored.\n\n## Layout\n\n```\n.claude/skills/model_patch/\n├── SKILL.md # this file\n└── model_patch.py # CLI + library\n```\n\n## Usage\n\n```bash\npython model_patch.py \\\n --base /path/to/base-model \\\n --extra /path/to/extra-model \\\n --out /path/to/merged-model\n```\n\n## What it writes\n\n- ``model-{i:05d}-of-{N:05d}.safetensors`` shards (total ``N`` = base shards +\n newly-packed shards holding the extra-only tensors)\n- a fresh ``model.safetensors.index.json``\n- **every non-tensor file from ``base``** — ``config.json``, tokenizer files,\n modeling/configuration ``.py``, ``generation_config.json``, README, etc. —\n copied verbatim so ``out`` is loadable on its own. Subdirectories are\n copied recursively. Existing entries in ``out`` are overwritten.\n\n``extra``'s aux files are deliberately **not** copied. The skill's overlap\nrule is \"base wins\" at the tensor level, and that extends to configs: extra\noften carries engine-specific edits or modality keys that don't match the\ntensor topology we keep. If you need fields from ``extra``'s ``config.json``\n(e.g. a new modality), merge them into ``out/config.json`` as a separate,\nexplicit step.\n\n## How shards are produced\n\n- **Base shards**: re-emitted under the new ``-of-N`` total. By default the\n bytes are fully copied so ``out`` is independent of ``base``. Pass\n ``--hardlink`` to use ``os.link`` instead (fast, but the two trees share\n inodes — editing one mutates the other; deleting one keeps the other\n intact since hardlinks are symmetric).\n- **Extra-only tensors**: greedily bin-packed into new shards of at most\n ``--shard-size-gb`` GiB (default 4), appended after the base shards.\n\nSo a base with 12 shards plus 5 GB of extra-only tensors yields:\n``model-00001-of-00014.safetensors`` … ``model-00012-of-00014.safetensors``\n(copies of the base shards) and\n``model-00013-of-00014.safetensors`` / ``model-00014-of-00014.safetensors``\n(freshly written, containing only the extra tensors).\n\n## When to use this skill\n\n| Situation | Use this? |\n| -------------------------------------------------------------------------------------- | --------- |\n| ``extra`` contains a few tensors ``base`` is missing; everything else is identical. | Yes |\n| You need to merge two checkpoints whose overlapping tensors **differ** in value. | No — this skill silently keeps base; you probably want a manual decision per key. |\n| ``extra`` is a full quantization of ``base`` and you want the quantized weights. | No — use ``model_normalize`` with ``--reference extra`` instead. |\n| You want a single ``model.safetensors`` instead of standard shards. | No — repack downstream. |\n\n## Required usage protocol (mandatory for the agent)\n\nBefore invoking ``model_patch.py``:\n\n1. Confirm the three paths (``--base``, ``--extra``, ``--out``) with the user\n via ``AskUserQuestion``. Never infer them from earlier conversation.\n2. Show the user the **diff summary** before writing, by reading the two\n ``model.safetensors.index.json`` files and reporting:\n - number of tensors in base\n - number of tensors in extra\n - number of extra-only keys that will actually be added\n - number of overlapping keys that will be dropped from extra\n3. Ask the user to confirm those numbers before launching the run.\n\nAfter a successful run, **do not** ask the user where the output should go —\n``--out`` is already the final destination the user named. Just report what\nlanded there (shard count, index path, total size, and that base's aux files\nwere mirrored over). Unlike ``model_normalize``, ``model_patch`` has no\nstaging-area concept: there is no temp dir, and the caller already made the\nplacement decision when they passed ``--out``.\n\nIf the user wanted any of ``extra``'s aux files (e.g. a ``config.json`` that\ndocuments a new modality), call that out — those are not copied by the\nskill and must be merged in by hand.\n",".claude/skills/sphinx-debug/SKILL.md":"---\nname: sphinx-debug\ndescription: >\n Use when `make html` or `sphinx-build` fails with\n `Extension error (sphinx.ext.autosummary)`,\n `ImportExceptionGroup`, or other Sphinx autosummary / autodoc\n import failures. Provides automated pdb diagnosis for\n `autodoc_mock_imports` issues.\n---\n\n# Sphinx Debug\n\n## Quick Diagnostic Command\n\nRun the build with maximum verbosity and pdb on exception:\n\n```bash\nmake html SPHINXOPTS=\"-vv -T -P\"\n```\n\nFlags:\n- `-vv`: Verbose autodoc mock logging\n- `-T`: Full traceback\n- `-P`: Drop into pdb on exception\n\n## For agents: automate pdb via monkey-patch\n\nIf you are an agent and cannot interact with an interactive pdb session,\n**do not redesign a subprocess/pty capture system**. Instead, monkey-patch\n`pdb.post_mortem` so it runs scripted commands and prints the output\ndirectly to stdout.\n\nRun this one-shot script from the docs directory (`docs/en` or `docs/zh_cn`):\n\n```bash\ncd docs/en # or docs/zh_cn\npython -c \"\nimport pdb, sys, io, traceback\n\n# Hijack pdb.post_mortem to auto-run diagnostic commands\n_original_post_mortem = pdb.post_mortem\n\ndef scripted_post_mortem(tb=None):\n out = io.StringIO()\n p = pdb.Pdb(stdout=out, stdin=io.StringIO(''))\n p.use_rawinput = False\n p.reset()\n if tb is None:\n tb = sys.exc_info()[2]\n p.setup(None, tb)\n\n old_stderr = sys.stderr\n sys.stderr = out\n\n # List all inner exceptions\n p.onecmd('for i, exc in enumerate(exceptions): print(f\\\"[{i}] {type(exc).__name__}: {exc}\\\")')\n # Print full traceback of the first TypeError (the real root cause)\n p.onecmd('type_errs = [exc for exc in exceptions if type(exc).__name__ == \\\"TypeError\\\"]')\n p.onecmd('if type_errs: import traceback; traceback.print_exception(type(type_errs[0]), type_errs[0], type_errs[0].__traceback__)')\n p.do_quit('')\n\n sys.stderr = old_stderr\n print('=== PDB AUTOMATED OUTPUT ===')\n print(out.getvalue())\n print('=== END OUTPUT ===')\n\npdb.post_mortem = scripted_post_mortem\n\nfrom sphinx.cmd.build import build_main\nbuild_main(['-b', 'html', '-vv', '-T', '-P', '.', '_build/html'])\n\"\n```\n\nThis runs the real Sphinx build with `-P`, but when pdb drops in it\nautomatically executes your commands and dumps the results to stdout.\nYou read the output exactly as if you had typed the pdb commands yourself.\n\n## What to look for in the automated output\n\nSphinx autosummary raises `ImportExceptionGroup` when it cannot resolve a module.\nThe **top-level message is often misleading** (e.g. `no module named xtuner.v1.ray.dataflow`).\nThe real root cause is hidden inside the grouped exceptions.\n\n### Common root-cause patterns\n\n1. **TypeError about `__version__`**\n ```\n TypeError: expected string or bytes-like object, got '__version__'\n ```\n This happens when a mocked module (e.g. `torch`) is imported for real by an\n intermediate library (e.g. `fla`) that calls `packaging.version.parse()` on\n the mocked `__version__` object.\n\n **Fix**: Add the intermediate library to `autodoc_mock_imports` in `conf.py`.\n To identify the exact library, read the traceback: the frame just above\n `packaging/version.py` is the culprit (e.g. `fla/utils.py`).\n\n ```python\n autodoc_mock_imports = [\n ...\n \"fla\", # or whatever library triggers the real import\n ]\n ```\n\n2. **Missing-comma string-concatenation bug**\n\n If `autodoc_mock_imports` contains an entry like `\"scipytorchvision\"`,\n a comma is missing between two string literals in `conf.py`:\n\n ```python\n # WRONG\n autodoc_mock_imports = [\n ...\n \"scipy\"\n \"torchvision\",\n ...\n ]\n\n # CORRECT\n autodoc_mock_imports = [\n ...\n \"scipy\",\n \"torchvision\",\n ...\n ]\n ```\n\n3. **AttributeError after the TypeErrors**\n\n ```\n AttributeError: module 'xtuner.v1.ray' has no attribute 'dataflow'\n ```\n This is usually a *secondary* failure caused by the earlier `TypeError`s.\n Fix the `TypeError` first, then rebuild.\n\n## Step-by-step workflow\n\n1. Run the automated pdb script (see \"For agents\" section above).\n2. Read the `=== PDB AUTOMATED OUTPUT ===` block.\n3. If you see `TypeError: expected string or bytes-like object, got '__version__'`,\n read the traceback to find the intermediate library name.\n4. Add the identified library to `autodoc_mock_imports` in **both**\n `docs/en/conf.py` and `docs/zh_cn/conf.py`.\n5. Also eyeball `autodoc_mock_imports` for missing commas while you are there.\n6. Re-run `make html` (without `-P`) to verify the fix.\n",".claude/skills/xtuner-custom-judger/SKILL.md":"---\nname: xtuner-custom-judger\ndescription: Use when creating, modifying, or reviewing XTuner RL judgers under xtuner/v1/rl/judger or configs that build custom judgers. Enforces the preferred Judger payload contract and the BaseJudger/ComposedJudger constraints.\n---\n\n# XTuner Custom Judger\n\nUse this skill whenever you implement or review an XTuner RL judger.\n\n## Preferred Pattern\n\nPrefer subclassing `Judger`, not `BaseJudger`.\n\nFor custom scoring logic, override these methods as needed:\n\n```python\nclass MyJudger(Judger):\n def preprocess(self, rollout_state: RolloutState) -> JudgerPayload:\n ...\n\n async def judge_payload(self, payload: JudgerPayloadBatch) -> JudgerOutputBatch:\n ...\n\n def postprocess(self, rollout_state: RolloutState, output: JudgerOutput) -> RolloutState:\n ...\n```\n\nDo not override `Judger.judge()` just to read extra fields from `RolloutState`.\nPut the field extraction in `preprocess()` instead.\n\n## Why This Contract Exists\n\n`preprocess -> judge_payload -> postprocess` keeps the scoring payload small and\ncomposable:\n\n- `preprocess()` extracts only the fields required for scoring.\n- `judge_payload()` can run locally, in a Ray actor, or behind a judger pool.\n- `postprocess()` writes the output back to the original `RolloutState`.\n\nThis avoids `deepcopy(RolloutState)` in composed and remote judging. Deep-copying\nrollout states can add serialization overhead and can be risky for large fields,\nRay object refs, tensors, numpy arrays, or externally owned objects.\n\n## BaseJudger Rule\n\nSubclass `BaseJudger` only when the implementation must own the full\n`judge()` / `batch_judge()` flow and cannot reasonably be expressed as a\npayload contract.\n\nBaseJudger-only judgers cannot be used as `ComposedJudger` branches.\n`ComposedJudger` intentionally calls branch `preprocess()` and\n`judge_payload()` rather than arbitrary `BaseJudger.judge()` implementations.\nBecause it does not deep-copy `RolloutState`, supporting arbitrary\nBaseJudger-only branches would allow multiple branches to mutate the same\nrollout state concurrently and overwrite `reward` or other fields.\n\n## ComposedJudger Checklist\n\nWhen adding a branch intended for `ComposedJudger`:\n\n1. Ensure the class inherits `Judger`.\n2. Ensure branch-specific fields are copied into the payload in `preprocess()`.\n3. Ensure `judge_payload()` returns a `JudgerOutput` for one payload, or a\n `list[JudgerOutput]` for a batch payload.\n4. Ensure `postprocess()` writes `rollout_state.reward`.\n5. For multi-branch composed judging, ensure `merge_fn` creates the final\n reward shape, including `reward[\"score\"]` when training expects it.\n\n## Tests\n\nFor custom judger changes, add focused tests for:\n\n- Single-sample `judge()`.\n- `batch_judge()` if batch support is implemented.\n- `ComposedJudger` routing if the judger is intended as a composed branch.\n- Rejection or explicit non-support if the implementation directly subclasses\n `BaseJudger`.\n",".claude/skills/xtuner-sync-supported-models/SKILL.md":"---\nname: xtuner-sync-supported-models\ndescription: Synchronize xtuner's supported model documentation (docs/en/pretrain_sft/advanced_tutorial/model.md and docs/zh_cn/pretrain_sft/advanced_tutorial/model.md) with the actual Config classes defined under xtuner/v1/model/. Use when (1) new TransformerConfig, MoEConfig, or BaseComposeConfig subclasses are added, removed, or renamed in xtuner/v1/model/, (2) existing model configs change their inheritance hierarchy, scale, or HuggingFace counterpart, or (3) a code review or user request points out that model.md is out of sync with the codebase.\n---\n\n# Update XTuner Supported Model Docs\n\nKeep the English and Chinese `model.md` files synchronized with the actual Config classes in `xtuner/v1/model/`.\n\n## Scan the Codebase\n\nRun the bundled scan script from the xtuner project root to discover all Config classes and their inheritance:\n\n```bash\npython3 .agents/skills/xtuner-sync-supported-models/scripts/scan_model_configs.py\n```\n\nThe script outputs JSON with two keys:\n- `configs`: list of every `*Config` class under `xtuner/v1/model/` with its parent classes and file path\n- `children`: parent-to-children mapping for the hierarchy tree\n\n## What to Update\n\nCompare the script output against the two files:\n- `docs/en/pretrain_sft/advanced_tutorial/model.md`\n- `docs/zh_cn/pretrain_sft/advanced_tutorial/model.md`\n\nBoth files share the same structure and must stay in sync:\n\n1. **Base Config Classes** — configs that directly inherit from `TransformerConfig` (or `MoEConfig`) and provide a `from_hf` classmethod for loading HuggingFace weights\n2. **Concrete Model Configs** — fixed-scale subclasses of the base configs above\n3. **Compose Models** — multimodal configs that inherit from `BaseComposeConfig`\n4. **Inheritance Hierarchy** — a text tree showing the full `XTunerBaseModelConfig` hierarchy\n\n### Rules for the Base Config table\n\nInclude these direct descendants of `TransformerConfig`/`MoEConfig`:\n- `Qwen2DenseConfig`\n- `Qwen3DenseConfig`\n- `DeepSeekV3Config`\n- `GptOssConfig`\n- `Qwen3MoEConfig`\n\nExclude from the base table:\n- `MoEConfig` — it is an intermediate base class, not a usable model family\n- `Qwen3_5_VLTextMoEConfig` — it is an intermediate base with only one concrete child; its child `Qwen3_5_VLTextMoE35BA3BConfig` belongs under the MoE concrete table\n\n### Rules for the Concrete Model table\n\nInclude every concrete subclass that has fixed parameter defaults. For each row note:\n- `Config Class`\n- `Base Class / Family`\n- `Architecture Type`: `Dense`, `MoE`, `Dense (VL backbone)`, `MoE (VL backbone)`\n- `Scale / Notes`: parameter count or total/activated size; for VL backbones note \"for multimodal\"\n\n`DeepSeekV3Config` appears here even though it has no separate base entry (it is both base and concrete).\n\n### Rules for the Compose Models section\n\nInclude three sub-tables:\n1. **Compose Base Config Classes** — `Qwen3VLBaseConfig`, `InternVLBaseConfig`, `InternS1BaseConfig`\n - `Qwen3VLBaseConfig`: VL model based on Qwen3 text backbone\n - `InternVLBaseConfig`: VL model based on InternViT + Qwen3\n - `InternS1BaseConfig`: Science multimodal model based on InternViT + Qwen3\n2. **Concrete Compose Model Configs** — every subclass of the above bases; for each row note the wrapped `Text Config` and scale\n\n### Rules for the Inheritance Hierarchy tree\n\nRebuild the tree from `XTunerBaseModelConfig` with two top-level branches:\n\n```text\nXTunerBaseModelConfig\n├── TransformerConfig\n│ ├── Dense Models\n│ │ ├── Qwen2DenseConfig\n│ │ │ └── Qwen2Dense7BConfig\n│ │ └── Qwen3DenseConfig\n│ │ ├── Qwen3Dense8BConfig\n│ │ │ └── Qwen3VLTextDense8BConfig\n│ │ ├── Qwen3Dense4BConfig\n│ │ │ └── Qwen3VLTextDense4BConfig\n│ │ └── Qwen3Dense0P6BConfig\n│ └── MoE Models (via MoEConfig)\n│ ├── DeepSeekV3Config\n│ ├── GptOssConfig\n│ │ ├── GptOss21BA3P6Config\n│ │ └── GptOss117BA5P8Config\n│ ├── Qwen3MoEConfig\n│ │ ├── Qwen3MoE30BA3Config\n│ │ │ └── Qwen3VLTextMoE30BA3Config\n│ │ ├── Qwen3MoE235BA22Config\n│ │ │ └── Qwen3VLTextMoE235BA22Config\n│ │ └── Qwen3MoEFoPEConfig\n│ └── Qwen3_5_VLTextMoEConfig\n│ └── Qwen3_5_VLTextMoE35BA3BConfig\n└── BaseComposeConfig\n ├── Qwen3VLBaseConfig\n │ ├── Qwen3VLMoE30BA3Config\n │ ├── Qwen3VLMoE235BA22Config\n │ ├── Qwen3VLDense4BConfig\n │ ├── Qwen3VLDense8BConfig\n │ └── Qwen3_5_BaseConfig\n │ └── Qwen3_5_VLMoE35BA3Config\n ├── InternVLBaseConfig\n │ ├── InternVL3P5Dense8BConfig\n │ ├── InternVL3P5MoE30BA3Config\n │ └── InternVL3P5Dense1BConfig\n └── InternS1BaseConfig\n ├── InternS1Config\n └── InternS1MiniConfig\n```\n\nWhen new configs are added, insert them into the appropriate branch following the same indentation style.\n\n## Translation Notes\n\nKeep the Chinese `model.md` (`docs/zh_cn/...`) structurally identical to the English one. Translate:\n- Section headings\n- Table header cells\n- Description cells (e.g., \"Image / Video + Text\" → \"图像/视频 + 文本\")\n- Scale descriptions (e.g., \"~7B parameters\" → \"约 7B 参数\", \"FoPE variant\" → \"FoPE 变体\")\n\nDo **not** translate Config class names, file paths, or code identifiers.\n",".claude/skills/xtuner-trace/SKILL.md":"---\nname: xtuner-trace\ndescription: Use when the user wants to instrument, inspect, or debug XTuner traces to reconstruct a sample/request execution path, follow cross-process call chains, or identify latency hotspots, bottleneck stages, and abnormal paths from span data.\n---\n\n# XTuner Trace\n\nUse this skill for the current trace layer in this repository: OpenTelemetry runtime setup, the basic public API, local setup scripts, and the trace viewer.\n\n## Current Boundary\n\nKeep this package infrastructure-only:\n\n- Runtime/configuration: `xtuner/v1/rl/trace/runtime.py`\n- OTel SDK adapter: `xtuner/v1/rl/trace/otel_utils.py`\n- Public facade: `xtuner/v1/rl/trace/__init__.py`\n- Basic API: `xtuner/v1/rl/trace/api.py`\n- Rollout starter preset: `xtuner/v1/rl/trace/rollout_api.py`\n- Local trace tooling and viewer: `recipe/trace/`\n\nDo not add rollout, agent, judger, Ray remote, HTTP proxy, reward, status, or session-server business semantics to `api.py`, `runtime.py`, or `otel_utils.py`. Keep rollout-specific starter behavior inside `rollout_api.py` and gated by `TraceConfig.enable_rollout_trace`.\n\n## Viewer Choice\n\nChoose the viewer by the observation question:\n\n- Use `recipe/trace/viewer` when the user wants rollout-oriented inspection, such as comparing samples within a training step by stage duration, status, reward/filter metadata, or recorded sample call chain.\n- Use Jaeger when the question is not sample/step-centric, such as inspecting raw spans, service boundaries, cross-process causality, backend/client calls, collector/export behavior, or custom instrumentation that does not emit the rollout/sample attributes expected by the recipe viewer.\n\nThe recipe viewer reads `traces.jsonl` and adds XTuner rollout/sample aggregation. Jaeger remains the raw trace drill-down surface for the same traced run.\n\n## Black-Box Agent Trace Routing\n\nWhen the trace request involves an agent, first ask whether this is a black-box\nagent trace. In this mode, XTuner exposes an OpenAI-compatible inference endpoint\nto the agent through `SessionServer`; the agent runs its native loop and calls\nthat endpoint for LLM generation. Tool execution, environment interaction, and\nobservation handling happen inside the agent runtime and are not directly\ninstrumented by XTuner.\n\nIf the answer is yes, or the request mentions trajectory reconstruction,\n`SessionServer` request gaps, `record_synthetic_span(...)`,\n`reconstructed_agent_run`, `reconstructed_tool_call.N`, XTuner viewer plus\nJaeger display of recovered tool calls, read\n[references/blackbox-agent-trace.md](references/blackbox-agent-trace.md) before\nplanning or editing.\n\n## Basic API\n\nThese are the interfaces defined in `xtuner/v1/rl/trace/api.py` and re-exported\nfrom `xtuner.v1.rl.trace`:\n\n- `trace_span(name, attributes=None, parent_carrier=None)`\n- `trace_function(name=None, attributes=None)`\n- `trace_event(name, attributes=None)`\n- `set_trace_attributes(attributes)`\n- `inject_trace_context(carrier=None)`\n\n## Runtime API\n\nUse these `xtuner/v1/rl/trace/runtime.py` interfaces for explicit trace\nruntime setup:\n\n- `TraceConfig`\n- `configure_trace(...)`\n- `close_trace()`\n\n## Rollout Starter Preset\n\n`TraceConfig(enable_rollout_trace=True)` enables the built-in rollout starter\ntrace. Its helpers live in `xtuner/v1/rl/trace/rollout_api.py`:\n\n- `trace_rollout_endpoint(...)`\n- `trace_rollout_remote(...)`\n\nKeep this layer narrow: it may depend on `RolloutState` and Ray call boundaries,\nbut it must not leak those dependencies into the basic API.\n\nRollout endpoint `xtuner.span_name_path` is a sample call chain, not just the\ncurrent process context stack. `trace_rollout_endpoint(...)` should append the\nendpoint span name to an internal trace-only chain carried in\n`RolloutState.extra_fields`, pass that state through remote rollout boundaries,\nand clean the internal field at the outermost endpoint. For example, after a\nsample generates and then judges, `judger.run` should record a path like\n`single_turn_agent_loop.run -> rollout.controller.generate -> rollout.worker.generate -> judger.run`\neven though the local active context at judge time may only be\n`single_turn_agent_loop.run -> judger.run`. Keep this behavior in\n`rollout_api.py` and viewer tests; do not add rollout call-chain semantics to\nthe basic trace API.\n\n## Add Trace Workflow\n\nWhen adding trace instrumentation to an XTuner run:\n\n1. Ask for or locate the launch script and training config before editing.\n2. In the launch script, source `recipe/trace/setup_trace.sh` when\n `XTUNER_TRACE_ENABLED=1`.\n3. In the training config, add `TraceConfig` and set `enabled=True`; set\n `xtuner_viewer_enabled=True` when the user needs interactive sample/step inspection. Set\n `enable_rollout_trace=True` only when the user wants the built-in rollout\n starter trace.\n4. Before adding any `trace_span(...)` instrumentation, you must ask the user which\n stages they want to observe and which metrics each stage should expose.\n Do not infer default stages unless the user explicitly asks you to choose.\n5. Add `trace_span(...)` only around the user-confirmed observed stages. Put fields known at span\n start in initial attributes, and update runtime or final fields with\n `set_trace_attributes(...)`.\n6. For cross-process or request boundaries, inject a carrier with\n `inject_trace_context(...)` and pass it to downstream\n `trace_span(..., parent_carrier=carrier)`.\n7. Keep transport-specific propagation at the caller boundary; do not move\n rollout, agent, judger, Ray, or HTTP semantics into the basic trace package.\n8. Ensure the main training log includes the trace output path, XTuner viewer URL,\n and equivalent manual viewer command when the XTuner viewer is enabled.\n\n## Guardrails\n\n- Do not recreate `trace_utils.py`, `context_propagation.py`, span-name registries, or business attribute builders under `xtuner/v1/rl/trace`.\n- Do not import `RolloutState`, agent item classes, judgers, rollout workers, Ray actors, aiohttp clients, or trainer configs from the basic trace package.\n- Do not call OpenTelemetry SDK directly from business code; use the basic API only when trace instrumentation is explicitly requested.\n- Do not record prompts, responses, full configs, secrets, raw headers, stack traces, or large payloads as attributes.\n- Viewer stage grouping must come from span attributes such as `xtuner.stage`, `stage`, or `stage.name`; otherwise fall back to the raw span name.\n\nFor concrete patterns, read [references/trace-patterns.md](references/trace-patterns.md) before editing trace code.\n\n## Verification\n\nUse focused checks:\n\n- `PYTHONPATH=. python -m compileall -q xtuner/v1/rl/trace recipe/trace`\n- `PYTHONPATH=. python -m unittest discover -s tests/rl -p 'test_trace*.py' -v` when the trace tests exist in the worktree.\n- `git diff --check`\n",".claude/skills/xtuner-trace/references/blackbox-agent-trace.md":"# Blackbox Agent Trace Reference\n\nThis reference captures the XTuner black-box agent trace reconstruction workflow. It is intended for implementation and review work, not as user-facing docs.\n\nRead this reference when `xtuner-trace/SKILL.md` routes an agent trace request here. The design background lives in `docs/design/blackbox_agent_trace_reconstruction.md`.\n\n## Contents\n\n- [Decision Flow](#decision-flow)\n- [Path A: Trajectory-Backed Reconstruction](#path-a-trajectory-backed-reconstruction)\n- [Path B: SessionServer-Only Fallback](#path-b-sessionserver-only-fallback)\n- [`record_synthetic_span(...)` Usage](#record_synthetic_span-usage)\n- [Reconstructed Span Shape](#reconstructed-span-shape)\n- [Viewer And Jaeger Requirements](#viewer-and-jaeger-requirements)\n- [Example: mini-SWE-agent Trajectory Conversion](#example-mini-swe-agent-trajectory-conversion)\n- [Verification](#verification)\n- [Common Failure Modes](#common-failure-modes)\n\n## Decision Flow\n\nWhen adding trace support for a black-box agent, classify the available evidence before deciding where to instrument.\n\n1. Confirm this is a black-box agent path: XTuner exposes an OpenAI-compatible inference endpoint to the agent through `SessionServer`; the agent runs its native loop and calls that endpoint for LLM generation. Tool execution, environment interaction, and observation handling happen inside the agent runtime and are not directly instrumented by XTuner.\n2. Confirm whether the agent can save a trajectory for each run.\n3. If a trajectory exists, check whether it contains tool-call metadata such as tool name, action/command, status, return code, error information, observation, or timestamp.\n4. Use Path A when the trajectory contains tool-call metadata. The trajectory supplies tool semantics, and `session_server.request` spans supply the LLM request timeline.\n5. Use Path B only when no useful trajectory exists. This can recover request gaps, but cannot recover reliable tool names, return codes, errors, or observations.\n\nDo not require the trajectory to contain complete LLM generate timing before using Path A. `SessionServer` remains the stable source for LLM request start/end times. If the trajectory also records LLM calls, use them as an optional consistency check.\n\n## Path A: Trajectory-Backed Reconstruction\n\nUse this path when the agent can save a trajectory with tool-call metadata.\n\n### Data Contract\n\nThe `session_server.request` spans must provide:\n\n- `xtuner.session_id` or `xtuner.rollout_id`\n- span start time\n- span end time\n\nThe trajectory should provide best-effort tool metadata:\n\n- tool event order, id, or enough context to infer order\n- tool name, action, or command\n- status, return code, or exception information\n- observation or output summary\n- tool start/end timestamps, or a single tool-related timestamp with documented semantics, if the agent records one\n\nDo not define a universal trajectory schema in trace core. Agent trajectory formats are agent-specific; keep parsing inside that agent's recipe or trajectory-to-trace adapter until multiple agents prove a shared model is worth extracting.\n\n### Where To Add Trace\n\nAdd trace at the three black-box observation boundaries. These instructions must be enough to re-create the instrumentation even when no example implementation exists in the checkout.\n\n#### 1. SessionServer Request Trace\n\nFirst check whether the `SessionServer` HTTP request handler already emits `session_server.request` spans. If not, add this instrumentation to the handler that receives the agent's OpenAI-compatible request.\n\nImplementation requirements:\n\n- Locate the outer request handler, such as an aiohttp/FastAPI handler named like `_handle_request(request)`.\n- Read enough request headers/body to determine:\n - `session_id`\n - optional rollout id\n - whether the request is streaming\n - optional trace carrier from standard headers or a request-body carrier field\n- Wrap the full inference request lifecycle in:\n\n```python\nwith trace_span(\n \"session_server.request\",\n attributes={\n \"xtuner.stage\": \"llm_generate\",\n \"xtuner.session_id\": session_id,\n \"xtuner.rollout_id\": rollout_id or session_id,\n \"session.stream\": is_streaming,\n },\n parent_carrier=parent_carrier,\n):\n ...\n```\n\n- Keep the outer `session_server.request` span around prepare, backend request, response read, response post-processing, and return-to-agent.\n- Use `set_trace_attributes(...)` before exiting the span to record request-level status or error metadata when available.\n- If adding child spans, use these names:\n - `session_server.prepare_request`\n - `session_server.send_request`\n - `session_server.read_response`\n - `session_server.record_response`\n- Put `xtuner.session_id` on every child span too. This makes later trajectory-to-trace adapters able to filter a single agent run reliably.\n\nDo not parse agent trajectory or infer tool calls inside `SessionServer`. Its black-box responsibility is to provide the LLM request timeline.\n\n#### 2. Agent Run Boundary Trace\n\nIn the black-box recipe or launcher, find the call site that invokes the agent's native run loop, such as `agent.run(...)`.\n\nWrap only the outer run boundary:\n\n```python\nwith trace_span(\n \".run\",\n attributes={\n \"xtuner.stage\": \"agent.run\",\n \"xtuner.session_id\": session_id,\n \"xtuner.rollout_id\": rollout_id or session_id,\n \"agent.name\": agent_name,\n },\n):\n agent.run(...)\n```\n\nThis span is a lifecycle boundary. Do not wrap the agent's internal model, environment, or tool methods as the default black-box trace path. LLM/tool detail should come from `SessionServer` spans plus trajectory reconstruction.\n\n#### 3. Trajectory-To-Trace Reconstruction\n\nAfter the native agent run finishes, call the trajectory-to-trace adapter before `close_trace()`. Prefer a `finally` block so partial trajectories can still be converted after failures:\n\n```python\nreturn_code = 1\ntry:\n with trace_span(\".run\", attributes=agent_run_attributes):\n return_code = agent.run(...)\nfinally:\n emit_reconstructed_trace(\n trace_jsonl=trace_jsonl_path,\n trajectory=trajectory_path,\n session_id=session_id,\n )\n close_trace()\nreturn return_code\n```\n\nThe adapter must read the trajectory, read same-session `session_server.request` spans, derive tool timing from trajectory timestamps when possible, use request gaps only for correlation or inferred fallback, and emit reconstructed synthetic spans with `record_synthetic_span(...)`.\n\n### Adapter Responsibilities\n\nWrite a trajectory-to-trace adapter for the specific agent. The adapter should:\n\n1. Load the agent trajectory.\n2. Extract ordered tool events and their metadata.\n3. Load `session_server.request` spans from the same `traces.jsonl`.\n4. Filter request spans by `session_id`.\n5. Sort request spans by start time and assign local indexes.\n6. Extract tool timestamps from the trajectory and interpret their semantics:\n - If the trajectory provides reliable tool start/end times, use them as the `reconstructed_tool_call.N` span boundaries and set `xtuner.inferred=false`.\n - If the trajectory provides only one timestamp, record what it means if known, such as tool start, tool end, observation write, or trajectory write. Use it to correlate the tool event with the surrounding `session_server.request` window.\n - If the trajectory timestamp semantics are unknown or incomplete, use request-gap boundaries only as an inferred fallback and set `xtuner.inferred=true`.\n7. Use `session_server.request` spans as the LLM timeline and correlation windows, not as the default source of trajectory-backed tool timing.\n8. When using request-gap fallback for a bounded tool event, use:\n\n```text\ntool_i.start = request_i.end\ntool_i.end = request_{i+1}.start\n```\n\n9. Emit `reconstructed_agent_run`, `reconstructed_llm_call.N`, and bounded `reconstructed_tool_call.N` spans with `record_synthetic_span(...)`.\n10. Treat final tool events without a next LLM request as unbounded unless the trajectory has trustworthy start/end times.\n\nWhen request-gap fallback is used, the interval is not pure tool runtime. It can include LLM-response parsing, user confirmation time, tool execution, observation formatting, trajectory write time, and loop overhead before the next LLM request.\n\n## Path B: SessionServer-Only Fallback\n\nUse this path only when the agent does not save a useful trajectory.\n\n### Where To Add Trace\n\nAdd trace to the `SessionServer` request path so every LLM request produces a reliable timeline. At minimum, `session_server.request` spans must include:\n\n- `xtuner.session_id`\n- `xtuner.rollout_id`, falling back to the session id when there is no rollout id\n- request start time\n- request end time\n\nSupporting spans such as `session_server.prepare_request`, `session_server.send_request`, `session_server.read_response`, and `session_server.record_response` may remain useful for debugging latency, but the fallback reconstruction only requires the outer `session_server.request` start/end times.\n\nDo not put tool inference inside `SessionServer`. `SessionServer` should not parse trajectories, infer tool names, track return codes, or maintain complex per-session reconstruction state.\n\n### Fallback Reconstruction\n\nAfter the agent run, a fallback adapter may infer intervals from adjacent request spans:\n\n```text\ninferred_tool_call_i.start = request_i.end\ninferred_tool_call_i.end = request_{i+1}.start\n```\n\nThe fallback can emit synthetic spans such as `inferred_tool_call.N` with:\n\n```text\nxtuner.stage=tool_call\nxtuner.synthetic=true\nxtuner.inferred=true\nxtuner.session_id=\nreconstruction.source=session_server_only\nreconstruction.method=session_server_gap\n```\n\nDo not claim unavailable metadata in this mode. SessionServer-only fallback cannot reliably recover tool name, command, return code, stderr/stdout, agent exception, skipped action, or observation content.\n\nDo not infer the final tool event without a next LLM request. The interval has no reliable end time.\n\n## `record_synthetic_span(...)` Usage\n\nUse `record_synthetic_span(...)` to write reconstructed historical intervals back into the active XTuner trace runtime.\n\n```python\nrecord_synthetic_span(\n name: str,\n *,\n start_time_unix_ns: int,\n end_time_unix_ns: int,\n attributes: Mapping[str, Any] | None = None,\n parent_carrier: Mapping[str, str] | None = None,\n status: str = \"completed\",\n error_message: str | None = None,\n) -> dict[str, str] | None\n```\n\nRules:\n\n- Call it before `close_trace()`.\n- Use absolute Unix nanoseconds, not monotonic timestamps.\n- Ensure `end_time_unix_ns > start_time_unix_ns`.\n- Let the API default `xtuner.synthetic=true`.\n- Set `xtuner.inferred=true` only when the time boundary is inferred, such as from `session_server.request` gaps.\n- Use `status=\"error\"` and `error_message=...` for real errors.\n- Use `parent_carrier` to attach reconstructed children under one synthetic root trace.\n- Treat `None` return as trace-disabled no-op.\n\nTypical root and child pattern:\n\n```python\nroot_ids = record_synthetic_span(\n \"reconstructed_agent_run\",\n start_time_unix_ns=first_request.start_time_unix_ns,\n end_time_unix_ns=last_request.end_time_unix_ns,\n attributes={\n \"xtuner.stage\": \"agent.run\",\n \"xtuner.session_id\": session_id,\n \"xtuner.inferred\": False,\n \"reconstruction.method\": \"trajectory_session_timeline\",\n },\n)\nroot_carrier = None\nif root_ids:\n root_carrier = {\"traceparent\": f\"00-{root_ids['trace_id']}-{root_ids['span_id']}-01\"}\n\nrecord_synthetic_span(\n \"reconstructed_tool_call.1\",\n start_time_unix_ns=request_1.end_time_unix_ns,\n end_time_unix_ns=request_2.start_time_unix_ns,\n attributes={\n \"xtuner.stage\": \"tool_call\",\n \"xtuner.session_id\": session_id,\n \"xtuner.inferred\": True,\n \"agent.tool_call_index\": 1,\n \"agent.tool.name\": \"bash\",\n \"reconstruction.source\": \"agent_trajectory\",\n \"reconstruction.method\": \"session_server_gap\",\n },\n parent_carrier=root_carrier,\n)\n```\n\nDo not call OpenTelemetry SDK directly from the adapter. Use XTuner trace APIs.\n\n## Reconstructed Span Shape\n\nTrajectory-backed reconstruction should generate one synthetic root trace:\n\n```text\nreconstructed_agent_run\n|-- reconstructed_llm_call.1\n|-- reconstructed_tool_call.1\n|-- reconstructed_llm_call.2\n|-- reconstructed_tool_call.2\n...\n```\n\nUse these naming rules:\n\n- `reconstructed_agent_run` for the synthetic root.\n- `reconstructed_llm_call.N` for LLM request intervals copied from `session_server.request`.\n- `reconstructed_tool_call.N` for trajectory-backed tool metadata.\n- `inferred_tool_call.N` only for SessionServer-only fallback without trajectory metadata.\n\nRecommended root attributes:\n\n```text\nxtuner.stage=agent.run\nxtuner.synthetic=true\nxtuner.inferred=false\nxtuner.session_id=\nxtuner.rollout_id=\nagent.name=\nagent.llm_call_count=\nagent.tool_call_count=\nreconstruction.source=\nreconstruction.method=trajectory_session_timeline\n```\n\nRecommended LLM attributes:\n\n```text\nxtuner.stage=llm_generate\nxtuner.synthetic=true\nxtuner.inferred=false\nagent.llm_call_index=N\nreconstruction.source=session_server_request\nreconstruction.method=session_server_request_copy\nreconstruction.original_trace_id=\nreconstruction.original_span_id=\nreconstruction.original_operation=session_server.request\n```\n\nRecommended tool attributes:\n\n```text\nxtuner.stage=tool_call\nxtuner.synthetic=true\nxtuner.inferred=true\nagent.tool_call_index=N\nagent.tool.name=\nagent.tool.status=\nagent.tool.returncode=\nagent.tool.command_preview=\nreconstruction.source=\nreconstruction.method=session_server_gap\nreconstruction.timestamp_check=\n```\n\nDo not call trajectory-backed spans `inferred_tool_call.N`. Use `reconstructed_tool_call.N` and put uncertainty in attributes.\n\n## Viewer And Jaeger Requirements\n\nXTuner viewer and Jaeger must consume the same reconstructed spans. Do not implement one inference path in the viewer and another inference path for Jaeger.\n\nViewer expectations:\n\n- Show one sample per agent run/session after live and completed records are merged.\n- Prefer the trace containing `reconstructed_agent_run` as the primary sample trace.\n- Keep raw `session_server.request` trace ids available for audit.\n- Display the reconstructed path:\n\n```text\nreconstructed_agent_run -> reconstructed_llm_call.1 -> reconstructed_tool_call.1 -> ...\n```\n\nJaeger expectations:\n\n- The viewer's Jaeger link should open the trace containing `reconstructed_agent_run`.\n- Opening the first raw `session_server.request` trace is a bug for reconstructed black-box samples.\n- A healthy reconstructed trace contains `1 + llm_count + bounded_tool_count` reconstructed spans.",".claude/skills/xtuner-trace/references/trace-patterns.md":"# XTuner Trace Patterns\n\nThese patterns describe the retained trace surface in this branch: runtime, basic API, viewer, and local setup tooling.\n\n## Basic API\n\nUse the public facade:\n\n```python\nfrom xtuner.v1.rl.trace import (\n inject_trace_context,\n set_trace_attributes,\n trace_event,\n trace_function,\n trace_span,\n)\n```\n\n### Local Span\n\n```python\nwith trace_span(\"phase.name\", attributes={\"xtuner.stage\": \"phase\"}):\n ...\n set_trace_attributes({\"phase.count\": 3})\n```\n\n### Decorated Function\n\n```python\n@trace_function(\"phase.load\")\ndef load_item(path: str) -> object:\n ...\n```\n\n### Parent Carrier\n\n```python\ncarrier = inject_trace_context()\n\nwith trace_span(\"child.phase\", parent_carrier=carrier):\n ...\n```\n\n`parent_carrier` is a basic W3C context carrier. It is not tied to `RolloutState`, Ray, aiohttp, or any XTuner business object.\n\n## End-to-End Trace Wiring\n\nUse this pattern when adding trace support to an XTuner training run.\n\n### Launch Script\n\nEnable trace bootstrap from the launcher, guarded by `XTUNER_TRACE_ENABLED`:\n\n```bash\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\nREPO_ROOT=\"$(cd \"${SCRIPT_DIR}/../../..\" && pwd)\"\nif [ \"${XTUNER_TRACE_ENABLED:-0}\" = \"1\" ]; then\n source \"${REPO_ROOT}/recipe/trace/setup_trace.sh\"\nfi\n```\n\nKeep training stdout/stderr in the main training log so runtime trace messages\nare visible:\n\n```bash\npython xtuner/v1/train/cli/rl.py \\\n --config \"$CONFIG_PATH\" \\\n --num-workers \"$XTUNER_RL_NUM_WORKERS\" \\\n 2>&1 | tee -a \"${WORK_DIR}/training_log_${current_time}.txt\"\n```\n\n### Training Config\n\nAdd `TraceConfig` to the training config and pass it to the trainer config:\n\n```python\nimport os\n\nfrom xtuner.v1.rl.trace import TraceConfig\n\n\ntrace_config = TraceConfig(\n enabled=os.environ.get(\"XTUNER_TRACE_ENABLED\") == \"1\",\n service_name=\"xtuner-agent-rollout\",\n xtuner_viewer_enabled=True,\n xtuner_viewer_host=\"0.0.0.0\",\n xtuner_viewer_port=18080,\n xtuner_viewer_jaeger_query_url=\"http://127.0.0.1:16686\",\n enable_rollout_trace=True,\n)\n\ntrainer = RLColocateTrainerConfig(\n ...,\n trace_config=trace_config,\n)\n```\n\nWhen `output_dir` is omitted in an RL trainer config, the trainer places trace\nartifacts under the current experiment directory, for example\n`work_dirs/name/20260713071356/otel/`.\n\n### Stage Spans\n\nBefore adding spans, ask the user which stages they want to observe and which\nmetrics each stage should expose.\n\nPut fields known when entering the stage, especially fields used for grouping\nor filtering, in `trace_span(..., attributes=...)`. Put results known only\nafter or during execution, such as status, duration, token counts, errors, and\nreward values, in `set_trace_attributes(...)`. This lets the viewer reconstruct\nthe call chain and identify latency hotspots or abnormal stages.\n\nInfer stage:\n\n```python\nimport time\n\n\nstarted_at = time.monotonic()\nattributes = {\n \"xtuner.stage\": \"infer\",\n \"xtuner.task_name\": task_name,\n \"xtuner.sample_id\": sample_id,\n \"rollout.backend\": backend,\n}\n\nwith trace_span(\"agent.infer\", attributes=attributes):\n result = await run_infer(...)\n set_trace_attributes(\n {\n \"xtuner.status\": \"ok\" if result.ok else \"error\",\n \"prompt.tokens\": result.prompt_tokens,\n \"completion.tokens\": result.completion_tokens,\n \"stage.duration_ms\": int((time.monotonic() - started_at) * 1000),\n }\n )\n```\n\nResource acquire stage:\n\n```python\nwith trace_span(\n \"sandbox.acquire\",\n attributes={\n \"xtuner.stage\": \"acquire\",\n \"xtuner.sample_id\": sample_id,\n \"sandbox.name\": sandbox_name,\n },\n):\n client = await pool.get(sandbox_name)\n set_trace_attributes(\n {\n \"sandbox.env_id\": env_id,\n \"sandbox.image\": image,\n \"sandbox.reused\": reused,\n }\n )\n```\n\nValidation or reward stage:\n\n```python\nwith trace_span(\n \"sample.validate\",\n attributes={\n \"xtuner.stage\": \"validate\",\n \"xtuner.sample_id\": sample_id,\n \"validator.name\": validator_name,\n },\n):\n record = await validate(...)\n set_trace_attributes(\n {\n \"xtuner.status\": \"ok\" if record.ok else \"error\",\n \"reward.score\": record.score,\n \"reward.passed\": record.passed,\n }\n )\n\n if record.error is not None:\n set_trace_attributes(\n {\n \"error\": True,\n \"error.type\": record.error.type,\n \"error.message\": record.error.message,\n }\n )\n```\n\nDo not record prompts, responses, full configs, secrets, raw headers, stack\ntraces, or large payloads.\n\n### Cross-Boundary Propagation\n\nWhen a traced operation crosses a process, actor, or HTTP boundary, keep the\npropagation helper in the business or transport module that owns that boundary.\nDo not put Ray, HTTP, rollout, agent, or judger helpers back into\n`xtuner/v1/rl/trace`.\n\n#### Ray Remote Calls\n\nFor Ray calls, inject the current context into a plain carrier and temporarily\nattach that carrier to a serializable domain object or explicit call argument.\n\n```python\nfrom collections.abc import Mapping\nfrom contextlib import contextmanager\nfrom typing import Any\n\nfrom xtuner.v1.rl.trace import inject_trace_context, trace_span\n\n\n_TRACE_CARRIER_FIELD = \"_xtuner_trace_carrier\"\n\n\n@contextmanager\ndef attach_trace_carrier_temporarily(target: Any, carrier: Mapping[str, str]):\n if not carrier:\n yield\n return\n\n extra_fields = getattr(target, \"extra_fields\", None)\n if extra_fields is None:\n extra_fields = {}\n target.extra_fields = extra_fields\n had_previous = _TRACE_CARRIER_FIELD in extra_fields\n previous = extra_fields.get(_TRACE_CARRIER_FIELD)\n extra_fields[_TRACE_CARRIER_FIELD] = dict(carrier)\n try:\n yield\n finally:\n if had_previous:\n extra_fields[_TRACE_CARRIER_FIELD] = previous\n else:\n extra_fields.pop(_TRACE_CARRIER_FIELD, None)\n\n\ndef call_ray_remote_with_trace(remote_method, *args, trace_target: Any, **kwargs):\n carrier: dict[str, str] = {}\n inject_trace_context(carrier)\n with attach_trace_carrier_temporarily(trace_target, carrier):\n return remote_method.remote(*args, **kwargs)\n\n\ndef pop_trace_parent_carrier(target: Any) -> dict[str, str] | None:\n extra_fields = getattr(target, \"extra_fields\", None)\n if not isinstance(extra_fields, dict):\n return None\n carrier = extra_fields.pop(_TRACE_CARRIER_FIELD, None)\n if not isinstance(carrier, Mapping):\n return None\n return {str(key): str(value) for key, value in carrier.items()}\n```\n\nOn the receiver side, attach the carrier to the new span:\n\n```python\nparent_carrier = pop_trace_parent_carrier(trace_target)\n\nwith trace_span(\"worker.stage\", attributes=attributes, parent_carrier=parent_carrier):\n ...\n```\n\nKeep the helper domain-specific. Validate that exactly one trace target is used\nwhen the transport needs one target object; reject collections if restoring the\ncarrier to the right child span would be ambiguous.\n\n#### HTTP Calls\n\nFor HTTP calls, prefer W3C headers. If a third-party client or proxy may drop\ncustom headers, also put a copied carrier into the JSON body under an internal\nfield and remove it before forwarding the payload downstream.\n\n```python\nfrom collections.abc import Mapping\nfrom typing import Any\n\nfrom xtuner.v1.rl.trace import inject_trace_context, trace_span\n\n\n_TRACE_CARRIER_FIELD = \"_xtuner_trace_carrier\"\n\n\ndef inject_trace_carrier_into_json_body(kwargs: dict[str, Any], carrier: Mapping[str, str]) -> None:\n if not carrier:\n return\n payload = kwargs.get(\"json\")\n if not isinstance(payload, dict):\n return\n payload = dict(payload)\n payload.setdefault(_TRACE_CARRIER_FIELD, dict(carrier))\n kwargs[\"json\"] = payload\n\n\ndef extract_trace_carrier_from_mapping(payload: Mapping[str, Any] | None) -> dict[str, str] | None:\n if not isinstance(payload, Mapping):\n return None\n carrier = payload.get(_TRACE_CARRIER_FIELD)\n if not isinstance(carrier, Mapping):\n return None\n return {str(key): str(value) for key, value in carrier.items()}\n\n\ndef remove_trace_carrier_from_mapping(payload: dict[str, Any]) -> None:\n payload.pop(_TRACE_CARRIER_FIELD, None)\n\n\ndef extract_http_parent_carrier(headers: Mapping[str, Any], payload: Mapping[str, Any] | None) -> dict[str, str] | None:\n header_carrier = {\n str(key): str(value)\n for key, value in headers.items()\n if str(key).lower() in {\"traceparent\", \"tracestate\", \"baggage\"}\n }\n return header_carrier or extract_trace_carrier_from_mapping(payload)\n```\n\nCaller:\n\n```python\nheaders = dict(headers or {})\nrequest_kwargs = {\"json\": payload}\ncarrier: dict[str, str] = {}\ninject_trace_context(carrier)\nheaders.update(carrier)\ninject_trace_carrier_into_json_body(request_kwargs, carrier)\n\nwith trace_span(\"http.client.request\", attributes={\"http.method\": \"POST\", \"http.url\": url}):\n response = await client.post(url, headers=headers, **request_kwargs)\n```\n\nReceiver:\n\n```python\nheaders = dict(request.headers)\npayload = await request.json()\nparent_carrier = extract_http_parent_carrier(headers, payload)\n\nwith trace_span(\"http.server.request\", attributes=attributes, parent_carrier=parent_carrier):\n remove_trace_carrier_from_mapping(payload)\n ...\n```\n\nDo not record raw headers or full payloads as span attributes. Only record\nderived fields such as method, route, status code, stage, IDs, and timing.\n\n### Viewer Output\n\nWhen `xtuner_viewer_enabled=True`, the trace runtime logs the trace JSONL path,\nthe XTuner viewer URL, and the equivalent manual viewer command. The viewer\ndefaults to port `18080`; `xtuner_viewer_port=0` is still available when a run\nshould pick a free port automatically. The viewer process inherits the training\nstdout/stderr; keep those runtime log lines visible in the main training log and\nreport them to the user after the run.\n\nUseful reference points from the full trace implementation:\n\n- `examples/v1/scripts/run_rl_run.sh`\n- `recipe/trace/setup_trace.sh`\n- `examples/v1/config/agentic_rl_qwen3p5vl_mtp_ep_code.py`\n\n## Runtime\n\nConfigure tracing explicitly:\n\n```python\nfrom xtuner.v1.rl.trace import TraceConfig, close_trace, configure_trace\n\nruntime = configure_trace(\n TraceConfig(\n enabled=True,\n output_dir=\"work_dirs/example/otel\",\n service_name=\"xtuner\",\n xtuner_viewer_enabled=True,\n )\n)\n\ntry:\n ...\nfinally:\n close_trace()\n```\n\nThe runtime owns OTel collector setup, trace JSONL output, and optional lightweight XTuner viewer startup.\n\n## Viewer\n\nThe viewer reads Jaeger-style traces or OTel JSONL converted into Jaeger-style payloads. It should not depend on hard-coded rollout/agent span registries.\n\nStage display rules:\n\n1. Use `xtuner.stage` when present.\n2. Else use `stage` when present.\n3. Else use `stage.name` when present.\n4. Else fall back to the raw span name.\n\nUseful stable attributes:\n\n- IDs: `xtuner.rollout_id`, `xtuner.group_id`, `xtuner.session_id`, `xtuner.task_name`\n- Status: `xtuner.status`\n- Stage: `xtuner.stage`, `stage`, `stage.name`\n- Counts/timing: `prompt.tokens`, `completion.tokens`, `http.status_code`\n- Errors: `error`, `error.message`, `exception.type`\n\nAvoid recording prompts, responses, full configs, secrets, raw headers, or large payloads.\n\n## Local Setup\n\n`recipe/trace/setup_trace.sh` and the rest of `recipe/trace/` are local helper tooling for installing OTel collector binaries, starting the local Jaeger dependency, and clearing stale `recipe.trace.viewer.server` processes on `XTUNER_TRACE_VIEWER_PORT` or the default viewer port `18080`. Keep these as setup assets; do not use them to add automatic trace behavior to training configs or launch scripts unless that integration is explicitly requested.\n\n## Rollout Starter Preset\n\nThe built-in rollout starter trace is opt-in:\n\n```python\ntrace_config = TraceConfig(\n enabled=True,\n xtuner_viewer_enabled=True,\n enable_rollout_trace=True,\n)\n```\n\nIts rollout-specific helpers live in `xtuner/v1/rl/trace/rollout_api.py`:\n\n- `trace_rollout_endpoint(...)`\n- `trace_rollout_remote(...)`\n\nUse this only for the starter rollout chain. For custom instrumentation, ask\nwhich stages and metrics the user wants before adding spans.\n\n## Removed Legacy Semantics\n\nDo not use or recreate these removed surfaces in this branch:\n\n- `trace_remote`\n- `traced_rollout_endpoint`\n- `traced_agent_item_endpoint`\n- `traced_judger_endpoint`\n- `xtuner/v1/rl/trace/trace_utils.py`\n- `xtuner/v1/rl/trace/context_propagation.py`\n- fixed `TRACE_SPAN_*` registries\n- rollout/agent/judger attribute builders\n",".codex/AGENTS.md":"../.claude/CLAUDE.md","xtuner/tools/llm_calc/README.md":"### Summary\n\nThis PR introduces a comprehensive GPU memory calculator for LLM training, which helps users estimate per-GPU memory consumption before actually running training jobs. This tool is essential for capacity planning and optimizing parallel configurations.\n\n### Features\n\n- **Memory Breakdown Analysis**: Calculates detailed memory usage including:\n - Model parameters (embedding, attention, MLP, head)\n - Gradients\n - Optimizer states (Adam momentum & variance)\n - Master parameters and gradients (for mixed-precision training)\n - Intermediate activations\n\n- **Parallel Strategy Support**:\n - Data Parallelism with ZeRO Stage 1(for Megatron)/3 (FSDP for xtuenr)\n - Tensor Parallelism (TP)\n - Pipeline Parallelism (PP) and Virtual Pipeline Parallelism (VPP) for Megatron\n - Pipeline parallelism memory accounts for flying micro-batches during 1F1B schedule\n - Expert Parallelism (EP) for MoE models\n\n- **Model Architecture Support**:\n - Dense models (e.g., Qwen3-8B)\n - MoE models (e.g., Qwen3-235B-A22B, DeepSeek-V3-671B)\n - Multi-Head Attention (MHA)\n - Multi-Latent Attention (MLA) used in DeepSeek\n\n- **Training Optimizations**:\n - Activation checkpointing (recomputation)\n - Flash Attention\n - Chunked loss computation\n\n### Usage\n\n```bash\npython xtuner/tools/llm_calc/llm_calculator.py -c [-o ]\n```\n\n### Example Configurations\n\n| Config File | Model | Scale | TrainFramework |\n|------------|-------|-------|-------|\n| `xtuner_qwen_dense_8b_1node.yaml` | Qwen3-8B (Dense) | 1 node | xtuner |\n| `xtuner_qwen_moe_30b_1node.yaml` | Qwen3-30B-A3B (MoE) | 1 node | xtuner |\n| `xtuner_qwen_moe_235b_64node.yaml` | Qwen3-235B-A22B (MoE) | 64 nodes | xtuner |\n| `xtuner_deepseekv3_671b_128node.yaml` | DeepSeek-V3-671B (MoE) | 128 nodes | xtuner |\n| `megatron_qwen_moe_235b_64node.yaml` | Qwen3-235B-A22B (MoE) | 64 nodes | Megatron |\n\n### Output Example\n\nThe tool outputs:\n1. Total model parameters breakdown\n2. Per-GPU memory consumption for different training phases (forward/backward)\n3. Detailed memory breakdown by component (params, grads, optimizer states, activations)\n4. Optional Excel export for further analysis\n \n```\n# python llm_calculator.py -c xtuner_qwen_dense_8b_1node.yaml\n----------------------------------------------setting:----------------------------------------------\nmodel:Qwen3-8B zero_stage:3 tp:1 pp:1 dp:8 ep:1 vpp:1 nodes:1 mbs:1 gbs:8 L:32768 recompute:True flash_attn:True capacity:1.0 LN:36\n----------------------------------------------------------------------------------------------------\nTotal_params_num: 7.628208160400391 B, embed_params_num: 0.57958984375 B, attn_params_num: 1.4063873291015625 B, mlp_params_num: 5.0626373291015625 B, head_params_num: 0.5795936584472656 B\nMax_mem for microbatch=1: 26.58784294128418 GiB. micro1_forward_last: 26.58784294128418 GiB, micro1_backward_last: 21.386390686035156 GiB\nMax_mem for microbatch>1: 30.401947021484375 GiB. micro1_forward_last: 26.58784294128418 GiB, micro1_backward_last: 21.386390686035156 GiB, micro2_forward_last: 30.401947021484375 GiB\n---------------------------------------------------------------------memory parts---------------------------------------------------------------------\n params grads opt_states master_params master_grads acts\nname\nembed 1.159180 1.159180 0.579590 0.289795 0.289795 0.000061\nhead 1.159187 1.159187 0.579594 0.289797 0.289797 0.625000\nperlayer_attn 0.078133 0.078133 0.039066 0.019533 0.019533 1.751953\nperlayer_mlp 0.281258 0.281258 0.140629 0.070314 0.070314 2.500000\nall_attn 0.000000 0.000000 1.406387 0.703194 0.703194 0.000000\nall_mlp 0.000000 0.000000 5.062637 2.531319 2.531319 0.000000\nin_layernorm 0.000000 0.000000 0.000000 0.000000 0.000000 0.250000\n\n------------------------------------------------------micro1_forward_last: 26.58784294128418 GiB------------------------------------------------------\n [Static OS&Param] embed, all_attn, all_mlp, head: master_params, opt_states = 11.442312240600586 GiB\n [Allgathered Last Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Head] head: params, acts = 1.7841873168945312 GiB\n [Recompute Checkpoint] in_layernorm.acts * (LN - 1) = 8.75 GiB\n\n-----------------------------------------------------micro1_backward_last: 21.386390686035156 GiB-----------------------------------------------------\n [Static OS&Param] embed, all_attn, all_mlp, head: master_params, opt_states = 11.442312240600586 GiB\n [Static Grad] embed, all_attn, all_mlp, head: master_grads = 3.8141040802001953 GiB\n [Allgathered First Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Embed] embed: params, acts = 1.15924072265625 GiB\n [Previous Layer Grad ReduceScatering] perlayer_attn, perlayer_mlp: grads = 0.3593902587890625 GiB\n [Recompute Checkpoint] None = 0 GiB\n\n-----------------------------------------------------micro2_forward_last: 30.401947021484375 GiB------------------------------------------------------\n [Static OS&Param] embed, all_attn, all_mlp, head: master_params, opt_states = 11.442312240600586 GiB\n [Static Grad] embed, all_attn, all_mlp, head: master_grads = 3.8141040802001953 GiB\n [Allgathered Last Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Head] head: params, acts = 1.7841873168945312 GiB\n [Recompute Checkpoint] in_layernorm.acts * (LN - 1) = 8.75 GiB\n```\n\n### Technical Notes for Xtuner FSDP (with recompute) Memory Breakdown of the above example\n\n1. At initilization, each GPU holds sharded master params and optimizer states (usually both in fp32).\n```\n [Static OS&Param] embed, all_attn, all_mlp, head: master_params, opt_states = 11.442312240600586 GiB\n```\n\n2. At the end of the first micro batch forward: \n2.1 The whole last decoder layer and lm head are gathered in each GPU (in bf16).\n```\n [Allgathered Last Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Head] head: params, acts = 1.7841873168945312 GiB\n```\n2.2 The checkpoint of all decoder layers also exist in each GPU\n```\n [Recompute Checkpoint] in_layernorm.acts * (LN - 1) = 8.75 GiB\n```\n\n3. At the end of the first micro batch backward\n3.1 Each GPU holds master params' graident now\n```\n [Static Grad] embed, all_attn, all_mlp, head: master_grads = 3.8141040802001953 GiB\n```\n3.2 The whole first decoder layer and embedding layer are gathered in each GPU\n```\n [Allgathered First Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Embed] embed: params, acts = 1.15924072265625 GiB\n```\n3.3 FSDP does ReduceScatter of previous layer's gradients at the same time (to overlap computation and communication)\n```\n [Previous Layer Grad ReduceScatering] perlayer_attn, perlayer_mlp: grads = 0.3593902587890625 GiB\n```\n3.4 The checkpoints for recomputing have been released, along with the forward process\n```\n [Recompute Checkpoint] None = 0 GiB\n```\n\n4. At the end of the second micro batch forward, the only difference with the first micro batch is that each GPU also holds master gradients now.\n```\n [Static Grad] embed, all_attn, all_mlp, head: master_grads = 3.8141040802001953 GiB\n```\n\nThe maximum memory usually occurs in these cases, but maybe different when under different configs such as `chunk_loss_size`.\n"},"files":{".claude/CLAUDE.md":"# XTuner Programming Standards\n\n## Python\n\n### Development Standards\n\n1. When adding methods or functions to a module or class, declare **public interfaces** as early as possible and place **private interfaces** toward the end.\n2. Python **>= 3.10** is required. Use modern syntax accordingly (e.g., `X | Y` union types).\n3. Use `{}` instead of `dict()` for creating dictionaries. Use `[]` instead of `list()` for creating lists.\n4. Use **double quotes** (`\"`) for strings instead of single quotes (`'`).\n5. Use `TypedDict` with `{}` syntax when a parameter requires a typed dictionary.\n6. Prefer `pathlib.Path` over `os.path` for filesystem operations.\n7. All new code must include **type hints** for function signatures (parameters and return types).\n8. Maximum line length is **119** characters (configured in `pyproject.toml`).\n\n### Docstring Standards\n\n1. Use **Google Style** docstrings.\n2. For classes, include `__init__` documentation directly under the **class docstring**; do not repeat it under the `__init__` method.\n3. Parameter types **must** be declared in docstrings — include the type hint in parentheses immediately following the parameter name.\n4. The **return type** must be explicitly declared in the docstring.\n5. Do **not** declare raised exceptions in the docstring.\n6. Do **not** declare Attributes in the class docstring.\n7. If the return value is `None`, omit the `Returns` section.\n8. Only provide docstrings for **public methods**. Private methods do not require docstrings.\n\n### Linting, Formatting & Type Checking\n\n- **Ruff** for linting and formatting, **mypy** for type checking — see `pyproject.toml` for configuration.\n- `mypy` is strict for `xtuner/v1/*` (`ignore_missing_imports = false`).\n- Pre-commit hooks apply **only** to `xtuner/v1/` and `autotest/` — run `pre-commit run --all-files` before submitting a PR.\n- Docstrings are wrapped at **119** characters by `docformatter`.\n\n### Error Handling\n\n1. Never use bare `except:` — always catch specific exception types.\n2. Avoid silencing exceptions with `pass` unless there is a documented reason.\n\n### Testing\n\n1. Every bug fix PR **must** include a regression test that reproduces the original bug.\n2. Test functions must be named `test_` and grouped in classes named `Test`.\n3. Use `pytest` as the test framework. Prefer `pytest.raises` for exception assertions.\n4. Tests should be self-contained — avoid reliance on external services or GPUs unless explicitly marked with `@pytest.mark.gpu`.\n\n---\n\n## Git\n\n### Branch Naming\n\n- Bug fixes: `fix/`\n- Features: `feat/`\n- Refactors: `refactor/`\n\n### Commit Message Standards\n\n1. Follow the style of previous commit messages to maintain a **consistent** project history.\n2. Use the format: `[Tag] Short description` (e.g., `[Fix] Fix OOM in MoE expert routing`).\n3. Common tags: `[Fix]`, `[Feature]`, `[Refactor]`, `[CI]`, `[Docs]`, `[Test]`.\n4. **Never** include metadata or watermarks unrelated to the code itself, such as \"generated by AI\" or \"Co-Authored-By: Claude.\"\n\n---\n\n## Pull Request Standards\n\n### Bug Fix PRs\n\n1. **Title**: `[Fix] `\n2. **Body** must include:\n - **Root Cause**: A clear explanation of why the bug occurred.\n - **Fix**: What was changed and why this is the correct fix.\n - **Test Plan**: How the fix was verified (unit test, manual reproduction, etc.).\n3. Keep the diff **minimal** — fix only the bug. Do not refactor surrounding code, add unrelated improvements, or change formatting of untouched lines.\n4. If the fix touches performance-critical paths (data loading, forward/backward pass, communication), include **before/after** benchmarks or justify why benchmarks are unnecessary.\n\n### General PR Rules\n\n1. One logical change per PR. Do not mix bug fixes with features or refactors.\n2. All CI checks must pass before requesting review.\n3. Resolve all review comments before merging — do not dismiss without explanation.\n\n---\n\n## Code Review Standards\n\n### What to Check\n\n1. **Correctness**: Does the code do what it claims? Are edge cases handled?\n2. **Concurrency & Distributed**: Check for race conditions, deadlocks, incorrect `dist.barrier()` placement, and mismatched collective operations across ranks.\n3. **Memory**: Watch for GPU memory leaks — tensors not freed, caches growing unbounded, missing `torch.no_grad()` in inference paths.\n4. **Numerical Stability**: Flag potential issues like `log(0)`, division by zero, `float32`/`float16` overflow in loss computation or gradient scaling.\n5. **Performance**: Unnecessary CPU-GPU synchronization (`item()`, `.cpu()` in hot paths), redundant data copies, inefficient collective communication patterns.\n6. **API Contracts**: Do public interface changes maintain backward compatibility? Are deprecation warnings added for breaking changes?\n7. **Resource Cleanup**: Are file handles, NCCL communicators, and CUDA streams properly cleaned up?\n\n### Area-Specific Review Rules\n\nIf a PR changes code under `xtuner/v1/rl`, read `.claude/rules/rl_review.md` before commenting or\nposting the final review summary.\n\n### Review Output Format\n\nWhen performing code reviews, use the following structure:\n\n```\n## Summary\n<1-2 sentence overview of the change>\n\n## Issues\n### Critical\n- [file:line] Description of the issue and suggested fix\n\n### Warning\n- [file:line] Description of the concern\n\n### Nit\n- [file:line] Minor style or readability suggestion\n\n## Verdict\nAPPROVE / REQUEST_CHANGES / COMMENT\n```\n\n### Review Principles\n\n1. **Be specific** — always reference the exact file and line number.\n2. **Explain why** — don't just say \"this is wrong\"; explain the failure mode.\n3. **Suggest a fix** — provide a concrete code suggestion when possible.\n4. **Distinguish severity** — separate critical bugs from style nits.\n5. Do not flag issues in code that is **outside the scope** of the PR diff.\n\n\n\n## Rules\n\nPlease refer to the docs in `.claude/rules` for the development guidelines.\n",".claude/rules/datasets_architecture.md":"---\npaths:\n - \"xtuner/v1/datasets/**/*\"\n---\n\n# XTuner v1 Datasets Architecture Rules\n\n## Core Data Contract (DataItem Protocol)\n\nEvery dataset layer must produce/consume `DataItem`:\n```python\n{\n \"input_ids\": list[int], # tokenized token IDs\n \"labels\": list[int], # same length; first token masked to -100\n \"num_tokens\": int, # len(input_ids); used for sampler and packing\n}\n```\n\n- `labels[0] = -100` always (pretrain: first token not predicted)\n- `HardPackDataset.__getitem__` returns `list[DataItem]` (one per source sample in pack)\n- All other datasets return a single `DataItem`\n\n## Tokenization Functions\n\n**Standard pretrain**: `PretrainTokenizeFunction`\n- Accepts: `{\"content\": \"text\"}` or `{\"messages\": [{\"role\": \"pretrain\", \"content\": \"text\"}]}`\n- Hash: `\"{tokenizer_hash}_{source_hash}_{add_bos_token}_{add_eos_token}\"`\n\n**Long text pretrain**: `LongTextPretrainTokenizeFunction`\n- Two-phase: cache phase returns chunk metadata, runtime phase tokenizes specific char ranges\n- Parameters: `chunk_size`, `tokenizer_chunk_chars`, `overlap_chars`, `min_chunk_tokens`\n- BOS only at `char_start==0`, EOS only at document end\n- Hash includes all chunking parameters\n\n## JsonlDataset Architecture\n\n- **Shared memory**: entire file loaded once at init\n- **Offset table**: `np.ndarray` for O(1) line access\n- **Chunked mode**: activated when `chunks.npy` exists in cache\n - Expands `sampled` to per-chunk entries\n - `__getitem__` calls `tokenize_fn(data, char_start=cs, char_end=ce)`\n - `max_length` filter skipped (would incorrectly discard valid long-text samples)\n - `__len__` returns chunk count (not line count)\n\n## Cache Directory Layout\n\n```\ncache_dir/\n .xpuyu-cache-meta.json\n {file_xxhash}/\n offsets.npy\n {tokenize_fn_hash}/\n num_tokens.npy # always present\n chunks.npy # only for LongText mode: (total_chunks, 4)\n # [line_idx, char_start, char_end, num_tokens]\n```\n\nCache key: `(file xxhash, tokenizer hash, tokenize_fn source hash)`\n- Tag-based lookup via `cache_tag` on `DatasetConfig`\n- Source hash auto-invalidates on code change\n\n## Packing Strategies\n\n**HardPackDataset**:\n- Static pack_infos computed at init (deterministic from seed)\n- Returns `list[DataItem]` per pack\n- `global_pack=False`: per-dataset packing (default)\n- `global_pack=True`: concat all datasets then pack\n\n**PresetPackDataset**:\n- User-provided pack config (NPY directory + `paths.json`)\n- Validates structure and token ranges via vectorized numpy\n- Strategies for length mismatch: error/padding (short), error/truncate (long)\n- No state (deterministic from config file)\n\n## Samplers\n\n**LengthGroupedSampler**:\n- Megabatch logic: group similar lengths, sort DESC, shuffle chunks\n- State: `{epoch, step, world_size, num_samples, total_size}`\n- `step` = `consumed_samples % total_size`\n\n**PresetSampler**:\n- User-provided order via `sampler_config_path` (`.npy` path only, mmap read)\n- Round-down to `global_batch_size * world_size` multiple (truncate tail; error if length too short)\n- Per-rank slice: `effective[local_rank::world_size]` on the truncated mmap-backed view (`self.global_order`)\n- `pack_level=\"preset\"` in `DataloaderConfig`; requires `pack_config_path` and `sampler_config_path`\n- State dict compatible with existing resume logic\n\n## Resume Contract\n\n```python\n# Save\nstate = dataloader.get_state_dict(consumed_samples)\n# → {\"sampler\": {...}, \"dataset\": {}}\n\n# Restore\ndataloader.load_state_dict(state)\n```\n\n- `consumed_samples` = total globally-seen samples (across all ranks)\n- Sampler converts to per-epoch `step` by modulo\n- Packing datasets have no state (deterministic)\n\n## Multi-Dataset Support\n\n- `DatasetConfigList` = list of `{dataset, tokenize_fn}` pairs\n- `build_datasets()` returns `list[JsonlDataset]` (1 file → 1 dataset)\n- `sample_ratio` applied per-DatasetConfig via `sampled[]` repetition\n\n## Key Implementation Files\n\n| File | Purpose |\n|------|---------|\n| `config.py` | Pipeline orchestration via `DataloaderConfig.build()` |\n| `pt_tokenize_fn/text.py` | `PretrainTokenizeFunction` |\n| `pt_tokenize_fn/long_text.py` | `LongTextPretrainTokenizeFunction` |\n| `jsonl.py` | `JsonlDataset` with shared memory + offset table |\n| `packing.py` | `HardPackDataset`, `ExpandSoftPackDataset` |\n| `sampler.py` | `LengthGroupedSampler`, `ParallelSampler` |\n| `preset_pack.py` | `PresetPackDataset` |\n| `preset_sampler.py` | `PresetSampler` |\n| `dataloader.py` | `Dataloader` wrapper |\n| `data_item.py` | `DataItem` schema |\n\n## Design Principles\n\n1. **Protocol-based**: All layers communicate via `DataItem` contract\n2. **Cache-aware**: Tokenization cached with automatic invalidation\n3. **Deterministic**: Packing and sampling reproducible from seed/config\n4. **Resumable**: State dict pattern for checkpoint/resume\n5. **Composable**: Tokenize → Dataset → Pack → Sample → Collate pipeline\n",".claude/rules/rl_review.md":"# RL Review Guidelines\n\nUse these rules whenever a PR changes code under `xtuner/v1/rl`. They extend the general review\nstandards in `.claude/CLAUDE.md`; they do not replace them.\n\n## Required Review Output\n\nEvery review for an RL change must state the ProduceBatchResult impact near the top-level summary:\n\n```text\nProduceBatchResult impact: \n```\n\nAlso include routed-experts impact when the PR touches routed-experts logic, rollout response\nhandling, `RolloutState.extra_fields`, object references, or memory ownership around rollout outputs:\n\n```text\nRoutedExperts impact: \n```\n\nAlso include Ray concurrency impact when the PR touches Ray actor methods, actor construction,\nmethod decorators or wrappers, concurrency-group definitions, or rollout control paths:\n\n```text\nRay concurrency impact: \n```\n\nDo not leave these impacts implicit in the finding text. If a finding is about rollout status,\npause/abort/timeout behavior, response handling, producer aggregation, routed-experts ownership, or\nRay actor concurrency, repeat the relevant impact line inside that finding so the downstream effect\nis visible at the point of review.\n\n## ProduceBatchResult Checklist\n\nReview whether the PR can change trainer-visible batch accounting, timing, or reward semantics in\n`ProduceBatchResult`.\n\nCheck this area when the PR changes any of these paths:\n\n- `RolloutState.status`, `finish_reason`, or status conversion.\n- Abort, filter, expire, retry, timeout, cancellation, or failure handling.\n- Producer, agent loop, judger, replay buffer, rollout worker, rollout controller, or backend pause\n cleanup logic.\n- Writers, readers, tests, or fake implementations that construct or consume `ProduceBatchResult`,\n including trainer-facing code outside `xtuner/v1/rl`.\n\nName the concrete field-level impact when any of these fields can change:\n\n- Batch status: `status`.\n- Returned groups: `rollout_states`.\n- Generation timing: `group_gen_count`, `group_gen_mean_s`, `group_gen_p50_s`, `group_gen_p99_s`,\n `group_gen_p99_p50_ratio`, `group_gen_pause_time_s`.\n- Replay-buffer leftovers: `leftover_init`, `leftover_completed`, `leftover_aborted`,\n `leftover_expired`, `leftover_failed`, `leftover_filtered`.\n- Reward accounting: `raw_rewards_sum`, `raw_rewards_count`.\n- Produced work counters: `produced_samples`, `produced_tokens`, `produce_time_s`.\n- Multi-task aggregation: `task_batch_sizes`, `task_results`.\n\nCommon impacts to call out explicitly:\n\n- A sample moving between `ABORTED`, `FAILED`, `FILTERED`, `EXPIRED`, and `COMPLETED` can change the\n corresponding `leftover_*` counts.\n- Pause, abort, timeout, and cancellation changes can inflate or deflate `group_gen_*` timing,\n especially `group_gen_pause_time_s`.\n- Reward or filter-path changes can change `raw_rewards_sum`, `raw_rewards_count`,\n `produced_samples`, and `produced_tokens`.\n\n## Ray Actor Concurrency Checklist\n\nReview whether high-volume rollout calls can starve pause, abort, health-check, restart, or shutdown\nRPCs.\n\nCheck this area when the PR changes any of these paths:\n\n- Ray actor methods decorated with `@ray.method(...)` or `@ray_method(...)`.\n- Decorators or wrappers stacked on Ray actor methods, especially wrappers using `functools.wraps`\n or exposing `__wrapped__`.\n- Actor construction through `ray.remote(...)`, including `concurrency_groups` and\n `max_concurrency`.\n- Generate, pause, continue, abort, health-check, restart, or shutdown methods.\n\nWhen a Ray method is combined with another decorator, keep the Ray method decorator closest to the\nfunction definition:\n\n```python\n@trace_rollout_endpoint(\"rollout.worker.generate\")\n@ray.method(concurrency_group=ROLLOUT_CONCURRENCY_GROUP_GENERATE)\nasync def generate(...):\n ...\n```\n\nDo not use the reverse order:\n\n```python\n@ray.method(concurrency_group=ROLLOUT_CONCURRENCY_GROUP_GENERATE)\n@trace_rollout_endpoint(\"rollout.worker.generate\")\nasync def generate(...):\n ...\n```\n\nThe current Ray actor-class processing unwraps decorated methods before reading Ray method metadata.\nIf `@ray.method(...)` is outside a wrapper, metadata such as `__ray_concurrency_group__` can remain\non the discarded wrapper, causing the method to fall back to the default concurrency group.\n\nReview the effective dispatch behavior, not only the visible decorator order:\n\n- After following `__wrapped__`, the callable inspected by Ray must retain the expected Ray method\n metadata.\n- Every named method concurrency group must be declared by the corresponding `ray.remote(...)`\n actor construction.\n- High-volume `generate` methods must not accidentally fall back to the default group.\n- Pause, abort, and other control RPCs must remain schedulable while generation is saturated.\n- Slow health or lifecycle methods must not occupy every slot needed by critical control RPCs.\n- Blocking work inside async actor methods must not block the actor event loop or concurrency lane.\n\n## RoutedExperts Checklist\n\nReview whether routed-experts ownership and cleanup remain correct.\n\nCheck this area when the PR changes any of these paths:\n\n- LMDeploy rollout response handling.\n- `return_routed_experts`, `routed_experts`, `RolloutState.extra_fields`, or object-ref plumbing.\n- Abort, cancellation, timeout, filter, retry, or failure paths before response handling completes.\n- Background tasks, replay-buffer storage, trainer batches, metrics, tests, or fake rollout\n responses that can retain routed-experts object refs.\n\nFor LMDeploy rollout, `rollout_worker` obtains routed-experts object refs from the LMDeploy shared\nstore. At that point, ownership moves from LMDeploy to XTuner. A review finding should call out both\nsides of the ownership boundary when relevant:\n\n- Leak before transfer: requests whose routed experts remain in LMDeploy because XTuner never\n obtains the object refs.\n- Leak after transfer: object refs that XTuner keeps alive too long through `RolloutState`, replay\n buffer, trainer batches, metrics, fake tests, or background tasks.\n",".claude/skills/add-chat-template/SKILL.md":"---\nname: add-chat-template\ndescription: Add or audit an XTuner chat-template implementation and its loss mask from an official Hugging Face model repository or local model directory. Use when supporting a new model under xtuner/v1/data_proto/messages, checking an existing implementation against tokenizer or processor apply_chat_template, deciding which assistant output and stop/EOS tokens receive loss, adding regression tests, or validating that exported training behavior matches vLLM and SGLang inference.\n---\n\n# Add Chat Template\n\n## Goal\n\nTreat the official Hugging Face tokenizer, or processor for a multimodal model,\nas the rendering oracle. Produce the same conversation token sequence in\nXTuner, supervise model-generated output by default, and supervise the\ncontext-appropriate token that ends each assistant generation. Prove the\nbehavior through XTuner's public tokenize API and, when available, the target\nvLLM and SGLang versions.\n\nKeep the implementation small and model-local. Do not infer a template or stop\ncontract from another model in the family.\n\n## 1. Establish the reference\n\n1. Record the official HF repo or local directory, immutable revision, tokenizer\n class, Transformers version, and whether `trust_remote_code=True` is required.\n2. Inspect the actual artifacts used by inference:\n `tokenizer_config.json`, `chat_template.jinja` or `chat_template`,\n `special_tokens_map.json`, `generation_config.json`, and `config.json`.\n Inspect `processor_config.json` and remote processor/template code when the\n official multimodal path uses `AutoProcessor.apply_chat_template`.\n3. Run the bundled inventory against the same revision:\n\n ```bash\n python .claude/skills/add-chat-template/scripts/audit_hf_chat_template.py \\\n --trust-remote-code\n ```\n\n Omit `--trust-remote-code` unless the official model requires it. For\n model-specific branches such as tools, thinking, or multimodal inputs, pass a\n JSON case file with `--cases `; see the script's `--help` output.\n The script inventories the tokenizer side; for a processor-owned multimodal\n template, run the corresponding `AutoProcessor` renderer separately.\n4. Render every supported branch with the same `tokenizer.apply_chat_template`\n or `processor.apply_chat_template` entry point used by the official example.\n For multimodal models, do not assume tokenizer-only rendering reproduces\n media placeholders or processor expansion. Record all required template\n kwargs and their defaults, including\n `add_generation_prompt`, `continue_final_message`, tool definitions,\n thinking/reasoning flags, named templates, and multimodal options. Do not\n copy rendered strings from a model card when the executable tokenizer is\n available.\n5. When the selected official template contains Jinja `{% generation %}`\n regions, call its `apply_chat_template` with `tokenize=True`,\n `return_dict=True`, and\n `return_assistant_tokens_mask=True`. Use its assistant mask as an additional\n oracle, then separately audit stop boundaries: a next-role stop token can sit\n outside the official generation region while still belonging to the previous\n assistant for SFT.\n6. Pin the reference revision in tests or CI configuration. A moving HF branch\n is not a stable oracle.\n\n## 2. Audit before implementing\n\nSearch these integration points first:\n\n- `xtuner/v1/data_proto/templates/__init__.py`\n- `xtuner/v1/data_proto/messages/`\n- `xtuner/v1/data_proto/messages/__init__.py`\n- `xtuner/v1/datasets/sft_tokenize_fn/openai.py`\n- multimodal tokenize functions when applicable\n- `tests/chat_template/` and `tests/datasets/`\n\nIf the model already exists, run its public tokenize path before changing it:\n\n```python\ntokenize_fn = OpenaiTokenizeFunctionConfig(chat_template=\"\").build(tokenizer)\nresult = tokenize_fn({\"messages\": messages, \"tools\": tools})\n```\n\nCompare the result with the official renderer and inspect existing tests. Find a\nconcrete failing message sequence before fixing a mismatch. Do not rewrite a\nworking implementation because its structure looks unusual.\n\nUse `HybridChatTemplate` plus `ChatMessages` only when fixed role wrappers fully\nexpress the official template. Add a dedicated\n`xtuner/v1/data_proto/messages/_chat.py` when rendering depends on\nmessage history, tools, reasoning, content types, or the next role. Follow\n`glm52_chat.py` and `qwen35_chat.py` as integration examples, not as rendering\nspecifications for another model.\n\n## 3. Write the stop contract first\n\nInventory three distinct sources; do not collapse them into one `eos_token`:\n\n1. **Configured generation stops**: `tokenizer.eos_token_id`, `config.json`,\n `generation_config.json`, including integer lists and stop strings.\n2. **Template boundaries**: the exact token or token sequence following an\n assistant when the next item is user, tool/observation, system/developer, or\n the end of the sample.\n3. **Engine stops**: the IDs and strings actually installed by the target vLLM\n and SGLang versions after loading the exported checkpoint.\n\nCreate an explicit table before writing the mask:\n\n| Transition | Official serialized boundary | Token IDs | Engine mechanism | Loss when assistant loss is true |\n|---|---|---|---|---|\n| assistant → user | exact role/EOT sequence | exact IDs | EOS ID, stop ID/string, or parser | yes for the generated stop target |\n| assistant tool call → tool | exact observation boundary | exact IDs | EOS ID, stop ID/string, or parser | yes for the generated stop target |\n| final assistant → end | exact EOT/EOS, or documented training-only EOS | exact IDs | EOS ID or stop ID/string | yes |\n\nVerify each token by both encoding the exact boundary and converting its ID back\nto a token. Reject unknown-token conversions. Handle a multi-token stop as a\nsequence rather than assuming every stop is one special token.\n\n### Role tokens can belong to the preceding assistant\n\nSome official templates have no dedicated per-turn EOT. The next role token is\nthen the token the model must generate to stop the previous assistant. In that\ncase, keep the official serialized order and assign loss only on that boundary\ntoken or sequence to the preceding assistant:\n\n```text\nassistant answer user text\n ^ assistant loss ^ masked\n```\n\nFor a GLM-like protocol this can mean supervising the user boundary after a\nnormal answer, the observation boundary after a tool call, and the configured\nterminal EOS only when no following role supplies a boundary.\n\nDo not append a generic EOS after every assistant when the official inference\ntemplate places a role/observation boundary there. That creates training history\nthat vLLM and SGLang do not render. Conversely, when the official full-message\nrenderer omits the final token that the model must generate to terminate,\nappend exactly that one terminal token for SFT and document this single\ntraining-only suffix.\n\nConfigured EOS IDs are alternatives, not a sequence to append together.\nSupervise the actual context-specific stop token present in each training path.\n\n## 4. Implement rendering and loss together\n\nPrefer one cohesive renderer that returns rendered text plus a character- or\ntoken-level loss mask. Keep context-dependent boundary ownership beside the\nrendering branch that emits the boundary; avoid a collection of shallow helper\nfunctions.\n\nApply these defaults:\n\n- Treat `assistant` messages with no `loss` field as `loss=True`.\n- Mask system/developer/user/tool inputs and role scaffolding unless the stop\n contract proves that a boundary is generated by the preceding assistant.\n- Supervise all model-generated assistant components: visible content,\n reasoning that remains in the rendered sample, reasoning closing syntax,\n tool calls, and the appropriate stop target.\n- Treat `loss=False` as masking the entire assistant output and its stop target.\n- Mask `add_generation_prompt` completely; it is prompt scaffolding, not a\n demonstrated model output.\n- Preserve explicit model behavior such as clearing historical reasoning only\n when the official template does so for the chosen kwargs.\n\nIf character offsets are used, verify special-token and Unicode boundaries with\nthe real tokenizer. A token overlapping a loss span needs a deliberate,\ntested rule. Keep a slow, independent token-level oracle for regression tests;\ndo not call the production renderer from both sides of the same parity test.\n\nMaintain exact rendering parity before any documented training suffix:\n\n```mermaid\nflowchart LR\n A[\"Official HF messages\"] --> B[\"tokenizer.apply_chat_template\"]\n A --> C[\"XTuner public tokenize API\"]\n B --> D[\"Exact text and token IDs\"]\n C --> D\n D --> E[\"Optional one terminal SFT stop\"]\n E --> F[\"Loss-mask contract\"]\n```\n\nDo not use decoded-text equality as the sole check: decoding can hide a token-ID\nmismatch.\n\n## 5. Register the smallest integration\n\nFor a dedicated message implementation, update only the required seams:\n\n1. Export the message class from `xtuner/v1/data_proto/messages/__init__.py`.\n2. Add the public name and serving stop metadata to `CHAT_TEMPLATE_MAP`.\n3. Dispatch that name in `OpenaiTokenizeFunction`; update multimodal dispatch\n only if the model uses it.\n4. Update typed CLI/config literals only where they restrict the new name.\n\nKeep `CHAT_TEMPLATE_MAP.stop_words` consistent with the discovered contract,\nbut do not treat it as proof of engine behavior. vLLM and SGLang load their\nruntime stops from the exported HF artifacts and request/engine configuration.\n\n## 6. Add good regression tests\n\nUse the real official tokenizer/processor and XTuner's public tokenize\nfunction. Mock only unavailable external services. Cover every supported\ntemplate branch with the smallest useful matrix:\n\n- user → assistant, with `loss` omitted;\n- assistant → user in a multi-turn conversation;\n- assistant tool call → tool/observation → assistant;\n- final assistant;\n- `loss=False` followed by another role and at sample end;\n- system/developer input;\n- `add_generation_prompt=True`;\n- reasoning enabled/disabled and historical reasoning behavior;\n- tools and multimodal content when supported;\n- repeated assistant text, Unicode, and adjacent special tokens to expose\n faulty substring or offset alignment.\n\nAssert public behavior:\n\n1. Official tokenizer/processor rendered text equals XTuner rendered text,\n except for the one explicitly documented final SFT suffix if required.\n2. Official token IDs equal XTuner `input_ids` before that suffix.\n3. `len(input_ids) == len(labels)` and every label is either the matching input\n ID or `IGNORE_INDEX`.\n4. Assistant output receives loss by default; `loss=False` removes all of it.\n5. Each transition's exact stop token IDs receive or do not receive loss as the\n stop table specifies.\n6. User/tool content and generation prompts remain masked.\n7. The fast implementation matches an independent slow oracle.\n\nTest token positions and IDs directly. A substring assertion can supplement but\nmust not replace the loss assertion. Avoid snapshots that pass after both the\nrenderer and expected string are changed to the same incorrect value.\n\n## 7. Validate vLLM and SGLang\n\nResolve exact target versions. Prefer the user's production versions; otherwise\ninspect the current installed versions and latest stable official source. For\neach engine:\n\n1. Load the same exported tokenizer, `config.json`, and\n `generation_config.json`.\n2. Compare the engine's chat-render/tokenize output with the official\n tokenizer/processor renderer for every stop-table transition.\n3. Inspect or log the final stop token IDs/strings installed in the generation\n request. Distinguish tokenizer EOS, model generation-config EOS, request stop\n strings, and model-specific parsers.\n4. Run generation smoke tests for a normal final answer and a tool call. Use a\n sufficiently high output limit and assert the engine reports a stop rather\n than a length limit, with no role leakage.\n\nObtain the repository GPU lock before a local GPU server or generation test. If\nthe exact runtime cannot be executed, audit the exact tagged engine source and\nreport the result as a static source audit, not runtime compatibility.\n\nSwitching from Chat Completions to Responses does not remove this requirement:\nfor ordinary HF models both interfaces ultimately render model messages through\nthe chat template.\n\n## 8. Completion checklist\n\nFinish only when the report includes:\n\n- official HF repo, revision, and Transformers version;\n- implementation path and whether it is generic or dedicated;\n- rendering-parity matrix;\n- stop-contract table with token strings, IDs, and loss ownership;\n- tests proving default assistant loss and `loss=False`;\n- vLLM/SGLang versions and runtime or exact-tag source validation;\n- every intentional difference from the official renderer, normally at most a\n final training stop suffix.\n\nIf auditing an existing implementation, state which requirements already had\ntests, which were missing, and the concrete mismatches found. Do not claim the\nimplementation is correct merely because a test file exists.\n",".claude/skills/add-chat-template/agents/openai.yaml":"interface:\n display_name: \"Add Chat Template\"\n short_description: \"Add and verify XTuner chat templates\"\n default_prompt: \"Use $add-chat-template to add or audit XTuner chat-template rendering, loss masks, and stop-token behavior for this official Hugging Face model.\"\n",".claude/skills/add_hf_model/SKILL.md":"---\nname: add_hf_model\ndescription: >\n Add support for a HuggingFace model in XTuner so it can be trained with the\n full set of parallelism strategies and optimization switches. The user can\n provide a HuggingFace Hub repo id, a local model directory, or a model family\n already supported by `transformers`. This skill walks through the full path:\n locating the reference implementation, classifying the model, splitting new\n code across the model / module / ops layers, implementing the XTuner model\n class and config, registering the entry point, validating bitwise numerical\n parity with HuggingFace, writing regression tests, and enabling the training\n optimizations (EP/SP, micro-batch, torch.compile, fp8, activation offload).\n---\n\n# Add a HuggingFace model to XTuner\n\nA complete handbook for porting any HuggingFace causal LM into XTuner with\ncorrect training behavior. Covers four buckets: **Dense LLM**, **MoE LLM**,\n**VLM / compose model**, and **`trust_remote_code` model**. Every step below is\ngrounded in concrete `file:line` references in this repository — read the cited\ncode before writing yours.\n\n> **Anti-spaghetti reminder** — XTuner separates *what the model is* (`*Config`),\n> *what it computes* (`Dense` / `MoE` / `BaseComposeModel` subclass), and *how its\n> weights map to HF safetensors* (`to_hf_key_list`, optional `safetensors_to_params`\n> / `param_to_safetensor`). Keep new code inside those three seams. **Never** add\n> `if model_name == \"...\"` branches to the base classes.\n\n---\n\n## 0. Locate the source code\n\nThe only goal of this step is to find the **HuggingFace reference implementation\nyou will port from**, and to note the `model_type` you will register under. Do\nnot pick attention/router/layer here — that happens naturally in §1 as you read\nthe code.\n\n1. If the user gives a **remote-code** model, the modeling code ships in the repo\n as `modeling_*.py` — read it directly.\n2. If the user gives **weights only**, confirm the target Python environment with\n the user, then find the matching implementation inside that env's\n `transformers` install. If no code exists for that `model_type`, stop and ask\n the user.\n3. Confirm the weights are **training-suitable** — at least bf16 — so training\n precision can be aligned later. Flag fp8/int4-only checkpoints to the user.\n4. Note two strings from the HF config — you will need both downstream:\n ```python\n from transformers import AutoConfig\n cfg = AutoConfig.from_pretrained(path, trust_remote_code=True)\n cfg.model_type # registration key (§4)\n cfg.architectures[0] # the modeling class you are mirroring\n ```\n\n---\n\n## 1. How to do the migration\n\nXTuner splits a model into three layers. You port by reusing what already exists\nat each layer and only adding what is genuinely new — so the work is \"read the HF\nmodeling code, then match each of its pieces to the XTuner component below\". The\nlocations are listed plainly; you do not need extra guidance to map your model's\nattention / router / layer onto the matching one.\n\n- **model layer** (`xtuner/v1/model`) — the model body.\n - Dense body: `xtuner/v1/model/dense/dense.py`\n - MoE body: `xtuner/v1/model/moe/moe.py`\n - Multi-modal (compose) bodies: `xtuner/v1/model/compose/` — reuse the language\n model and add only the ViT / projector pieces.\n- **module layer** (`xtuner/v1/module`):\n - Decoder layers: `DenseDecoderLayer`, `MoEDecoderLayer` —\n `xtuner/v1/module/decoder_layer`. Small changes add a switch on the existing\n layer; a genuinely novel architecture may add `{model_name}_decoder_layer`.\n - Attention configs: `MHAConfig` (`xtuner/v1/module/attention/mha.py`),\n `MLAConfig` (`.../mla.py`, DeepSeek-style latent),\n `GatedDeltaNetConfig` (`.../gated_deltanet.py`, linear).\n - Router configs (MoE): `GreedyRouterConfig`\n (`xtuner/v1/module/router/greedy.py`, softmax top-k — Qwen3 MoE, GptOss),\n `NoAuxRouterConfig` (`xtuner/v1/module/router/noaux_router.py`, grouped sigmoid\n aux-loss-free — DeepSeek V3).\n - Also: rope, rms_norm, etc.\n- **ops layer** (`xtuner/v1/ops`) — kernels such as attention and rms_norm.\n\n### Existing models to copy from\n\nPick the one whose attention + (router) match yours; the closer it is, the\nsmaller your diff.\n\n| Your model is… | Copy from |\n|--------------------------------------|--------------------------------------|\n| Dense LLM | `xtuner/v1/model/dense/qwen3.py` |\n| MoE LLM | `xtuner/v1/model/moe/qwen3.py` |\n| MoE with MLA + aux-loss-free router | `xtuner/v1/model/moe/deepseek_v3.py` |\n| MoE needing weight-layout transform | `xtuner/v1/model/moe/gpt_oss.py` |\n| Compose / VLM | `xtuner/v1/model/compose/qwen3_vl/` |\n\n\nAfter you have a plan, **write an integration design doc into `docs/design/model`**\nand confirm with the user that the decomposition is reasonable before\nimplementing.\n\n### File layout for the new model\n\n| Bucket | Model file | Public re-export | Test file |\n|----------------------|-------------------------------------------------------------|-----------------------------------------------|--------------------------------------------|\n| Dense LLM | `xtuner/v1/model/dense/.py` | `xtuner/v1/model/__init__.py` | `tests/model/test__dense.py` |\n| MoE LLM | `xtuner/v1/model/moe/.py` | `xtuner/v1/model/__init__.py` | `tests/model/test__moe.py` |\n| Compose / VLM | `xtuner/v1/model/compose//...` | `xtuner/v1/model/__init__.py` | `tests/model/test_.py` |\n\n---\n\n## 2. Implement the model class\n\nSubclass `Dense`, `MoE`, or `BaseComposeModel`. You typically only need to\noverride one method.\n\n### 2.1 `to_hf_key_list(self, key: str) -> list[str]` — **mandatory, abstract on `BaseModel`**\n\nDeclared at `xtuner/v1/model/base.py`. Translates an **XTuner-side** parameter\nname into one or more **HF-side** safetensors keys. Return-list cardinality:\n\n- **1 → 1** for plain weights (most params).\n- **1 → N** for fused MoE experts (one fused param explodes into N per-expert\n HF keys; see `xtuner/v1/model/moe/qwen3.py`).\n\nPatterns you almost always need (study `xtuner/v1/model/dense/qwen3.py`\nand `xtuner/v1/model/moe/qwen3.py`):\n\n```python\n# 1. Tied embeddings: redirect lm_head → embed_tokens before adding the model. prefix.\nif self.config.tie_word_embeddings and \"lm_head\" in key:\n key = key.replace(\"lm_head\", \"embed_tokens\")\n\n# 2. Add the \"model.\" prefix that HF wraps everything in.\nif \"layers\" in key or \"embed_tokens\" in key:\n key = \"model.\" + key\n\n# 3. HF MoE nests experts/gate under .mlp.; XTuner does not.\nif \"layers\" in key:\n key = re.sub(r\"layers\\.(\\d+)\\.(experts|gate)\", r\"layers.\\1.mlp.\\2\", key)\n\n# 4. Top-level norm.\nif key.startswith(\"norm.\"):\n return [key.replace(\"norm.\", \"model.norm.\")]\n\n# 5. MoE expert fusion: one fused param → N HF keys.\nif \"fused_w1w3.weight\" in key:\n out = []\n for i in range(self.config.n_routed_experts):\n out.append(key.replace(\"fused_w1w3.weight\", f\"{i}.gate_proj.weight\"))\n out.append(key.replace(\"fused_w1w3.weight\", f\"{i}.up_proj.weight\"))\n return out\n\n# 6. Model-specific buffers (e.g. FoPE: rotary_emb.sin_coef / cos_coef are persistent buffers).\nif key.startswith(\"rotary_emb.\"):\n return [key.replace(\"rotary_emb.\", \"model.rotary_emb.\")]\n```\n\nA prefix-region remap that differs by *deployment* (e.g. a text tower nested under a\nVLM as `model.language_model.`) does **not** belong here — put it in the config's\n`hf_key_mapping`. See §5.1.\n\n### 2.2 `safetensors_to_params` / `param_to_safetensor` — **optional, only for layout mismatches**\n\nIf the HF storage layout cannot be expressed as “same tensor, different name”,\nyou must transform the bytes. Canonical example: GptOss stores expert weights as\n`(num_experts, hidden_size, expert_dim * 2)` but XTuner fuses them as\n`(num_experts * 2 * expert_dim, hidden_size)` — see overrides at\n`xtuner/v1/model/moe/gpt_oss.py`.\n\nUse this hook only when transposing/reshaping is unavoidable. Renames go in\n`to_hf_key_list`; numerics go here.\n\n### 2.3 Do **not** override\n\n- `build_layers`, `build_embeddings`, `_init_weights` — the base handles them\n via the config. Override only if the new model adds a layer type the base\n cannot express (rare).\n- `forward` — same reasoning. If you find yourself touching it, re-evaluate the\n bucket choice; the problem is probably in the config (`attention`, `router`,\n `first_k_dense_replace`).\n\n---\n\n## 3. Implement the config\n\nTwo layers: a **base config** that reads HF and a **size-specific subclass** per\nreleased checkpoint.\n\n### 3.1 Base config — three required members\n\nMirror `Qwen3MoEConfig` (`xtuner/v1/model/moe/qwen3.py`) or\n`Qwen3DenseConfig` (`xtuner/v1/model/dense/qwen3.py`).\n\n1. **`build(self) -> `** — return `(self)`.\n\n2. **`@classmethod from_hf(cls, hf_path) -> Self`** — read the HF config and map\n its fields one-to-one onto your config. The field set is whatever the live\n config class declares (`TransformerConfig` in `xtuner/v1/model/base.py`,\n `MoEConfig` in `xtuner/v1/model/moe/moe.py`); read it, don't work from a\n hard-coded list. A few non-obvious traps:\n - Use `RopeParametersConfig.from_hf_config(hf_config)` from\n `xtuner/v1/module/rope/rope.py`. It already handles both the\n `rope_parameters` dict (HF ≥ 5.2.0) and the legacy `rope_scaling` dict\n (HF 4.57.x), plus YARN / FoPE / mrope special fields. **Do not** parse\n rope yourself.\n - For optional HF fields, use `getattr(hf_config, \"\", )`\n rather than `hf_config.` — older checkpoints drop fields.\n\n3. **`@property hf_config(self) -> | None`** — the inverse of\n `from_hf`: every field `from_hf` reads must be re-emitted so `save_hf`\n round-trips.\n\n**Built-in vs. `trust_remote_code` — the only axis that changes `from_hf` /\n`hf_config`:** a `trust_remote_code` model is structurally still Dense / MoE /\nVLM; the *only* difference is whether `transformers` ships a built-in\n`Config` class. Decide which case you are in and follow the matching column\n— this is the single place that contract is defined.\n\n| | Built-in config (e.g. Qwen3) | `trust_remote_code` (no built-in config) |\n|---|---|---|\n| `from_hf` reads via | `.from_pretrained(hf_path)` | `AutoConfig.from_pretrained(hf_path, trust_remote_code=True)`, then `assert hf_config.model_type == \"\"` |\n| `hf_config` returns | a populated `` re-emitting every field | `None` |\n| `save_hf` weight-map | written from `hf_config` | same |\n| `save_hf` side files | derived from `hf_config` | falls back to copying `config.json` / tokenizer / `*.py` from `self._hf_path` so the dir stays loadable with `trust_remote_code=True`; module-cache test in the matrix below |\n\nCompare `Qwen3MoEConfig.hf_config` (built-in) and `Qwen3MoEFoPEConfig.hf_config`\n(returns `None`), both in `xtuner/v1/model/moe/qwen3.py`. **Do not** invent a\nbuilt-in HF config class for a remote-code model — return `None`.\n\n**Invariants:**\n\n- `from_hf` ↔ `hf_config` must round-trip. The `test_save_hf` test (§7) is the\n enforcement mechanism — every key in the original index must appear in the\n saved index with byte-equal tensors.\n- The config must be **pickleable and side-effect free**. No env vars, no file\n I/O, no `torch.cuda.*` at construction time.\n\n### 3.2 Size-specific subclasses\n\nAdd one subclass per released checkpoint that hard-codes the published\ndimensions. Mirror `Qwen3MoE30BA3Config` / `Qwen3MoE235BA22Config` in\n`xtuner/v1/model/moe/qwen3.py`. Even if only one size ships, define at least one\n— it is the entry point for `get_model_config` and for tests.\n\n---\n\n## 4. Register the entry point\n\nOpen `xtuner/v1/model/__init__.py` — all four edits are short:\n\n1. **Import** — bring in the new config classes.\n2. **Alias** — add `\"\": ()` to `model_mapping`. Lookup\n normalizes case and `-`/`_`, so `\"my-model-7b\"` and `\"my_model_7b\"` are the\n same key.\n3. **Dispatch** — add `elif cfg.model_type == \"\":` to\n `get_model_config_from_hf` returning `.from_hf(model_path)`.\n The catch-all `raise ValueError(...)` must stay last.\n4. **`__all__`** — re-export the new public config names.\n\n**Never** key dispatch on parameter names, file paths, or architecture strings.\nOnly `model_type`.\n\n---\n\n## 5. Compose / VLM specifics\n\nCompose models live under `xtuner/v1/model/compose//` and follow a\nthree-tower pattern. See `xtuner/v1/model/compose/base.py` and\n`xtuner/v1/model/compose/qwen3_vl/qwen3_vl_config.py`.\n\n### 5.1 Config shape\n\n`BaseComposeConfig` (`compose/base.py`) carries three sub-configs:\n\n```python\nvision_config: XTunerBaseModelConfig # e.g. Qwen3VLVisionConfig\nprojector_config: XTunerBaseModelConfig # e.g. Qwen3VLProjectorConfig\ntext_config: XTunerBaseModelConfig # any Dense/MoEConfig subclass\nfreeze_vision: bool = False\nfreeze_projector: bool = False\nfreeze_language: bool = False\n```\n\nReuse existing text-tower configs where possible. `Qwen3VLDense*` delegates to\nthe Dense path via `xtuner/v1/model/dense/qwen3vl_text.py`. **Do not** copy-paste\nthe entire dense/MoE config — compose over them.\n\n**Reused text-tower key prefixes — use `hf_key_mapping`, not `to_hf_key_list`.** A\ntext tower reused inside a VLM keeps its *standalone* `to_hf_key_list` (it emits\n`model.<...>` as if it were a plain LLM). The compose checkpoint nests it one level\ndeeper (`model.language_model.<...>`). Do **not** teach the text tower's\n`to_hf_key_list` about that prefix — that leaks the compose context into a tower that\nmust also work standalone. Set the remap on the **text config** instead:\n`hf_key_mapping = {r\"^model\\.\": \"model.language_model.\"}` (applied in\n`_init_load_spec`). Keep *structural* renames in `to_hf_key_list`; the\ndeployment-dependent prefix region belongs in `hf_key_mapping`.\n\n### 5.2 Model shape\n\n`BaseComposeModel` (`compose/base.py`) builds three sub-models:\n`vision_tower`, `multi_modal_projector`, `language_model`.\n\n- **`from_hf(hf_path)`** — delegates to each sub-model with\n `strict=False` (vision keys are missing from the language-model state dict\n and vice versa) and unions the missing-keys sets.\n- **`save_hf(hf_dir)`** — saves each tower with a distinct prefix\n (`\"model-language\"`, `\"model-vision\"`, `\"model-projector\"`) and merges the\n three `weight_map`s into one `model.safetensors.index.json`.\n\n### 5.3 `hf_config` for VLMs\n\nMany VLMs ship without a stable HF top-level config class; `hf_config` returns\n`None` (`compose/qwen3_vl/qwen3_vl_config.py`). In that case, `save_hf`\nfalls back to copying source files from `self._hf_path` (see §3.1).\n\n### 5.4 Vision-side quirks to watch for\n\n- `attn_impl` selectable per checkpoint: `\"flash_attention\" | \"flex_attention\" |\n \"eager_attention\"`.\n- `deepstack_visual_indexes` (e.g. `[8, 16, 24]`) — auxiliary supervision depths\n on the vision tower; must round-trip.\n- Specialized rope: `rope_type=\"qwen3_vl\"`, `mrope_section=[24, 20, 20]` —\n delegated to `RopeParametersConfig.from_hf_config`.\n- The projector’s `torch.compile` path is enabled on torch ≥ 2.9.1 (see commit\n `f6d74efb`). Tests must validate both compiled and eager outputs match.\n\n---\n\n## 6. Baseline parity — no complex parallelism yet\n\nFirst make the model correct on the **simplest execution path**: single rank, no\nEP/SP, no `torch.compile`, no fp8, no activation offload. Align precision from\nthe inside out — ops → decoder layer → full model — reusing XTuner's own modules\nat every level. Only once this baseline holds do you add the parallel and\noptimization features (§8).\n\n1. **Reuse, don't reimplement.** Use the existing XTuner modules — MHA, MLA,\n GatedDeltaNet, GroupLinear, etc. Do not re-implement them with naive torch.\n\n2. **Route op differences through `XTUNER_HF_IMPL`.** This env var already exists\n (`xtuner/v1/utils/misc.py`) and selects HF-exact ops via per-op selectors:\n `xtuner/v1/ops/attn_imp.py::get_attn_impl_fn` (eager attention),\n `xtuner/v1/ops/rms_norm/__init__.py::get_rms_norm_fn` (native torch rms_norm),\n and `xtuner/v1/ops/gated_deltanet/__init__.py::{get_chunk_gated_delta_rule_fn,\n get_causal_conv1d_fn}` (return the canonical fla / causal_conv1d wrappers HF\n calls instead of XTuner's compile-friendly `torch.library.custom_op` wraps —\n same forward output, but the backward op graph now matches HF). Selectors\n read the env var **live** so a test can toggle it per model instance. If your\n model introduces a new op whose fast path is not bitwise against HF (typical\n for fla-backed linear-attention variants), add its HF-exact branch in the\n corresponding op selector the same way. **You may only patch at the ops\n level** — never take the shortcut of patching the entire\n `XTunerDecoderLayer.forward`.\n\n Pair `XTUNER_HF_IMPL` with `XTUNER_DETERMINISTIC=true`. The latter gates an\n autotune-pin block in `xtuner/v1/__init__.py` that monkey-patches\n `triton.autotune` to lock every kernel to its first config and disable the\n result cache. Without the pin, fla's `@triton.autotune` picks tiling /\n reduction order per kernel and the choice drifts between runs, producing\n 1 ULP per linear-attention layer that the chain rule amplifies to ~1.0 at\n the model boundary (§12). `tests/conftest.py` sets the env var and runs\n `import xtuner.v1` so the patch installs before fla is imported. The patch\n lives on the `xtuner.v1.*` import chain (not in the conftest) because\n `multiprocessing.spawn` children of `MultiProcessTestCase` don't go through\n pytest's conftest — they re-import the test class top-level (which pulls\n `xtuner.v1.model.*`) and the patch must run in those processes too.\n Production training doesn't set the env var, so the block is inert there.\n\n3. **Decoder-layer bitwise parity — forward AND backward, both required.** Build\n one XTuner `DecoderLayer` (one per distinct layer type — e.g. a linear and a\n full layer for a hybrid model) and the matching HF layer, set `XTUNER_HF_IMPL`\n and `XTUNER_DETERMINISTIC`, feed identical inputs, and align **bitwise** — not a\n tolerance. **Backward is not optional**: a `single layer → final norm → lm_head\n → CE loss → loss.backward()` test asserts the **layer output, the loss, and the\n input gradient `dL/dx`** all bitwise-equal to HF — test one full and one linear\n layer separately (reference: `test_decoder_layer_bitwise_parity`). A bitwise\n *forward* does not imply a bitwise backward — the forward values can round\n identically while the backward graph differs (see the fp32-softmax note in §12).\n This is the **primary** path, required at all scales, and the only feasible one\n for large models (it never loads the full model — see \"load only what you use\"\n below). When you hand-build a single layer's\n inputs, the two sides' `causal mask` / `seq_ctx` must match exactly — see §12.\n **Load only what you use** so the test fits any model size: build the XTuner\n tower on `meta` and materialize (`to_empty` + selective load) *only* the tested\n layer + `norm` + `lm_head` via `HFCheckpointLoader.load(key)` (it reads just the\n needed safetensors shard); build a *standalone* HF `DecoderLayer` + norm +\n lm_head and load the same keys. Never `from_pretrained` the whole model for a\n single-layer test. (`build_rotary_embedding` builds the rotary on CPU with real\n buffers even under meta, so `model.rotary_emb` is usable without materializing\n the layers.) For < 40B you may also read per-layer hidden states from one\n full-model forward (HF `output_hidden_states` + XTuner `return_hidden_states`);\n mind the off-by-one in §12.\n\n4. **Model config matches HF hyperparameters** — see the §3.1 built-in vs.\n remote-code contract for how `from_hf` / `hf_config` differ between the two.\n\n5. **Full-model forward parity for models < 40B**, bitwise against HF, on the\n single-rank path (FSDP on or off; no EP / compile / fp8 yet).\n\n6. **`save_hf` / `from_hf` correctness** (round-trip; see §7).\n\n7. **Loss-convergence trace + engine test.** Once model forward and decoder-layer\n forward/backward are aligned, record a loss-convergence trajectory and add an\n engine test case.\n\n**Commit the baseline — do not stop to ask.** Committing is part of this\nworkflow: invoking this skill authorizes the in-workflow commits. Once parity\nholds on the simple path and lint/tests pass, commit immediately so this correct\nbaseline is preserved before any parallel/optimization work begins (the §8\nsub-agents branch off this commit). Commit each logical step as you finish it\n(§10) rather than batching everything to the end or waiting for a separate\ngo-ahead.\n\nReport back to the user: whether bitwise parity was achieved, and the residual\nerror (decoder-layer level and model level) once XTuner's internal components are\nused.\n\n---\n\n## 7. Baseline tests — add these first\n\nBefore any parallel feature, add the most basic regression tests so the baseline\n(§6) is locked and the §8 work has a fixed reference to measure against. Add\n`tests/model/test__.py`, mirroring\n`tests/model/test_qwen3_moe.py` (MoE) or `tests/model/test_qwen3_dense.py`\n(Dense). All tests extend `xtuner._testing.DeterministicDDPTestCase` for\ndeterministic distributed setup.\n\nReference test cases:\n- decoder-layer bitwise parity + save_hf round-trip: `tests/model/test_qwen3_5_dense.py`\n (hybrid linear/full dense VLM; toggles `XTUNER_HF_IMPL` per case)\n- model forward & save & load parity: `tests/model/test_qwen3_moe.py`\n- engine training test: `tests/engine/test_moe_train_engine.py`\n\n### 7.1 Baseline test matrix\n\nAll single execution path — no EP / dispatcher / compile / fp8 here; those are\n§8 tests, written against this baseline.\n\n| Case | Dense | MoE | Compose | Notes |\n|-------------------------------------------------|:-----:|:---:|:-------:|------------------------------------------------------------------------------------|\n| Decoder-layer parity — output + loss + `dL/dx` | ✅ | ✅ | ✅ | **Required at all scales, bitwise.** Test one full and one linear layer *separately*. Standalone & memory-light: build the XTuner tower on `meta` and `materialize_xtuner_submodule` only {tested layer, `norm`, `lm_head`}; build a standalone HF `DecoderLayer`; load via `HFCheckpointLoader`. Under `XTUNER_HF_IMPL` assert the **layer output, loss, and `dL/dx` all bitwise** (forward AND backward at the layer level — a bitwise forward ≠ bitwise backward, see §12). Reference: `test_decoder_layer_bitwise_parity`. |\n| Whole-model forward + backward parity (by scale) | ✅(4B)| scale | ✅(4B) | Run the **top-level model** end-to-end vs HF, eager both sides (`XTUNER_HF_IMPL` + `XTUNER_DETERMINISTIC` + HF `eager`). For a **VLM** that's the compose model on an **image** prompt (vision + projector + text — subsumes \"VLM forward parity\"); for a plain LLM, the text/LM model. With both env switches on plus the autotune pin loaded (§6.2), **forward AND backward are bitwise**: assert logits + loss + `dL/d(pixel_values)` / `dL/d(inputs_embeds)` all `== 0.0`. **Scale**: small = fwd+bwd; medium = forward only; large (can't e2e-forward) = skip → integration. Reference: `test_vl_forward_parity`. |\n| Vision-tower bitwise parity | — | — | ✅ | **Compose/VLM only.** Compare HF's `visual` output (`pooler_output`, which is **post-merger**) bitwise (0.0), both eager. XTuner splits that boundary into `vision_tower` (patches → merged hidden) + `projector` (the merger MLP → text dim), so load **both** standalone to match it — don't build the whole VLM just for the vision path. Build them on the **real device, not `meta`**: the vision rotary is computed into a buffer at forward time, so `to_empty` would leave garbage (unlike the text rotary, which `build_rotary_embedding` builds real even under meta — §6.3). Vision attention is *non-causal* → `eager_attention(causal=False)`, which `XTUNER_HF_IMPL` selects. Reference: `test_vision_tower_bitwise_parity`. |\n| FSDP forward parity | ✅ | ✅ | ✅ | Looser tolerance (≈ 3e-2 is the established budget). |\n| `save_hf` round-trip (byte-equal tensors) | ✅ | ✅ | ✅ | See `tests/model/test_qwen3_moe.py`. |\n\n### 7.2 Parity bar by model scale\n\nTwo independent axes:\n\n**Per-layer (op) parity — required at _all_ scales, bitwise.** Test one full and\none linear decoder layer separately; for each assert layer **output + loss + `dL/dx`**\nbitwise vs HF under `XTUNER_HF_IMPL`. Keep it memory-light (XTuner `meta`-build +\nmaterialize only the tested layer + `norm` + `lm_head`; standalone HF layer; load\nvia `HFCheckpointLoader`) so it runs even for models too large to forward\nend-to-end. This is the bitwise guarantee for both forward and backward ops.\n\n**Whole-model (integration) parity — on the _top-level_ model, graded by what fits\none GPU.** Run the real model end-to-end: for a VLM that's the **compose model on an\nimage prompt** (one test covers vision + projector + text — don't add a separate\ntext-tower model test); for a plain LLM, the text/LM model.\n- **Small** (e.g. dense 4B): both forward (logits/loss) AND backward\n (`dL/d(pixel_values)` for a VLM, `dL/d(inputs_embeds)` for text) **bitwise**.\n Prerequisites: `XTUNER_HF_IMPL=true` so every op selector returns the\n HF-canonical callable (§6.2) AND `XTUNER_DETERMINISTIC=true` so Triton\n autotune is pinned (§6.2). With both on, even hybrid linear/full models match\n HF byte-for-byte through the whole compose chain. Don't check weight\n gradients — they accumulate across positions + tied `lm_head` and aren't the\n right granularity for parity.\n- **Medium**: forward only (logits/loss bitwise).\n- **Large** (cannot forward end-to-end on the test GPUs): skip the whole-model\n test; leave it to integration tests. The per-layer test still gives op-level\n forward+backward bitwise.\n\n### 7.3 Required test idioms\n\n`tests/model/test_qwen3_5_dense.py` is the reference template; extend\n`DeterministicDDPTestCase` (`xtuner._testing`), which provides the shared\nparity scaffolding so the test file only holds model-specific logic:\n\n- `with self.hf_impl():` — sets/restores `XTUNER_HF_IMPL=true`; build the XTuner\n model **inside** the block so its attention picks the eager op at construction.\n- `self.materialize_xtuner_submodule(model, submodule, prefix, loader)` — for a\n model built on `meta`, materialize *only* that submodule on CUDA and load its\n weights (so a single-layer test fits any model size).\n- `self.load_params_from_hf(module, key_for, loader)` /\n `self.xtuner_ckpt_key(model, name)` — selective per-param load by checkpoint key\n (`HFCheckpointLoader` reads only the needed shard); `xtuner_ckpt_key` applies\n `to_hf_key_list` + the config's `hf_key_mapping`.\n\n```python\n# Construct on meta, materialize on device — single-rank baseline, eager, no EP.\nwith torch.device(\"meta\"):\n cfg = get_model_config_from_hf(hf_model_path)\n cfg.compile_cfg = False # baseline tests run eager\n model = cfg.build()._to_device_dtype(dtype=torch.bfloat16, skip_buffers_dtype=True)\n\n# Optionally shard with FSDP (data-parallel only), then load.\nmodel.fully_shard(fsdp_config=FSDPConfig(cpu_offload=False))\nmodel.from_hf(hf_model_path)\n```\n\nOlder `patch_hf_rms_norm` / `patch_hf_rope` helpers also live in\n`xtuner._testing`. **Check they actually fit your model before using them** —\nthey were written for earlier models and match HF modules\nby class-name substring and attribute name (e.g. `variance_epsilon`). On a newer\nmodel they can silently mis-patch (wrong eps attribute, or replacing a\n`zero_centered` RMSNorm with a default one). Often the native XTuner ops already\nmatch HF bitwise (see §12), so no patching is needed; prefer `XTUNER_HF_IMPL`\nover these helpers for parity. The §8 parallel-feature tests reuse this same\nidiom with `cfg.dispatcher` / `cfg.ep_size` (and `FSDPConfig(ep_size=...)`) set.\n\n### 7.4 Checkpoint paths\n\nRead from env vars; **never** hard-code. Example:\n`QWEN3_MOE_PATH = os.environ[\"QWEN3_MOE_PATH\"]` at\n`tests/model/test_qwen3_moe.py`. Document new env vars in the PR body.\n\n### 7.5 Drop-in training config\n\nShip a runnable training config at `ci/config/.py` so a developer (and CI) can verify\nthe port end-to-end with one command: load the HF checkpoint, run a few real training steps,\nwatch the loss curve descend as expected. This is the smoke test that catches anything the\nunit tests miss — the full `from_hf` → `model.forward` → loss → backward → optimizer step →\nFSDP shard/reduce chain — and the file doubles as the example users copy when wiring the\nmodel into their own training pipeline. Mirror `ci/config/qwen3_moe_30BA3.py` (MoE) or\n`ci/config/qwen3_dense.py` (dense), keeping its structure: one `()` (from §3.2)\nfed into a `TrainerConfig` alongside `optim_cfg` / `lr_cfg` / `fsdp_cfg` / `dataset_cfg` /\n`dataloader_cfg` / `loss_cfg`. `load_from` and `tokenizer_path` read from an env var (§7.4 —\ntypically the same one as the parity test). Verify by running ~50 steps and confirming the\nloss drops monotonically into a plausible range for that model size; record the trajectory\nin the PR body alongside the §6 convergence trace.\n\n---\n\n## 8. Training optimizations\n\nOnly after the §6 baseline is committed and the §7 tests are green: verify each\nswitch runs correctly and keeps precision within budget **against that baseline**.\n\n1. EP and SP can be enabled normally.\n2. With `ep_size > 1`, `intra_layer_micro_batch` enables correctly and stays\n precise.\n3. `torch.compile` runs correctly.\n4. fp8 can be enabled.\n5. `XTUNER_ACTIVATION_OFFLOAD` can be enabled.\n\nEach switch needs a test that compares its output to the §6 baseline — e.g. the\ndispatcher × ep_size parity matrix (at minimum `{(None,1), (\"all2all\",4),\n(\"all2all\",8)}`, add `\"deepep\"` if supported), compiled-vs-eager, or fp8-vs-bf16\nwithin tolerance.\n\nThese items are well-suited to the multi-agent workflow in §11: the main agent\nanalyzes and creates tasks (with test cases), sub-agents develop in parallel, and\nthe main agent merges. Report the precision error introduced by each switch.\n\n---\n\n## 9. Reference cheat-sheet\n\n| Concept | Concrete location |\n|-------------------------|--------------------------------------------------------------------------------|\n| Dense base config | `xtuner/v1/model/base.py` (`TransformerConfig`) |\n| MoE base config | `xtuner/v1/model/moe/moe.py` (`MoEConfig`) |\n| Compose base config | `xtuner/v1/model/compose/base.py` (`BaseComposeConfig`) |\n| `from_hf` (model) | `xtuner/v1/model/base.py` (`from_hf`) |\n| `save_hf` (model) | `xtuner/v1/model/base.py` (`save_hf`) |\n| `to_hf_key_list` proto | `xtuner/v1/model/base.py` |\n| MoE layer split | `xtuner/v1/model/moe/moe.py` (`first_k_dense_replace`) |\n| RoPE auto-parse | `xtuner/v1/module/rope/rope.py` (`RopeParametersConfig.from_hf_config`)|\n| Dense reference | `xtuner/v1/model/dense/qwen3.py` |\n| MoE reference | `xtuner/v1/model/moe/qwen3.py` |\n| MoE w/ MLA + NoAux | `xtuner/v1/model/moe/deepseek_v3.py` |\n| MoE w/ layout transform | `xtuner/v1/model/moe/gpt_oss.py` |\n| Remote-code reference | `xtuner/v1/model/moe/qwen3.py` (`Qwen3MoEFoPEConfig`) |\n| Compose reference (VLM) | `xtuner/v1/model/compose/qwen3_vl/` |\n| Dense test reference | `tests/model/test_qwen3_dense.py` |\n| MoE test reference | `tests/model/test_qwen3_moe.py` |\n| Engine test reference | `tests/engine/test_moe_train_engine.py` |\n\n---\n\n## 10. Commit discipline\n\n0. **Commit as you go — this is authorized, do not pause for a separate\n go-ahead.** The user invoking this skill authorizes the commits this workflow\n produces. Finish a logical step → lint/test → commit, then move on. Do not\n leave the whole port uncommitted and end with \"should I commit?\"; that strands\n the work. (Hard-to-reverse or outward actions like `git push` / opening a PR\n still need explicit confirmation — committing locally does not.)\n1. **Follow the stacked-PR convention.** Plan upfront how many commits the model\n port needs — one per logical step (e.g. shared-path fixes; model class +\n config + registration; the `XTUNER_HF_IMPL` switch; baseline tests; then one\n per parallel/optimization feature). Order them by dependency (a shared-path fix\n the baseline relies on goes first). Land every later change as\n `git commit --fixup=` + `git rebase --autosquash` into the commit it\n belongs to. **Never grow the history with endless patch commits just to keep\n fixing things up.**\n2. **Every commit must pass lint:**\n ```bash\n pre-commit run --files $(find xtuner/v1)\n ```\n\n---\n\n## 11. Multi-agent workflow\n\nThis applies **only to the §8 parallel/optimization features** — and only once\nthe §6 baseline and §7 baseline tests are done and committed (§10). The baseline\nitself is built sequentially, not multi-agent; the §8 features are independent of\neach other, which is what makes them safe to develop in parallel.\n\n1. The **main agent** splits the §8 features into tasks and, per §8, writes each\n feature's comparison test against the committed baseline and commits the tests.\n2. **Sub-agents** branch off the baseline/test commit and develop one feature\n each (e.g. one switch / one tower) in parallel, making its test pass.\n3. **The two sides adjust each other** — a shared-interface change is made once by\n the main agent (it updates the test/baseline; sub-agents rebase), and a failing\n test pushes the sub-agent to fix its feature, not to weaken the test.\n4. The **main agent** merges, resolves conflicts, and reports the precision error\n each feature introduces relative to the baseline.\n\n---\n\n## 12. Parity debugging pitfalls\n\nHard-won notes from aligning hybrid (GatedDeltaNet + gated-MHA) models bitwise.\nThe meta-rule: **measure on real GPU and bisect to localize the divergence before\nconcluding anything** — most \"bugs\" here are measurement artifacts, not model\nerrors. A clean way to bisect is to feed one side's intermediate tensors into\nboth modules and compare step by step (norm → q/k/v → qk_norm → rope → attention\ncore → o_proj).\n\n- **Single-layer parity: the two sides' inputs must match exactly — especially\n the causal mask.** When you hand-build one layer's inputs, passing\n `attention_mask=None` to an HF attention/layer makes HF apply **no** causal\n mask while XTuner stays causal, producing a large *false* diff. Build a causal\n mask equivalent to XTuner's `seq_ctx`, or drive the layer through the model so\n the mask is constructed internally.\n\n- **`flash` ≠ `eager`, and HF eager upcasts softmax to fp32.** XTuner defaults to\n flash attention; HF reference eager does softmax in fp32. They are numerically\n close but not bitwise equal, so bitwise parity requires forcing eager on the\n XTuner side (that is exactly what `XTUNER_HF_IMPL` does). XTuner's\n `eager_attention` matches HF's `eager_attention_forward` bitwise.\n\n- **A bitwise forward does not imply a bitwise backward.** The eager softmax must\n run in **fp32** (`softmax(..., dtype=torch.float32).to(x.dtype)`, like HF), not\n bf16. With a bf16 softmax the forward logits can still round identically to HF\n (bitwise), but the *backward* differs (~1e-5 on `dL/dx`) because the gradient\n accumulates at bf16 precision. So always check backward bitwise separately\n (the `loss.backward()` test), not just forward — and align op dtypes (softmax,\n norms) in the backward-sensitive direction, fp32 where HF uses fp32.\n\n- **Whole-model backward is bitwise — but only with the full env stack.**\n Per-layer bitwise backward does not on its own imply whole-model bitwise\n backward; the missing pieces are `XTUNER_HF_IMPL=true` AND\n `XTUNER_DETERMINISTIC=true` AND the conftest must `import xtuner.v1` so the\n autotune-pin block in `xtuner/v1/__init__.py` runs before any test reaches\n fla. Without those, two distinct ULP-level sources compound through the LM\n chain rule and look indistinguishable from \"fundamental bf16 noise\":\n\n 1. **Triton autotune drift in fla.** fla's `@triton.autotune` picks\n tiling / num_warps / reduction-order per kernel. Across runs the choice\n can change, producing a ~1-ULP backward diff per linear-attention layer\n that the chain rule amplifies — a 32-layer LM stack turned a per-layer\n ~2e-3 abs diff at grad magnitude ~30 into a ~1.0 diff on the\n massive-activation channels of `dL/d(pixel_values)`. The pin (configs\n held to the first one, cache disabled) is in `xtuner/v1/__init__.py`\n under an `XTUNER_DETERMINISTIC` gate. It must run **before any module\n imports fla** — fla's `@triton.autotune` decorators evaluate at import\n time, so patching after the fact has no effect on kernels already\n constructed. That's why the patch is on the `xtuner.v1.*` import chain\n rather than in the conftest: `multiprocessing.spawn` children of\n `MultiProcessTestCase` re-import the test class top-level (which pulls\n `xtuner.v1.model.*`) without running conftest, so the conftest alone\n can't install the patch in those processes.\n 2. **`torch.library.custom_op` wraps for the linear-attention path**\n (`GatedDeltaNet`, `causal_conv1d`) produce a different backward autograd\n graph from the canonical fla / causal_conv1d call HF uses, even though\n the underlying kernel and forward output are identical. `XTUNER_HF_IMPL`\n routes through `get_chunk_gated_delta_rule_fn` / `get_causal_conv1d_fn`\n (§6.2) so the parity path matches HF byte-for-byte.\n\n Production train doesn't set either env var; the custom_op wraps + autotune\n still operate. These switches are **parity-test only**. If the whole-model\n backward is not bitwise but the per-layer backward is, suspect one of the\n two before suspecting depth-accumulated bf16 noise.\n\n- **`flash`(XTuner varlen) vs `flash_attention_2`(HF) is bitwise only for short,\n single-tile sequences.** They are the same kernel, so at a short seqlen (e.g.\n 32) the logits are bitwise (0.0); at longer sequences the tiling /\n accumulation order differs and they drift ~1e-2. So a short-seq text test can\n assert flash-vs-FA2 bitwise, but anything realistic-length (or a VLM whose\n image tokens make the sequence long) needs the **eager** path for\n seqlen-independent bitwise. Don't conclude \"flash matches FA2\" from one short\n input.\n\n- **Shared kernels are already bitwise; focus on what isn't.** GatedDeltaNet /\n linear attention go through the same `fla` / `causal_conv1d` kernels as HF, and\n native rms_norm / rope match HF bitwise — these come out at 0.0 with no work.\n Spend your effort on the path that actually differs (usually full attention).\n\n- **`output_hidden_states` off-by-one (full-forward route, < 40B only).** HF's\n tuple is `[emb, L0_out, …, L{n-2}_out, post_norm]`: index `i+1` is layer `i`'s\n pre-norm output only for `i ≤ n-2`; the **last** entry is post-norm, not the\n last layer's pre-norm output. Compare the last layer via `model.norm(...)` or\n you will see the final layer \"explode\".\n\n- **A big max-diff is not automatically a bug.** Late layers have *massive\n activations* (a few channels in the tens/hundreds); a tiny relative error there\n shows up as a large absolute max-diff. Look at the mean and the trend, not just\n the max. Conversely, feeding `randn` inputs to a single attention layer is\n ill-conditioned and misleads — use real (or model-produced) hidden states.\n\n- **A single-model failure is often a general-path bug.** E.g. `Dense.build_layers`\n not forwarding `rms_norm_type` to `DenseDecoderLayer` broke any `zero_centered`\n dense model, not just this one. Fix the shared invariant, not the symptom.\n\n- **Loading a standalone sub-tower uses `strict=False`.** A text tower loaded\n directly from a full VLM checkpoint will see vision (and deferred `mtp.*`) keys\n as unexpected; load with `strict=False` so they are ignored.\n\n- **Test the whole VLM, not just the text tower.** Text-tower parity leaves the\n vision tower and projector completely untested. A compose port needs a\n full-model forward-parity test with real image inputs (§7.1). The text path of\n the compose forward still runs a *dummy* vision forward to keep the tower in the\n autograd graph, so vision bugs surface even on text prompts.\n\n- **Vision attention is non-causal — give `eager_attention` a `causal=False`\n path.** XTuner's `eager_attention` defaulted to a causal block-diagonal mask;\n that is wrong for a vision tower (bidirectional within each image), which is why\n forcing vision to the causal eager op produced large diffs. The op now takes\n `causal=True|False` (False → non-causal block-diagonal mask), the vision module\n already calls it with `causal=False`, and the vision tower routes through\n `get_attn_impl_fn` so `XTUNER_HF_IMPL` selects eager. With that, the *whole* VLM\n is bitwise in one run, everything eager: `XTUNER_HF_IMPL` on the XTuner side\n (text causal-eager + vision non-causal-eager) and HF loaded with a single\n `attn_implementation=\"eager\"`. Prefer eager for parity — it is\n seqlen-independent, unlike flash/FA2 (see the flash seqlen bullet above).\n\n- **Vision pos-embed dtype when reusing the Qwen3-VL tower.** HF's\n `fast_pos_embed_interpolate` returns fp32 (bilinear weights are fp32) and HF's\n vision forward casts it (`pos_embeds.to(hidden_states.dtype)`); XTuner's vision\n forward adds it without a cast. If you patch in the HF method for parity, cast\n the result back to the param dtype, or you get an fp32/bf16 `LayerNorm` mismatch\n (`expected scalar type Float but found BFloat16`).\n\n- **Hybrid-model `torch.compile`: choose `fullgraph` per layer — verify, then fall\n back; don't gate *whether* to compile.** When a decoder layer's forward is wrapped\n into a higher-order op (HOP) — e.g. by an activation-checkpoint wrapper, which\n `torch.compile` lowers to a HOP whose body must be functional so it can be\n recomputed in backward — *and* that layer is a linear-attention layer\n (GatedDeltaNet), the layer mutates an outer-scope variable inside the HOP\n (GatedDeltaNet writes back into `seq_ctx`). Under `fullgraph=True` that raises\n `HigherOrderOperator ... Mutating a variable not in the current scope (SideEffects)`.\n This is **not** a blanket \"linear attention ⇒ `fullgraph=False`\" rule — whether it\n triggers depends on whether the layer is actually HOP-wrapped and whether that\n linear impl really writes outer state. So for such a HOP + linear combination,\n **first test whether `fullgraph=True` runs**; keep it if it does (it compiles more\n of the layer), and only **fall back to `fullgraph=False`** if it doesn't —\n `fullgraph=False` survives by graph-breaking at the mutation and running that\n side-effecting write in eager. Make this a **per-layer** choice in\n `Dense.fully_shard` (decide each layer's `fullgraph`), not a decision about whether\n a layer is compiled at all.\n\n",".claude/skills/check-hf-config-save/SKILL.md":"---\nname: check-hf-config-save\ndescription: Implement a missing XTuner Hugging Face config export from the official HF model when possible, then validate it against the installed Transformers round-trip and versioned inference-engine field contracts. Use when adding a model, when hf_config is missing or returns None, when changing from_hf/hf_config/save_hf, when reviewing config.json differences, or when debugging vLLM/SGLang checkpoint-load or inference failures. Also use for requests named check_hf_config_save.\n---\n\n# Check HF Config Save\n\n## Overview\n\nFirst ensure that a model with an official built-in Transformers config has an\ninverse `from_hf <-> hf_config` mapping. If that export is missing, implement it\nfrom the official HF model before using `xtuner._testing.check_hf_config_save`\nto test the real public `from_hf -> save_hf` path. Separate changes forced by\nthe installed Transformers version from fields dropped by XTuner, then protect\nfields that exact inference engine versions use for model construction or\nweight loading.\n\n## Workflow\n\n### 1. Fix the scope and version matrix\n\n1. Identify the source HF model directory, XTuner config class, export path, and\n any engine/version named by the user or failure log.\n2. Record the executable environment versions for Python, Transformers, and any\n installed engines. Use the requested environment; otherwise follow the repo's\n default environment instructions.\n3. A user-specified or production-log version takes precedence. If a component\n version is unspecified, look up the latest stable release from its official\n PyPI project/API or official release page and inspect the matching source tag.\n By default audit both vLLM and SGLang.\n4. List every checked version in the result. Distinguish an installed runtime\n test from a static audit of an exact source tag.\n\nUse a table with at least these columns:\n\n| Component | Version | How selected | Validation |\n|---|---:|---|---|\n| Transformers | exact version | active environment | executable round-trip |\n| vLLM | exact version | user/log/latest official | runtime or exact-tag source audit |\n| SGLang | exact version | user/log/latest official | runtime or exact-tag source audit |\n\nDo not write `latest` without resolving it to an exact version and source.\n\n### 2. Implement a missing HF config export\n\nExercise the supported model's public conversion path first:\n\n```python\nconfig = get_model_config_from_hf(SOURCE_HF_DIR)\nexported_config = config.hf_config\n```\n\n`exported_config is None`, or the corresponding public `config.save_hf(...)`\nraising the base missing-`hf_config` `NotImplementedError`, means the config-only\nexport is not implemented. Classify the official model before changing code:\n\n```mermaid\nflowchart TD\n A[\"Official HF model and exact revision\"] --> B{\"Built-in official Transformers config?\"}\n B -->|Yes| C{\"XTuner hf_config implemented?\"}\n C -->|No| D[\"Implement inverse from_hf and hf_config mapping\"]\n C -->|Yes| E[\"Continue round-trip validation\"]\n D --> E\n B -->|No; trust_remote_code only| F[\"Keep hf_config as None and validate source-config copying\"]\n```\n\nFor a built-in official Transformers config:\n\n1. Read the official checkpoint's raw `config.json` and resolve its exact\n `model_type`, `architectures`, repository revision, and official config\n implementation. Load it with the selected Transformers version and record\n the concrete `PretrainedConfig` subclass. Do not use a third-party model\n implementation as the reference.\n2. Implement or complete `from_hf` with that official class. Map every\n architecture value needed by XTuner, use `getattr` for genuinely optional\n versioned fields, and use `RopeParametersConfig.from_hf_config` for RoPE.\n3. Implement `hf_config` by constructing the same official config class from\n the XTuner config's current values. It must be the inverse of `from_hf`, not\n a cached copy of the source object. Re-emit every field consumed by\n `from_hf`, plus raw compatibility fields required by vLLM or SGLang even when\n Transformers treats them as optional or legacy.\n4. Preserve the official `model_type` and architecture name. Do not substitute\n a generic `PretrainedConfig`, duplicate the official config class in XTuner,\n or return a hand-written dictionary.\n5. Verify through public behavior that `config.hf_config` has the expected\n official type and that `config.save_hf(...)` succeeds, then continue with\n the helper below.\n\nIf the official repository only works through `trust_remote_code` and the\nselected Transformers versions have no built-in config class, do not invent an\n`hf_config` implementation. Keep `hf_config=None`, retain the source `_hf_path`,\nand test the model's public HF save path that copies the original config,\ntokenizer, and required remote-code files. If the entire model or dispatch is\nnot yet supported by XTuner, follow `$add_hf_model`; this skill only fills a\nmissing exporter for an otherwise supported model.\n\n### 3. Establish the Transformers reference\n\nCompare three raw JSON states:\n\n```mermaid\nflowchart LR\n A[\"Source config.json\"] --> B[\"Current Transformers load + save\"]\n A --> C[\"XTuner from_hf + save_hf\"]\n B --> D[\"Expected serialized reference\"]\n C --> E[\"XTuner export\"]\n D --> F[\"Public helper comparison\"]\n E --> F\n```\n\n- Read the source `config.json` before `AutoConfig` normalization.\n- Load and save it directly with the active Transformers version. This is the\n serialized reference and captures forced defaults or `__post_init__` changes.\n- Build through XTuner's public `get_model_config_from_hf`/`from_hf` API and call\n the public `save_hf` API.\n- Compare the Transformers reference with the XTuner export. Do not use direct\n source-versus-export equality as the primary assertion: it misclassifies\n Transformers normalization as an XTuner bug.\n\nInspect the exact Transformers config implementation, including generated or\nmodular source and `__post_init__`, for every source-to-reference difference.\nTypical examples are derived `head_dim`, generated `layer_types`, new defaults,\nand serializer metadata, but never assume these examples cover a new model.\n\n### 4. Audit inference-engine dependencies\n\nFor every changed, missing, newly defaulted, or architecture-selecting field:\n\n1. Search the exact engine tag, not an arbitrary installed or main-branch copy.\n2. Follow model registration, config access, module construction, and the weight\n loader. Search aliases, `getattr` defaults, direct attribute access, and tensor\n name/shape conditions.\n3. Classify the use as one of:\n - module/parameter registration;\n - checkpoint key or tensor-shape selection;\n - layer topology or MoE routing;\n - attention/RoPE behavior;\n - ignored or default-compatible.\n4. Encode every value-sensitive dependency as `HFConfigFieldDependency`, with\n exact engine version, JSON pointer, expected value, reason, and an official\n source permalink.\n\nA field can be HF-equivalent yet engine-critical. For example, an engine may\nregister a checkpoint parameter only when a legacy routing field has one exact\nvalue; omitting that field then becomes a real weight-loader failure.\n\nStatic source inspection proves the dependency but not end-to-end engine\ncompatibility. Run an engine checkpoint-load smoke test when the exact runtime\nand required hardware are available. Otherwise report `exact-tag source audit`\nand do not claim runtime success. Follow the repo GPU-lock instructions before\nany local GPU run.\n\n### 5. Add the model regression test\n\nDelete narrow hand-written assertions that duplicate this contract, then add a\nmodel test on the real public conversion path:\n\n```python\nimport transformers\n\nfrom xtuner._testing import HFConfigFieldDependency, check_hf_config_save\nfrom xtuner.v1.model import get_model_config_from_hf\n\n\ndef test_save_hf_matches_transformers_and_engine_contracts():\n config = get_model_config_from_hf(SOURCE_HF_DIR)\n assert isinstance(config.hf_config, OFFICIAL_HF_CONFIG_CLASS)\n report = check_hf_config_save(\n config,\n SOURCE_HF_DIR,\n engine_dependencies=(\n HFConfigFieldDependency(\n engine=\"vllm\",\n version=\"\",\n path=\"/\",\n expected=\"\",\n reason=\"\",\n source=\"\",\n ),\n ),\n )\n\n assert report.transformers_version == transformers.__version__\n assert report.checked_engine_versions == (\"vllm==\",)\n```\n\nThe helper performs two independent checks:\n\n- XTuner export matches the active Transformers direct round-trip.\n- Exported values satisfy the declared inference-engine contracts, even when\n Transformers itself does not consume those fields.\n\nUse `allowed_export_differences={\"/json/pointer\": \"specific reason\"}` only for\nan intentional XTuner difference that is not already an engine dependency.\nEvery exception needs a non-empty model-level reason.\n\n### 6. Prove the regression and run the matrix\n\n1. Run the helper's own behavior tests.\n2. Run the generated model test in the project's pinned Transformers version.\n3. Run it in each user-requested/current upgraded Transformers environment.\n4. If the old broken export is available, pass its `config.json` through the\n same helper and show that the expected missing/extra paths fail. This is the\n minimal proof that the test catches the original bug.\n5. Run formatting and the smallest relevant model test suite.\n\nReport:\n\n- whether `hf_config` already existed, was implemented from which official\n config class/revision, or correctly remained `None` for remote code;\n- source-to-Transformers normalization paths;\n- Transformers-reference-to-XTuner paths (normally none, apart from documented\n allowed contracts);\n- engine dependency, expected value, exact version, and effect;\n- which checks were executable and which were source-only.\n\n## Guardrails\n\n- Compare raw JSON, not only `AutoConfig` attributes; unknown compatibility\n fields may disappear from a newer config class while engines still read them.\n- Exercise public APIs and real conversion behavior. Do not mock XTuner model\n internals.\n- Do not replace the helper with a blanket list of expected keys.\n- Do not silently refresh an engine version in a test. Re-audit its exact source\n before updating the version and permalink.\n- HF semantic equivalence is not evidence of vLLM/SGLang compatibility.\n- Do not set `hf_config=None` for a model with a built-in official config merely\n to bypass config reconstruction or round-trip failures.\n- Preserve unrelated worktree changes and keep the implementation model-agnostic.\n\n## Completion criteria\n\nThe work is complete only when the official config export path exists where it\nshould, the public round-trip matches Transformers, every audited engine\ncontract is preserved, and the report identifies the exact official model\nrevision and component versions used.\n",".claude/skills/check-hf-config-save/agents/openai.yaml":"interface:\n display_name: \"Check HF Config Save\"\n short_description: \"Implement and validate exported HF configs\"\n default_prompt: \"Use $check-hf-config-save to implement any missing official Hugging Face config export and validate it across Transformers and inference engines.\"\n",".claude/skills/model_normalize/SKILL.md":"---\nname: model_normalize\ndescription: Normalize an HF model checkpoint — quantize bf16/fp16 weights to per-block FP8, and/or repack engine-named safetensors into HF-standard ~4GB shards.\n---\n\n# model_normalize\n\nTwo complementary capabilities for getting a training-engine checkpoint into a\nformat that HF / vLLM / SGLang can consume:\n\n1. **FP8 quantization** of large linears, in two modes:\n - **Reference-guided** — point at an existing FP8 checkpoint of the same\n architecture; the tool reads its index and reproduces the exact same\n quantization decisions on the source.\n - **Heuristic** — apply a built-in rule set that quantizes only the large\n matrix multiplications (attention, MLP, MoE experts) and keeps norms,\n routers, biases, embeddings, and vision tower in their original dtype.\n2. **Standard repack** — re-shard arbitrary safetensors directories into the\n HF-standard ``model-{i:05d}-of-{n:05d}.safetensors`` layout with ~4GB\n shards and a fresh ``model.safetensors.index.json``.\n\nWhen ``to-fp8`` detects non-standard source shard names, it automatically\nruns the repack step on the quantized output, so the user-visible result is\nalways a clean HF-conformant directory.\n\n## Layout\n\n```\n.claude/skills/model_normalize/\n├── SKILL.md # this file\n├── model_normalize.py # top-level CLI (to-fp8 / repack)\n├── hf_to_fp8.py # FP8 conversion library + standalone CLI\n├── heuristics.py # codified default quantization rule set\n├── repack_hf.py # 4GB-shard repacker library + CLI\n└── convert_interns2_to_fp8.sh # example invocation for InternS2\n```\n\n## Usage\n\n### Mode 1 — Reference-guided FP8 conversion\n\n```bash\npython model_normalize.py to-fp8 \\\n --source /path/to/bf16-model \\\n --output /path/to/fp8-model \\\n --reference /path/to/fp8-reference\n```\n\nBehavior:\n- Loads the reference's ``model.safetensors.index.json``.\n- For each ``_scale_inv`` entry in the reference index, marks ````\n as an FP8-quantized tensor.\n- Applies that exact set to the source as a literal name match (no regex\n surprise; works even when names don't follow any pattern).\n\n### Mode 2 — Heuristic FP8 conversion (no reference)\n\n```bash\npython model_normalize.py to-fp8 \\\n --source /path/to/bf16-model \\\n --output /path/to/fp8-model\n```\n\nThe rule set is defined in ``heuristics.py``. Concretely, **quantize iff** the\ntensor name matches one of:\n\n| Group | Pattern (anchored) |\n| -------------------------------- | ------------------------------------------------------------------------------------------------------------------- |\n| Self-attention projections | ``[…]layers..self_attn.{q,k,v,o}_proj.weight`` |\n| Dense MLP | ``[…]layers..mlp.{gate,up,down}_proj.weight`` |\n| MoE per-expert (unfused) | ``[…]layers..mlp.experts..{gate,up,down}_proj.weight`` |\n| MoE fused-expert (3D tensors) | ``[…]layers..mlp.experts.{gate_up_proj, down_proj}`` |\n| MoE shared expert | ``[…]layers..mlp.shared_expert.{gate,up,down}_proj.weight`` |\n| Linear-attn wide projections | ``[…]layers..linear_attn.{in_proj_qkv, in_proj_z, out_proj}.weight`` |\n\nThe ``[…]`` prefix allows ``model.`` and ``model.language_model.``; the same\npatterns repeat under ``mtp.layers..`` for multi-token-prediction blocks.\n\n**Everything else is kept in the original dtype**, in particular:\n- all norms (``*norm*``, ``*_layernorm.*``)\n- MoE routers (``mlp.gate.weight``, ``mlp.shared_expert_gate.weight``)\n- embeddings (``embed_tokens``, ``lm_head``, ``pos_embed``, ``patch_embed``, ``mtp.fc``)\n- vision tower (``model.visual.*``)\n- all ``*.bias``\n- ``linear_attn`` control-flow tensors (``A_log``, ``conv1d.weight``, ``dt_bias``, ``in_proj_a/b.weight``, ``norm.weight``)\n\nThis rule set was validated against the InternS2 / Qwen3-MoE FP8 references:\nit reproduces their quantization decisions on every tensor (no false\npositives or false negatives).\n\n### Mode 3 — Standalone repack (no quantization)\n\n```bash\npython model_normalize.py repack \\\n --source /path/to/messy-shards \\\n --output /path/to/standard-shards \\\n --shard-size-gb 4\n```\n\nGreedy bin-packs tensors into ``model-{i:05d}-of-{n:05d}.safetensors`` shards\nof at most ``--shard-size-gb`` GiB, writes a fresh\n``model.safetensors.index.json`` (including ``metadata.total_size``), and\ncopies all non-safetensors files (config, tokenizer, etc.) through.\n\n## When to use which mode\n\n| Situation | Mode |\n| ---------------------------------------------------------------------------- | ------------------ |\n| You already have an FP8 reference for this architecture. | Reference-guided |\n| New architecture / first-time conversion; standard naming. | Heuristic |\n| Quantized model has engine-specific shard names; need HF-conformant output. | (auto, see below) |\n| You only need to standardize shard naming; weights are already FP8 / bf16. | ``repack`` |\n\nThe ``to-fp8`` subcommand auto-detects non-standard source shard names (anything\nthat isn't ``model.safetensors`` or ``model-NNNNN-of-NNNNN.safetensors``) and\nchains the repack step. Pass ``--no-repack`` to keep the source naming.\n\n## Quantization details\n\n- Block size **128 × 128**, dtype ``float8_e4m3fn`` with saturating cast.\n- Per-block scale stored as ``_scale_inv`` (``float32``) in the same\n shard as the parent weight. Layout matches what vLLM / SGLang load for\n per-block FP8.\n- 3D MoE-expert tensors (shape ``(E, D0, D1)``) are quantized per-expert; the\n per-expert scales are stacked along a leading dim.\n- ``config.json`` receives a ``quantization_config`` block with\n ``quant_method=fp8``, ``weight_block_size=[128, 128]``,\n ``scale_fmt=ue8m0``, and an explicit ``modules_to_not_convert`` list\n derived from the actual quantization decisions.\n\n## Post-conversion handoff (mandatory for the agent)\n\nAfter **any** successful run of `to-fp8` or `repack`, the agent must ask the\nuser — via `AskUserQuestion` — where to **place** the produced\n`*.safetensors` and `model.safetensors.index.json` files. The conversion\noutput is treated as a staging area; the user decides the final home.\n\nOffer at least these options:\n1. **Keep in place** — leave the files at the `--output` path as-is.\n2. **Copy to a new directory** — let the user specify a target; copy the\n shards + index there. Always **copy** (`cp`), never move — the\n `--output` staging directory must remain intact unless the user\n explicitly asks to delete it as a separate step.\n3. **Copy into an existing model directory** — copy shards/index into a\n user-specified directory while preserving its other files (tokenizer,\n config, etc.).\n\nIf the user picks (2) or (3), execute the copy with `cp` (confirming the\ndestination path back to the user before any overwrite). Do not infer the\ndestination from earlier turns — the user may have a fresh target in mind\neach time. Cleaning up the staging directory afterwards is a separate,\nuser-initiated step — never delete it as part of the handoff.\n\n## Example: InternS2 Preview\n\n```bash\npython model_normalize.py to-fp8 \\\n --source /mnt/shared-storage-user/puyudelivery/user/yehaochen/models/InternS2PreviewCandidate5 \\\n --output /tmp/InternS2-FP8 \\\n --reference /mnt/shared-storage-user/puyudelivery/user/yehaochen/models/InternS2PreviewCandidate5-FP8\n```\n\nThe source uses engine-specific names like\n``model-language-0001-fused-save_rank0.safetensors``; the output will be\nrepacked into the standard ``model-00001-of-NNNNN.safetensors`` layout.\n",".claude/skills/model_patch/SKILL.md":"---\nname: model_patch\ndescription: Merge a base HF model dir with an extra HF model dir by index diff — take tensors only present in extra, append them to a new output dir in HF-standard shard layout. Base wins on overlap.\n---\n\n# model_patch\n\nProduce ``out = base ∪ (extra \\ base)`` at the **tensor level**, by diffing the\ntwo ``model.safetensors.index.json`` files. Useful when ``extra`` is a sibling\ncheckpoint (different training run, fine-tune, or fp8 conversion) that holds a\nfew tensors the base is missing — e.g. an updated head, a newly-added MTP\nblock, or auxiliary projections — and you want a self-contained merged dir\nwithout re-running the whole conversion pipeline.\n\nOverlap rule: **base wins**. Any tensor present in both indexes is taken from\n``base``; the corresponding entry in ``extra`` is ignored.\n\n## Layout\n\n```\n.claude/skills/model_patch/\n├── SKILL.md # this file\n└── model_patch.py # CLI + library\n```\n\n## Usage\n\n```bash\npython model_patch.py \\\n --base /path/to/base-model \\\n --extra /path/to/extra-model \\\n --out /path/to/merged-model\n```\n\n## What it writes\n\n- ``model-{i:05d}-of-{N:05d}.safetensors`` shards (total ``N`` = base shards +\n newly-packed shards holding the extra-only tensors)\n- a fresh ``model.safetensors.index.json``\n- **every non-tensor file from ``base``** — ``config.json``, tokenizer files,\n modeling/configuration ``.py``, ``generation_config.json``, README, etc. —\n copied verbatim so ``out`` is loadable on its own. Subdirectories are\n copied recursively. Existing entries in ``out`` are overwritten.\n\n``extra``'s aux files are deliberately **not** copied. The skill's overlap\nrule is \"base wins\" at the tensor level, and that extends to configs: extra\noften carries engine-specific edits or modality keys that don't match the\ntensor topology we keep. If you need fields from ``extra``'s ``config.json``\n(e.g. a new modality), merge them into ``out/config.json`` as a separate,\nexplicit step.\n\n## How shards are produced\n\n- **Base shards**: re-emitted under the new ``-of-N`` total. By default the\n bytes are fully copied so ``out`` is independent of ``base``. Pass\n ``--hardlink`` to use ``os.link`` instead (fast, but the two trees share\n inodes — editing one mutates the other; deleting one keeps the other\n intact since hardlinks are symmetric).\n- **Extra-only tensors**: greedily bin-packed into new shards of at most\n ``--shard-size-gb`` GiB (default 4), appended after the base shards.\n\nSo a base with 12 shards plus 5 GB of extra-only tensors yields:\n``model-00001-of-00014.safetensors`` … ``model-00012-of-00014.safetensors``\n(copies of the base shards) and\n``model-00013-of-00014.safetensors`` / ``model-00014-of-00014.safetensors``\n(freshly written, containing only the extra tensors).\n\n## When to use this skill\n\n| Situation | Use this? |\n| -------------------------------------------------------------------------------------- | --------- |\n| ``extra`` contains a few tensors ``base`` is missing; everything else is identical. | Yes |\n| You need to merge two checkpoints whose overlapping tensors **differ** in value. | No — this skill silently keeps base; you probably want a manual decision per key. |\n| ``extra`` is a full quantization of ``base`` and you want the quantized weights. | No — use ``model_normalize`` with ``--reference extra`` instead. |\n| You want a single ``model.safetensors`` instead of standard shards. | No — repack downstream. |\n\n## Required usage protocol (mandatory for the agent)\n\nBefore invoking ``model_patch.py``:\n\n1. Confirm the three paths (``--base``, ``--extra``, ``--out``) with the user\n via ``AskUserQuestion``. Never infer them from earlier conversation.\n2. Show the user the **diff summary** before writing, by reading the two\n ``model.safetensors.index.json`` files and reporting:\n - number of tensors in base\n - number of tensors in extra\n - number of extra-only keys that will actually be added\n - number of overlapping keys that will be dropped from extra\n3. Ask the user to confirm those numbers before launching the run.\n\nAfter a successful run, **do not** ask the user where the output should go —\n``--out`` is already the final destination the user named. Just report what\nlanded there (shard count, index path, total size, and that base's aux files\nwere mirrored over). Unlike ``model_normalize``, ``model_patch`` has no\nstaging-area concept: there is no temp dir, and the caller already made the\nplacement decision when they passed ``--out``.\n\nIf the user wanted any of ``extra``'s aux files (e.g. a ``config.json`` that\ndocuments a new modality), call that out — those are not copied by the\nskill and must be merged in by hand.\n",".claude/skills/sphinx-debug/SKILL.md":"---\nname: sphinx-debug\ndescription: >\n Use when `make html` or `sphinx-build` fails with\n `Extension error (sphinx.ext.autosummary)`,\n `ImportExceptionGroup`, or other Sphinx autosummary / autodoc\n import failures. Provides automated pdb diagnosis for\n `autodoc_mock_imports` issues.\n---\n\n# Sphinx Debug\n\n## Quick Diagnostic Command\n\nRun the build with maximum verbosity and pdb on exception:\n\n```bash\nmake html SPHINXOPTS=\"-vv -T -P\"\n```\n\nFlags:\n- `-vv`: Verbose autodoc mock logging\n- `-T`: Full traceback\n- `-P`: Drop into pdb on exception\n\n## For agents: automate pdb via monkey-patch\n\nIf you are an agent and cannot interact with an interactive pdb session,\n**do not redesign a subprocess/pty capture system**. Instead, monkey-patch\n`pdb.post_mortem` so it runs scripted commands and prints the output\ndirectly to stdout.\n\nRun this one-shot script from the docs directory (`docs/en` or `docs/zh_cn`):\n\n```bash\ncd docs/en # or docs/zh_cn\npython -c \"\nimport pdb, sys, io, traceback\n\n# Hijack pdb.post_mortem to auto-run diagnostic commands\n_original_post_mortem = pdb.post_mortem\n\ndef scripted_post_mortem(tb=None):\n out = io.StringIO()\n p = pdb.Pdb(stdout=out, stdin=io.StringIO(''))\n p.use_rawinput = False\n p.reset()\n if tb is None:\n tb = sys.exc_info()[2]\n p.setup(None, tb)\n\n old_stderr = sys.stderr\n sys.stderr = out\n\n # List all inner exceptions\n p.onecmd('for i, exc in enumerate(exceptions): print(f\\\"[{i}] {type(exc).__name__}: {exc}\\\")')\n # Print full traceback of the first TypeError (the real root cause)\n p.onecmd('type_errs = [exc for exc in exceptions if type(exc).__name__ == \\\"TypeError\\\"]')\n p.onecmd('if type_errs: import traceback; traceback.print_exception(type(type_errs[0]), type_errs[0], type_errs[0].__traceback__)')\n p.do_quit('')\n\n sys.stderr = old_stderr\n print('=== PDB AUTOMATED OUTPUT ===')\n print(out.getvalue())\n print('=== END OUTPUT ===')\n\npdb.post_mortem = scripted_post_mortem\n\nfrom sphinx.cmd.build import build_main\nbuild_main(['-b', 'html', '-vv', '-T', '-P', '.', '_build/html'])\n\"\n```\n\nThis runs the real Sphinx build with `-P`, but when pdb drops in it\nautomatically executes your commands and dumps the results to stdout.\nYou read the output exactly as if you had typed the pdb commands yourself.\n\n## What to look for in the automated output\n\nSphinx autosummary raises `ImportExceptionGroup` when it cannot resolve a module.\nThe **top-level message is often misleading** (e.g. `no module named xtuner.v1.ray.dataflow`).\nThe real root cause is hidden inside the grouped exceptions.\n\n### Common root-cause patterns\n\n1. **TypeError about `__version__`**\n ```\n TypeError: expected string or bytes-like object, got '__version__'\n ```\n This happens when a mocked module (e.g. `torch`) is imported for real by an\n intermediate library (e.g. `fla`) that calls `packaging.version.parse()` on\n the mocked `__version__` object.\n\n **Fix**: Add the intermediate library to `autodoc_mock_imports` in `conf.py`.\n To identify the exact library, read the traceback: the frame just above\n `packaging/version.py` is the culprit (e.g. `fla/utils.py`).\n\n ```python\n autodoc_mock_imports = [\n ...\n \"fla\", # or whatever library triggers the real import\n ]\n ```\n\n2. **Missing-comma string-concatenation bug**\n\n If `autodoc_mock_imports` contains an entry like `\"scipytorchvision\"`,\n a comma is missing between two string literals in `conf.py`:\n\n ```python\n # WRONG\n autodoc_mock_imports = [\n ...\n \"scipy\"\n \"torchvision\",\n ...\n ]\n\n # CORRECT\n autodoc_mock_imports = [\n ...\n \"scipy\",\n \"torchvision\",\n ...\n ]\n ```\n\n3. **AttributeError after the TypeErrors**\n\n ```\n AttributeError: module 'xtuner.v1.ray' has no attribute 'dataflow'\n ```\n This is usually a *secondary* failure caused by the earlier `TypeError`s.\n Fix the `TypeError` first, then rebuild.\n\n## Step-by-step workflow\n\n1. Run the automated pdb script (see \"For agents\" section above).\n2. Read the `=== PDB AUTOMATED OUTPUT ===` block.\n3. If you see `TypeError: expected string or bytes-like object, got '__version__'`,\n read the traceback to find the intermediate library name.\n4. Add the identified library to `autodoc_mock_imports` in **both**\n `docs/en/conf.py` and `docs/zh_cn/conf.py`.\n5. Also eyeball `autodoc_mock_imports` for missing commas while you are there.\n6. Re-run `make html` (without `-P`) to verify the fix.\n",".claude/skills/xtuner-custom-judger/SKILL.md":"---\nname: xtuner-custom-judger\ndescription: Use when creating, modifying, or reviewing XTuner RL judgers under xtuner/v1/rl/judger or configs that build custom judgers. Enforces the preferred Judger payload contract and the BaseJudger/ComposedJudger constraints.\n---\n\n# XTuner Custom Judger\n\nUse this skill whenever you implement or review an XTuner RL judger.\n\n## Preferred Pattern\n\nPrefer subclassing `Judger`, not `BaseJudger`.\n\nFor custom scoring logic, override these methods as needed:\n\n```python\nclass MyJudger(Judger):\n def preprocess(self, rollout_state: RolloutState) -> JudgerPayload:\n ...\n\n async def judge_payload(self, payload: JudgerPayloadBatch) -> JudgerOutputBatch:\n ...\n\n def postprocess(self, rollout_state: RolloutState, output: JudgerOutput) -> RolloutState:\n ...\n```\n\nDo not override `Judger.judge()` just to read extra fields from `RolloutState`.\nPut the field extraction in `preprocess()` instead.\n\n## Why This Contract Exists\n\n`preprocess -> judge_payload -> postprocess` keeps the scoring payload small and\ncomposable:\n\n- `preprocess()` extracts only the fields required for scoring.\n- `judge_payload()` can run locally, in a Ray actor, or behind a judger pool.\n- `postprocess()` writes the output back to the original `RolloutState`.\n\nThis avoids `deepcopy(RolloutState)` in composed and remote judging. Deep-copying\nrollout states can add serialization overhead and can be risky for large fields,\nRay object refs, tensors, numpy arrays, or externally owned objects.\n\n## BaseJudger Rule\n\nSubclass `BaseJudger` only when the implementation must own the full\n`judge()` / `batch_judge()` flow and cannot reasonably be expressed as a\npayload contract.\n\nBaseJudger-only judgers cannot be used as `ComposedJudger` branches.\n`ComposedJudger` intentionally calls branch `preprocess()` and\n`judge_payload()` rather than arbitrary `BaseJudger.judge()` implementations.\nBecause it does not deep-copy `RolloutState`, supporting arbitrary\nBaseJudger-only branches would allow multiple branches to mutate the same\nrollout state concurrently and overwrite `reward` or other fields.\n\n## ComposedJudger Checklist\n\nWhen adding a branch intended for `ComposedJudger`:\n\n1. Ensure the class inherits `Judger`.\n2. Ensure branch-specific fields are copied into the payload in `preprocess()`.\n3. Ensure `judge_payload()` returns a `JudgerOutput` for one payload, or a\n `list[JudgerOutput]` for a batch payload.\n4. Ensure `postprocess()` writes `rollout_state.reward`.\n5. For multi-branch composed judging, ensure `merge_fn` creates the final\n reward shape, including `reward[\"score\"]` when training expects it.\n\n## Tests\n\nFor custom judger changes, add focused tests for:\n\n- Single-sample `judge()`.\n- `batch_judge()` if batch support is implemented.\n- `ComposedJudger` routing if the judger is intended as a composed branch.\n- Rejection or explicit non-support if the implementation directly subclasses\n `BaseJudger`.\n",".claude/skills/xtuner-sync-supported-models/SKILL.md":"---\nname: xtuner-sync-supported-models\ndescription: Synchronize xtuner's supported model documentation (docs/en/pretrain_sft/advanced_tutorial/model.md and docs/zh_cn/pretrain_sft/advanced_tutorial/model.md) with the actual Config classes defined under xtuner/v1/model/. Use when (1) new TransformerConfig, MoEConfig, or BaseComposeConfig subclasses are added, removed, or renamed in xtuner/v1/model/, (2) existing model configs change their inheritance hierarchy, scale, or HuggingFace counterpart, or (3) a code review or user request points out that model.md is out of sync with the codebase.\n---\n\n# Update XTuner Supported Model Docs\n\nKeep the English and Chinese `model.md` files synchronized with the actual Config classes in `xtuner/v1/model/`.\n\n## Scan the Codebase\n\nRun the bundled scan script from the xtuner project root to discover all Config classes and their inheritance:\n\n```bash\npython3 .agents/skills/xtuner-sync-supported-models/scripts/scan_model_configs.py\n```\n\nThe script outputs JSON with two keys:\n- `configs`: list of every `*Config` class under `xtuner/v1/model/` with its parent classes and file path\n- `children`: parent-to-children mapping for the hierarchy tree\n\n## What to Update\n\nCompare the script output against the two files:\n- `docs/en/pretrain_sft/advanced_tutorial/model.md`\n- `docs/zh_cn/pretrain_sft/advanced_tutorial/model.md`\n\nBoth files share the same structure and must stay in sync:\n\n1. **Base Config Classes** — configs that directly inherit from `TransformerConfig` (or `MoEConfig`) and provide a `from_hf` classmethod for loading HuggingFace weights\n2. **Concrete Model Configs** — fixed-scale subclasses of the base configs above\n3. **Compose Models** — multimodal configs that inherit from `BaseComposeConfig`\n4. **Inheritance Hierarchy** — a text tree showing the full `XTunerBaseModelConfig` hierarchy\n\n### Rules for the Base Config table\n\nInclude these direct descendants of `TransformerConfig`/`MoEConfig`:\n- `Qwen2DenseConfig`\n- `Qwen3DenseConfig`\n- `DeepSeekV3Config`\n- `GptOssConfig`\n- `Qwen3MoEConfig`\n\nExclude from the base table:\n- `MoEConfig` — it is an intermediate base class, not a usable model family\n- `Qwen3_5_VLTextMoEConfig` — it is an intermediate base with only one concrete child; its child `Qwen3_5_VLTextMoE35BA3BConfig` belongs under the MoE concrete table\n\n### Rules for the Concrete Model table\n\nInclude every concrete subclass that has fixed parameter defaults. For each row note:\n- `Config Class`\n- `Base Class / Family`\n- `Architecture Type`: `Dense`, `MoE`, `Dense (VL backbone)`, `MoE (VL backbone)`\n- `Scale / Notes`: parameter count or total/activated size; for VL backbones note \"for multimodal\"\n\n`DeepSeekV3Config` appears here even though it has no separate base entry (it is both base and concrete).\n\n### Rules for the Compose Models section\n\nInclude three sub-tables:\n1. **Compose Base Config Classes** — `Qwen3VLBaseConfig`, `InternVLBaseConfig`, `InternS1BaseConfig`\n - `Qwen3VLBaseConfig`: VL model based on Qwen3 text backbone\n - `InternVLBaseConfig`: VL model based on InternViT + Qwen3\n - `InternS1BaseConfig`: Science multimodal model based on InternViT + Qwen3\n2. **Concrete Compose Model Configs** — every subclass of the above bases; for each row note the wrapped `Text Config` and scale\n\n### Rules for the Inheritance Hierarchy tree\n\nRebuild the tree from `XTunerBaseModelConfig` with two top-level branches:\n\n```text\nXTunerBaseModelConfig\n├── TransformerConfig\n│ ├── Dense Models\n│ │ ├── Qwen2DenseConfig\n│ │ │ └── Qwen2Dense7BConfig\n│ │ └── Qwen3DenseConfig\n│ │ ├── Qwen3Dense8BConfig\n│ │ │ └── Qwen3VLTextDense8BConfig\n│ │ ├── Qwen3Dense4BConfig\n│ │ │ └── Qwen3VLTextDense4BConfig\n│ │ └── Qwen3Dense0P6BConfig\n│ └── MoE Models (via MoEConfig)\n│ ├── DeepSeekV3Config\n│ ├── GptOssConfig\n│ │ ├── GptOss21BA3P6Config\n│ │ └── GptOss117BA5P8Config\n│ ├── Qwen3MoEConfig\n│ │ ├── Qwen3MoE30BA3Config\n│ │ │ └── Qwen3VLTextMoE30BA3Config\n│ │ ├── Qwen3MoE235BA22Config\n│ │ │ └── Qwen3VLTextMoE235BA22Config\n│ │ └── Qwen3MoEFoPEConfig\n│ └── Qwen3_5_VLTextMoEConfig\n│ └── Qwen3_5_VLTextMoE35BA3BConfig\n└── BaseComposeConfig\n ├── Qwen3VLBaseConfig\n │ ├── Qwen3VLMoE30BA3Config\n │ ├── Qwen3VLMoE235BA22Config\n │ ├── Qwen3VLDense4BConfig\n │ ├── Qwen3VLDense8BConfig\n │ └── Qwen3_5_BaseConfig\n │ └── Qwen3_5_VLMoE35BA3Config\n ├── InternVLBaseConfig\n │ ├── InternVL3P5Dense8BConfig\n │ ├── InternVL3P5MoE30BA3Config\n │ └── InternVL3P5Dense1BConfig\n └── InternS1BaseConfig\n ├── InternS1Config\n └── InternS1MiniConfig\n```\n\nWhen new configs are added, insert them into the appropriate branch following the same indentation style.\n\n## Translation Notes\n\nKeep the Chinese `model.md` (`docs/zh_cn/...`) structurally identical to the English one. Translate:\n- Section headings\n- Table header cells\n- Description cells (e.g., \"Image / Video + Text\" → \"图像/视频 + 文本\")\n- Scale descriptions (e.g., \"~7B parameters\" → \"约 7B 参数\", \"FoPE variant\" → \"FoPE 变体\")\n\nDo **not** translate Config class names, file paths, or code identifiers.\n",".claude/skills/xtuner-trace/SKILL.md":"---\nname: xtuner-trace\ndescription: Use when the user wants to instrument, inspect, or debug XTuner traces to reconstruct a sample/request execution path, follow cross-process call chains, or identify latency hotspots, bottleneck stages, and abnormal paths from span data.\n---\n\n# XTuner Trace\n\nUse this skill for the current trace layer in this repository: OpenTelemetry runtime setup, the basic public API, local setup scripts, and the trace viewer.\n\n## Current Boundary\n\nKeep this package infrastructure-only:\n\n- Runtime/configuration: `xtuner/v1/rl/trace/runtime.py`\n- OTel SDK adapter: `xtuner/v1/rl/trace/otel_utils.py`\n- Public facade: `xtuner/v1/rl/trace/__init__.py`\n- Basic API: `xtuner/v1/rl/trace/api.py`\n- Rollout starter preset: `xtuner/v1/rl/trace/rollout_api.py`\n- Local trace tooling and viewer: `recipe/trace/`\n\nDo not add rollout, agent, judger, Ray remote, HTTP proxy, reward, status, or session-server business semantics to `api.py`, `runtime.py`, or `otel_utils.py`. Keep rollout-specific starter behavior inside `rollout_api.py` and gated by `TraceConfig.enable_rollout_trace`.\n\n## Viewer Choice\n\nChoose the viewer by the observation question:\n\n- Use `recipe/trace/viewer` when the user wants rollout-oriented inspection, such as comparing samples within a training step by stage duration, status, reward/filter metadata, or recorded sample call chain.\n- Use Jaeger when the question is not sample/step-centric, such as inspecting raw spans, service boundaries, cross-process causality, backend/client calls, collector/export behavior, or custom instrumentation that does not emit the rollout/sample attributes expected by the recipe viewer.\n\nThe recipe viewer reads `traces.jsonl` and adds XTuner rollout/sample aggregation. Jaeger remains the raw trace drill-down surface for the same traced run.\n\n## Black-Box Agent Trace Routing\n\nWhen the trace request involves an agent, first ask whether this is a black-box\nagent trace. In this mode, XTuner exposes an OpenAI-compatible inference endpoint\nto the agent through `SessionServer`; the agent runs its native loop and calls\nthat endpoint for LLM generation. Tool execution, environment interaction, and\nobservation handling happen inside the agent runtime and are not directly\ninstrumented by XTuner.\n\nIf the answer is yes, or the request mentions trajectory reconstruction,\n`SessionServer` request gaps, `record_synthetic_span(...)`,\n`reconstructed_agent_run`, `reconstructed_tool_call.N`, XTuner viewer plus\nJaeger display of recovered tool calls, read\n[references/blackbox-agent-trace.md](references/blackbox-agent-trace.md) before\nplanning or editing.\n\n## Basic API\n\nThese are the interfaces defined in `xtuner/v1/rl/trace/api.py` and re-exported\nfrom `xtuner.v1.rl.trace`:\n\n- `trace_span(name, attributes=None, parent_carrier=None)`\n- `trace_function(name=None, attributes=None)`\n- `trace_event(name, attributes=None)`\n- `set_trace_attributes(attributes)`\n- `inject_trace_context(carrier=None)`\n\n## Runtime API\n\nUse these `xtuner/v1/rl/trace/runtime.py` interfaces for explicit trace\nruntime setup:\n\n- `TraceConfig`\n- `configure_trace(...)`\n- `close_trace()`\n\n## Rollout Starter Preset\n\n`TraceConfig(enable_rollout_trace=True)` enables the built-in rollout starter\ntrace. Its helpers live in `xtuner/v1/rl/trace/rollout_api.py`:\n\n- `trace_rollout_endpoint(...)`\n- `trace_rollout_remote(...)`\n\nKeep this layer narrow: it may depend on `RolloutState` and Ray call boundaries,\nbut it must not leak those dependencies into the basic API.\n\nRollout endpoint `xtuner.span_name_path` is a sample call chain, not just the\ncurrent process context stack. `trace_rollout_endpoint(...)` should append the\nendpoint span name to an internal trace-only chain carried in\n`RolloutState.extra_fields`, pass that state through remote rollout boundaries,\nand clean the internal field at the outermost endpoint. For example, after a\nsample generates and then judges, `judger.run` should record a path like\n`single_turn_agent_loop.run -> rollout.controller.generate -> rollout.worker.generate -> judger.run`\neven though the local active context at judge time may only be\n`single_turn_agent_loop.run -> judger.run`. Keep this behavior in\n`rollout_api.py` and viewer tests; do not add rollout call-chain semantics to\nthe basic trace API.\n\n## Add Trace Workflow\n\nWhen adding trace instrumentation to an XTuner run:\n\n1. Ask for or locate the launch script and training config before editing.\n2. In the launch script, source `recipe/trace/setup_trace.sh` when\n `XTUNER_TRACE_ENABLED=1`.\n3. In the training config, add `TraceConfig` and set `enabled=True`; set\n `xtuner_viewer_enabled=True` when the user needs interactive sample/step inspection. Set\n `enable_rollout_trace=True` only when the user wants the built-in rollout\n starter trace.\n4. Before adding any `trace_span(...)` instrumentation, you must ask the user which\n stages they want to observe and which metrics each stage should expose.\n Do not infer default stages unless the user explicitly asks you to choose.\n5. Add `trace_span(...)` only around the user-confirmed observed stages. Put fields known at span\n start in initial attributes, and update runtime or final fields with\n `set_trace_attributes(...)`.\n6. For cross-process or request boundaries, inject a carrier with\n `inject_trace_context(...)` and pass it to downstream\n `trace_span(..., parent_carrier=carrier)`.\n7. Keep transport-specific propagation at the caller boundary; do not move\n rollout, agent, judger, Ray, or HTTP semantics into the basic trace package.\n8. Ensure the main training log includes the trace output path, XTuner viewer URL,\n and equivalent manual viewer command when the XTuner viewer is enabled.\n\n## Guardrails\n\n- Do not recreate `trace_utils.py`, `context_propagation.py`, span-name registries, or business attribute builders under `xtuner/v1/rl/trace`.\n- Do not import `RolloutState`, agent item classes, judgers, rollout workers, Ray actors, aiohttp clients, or trainer configs from the basic trace package.\n- Do not call OpenTelemetry SDK directly from business code; use the basic API only when trace instrumentation is explicitly requested.\n- Do not record prompts, responses, full configs, secrets, raw headers, stack traces, or large payloads as attributes.\n- Viewer stage grouping must come from span attributes such as `xtuner.stage`, `stage`, or `stage.name`; otherwise fall back to the raw span name.\n\nFor concrete patterns, read [references/trace-patterns.md](references/trace-patterns.md) before editing trace code.\n\n## Verification\n\nUse focused checks:\n\n- `PYTHONPATH=. python -m compileall -q xtuner/v1/rl/trace recipe/trace`\n- `PYTHONPATH=. python -m unittest discover -s tests/rl -p 'test_trace*.py' -v` when the trace tests exist in the worktree.\n- `git diff --check`\n",".claude/skills/xtuner-trace/references/blackbox-agent-trace.md":"# Blackbox Agent Trace Reference\n\nThis reference captures the XTuner black-box agent trace reconstruction workflow. It is intended for implementation and review work, not as user-facing docs.\n\nRead this reference when `xtuner-trace/SKILL.md` routes an agent trace request here. The design background lives in `docs/design/blackbox_agent_trace_reconstruction.md`.\n\n## Contents\n\n- [Decision Flow](#decision-flow)\n- [Path A: Trajectory-Backed Reconstruction](#path-a-trajectory-backed-reconstruction)\n- [Path B: SessionServer-Only Fallback](#path-b-sessionserver-only-fallback)\n- [`record_synthetic_span(...)` Usage](#record_synthetic_span-usage)\n- [Reconstructed Span Shape](#reconstructed-span-shape)\n- [Viewer And Jaeger Requirements](#viewer-and-jaeger-requirements)\n- [Example: mini-SWE-agent Trajectory Conversion](#example-mini-swe-agent-trajectory-conversion)\n- [Verification](#verification)\n- [Common Failure Modes](#common-failure-modes)\n\n## Decision Flow\n\nWhen adding trace support for a black-box agent, classify the available evidence before deciding where to instrument.\n\n1. Confirm this is a black-box agent path: XTuner exposes an OpenAI-compatible inference endpoint to the agent through `SessionServer`; the agent runs its native loop and calls that endpoint for LLM generation. Tool execution, environment interaction, and observation handling happen inside the agent runtime and are not directly instrumented by XTuner.\n2. Confirm whether the agent can save a trajectory for each run.\n3. If a trajectory exists, check whether it contains tool-call metadata such as tool name, action/command, status, return code, error information, observation, or timestamp.\n4. Use Path A when the trajectory contains tool-call metadata. The trajectory supplies tool semantics, and `session_server.request` spans supply the LLM request timeline.\n5. Use Path B only when no useful trajectory exists. This can recover request gaps, but cannot recover reliable tool names, return codes, errors, or observations.\n\nDo not require the trajectory to contain complete LLM generate timing before using Path A. `SessionServer` remains the stable source for LLM request start/end times. If the trajectory also records LLM calls, use them as an optional consistency check.\n\n## Path A: Trajectory-Backed Reconstruction\n\nUse this path when the agent can save a trajectory with tool-call metadata.\n\n### Data Contract\n\nThe `session_server.request` spans must provide:\n\n- `xtuner.session_id` or `xtuner.rollout_id`\n- span start time\n- span end time\n\nThe trajectory should provide best-effort tool metadata:\n\n- tool event order, id, or enough context to infer order\n- tool name, action, or command\n- status, return code, or exception information\n- observation or output summary\n- tool start/end timestamps, or a single tool-related timestamp with documented semantics, if the agent records one\n\nDo not define a universal trajectory schema in trace core. Agent trajectory formats are agent-specific; keep parsing inside that agent's recipe or trajectory-to-trace adapter until multiple agents prove a shared model is worth extracting.\n\n### Where To Add Trace\n\nAdd trace at the three black-box observation boundaries. These instructions must be enough to re-create the instrumentation even when no example implementation exists in the checkout.\n\n#### 1. SessionServer Request Trace\n\nFirst check whether the `SessionServer` HTTP request handler already emits `session_server.request` spans. If not, add this instrumentation to the handler that receives the agent's OpenAI-compatible request.\n\nImplementation requirements:\n\n- Locate the outer request handler, such as an aiohttp/FastAPI handler named like `_handle_request(request)`.\n- Read enough request headers/body to determine:\n - `session_id`\n - optional rollout id\n - whether the request is streaming\n - optional trace carrier from standard headers or a request-body carrier field\n- Wrap the full inference request lifecycle in:\n\n```python\nwith trace_span(\n \"session_server.request\",\n attributes={\n \"xtuner.stage\": \"llm_generate\",\n \"xtuner.session_id\": session_id,\n \"xtuner.rollout_id\": rollout_id or session_id,\n \"session.stream\": is_streaming,\n },\n parent_carrier=parent_carrier,\n):\n ...\n```\n\n- Keep the outer `session_server.request` span around prepare, backend request, response read, response post-processing, and return-to-agent.\n- Use `set_trace_attributes(...)` before exiting the span to record request-level status or error metadata when available.\n- If adding child spans, use these names:\n - `session_server.prepare_request`\n - `session_server.send_request`\n - `session_server.read_response`\n - `session_server.record_response`\n- Put `xtuner.session_id` on every child span too. This makes later trajectory-to-trace adapters able to filter a single agent run reliably.\n\nDo not parse agent trajectory or infer tool calls inside `SessionServer`. Its black-box responsibility is to provide the LLM request timeline.\n\n#### 2. Agent Run Boundary Trace\n\nIn the black-box recipe or launcher, find the call site that invokes the agent's native run loop, such as `agent.run(...)`.\n\nWrap only the outer run boundary:\n\n```python\nwith trace_span(\n \".run\",\n attributes={\n \"xtuner.stage\": \"agent.run\",\n \"xtuner.session_id\": session_id,\n \"xtuner.rollout_id\": rollout_id or session_id,\n \"agent.name\": agent_name,\n },\n):\n agent.run(...)\n```\n\nThis span is a lifecycle boundary. Do not wrap the agent's internal model, environment, or tool methods as the default black-box trace path. LLM/tool detail should come from `SessionServer` spans plus trajectory reconstruction.\n\n#### 3. Trajectory-To-Trace Reconstruction\n\nAfter the native agent run finishes, call the trajectory-to-trace adapter before `close_trace()`. Prefer a `finally` block so partial trajectories can still be converted after failures:\n\n```python\nreturn_code = 1\ntry:\n with trace_span(\".run\", attributes=agent_run_attributes):\n return_code = agent.run(...)\nfinally:\n emit_reconstructed_trace(\n trace_jsonl=trace_jsonl_path,\n trajectory=trajectory_path,\n session_id=session_id,\n )\n close_trace()\nreturn return_code\n```\n\nThe adapter must read the trajectory, read same-session `session_server.request` spans, derive tool timing from trajectory timestamps when possible, use request gaps only for correlation or inferred fallback, and emit reconstructed synthetic spans with `record_synthetic_span(...)`.\n\n### Adapter Responsibilities\n\nWrite a trajectory-to-trace adapter for the specific agent. The adapter should:\n\n1. Load the agent trajectory.\n2. Extract ordered tool events and their metadata.\n3. Load `session_server.request` spans from the same `traces.jsonl`.\n4. Filter request spans by `session_id`.\n5. Sort request spans by start time and assign local indexes.\n6. Extract tool timestamps from the trajectory and interpret their semantics:\n - If the trajectory provides reliable tool start/end times, use them as the `reconstructed_tool_call.N` span boundaries and set `xtuner.inferred=false`.\n - If the trajectory provides only one timestamp, record what it means if known, such as tool start, tool end, observation write, or trajectory write. Use it to correlate the tool event with the surrounding `session_server.request` window.\n - If the trajectory timestamp semantics are unknown or incomplete, use request-gap boundaries only as an inferred fallback and set `xtuner.inferred=true`.\n7. Use `session_server.request` spans as the LLM timeline and correlation windows, not as the default source of trajectory-backed tool timing.\n8. When using request-gap fallback for a bounded tool event, use:\n\n```text\ntool_i.start = request_i.end\ntool_i.end = request_{i+1}.start\n```\n\n9. Emit `reconstructed_agent_run`, `reconstructed_llm_call.N`, and bounded `reconstructed_tool_call.N` spans with `record_synthetic_span(...)`.\n10. Treat final tool events without a next LLM request as unbounded unless the trajectory has trustworthy start/end times.\n\nWhen request-gap fallback is used, the interval is not pure tool runtime. It can include LLM-response parsing, user confirmation time, tool execution, observation formatting, trajectory write time, and loop overhead before the next LLM request.\n\n## Path B: SessionServer-Only Fallback\n\nUse this path only when the agent does not save a useful trajectory.\n\n### Where To Add Trace\n\nAdd trace to the `SessionServer` request path so every LLM request produces a reliable timeline. At minimum, `session_server.request` spans must include:\n\n- `xtuner.session_id`\n- `xtuner.rollout_id`, falling back to the session id when there is no rollout id\n- request start time\n- request end time\n\nSupporting spans such as `session_server.prepare_request`, `session_server.send_request`, `session_server.read_response`, and `session_server.record_response` may remain useful for debugging latency, but the fallback reconstruction only requires the outer `session_server.request` start/end times.\n\nDo not put tool inference inside `SessionServer`. `SessionServer` should not parse trajectories, infer tool names, track return codes, or maintain complex per-session reconstruction state.\n\n### Fallback Reconstruction\n\nAfter the agent run, a fallback adapter may infer intervals from adjacent request spans:\n\n```text\ninferred_tool_call_i.start = request_i.end\ninferred_tool_call_i.end = request_{i+1}.start\n```\n\nThe fallback can emit synthetic spans such as `inferred_tool_call.N` with:\n\n```text\nxtuner.stage=tool_call\nxtuner.synthetic=true\nxtuner.inferred=true\nxtuner.session_id=\nreconstruction.source=session_server_only\nreconstruction.method=session_server_gap\n```\n\nDo not claim unavailable metadata in this mode. SessionServer-only fallback cannot reliably recover tool name, command, return code, stderr/stdout, agent exception, skipped action, or observation content.\n\nDo not infer the final tool event without a next LLM request. The interval has no reliable end time.\n\n## `record_synthetic_span(...)` Usage\n\nUse `record_synthetic_span(...)` to write reconstructed historical intervals back into the active XTuner trace runtime.\n\n```python\nrecord_synthetic_span(\n name: str,\n *,\n start_time_unix_ns: int,\n end_time_unix_ns: int,\n attributes: Mapping[str, Any] | None = None,\n parent_carrier: Mapping[str, str] | None = None,\n status: str = \"completed\",\n error_message: str | None = None,\n) -> dict[str, str] | None\n```\n\nRules:\n\n- Call it before `close_trace()`.\n- Use absolute Unix nanoseconds, not monotonic timestamps.\n- Ensure `end_time_unix_ns > start_time_unix_ns`.\n- Let the API default `xtuner.synthetic=true`.\n- Set `xtuner.inferred=true` only when the time boundary is inferred, such as from `session_server.request` gaps.\n- Use `status=\"error\"` and `error_message=...` for real errors.\n- Use `parent_carrier` to attach reconstructed children under one synthetic root trace.\n- Treat `None` return as trace-disabled no-op.\n\nTypical root and child pattern:\n\n```python\nroot_ids = record_synthetic_span(\n \"reconstructed_agent_run\",\n start_time_unix_ns=first_request.start_time_unix_ns,\n end_time_unix_ns=last_request.end_time_unix_ns,\n attributes={\n \"xtuner.stage\": \"agent.run\",\n \"xtuner.session_id\": session_id,\n \"xtuner.inferred\": False,\n \"reconstruction.method\": \"trajectory_session_timeline\",\n },\n)\nroot_carrier = None\nif root_ids:\n root_carrier = {\"traceparent\": f\"00-{root_ids['trace_id']}-{root_ids['span_id']}-01\"}\n\nrecord_synthetic_span(\n \"reconstructed_tool_call.1\",\n start_time_unix_ns=request_1.end_time_unix_ns,\n end_time_unix_ns=request_2.start_time_unix_ns,\n attributes={\n \"xtuner.stage\": \"tool_call\",\n \"xtuner.session_id\": session_id,\n \"xtuner.inferred\": True,\n \"agent.tool_call_index\": 1,\n \"agent.tool.name\": \"bash\",\n \"reconstruction.source\": \"agent_trajectory\",\n \"reconstruction.method\": \"session_server_gap\",\n },\n parent_carrier=root_carrier,\n)\n```\n\nDo not call OpenTelemetry SDK directly from the adapter. Use XTuner trace APIs.\n\n## Reconstructed Span Shape\n\nTrajectory-backed reconstruction should generate one synthetic root trace:\n\n```text\nreconstructed_agent_run\n|-- reconstructed_llm_call.1\n|-- reconstructed_tool_call.1\n|-- reconstructed_llm_call.2\n|-- reconstructed_tool_call.2\n...\n```\n\nUse these naming rules:\n\n- `reconstructed_agent_run` for the synthetic root.\n- `reconstructed_llm_call.N` for LLM request intervals copied from `session_server.request`.\n- `reconstructed_tool_call.N` for trajectory-backed tool metadata.\n- `inferred_tool_call.N` only for SessionServer-only fallback without trajectory metadata.\n\nRecommended root attributes:\n\n```text\nxtuner.stage=agent.run\nxtuner.synthetic=true\nxtuner.inferred=false\nxtuner.session_id=\nxtuner.rollout_id=\nagent.name=\nagent.llm_call_count=\nagent.tool_call_count=\nreconstruction.source=\nreconstruction.method=trajectory_session_timeline\n```\n\nRecommended LLM attributes:\n\n```text\nxtuner.stage=llm_generate\nxtuner.synthetic=true\nxtuner.inferred=false\nagent.llm_call_index=N\nreconstruction.source=session_server_request\nreconstruction.method=session_server_request_copy\nreconstruction.original_trace_id=\nreconstruction.original_span_id=\nreconstruction.original_operation=session_server.request\n```\n\nRecommended tool attributes:\n\n```text\nxtuner.stage=tool_call\nxtuner.synthetic=true\nxtuner.inferred=true\nagent.tool_call_index=N\nagent.tool.name=\nagent.tool.status=\nagent.tool.returncode=\nagent.tool.command_preview=\nreconstruction.source=\nreconstruction.method=session_server_gap\nreconstruction.timestamp_check=\n```\n\nDo not call trajectory-backed spans `inferred_tool_call.N`. Use `reconstructed_tool_call.N` and put uncertainty in attributes.\n\n## Viewer And Jaeger Requirements\n\nXTuner viewer and Jaeger must consume the same reconstructed spans. Do not implement one inference path in the viewer and another inference path for Jaeger.\n\nViewer expectations:\n\n- Show one sample per agent run/session after live and completed records are merged.\n- Prefer the trace containing `reconstructed_agent_run` as the primary sample trace.\n- Keep raw `session_server.request` trace ids available for audit.\n- Display the reconstructed path:\n\n```text\nreconstructed_agent_run -> reconstructed_llm_call.1 -> reconstructed_tool_call.1 -> ...\n```\n\nJaeger expectations:\n\n- The viewer's Jaeger link should open the trace containing `reconstructed_agent_run`.\n- Opening the first raw `session_server.request` trace is a bug for reconstructed black-box samples.\n- A healthy reconstructed trace contains `1 + llm_count + bounded_tool_count` reconstructed spans.",".claude/skills/xtuner-trace/references/trace-patterns.md":"# XTuner Trace Patterns\n\nThese patterns describe the retained trace surface in this branch: runtime, basic API, viewer, and local setup tooling.\n\n## Basic API\n\nUse the public facade:\n\n```python\nfrom xtuner.v1.rl.trace import (\n inject_trace_context,\n set_trace_attributes,\n trace_event,\n trace_function,\n trace_span,\n)\n```\n\n### Local Span\n\n```python\nwith trace_span(\"phase.name\", attributes={\"xtuner.stage\": \"phase\"}):\n ...\n set_trace_attributes({\"phase.count\": 3})\n```\n\n### Decorated Function\n\n```python\n@trace_function(\"phase.load\")\ndef load_item(path: str) -> object:\n ...\n```\n\n### Parent Carrier\n\n```python\ncarrier = inject_trace_context()\n\nwith trace_span(\"child.phase\", parent_carrier=carrier):\n ...\n```\n\n`parent_carrier` is a basic W3C context carrier. It is not tied to `RolloutState`, Ray, aiohttp, or any XTuner business object.\n\n## End-to-End Trace Wiring\n\nUse this pattern when adding trace support to an XTuner training run.\n\n### Launch Script\n\nEnable trace bootstrap from the launcher, guarded by `XTUNER_TRACE_ENABLED`:\n\n```bash\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\nREPO_ROOT=\"$(cd \"${SCRIPT_DIR}/../../..\" && pwd)\"\nif [ \"${XTUNER_TRACE_ENABLED:-0}\" = \"1\" ]; then\n source \"${REPO_ROOT}/recipe/trace/setup_trace.sh\"\nfi\n```\n\nKeep training stdout/stderr in the main training log so runtime trace messages\nare visible:\n\n```bash\npython xtuner/v1/train/cli/rl.py \\\n --config \"$CONFIG_PATH\" \\\n --num-workers \"$XTUNER_RL_NUM_WORKERS\" \\\n 2>&1 | tee -a \"${WORK_DIR}/training_log_${current_time}.txt\"\n```\n\n### Training Config\n\nAdd `TraceConfig` to the training config and pass it to the trainer config:\n\n```python\nimport os\n\nfrom xtuner.v1.rl.trace import TraceConfig\n\n\ntrace_config = TraceConfig(\n enabled=os.environ.get(\"XTUNER_TRACE_ENABLED\") == \"1\",\n service_name=\"xtuner-agent-rollout\",\n xtuner_viewer_enabled=True,\n xtuner_viewer_host=\"0.0.0.0\",\n xtuner_viewer_port=18080,\n xtuner_viewer_jaeger_query_url=\"http://127.0.0.1:16686\",\n enable_rollout_trace=True,\n)\n\ntrainer = RLColocateTrainerConfig(\n ...,\n trace_config=trace_config,\n)\n```\n\nWhen `output_dir` is omitted in an RL trainer config, the trainer places trace\nartifacts under the current experiment directory, for example\n`work_dirs/name/20260713071356/otel/`.\n\n### Stage Spans\n\nBefore adding spans, ask the user which stages they want to observe and which\nmetrics each stage should expose.\n\nPut fields known when entering the stage, especially fields used for grouping\nor filtering, in `trace_span(..., attributes=...)`. Put results known only\nafter or during execution, such as status, duration, token counts, errors, and\nreward values, in `set_trace_attributes(...)`. This lets the viewer reconstruct\nthe call chain and identify latency hotspots or abnormal stages.\n\nInfer stage:\n\n```python\nimport time\n\n\nstarted_at = time.monotonic()\nattributes = {\n \"xtuner.stage\": \"infer\",\n \"xtuner.task_name\": task_name,\n \"xtuner.sample_id\": sample_id,\n \"rollout.backend\": backend,\n}\n\nwith trace_span(\"agent.infer\", attributes=attributes):\n result = await run_infer(...)\n set_trace_attributes(\n {\n \"xtuner.status\": \"ok\" if result.ok else \"error\",\n \"prompt.tokens\": result.prompt_tokens,\n \"completion.tokens\": result.completion_tokens,\n \"stage.duration_ms\": int((time.monotonic() - started_at) * 1000),\n }\n )\n```\n\nResource acquire stage:\n\n```python\nwith trace_span(\n \"sandbox.acquire\",\n attributes={\n \"xtuner.stage\": \"acquire\",\n \"xtuner.sample_id\": sample_id,\n \"sandbox.name\": sandbox_name,\n },\n):\n client = await pool.get(sandbox_name)\n set_trace_attributes(\n {\n \"sandbox.env_id\": env_id,\n \"sandbox.image\": image,\n \"sandbox.reused\": reused,\n }\n )\n```\n\nValidation or reward stage:\n\n```python\nwith trace_span(\n \"sample.validate\",\n attributes={\n \"xtuner.stage\": \"validate\",\n \"xtuner.sample_id\": sample_id,\n \"validator.name\": validator_name,\n },\n):\n record = await validate(...)\n set_trace_attributes(\n {\n \"xtuner.status\": \"ok\" if record.ok else \"error\",\n \"reward.score\": record.score,\n \"reward.passed\": record.passed,\n }\n )\n\n if record.error is not None:\n set_trace_attributes(\n {\n \"error\": True,\n \"error.type\": record.error.type,\n \"error.message\": record.error.message,\n }\n )\n```\n\nDo not record prompts, responses, full configs, secrets, raw headers, stack\ntraces, or large payloads.\n\n### Cross-Boundary Propagation\n\nWhen a traced operation crosses a process, actor, or HTTP boundary, keep the\npropagation helper in the business or transport module that owns that boundary.\nDo not put Ray, HTTP, rollout, agent, or judger helpers back into\n`xtuner/v1/rl/trace`.\n\n#### Ray Remote Calls\n\nFor Ray calls, inject the current context into a plain carrier and temporarily\nattach that carrier to a serializable domain object or explicit call argument.\n\n```python\nfrom collections.abc import Mapping\nfrom contextlib import contextmanager\nfrom typing import Any\n\nfrom xtuner.v1.rl.trace import inject_trace_context, trace_span\n\n\n_TRACE_CARRIER_FIELD = \"_xtuner_trace_carrier\"\n\n\n@contextmanager\ndef attach_trace_carrier_temporarily(target: Any, carrier: Mapping[str, str]):\n if not carrier:\n yield\n return\n\n extra_fields = getattr(target, \"extra_fields\", None)\n if extra_fields is None:\n extra_fields = {}\n target.extra_fields = extra_fields\n had_previous = _TRACE_CARRIER_FIELD in extra_fields\n previous = extra_fields.get(_TRACE_CARRIER_FIELD)\n extra_fields[_TRACE_CARRIER_FIELD] = dict(carrier)\n try:\n yield\n finally:\n if had_previous:\n extra_fields[_TRACE_CARRIER_FIELD] = previous\n else:\n extra_fields.pop(_TRACE_CARRIER_FIELD, None)\n\n\ndef call_ray_remote_with_trace(remote_method, *args, trace_target: Any, **kwargs):\n carrier: dict[str, str] = {}\n inject_trace_context(carrier)\n with attach_trace_carrier_temporarily(trace_target, carrier):\n return remote_method.remote(*args, **kwargs)\n\n\ndef pop_trace_parent_carrier(target: Any) -> dict[str, str] | None:\n extra_fields = getattr(target, \"extra_fields\", None)\n if not isinstance(extra_fields, dict):\n return None\n carrier = extra_fields.pop(_TRACE_CARRIER_FIELD, None)\n if not isinstance(carrier, Mapping):\n return None\n return {str(key): str(value) for key, value in carrier.items()}\n```\n\nOn the receiver side, attach the carrier to the new span:\n\n```python\nparent_carrier = pop_trace_parent_carrier(trace_target)\n\nwith trace_span(\"worker.stage\", attributes=attributes, parent_carrier=parent_carrier):\n ...\n```\n\nKeep the helper domain-specific. Validate that exactly one trace target is used\nwhen the transport needs one target object; reject collections if restoring the\ncarrier to the right child span would be ambiguous.\n\n#### HTTP Calls\n\nFor HTTP calls, prefer W3C headers. If a third-party client or proxy may drop\ncustom headers, also put a copied carrier into the JSON body under an internal\nfield and remove it before forwarding the payload downstream.\n\n```python\nfrom collections.abc import Mapping\nfrom typing import Any\n\nfrom xtuner.v1.rl.trace import inject_trace_context, trace_span\n\n\n_TRACE_CARRIER_FIELD = \"_xtuner_trace_carrier\"\n\n\ndef inject_trace_carrier_into_json_body(kwargs: dict[str, Any], carrier: Mapping[str, str]) -> None:\n if not carrier:\n return\n payload = kwargs.get(\"json\")\n if not isinstance(payload, dict):\n return\n payload = dict(payload)\n payload.setdefault(_TRACE_CARRIER_FIELD, dict(carrier))\n kwargs[\"json\"] = payload\n\n\ndef extract_trace_carrier_from_mapping(payload: Mapping[str, Any] | None) -> dict[str, str] | None:\n if not isinstance(payload, Mapping):\n return None\n carrier = payload.get(_TRACE_CARRIER_FIELD)\n if not isinstance(carrier, Mapping):\n return None\n return {str(key): str(value) for key, value in carrier.items()}\n\n\ndef remove_trace_carrier_from_mapping(payload: dict[str, Any]) -> None:\n payload.pop(_TRACE_CARRIER_FIELD, None)\n\n\ndef extract_http_parent_carrier(headers: Mapping[str, Any], payload: Mapping[str, Any] | None) -> dict[str, str] | None:\n header_carrier = {\n str(key): str(value)\n for key, value in headers.items()\n if str(key).lower() in {\"traceparent\", \"tracestate\", \"baggage\"}\n }\n return header_carrier or extract_trace_carrier_from_mapping(payload)\n```\n\nCaller:\n\n```python\nheaders = dict(headers or {})\nrequest_kwargs = {\"json\": payload}\ncarrier: dict[str, str] = {}\ninject_trace_context(carrier)\nheaders.update(carrier)\ninject_trace_carrier_into_json_body(request_kwargs, carrier)\n\nwith trace_span(\"http.client.request\", attributes={\"http.method\": \"POST\", \"http.url\": url}):\n response = await client.post(url, headers=headers, **request_kwargs)\n```\n\nReceiver:\n\n```python\nheaders = dict(request.headers)\npayload = await request.json()\nparent_carrier = extract_http_parent_carrier(headers, payload)\n\nwith trace_span(\"http.server.request\", attributes=attributes, parent_carrier=parent_carrier):\n remove_trace_carrier_from_mapping(payload)\n ...\n```\n\nDo not record raw headers or full payloads as span attributes. Only record\nderived fields such as method, route, status code, stage, IDs, and timing.\n\n### Viewer Output\n\nWhen `xtuner_viewer_enabled=True`, the trace runtime logs the trace JSONL path,\nthe XTuner viewer URL, and the equivalent manual viewer command. The viewer\ndefaults to port `18080`; `xtuner_viewer_port=0` is still available when a run\nshould pick a free port automatically. The viewer process inherits the training\nstdout/stderr; keep those runtime log lines visible in the main training log and\nreport them to the user after the run.\n\nUseful reference points from the full trace implementation:\n\n- `examples/v1/scripts/run_rl_run.sh`\n- `recipe/trace/setup_trace.sh`\n- `examples/v1/config/agentic_rl_qwen3p5vl_mtp_ep_code.py`\n\n## Runtime\n\nConfigure tracing explicitly:\n\n```python\nfrom xtuner.v1.rl.trace import TraceConfig, close_trace, configure_trace\n\nruntime = configure_trace(\n TraceConfig(\n enabled=True,\n output_dir=\"work_dirs/example/otel\",\n service_name=\"xtuner\",\n xtuner_viewer_enabled=True,\n )\n)\n\ntry:\n ...\nfinally:\n close_trace()\n```\n\nThe runtime owns OTel collector setup, trace JSONL output, and optional lightweight XTuner viewer startup.\n\n## Viewer\n\nThe viewer reads Jaeger-style traces or OTel JSONL converted into Jaeger-style payloads. It should not depend on hard-coded rollout/agent span registries.\n\nStage display rules:\n\n1. Use `xtuner.stage` when present.\n2. Else use `stage` when present.\n3. Else use `stage.name` when present.\n4. Else fall back to the raw span name.\n\nUseful stable attributes:\n\n- IDs: `xtuner.rollout_id`, `xtuner.group_id`, `xtuner.session_id`, `xtuner.task_name`\n- Status: `xtuner.status`\n- Stage: `xtuner.stage`, `stage`, `stage.name`\n- Counts/timing: `prompt.tokens`, `completion.tokens`, `http.status_code`\n- Errors: `error`, `error.message`, `exception.type`\n\nAvoid recording prompts, responses, full configs, secrets, raw headers, or large payloads.\n\n## Local Setup\n\n`recipe/trace/setup_trace.sh` and the rest of `recipe/trace/` are local helper tooling for installing OTel collector binaries, starting the local Jaeger dependency, and clearing stale `recipe.trace.viewer.server` processes on `XTUNER_TRACE_VIEWER_PORT` or the default viewer port `18080`. Keep these as setup assets; do not use them to add automatic trace behavior to training configs or launch scripts unless that integration is explicitly requested.\n\n## Rollout Starter Preset\n\nThe built-in rollout starter trace is opt-in:\n\n```python\ntrace_config = TraceConfig(\n enabled=True,\n xtuner_viewer_enabled=True,\n enable_rollout_trace=True,\n)\n```\n\nIts rollout-specific helpers live in `xtuner/v1/rl/trace/rollout_api.py`:\n\n- `trace_rollout_endpoint(...)`\n- `trace_rollout_remote(...)`\n\nUse this only for the starter rollout chain. For custom instrumentation, ask\nwhich stages and metrics the user wants before adding spans.\n\n## Removed Legacy Semantics\n\nDo not use or recreate these removed surfaces in this branch:\n\n- `trace_remote`\n- `traced_rollout_endpoint`\n- `traced_agent_item_endpoint`\n- `traced_judger_endpoint`\n- `xtuner/v1/rl/trace/trace_utils.py`\n- `xtuner/v1/rl/trace/context_propagation.py`\n- fixed `TRACE_SPAN_*` registries\n- rollout/agent/judger attribute builders\n",".codex/AGENTS.md":"../.claude/CLAUDE.md","xtuner/tools/llm_calc/README.md":"### Summary\n\nThis PR introduces a comprehensive GPU memory calculator for LLM training, which helps users estimate per-GPU memory consumption before actually running training jobs. This tool is essential for capacity planning and optimizing parallel configurations.\n\n### Features\n\n- **Memory Breakdown Analysis**: Calculates detailed memory usage including:\n - Model parameters (embedding, attention, MLP, head)\n - Gradients\n - Optimizer states (Adam momentum & variance)\n - Master parameters and gradients (for mixed-precision training)\n - Intermediate activations\n\n- **Parallel Strategy Support**:\n - Data Parallelism with ZeRO Stage 1(for Megatron)/3 (FSDP for xtuenr)\n - Tensor Parallelism (TP)\n - Pipeline Parallelism (PP) and Virtual Pipeline Parallelism (VPP) for Megatron\n - Pipeline parallelism memory accounts for flying micro-batches during 1F1B schedule\n - Expert Parallelism (EP) for MoE models\n\n- **Model Architecture Support**:\n - Dense models (e.g., Qwen3-8B)\n - MoE models (e.g., Qwen3-235B-A22B, DeepSeek-V3-671B)\n - Multi-Head Attention (MHA)\n - Multi-Latent Attention (MLA) used in DeepSeek\n\n- **Training Optimizations**:\n - Activation checkpointing (recomputation)\n - Flash Attention\n - Chunked loss computation\n\n### Usage\n\n```bash\npython xtuner/tools/llm_calc/llm_calculator.py -c [-o ]\n```\n\n### Example Configurations\n\n| Config File | Model | Scale | TrainFramework |\n|------------|-------|-------|-------|\n| `xtuner_qwen_dense_8b_1node.yaml` | Qwen3-8B (Dense) | 1 node | xtuner |\n| `xtuner_qwen_moe_30b_1node.yaml` | Qwen3-30B-A3B (MoE) | 1 node | xtuner |\n| `xtuner_qwen_moe_235b_64node.yaml` | Qwen3-235B-A22B (MoE) | 64 nodes | xtuner |\n| `xtuner_deepseekv3_671b_128node.yaml` | DeepSeek-V3-671B (MoE) | 128 nodes | xtuner |\n| `megatron_qwen_moe_235b_64node.yaml` | Qwen3-235B-A22B (MoE) | 64 nodes | Megatron |\n\n### Output Example\n\nThe tool outputs:\n1. Total model parameters breakdown\n2. Per-GPU memory consumption for different training phases (forward/backward)\n3. Detailed memory breakdown by component (params, grads, optimizer states, activations)\n4. Optional Excel export for further analysis\n \n```\n# python llm_calculator.py -c xtuner_qwen_dense_8b_1node.yaml\n----------------------------------------------setting:----------------------------------------------\nmodel:Qwen3-8B zero_stage:3 tp:1 pp:1 dp:8 ep:1 vpp:1 nodes:1 mbs:1 gbs:8 L:32768 recompute:True flash_attn:True capacity:1.0 LN:36\n----------------------------------------------------------------------------------------------------\nTotal_params_num: 7.628208160400391 B, embed_params_num: 0.57958984375 B, attn_params_num: 1.4063873291015625 B, mlp_params_num: 5.0626373291015625 B, head_params_num: 0.5795936584472656 B\nMax_mem for microbatch=1: 26.58784294128418 GiB. micro1_forward_last: 26.58784294128418 GiB, micro1_backward_last: 21.386390686035156 GiB\nMax_mem for microbatch>1: 30.401947021484375 GiB. micro1_forward_last: 26.58784294128418 GiB, micro1_backward_last: 21.386390686035156 GiB, micro2_forward_last: 30.401947021484375 GiB\n---------------------------------------------------------------------memory parts---------------------------------------------------------------------\n params grads opt_states master_params master_grads acts\nname\nembed 1.159180 1.159180 0.579590 0.289795 0.289795 0.000061\nhead 1.159187 1.159187 0.579594 0.289797 0.289797 0.625000\nperlayer_attn 0.078133 0.078133 0.039066 0.019533 0.019533 1.751953\nperlayer_mlp 0.281258 0.281258 0.140629 0.070314 0.070314 2.500000\nall_attn 0.000000 0.000000 1.406387 0.703194 0.703194 0.000000\nall_mlp 0.000000 0.000000 5.062637 2.531319 2.531319 0.000000\nin_layernorm 0.000000 0.000000 0.000000 0.000000 0.000000 0.250000\n\n------------------------------------------------------micro1_forward_last: 26.58784294128418 GiB------------------------------------------------------\n [Static OS&Param] embed, all_attn, all_mlp, head: master_params, opt_states = 11.442312240600586 GiB\n [Allgathered Last Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Head] head: params, acts = 1.7841873168945312 GiB\n [Recompute Checkpoint] in_layernorm.acts * (LN - 1) = 8.75 GiB\n\n-----------------------------------------------------micro1_backward_last: 21.386390686035156 GiB-----------------------------------------------------\n [Static OS&Param] embed, all_attn, all_mlp, head: master_params, opt_states = 11.442312240600586 GiB\n [Static Grad] embed, all_attn, all_mlp, head: master_grads = 3.8141040802001953 GiB\n [Allgathered First Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Embed] embed: params, acts = 1.15924072265625 GiB\n [Previous Layer Grad ReduceScatering] perlayer_attn, perlayer_mlp: grads = 0.3593902587890625 GiB\n [Recompute Checkpoint] None = 0 GiB\n\n-----------------------------------------------------micro2_forward_last: 30.401947021484375 GiB------------------------------------------------------\n [Static OS&Param] embed, all_attn, all_mlp, head: master_params, opt_states = 11.442312240600586 GiB\n [Static Grad] embed, all_attn, all_mlp, head: master_grads = 3.8141040802001953 GiB\n [Allgathered Last Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Head] head: params, acts = 1.7841873168945312 GiB\n [Recompute Checkpoint] in_layernorm.acts * (LN - 1) = 8.75 GiB\n```\n\n### Technical Notes for Xtuner FSDP (with recompute) Memory Breakdown of the above example\n\n1. At initilization, each GPU holds sharded master params and optimizer states (usually both in fp32).\n```\n [Static OS&Param] embed, all_attn, all_mlp, head: master_params, opt_states = 11.442312240600586 GiB\n```\n\n2. At the end of the first micro batch forward: \n2.1 The whole last decoder layer and lm head are gathered in each GPU (in bf16).\n```\n [Allgathered Last Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Head] head: params, acts = 1.7841873168945312 GiB\n```\n2.2 The checkpoint of all decoder layers also exist in each GPU\n```\n [Recompute Checkpoint] in_layernorm.acts * (LN - 1) = 8.75 GiB\n```\n\n3. At the end of the first micro batch backward\n3.1 Each GPU holds master params' graident now\n```\n [Static Grad] embed, all_attn, all_mlp, head: master_grads = 3.8141040802001953 GiB\n```\n3.2 The whole first decoder layer and embedding layer are gathered in each GPU\n```\n [Allgathered First Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Embed] embed: params, acts = 1.15924072265625 GiB\n```\n3.3 FSDP does ReduceScatter of previous layer's gradients at the same time (to overlap computation and communication)\n```\n [Previous Layer Grad ReduceScatering] perlayer_attn, perlayer_mlp: grads = 0.3593902587890625 GiB\n```\n3.4 The checkpoints for recomputing have been released, along with the forward process\n```\n [Recompute Checkpoint] None = 0 GiB\n```\n\n4. At the end of the second micro batch forward, the only difference with the first micro batch is that each GPU also holds master gradients now.\n```\n [Static Grad] embed, all_attn, all_mlp, head: master_grads = 3.8141040802001953 GiB\n```\n\nThe maximum memory usually occurs in these cases, but maybe different when under different configs such as `chunk_loss_size`.\n"},"items":[{"name":"AGENTS.md","path":".codex/AGENTS.md","rawUrl":"https://raw.githubusercontent.com/InternLM/xtuner/HEAD/.codex/AGENTS.md","title":".codex - AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"../.claude/CLAUDE.md","isInternal":false,"tokens":9,"sizeBytes":20},{"name":"CLAUDE.md","path":".claude/CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/InternLM/xtuner/HEAD/.claude/CLAUDE.md","title":".claude - Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"# XTuner Programming Standards\n\n## Python\n\n### Development Standards\n\n1. When adding methods or functions to a module or class, declare **public interfaces** as early as possible and place **private interfaces** toward the end.\n2. Python **>= 3.10** is required. Use modern syntax accordingly (e.g., `X | Y` union types).\n3. Use `{}` instead of `dict()` for creating dictionaries. Use `[]` instead of `list()` for creating lists.\n4. Use **double quotes** (`\"`) for strings instead of single quotes (`'`).\n5. Use `TypedDict` with `{}` syntax when a parameter requires a typed dictionary.\n6. Prefer `pathlib.Path` over `os.path` for filesystem operations.\n7. All new code must include **type hints** for function signatures (parameters and return types).\n8. Maximum line length is **119** characters (configured in `pyproject.toml`).\n\n### Docstring Standards\n\n1. Use **Google Style** docstrings.\n2. For classes, include `__init__` documentation directly under the **class docstring**; do not repeat it under the `__init__` method.\n3. Parameter types **must** be declared in docstrings — include the type hint in parentheses immediately following the parameter name.\n4. The **return type** must be explicitly declared in the docstring.\n5. Do **not** declare raised exceptions in the docstring.\n6. Do **not** declare Attributes in the class docstring.\n7. If the return value is `None`, omit the `Returns` section.\n8. Only provide docstrings for **public methods**. Private methods do not require docstrings.\n\n### Linting, Formatting & Type Checking\n\n- **Ruff** for linting and formatting, **mypy** for type checking — see `pyproject.toml` for configuration.\n- `mypy` is strict for `xtuner/v1/*` (`ignore_missing_imports = false`).\n- Pre-commit hooks apply **only** to `xtuner/v1/` and `autotest/` — run `pre-commit run --all-files` before submitting a PR.\n- Docstrings are wrapped at **119** characters by `docformatter`.\n\n### Error Handling\n\n1. Never use bare `except:` — always catch specific exception types.\n2. Avoid silencing exceptions with `pass` unless there is a documented reason.\n\n### Testing\n\n1. Every bug fix PR **must** include a regression test that reproduces the original bug.\n2. Test functions must be named `test_` and grouped in classes named `Test`.\n3. Use `pytest` as the test framework. Prefer `pytest.raises` for exception assertions.\n4. Tests should be self-contained — avoid reliance on external services or GPUs unless explicitly marked with `@pytest.mark.gpu`.\n\n---\n\n## Git\n\n### Branch Naming\n\n- Bug fixes: `fix/`\n- Features: `feat/`\n- Refactors: `refactor/`\n\n### Commit Message Standards\n\n1. Follow the style of previous commit messages to maintain a **consistent** project history.\n2. Use the format: `[Tag] Short description` (e.g., `[Fix] Fix OOM in MoE expert routing`).\n3. Common tags: `[Fix]`, `[Feature]`, `[Refactor]`, `[CI]`, `[Docs]`, `[Test]`.\n4. **Never** include metadata or watermarks unrelated to the code itself, such as \"generated by AI\" or \"Co-Authored-By: Claude.\"\n\n---\n\n## Pull Request Standards\n\n### Bug Fix PRs\n\n1. **Title**: `[Fix] `\n2. **Body** must include:\n - **Root Cause**: A clear explanation of why the bug occurred.\n - **Fix**: What was changed and why this is the correct fix.\n - **Test Plan**: How the fix was verified (unit test, manual reproduction, etc.).\n3. Keep the diff **minimal** — fix only the bug. Do not refactor surrounding code, add unrelated improvements, or change formatting of untouched lines.\n4. If the fix touches performance-critical paths (data loading, forward/backward pass, communication), include **before/after** benchmarks or justify why benchmarks are unnecessary.\n\n### General PR Rules\n\n1. One logical change per PR. Do not mix bug fixes with features or refactors.\n2. All CI checks must pass before requesting review.\n3. Resolve all review comments before merging — do not dismiss without explanation.\n\n---\n\n## Code Review Standards\n\n### What to Check\n\n1. **Correctness**: Does the code do what it claims? Are edge cases handled?\n2. **Concurrency & Distributed**: Check for race conditions, deadlocks, incorrect `dist.barrier()` placement, and mismatched collective operations across ranks.\n3. **Memory**: Watch for GPU memory leaks — tensors not freed, caches growing unbounded, missing `torch.no_grad()` in inference paths.\n4. **Numerical Stability**: Flag potential issues like `log(0)`, division by zero, `float32`/`float16` overflow in loss computation or gradient scaling.\n5. **Performance**: Unnecessary CPU-GPU synchronization (`item()`, `.cpu()` in hot paths), redundant data copies, inefficient collective communication patterns.\n6. **API Contracts**: Do public interface changes maintain backward compatibility? Are deprecation warnings added for breaking changes?\n7. **Resource Cleanup**: Are file handles, NCCL communicators, and CUDA streams properly cleaned up?\n\n### Area-Specific Review Rules\n\nIf a PR changes code under `xtuner/v1/rl`, read `.claude/rules/rl_review.md` before commenting or\nposting the final review summary.\n\n### Review Output Format\n\nWhen performing code reviews, use the following structure:\n\n```\n## Summary\n<1-2 sentence overview of the change>\n\n## Issues\n### Critical\n- [file:line] Description of the issue and suggested fix\n\n### Warning\n- [file:line] Description of the concern\n\n### Nit\n- [file:line] Minor style or readability suggestion\n\n## Verdict\nAPPROVE / REQUEST_CHANGES / COMMENT\n```\n\n### Review Principles\n\n1. **Be specific** — always reference the exact file and line number.\n2. **Explain why** — don't just say \"this is wrong\"; explain the failure mode.\n3. **Suggest a fix** — provide a concrete code suggestion when possible.\n4. **Distinguish severity** — separate critical bugs from style nits.\n5. Do not flag issues in code that is **outside the scope** of the PR diff.\n\n\n\n## Rules\n\nPlease refer to the docs in `.claude/rules` for the development guidelines.\n","isInternal":false,"tokens":1409,"sizeBytes":6040},{"name":"datasets_architecture.md","path":".claude/rules/datasets_architecture.md","rawUrl":"https://raw.githubusercontent.com/InternLM/xtuner/HEAD/.claude/rules/datasets_architecture.md","title":"Rules Skill","category":"anthropic-skill","format":"markdown","content":"---\npaths:\n - \"xtuner/v1/datasets/**/*\"\n---\n\n# XTuner v1 Datasets Architecture Rules\n\n## Core Data Contract (DataItem Protocol)\n\nEvery dataset layer must produce/consume `DataItem`:\n```python\n{\n \"input_ids\": list[int], # tokenized token IDs\n \"labels\": list[int], # same length; first token masked to -100\n \"num_tokens\": int, # len(input_ids); used for sampler and packing\n}\n```\n\n- `labels[0] = -100` always (pretrain: first token not predicted)\n- `HardPackDataset.__getitem__` returns `list[DataItem]` (one per source sample in pack)\n- All other datasets return a single `DataItem`\n\n## Tokenization Functions\n\n**Standard pretrain**: `PretrainTokenizeFunction`\n- Accepts: `{\"content\": \"text\"}` or `{\"messages\": [{\"role\": \"pretrain\", \"content\": \"text\"}]}`\n- Hash: `\"{tokenizer_hash}_{source_hash}_{add_bos_token}_{add_eos_token}\"`\n\n**Long text pretrain**: `LongTextPretrainTokenizeFunction`\n- Two-phase: cache phase returns chunk metadata, runtime phase tokenizes specific char ranges\n- Parameters: `chunk_size`, `tokenizer_chunk_chars`, `overlap_chars`, `min_chunk_tokens`\n- BOS only at `char_start==0`, EOS only at document end\n- Hash includes all chunking parameters\n\n## JsonlDataset Architecture\n\n- **Shared memory**: entire file loaded once at init\n- **Offset table**: `np.ndarray` for O(1) line access\n- **Chunked mode**: activated when `chunks.npy` exists in cache\n - Expands `sampled` to per-chunk entries\n - `__getitem__` calls `tokenize_fn(data, char_start=cs, char_end=ce)`\n - `max_length` filter skipped (would incorrectly discard valid long-text samples)\n - `__len__` returns chunk count (not line count)\n\n## Cache Directory Layout\n\n```\ncache_dir/\n .xpuyu-cache-meta.json\n {file_xxhash}/\n offsets.npy\n {tokenize_fn_hash}/\n num_tokens.npy # always present\n chunks.npy # only for LongText mode: (total_chunks, 4)\n # [line_idx, char_start, char_end, num_tokens]\n```\n\nCache key: `(file xxhash, tokenizer hash, tokenize_fn source hash)`\n- Tag-based lookup via `cache_tag` on `DatasetConfig`\n- Source hash auto-invalidates on code change\n\n## Packing Strategies\n\n**HardPackDataset**:\n- Static pack_infos computed at init (deterministic from seed)\n- Returns `list[DataItem]` per pack\n- `global_pack=False`: per-dataset packing (default)\n- `global_pack=True`: concat all datasets then pack\n\n**PresetPackDataset**:\n- User-provided pack config (NPY directory + `paths.json`)\n- Validates structure and token ranges via vectorized numpy\n- Strategies for length mismatch: error/padding (short), error/truncate (long)\n- No state (deterministic from config file)\n\n## Samplers\n\n**LengthGroupedSampler**:\n- Megabatch logic: group similar lengths, sort DESC, shuffle chunks\n- State: `{epoch, step, world_size, num_samples, total_size}`\n- `step` = `consumed_samples % total_size`\n\n**PresetSampler**:\n- User-provided order via `sampler_config_path` (`.npy` path only, mmap read)\n- Round-down to `global_batch_size * world_size` multiple (truncate tail; error if length too short)\n- Per-rank slice: `effective[local_rank::world_size]` on the truncated mmap-backed view (`self.global_order`)\n- `pack_level=\"preset\"` in `DataloaderConfig`; requires `pack_config_path` and `sampler_config_path`\n- State dict compatible with existing resume logic\n\n## Resume Contract\n\n```python\n# Save\nstate = dataloader.get_state_dict(consumed_samples)\n# → {\"sampler\": {...}, \"dataset\": {}}\n\n# Restore\ndataloader.load_state_dict(state)\n```\n\n- `consumed_samples` = total globally-seen samples (across all ranks)\n- Sampler converts to per-epoch `step` by modulo\n- Packing datasets have no state (deterministic)\n\n## Multi-Dataset Support\n\n- `DatasetConfigList` = list of `{dataset, tokenize_fn}` pairs\n- `build_datasets()` returns `list[JsonlDataset]` (1 file → 1 dataset)\n- `sample_ratio` applied per-DatasetConfig via `sampled[]` repetition\n\n## Key Implementation Files\n\n| File | Purpose |\n|------|---------|\n| `config.py` | Pipeline orchestration via `DataloaderConfig.build()` |\n| `pt_tokenize_fn/text.py` | `PretrainTokenizeFunction` |\n| `pt_tokenize_fn/long_text.py` | `LongTextPretrainTokenizeFunction` |\n| `jsonl.py` | `JsonlDataset` with shared memory + offset table |\n| `packing.py` | `HardPackDataset`, `ExpandSoftPackDataset` |\n| `sampler.py` | `LengthGroupedSampler`, `ParallelSampler` |\n| `preset_pack.py` | `PresetPackDataset` |\n| `preset_sampler.py` | `PresetSampler` |\n| `dataloader.py` | `Dataloader` wrapper |\n| `data_item.py` | `DataItem` schema |\n\n## Design Principles\n\n1. **Protocol-based**: All layers communicate via `DataItem` contract\n2. **Cache-aware**: Tokenization cached with automatic invalidation\n3. **Deterministic**: Packing and sampling reproducible from seed/config\n4. **Resumable**: State dict pattern for checkpoint/resume\n5. **Composable**: Tokenize → Dataset → Pack → Sample → Collate pipeline\n","frontmatter":{"paths":["xtuner/v1/datasets/**/*"]},"isInternal":false,"tokens":1266,"sizeBytes":4924},{"name":"rl_review.md","path":".claude/rules/rl_review.md","rawUrl":"https://raw.githubusercontent.com/InternLM/xtuner/HEAD/.claude/rules/rl_review.md","title":"Rules Skill","category":"anthropic-skill","format":"markdown","content":"# RL Review Guidelines\n\nUse these rules whenever a PR changes code under `xtuner/v1/rl`. They extend the general review\nstandards in `.claude/CLAUDE.md`; they do not replace them.\n\n## Required Review Output\n\nEvery review for an RL change must state the ProduceBatchResult impact near the top-level summary:\n\n```text\nProduceBatchResult impact: \n```\n\nAlso include routed-experts impact when the PR touches routed-experts logic, rollout response\nhandling, `RolloutState.extra_fields`, object references, or memory ownership around rollout outputs:\n\n```text\nRoutedExperts impact: \n```\n\nAlso include Ray concurrency impact when the PR touches Ray actor methods, actor construction,\nmethod decorators or wrappers, concurrency-group definitions, or rollout control paths:\n\n```text\nRay concurrency impact: \n```\n\nDo not leave these impacts implicit in the finding text. If a finding is about rollout status,\npause/abort/timeout behavior, response handling, producer aggregation, routed-experts ownership, or\nRay actor concurrency, repeat the relevant impact line inside that finding so the downstream effect\nis visible at the point of review.\n\n## ProduceBatchResult Checklist\n\nReview whether the PR can change trainer-visible batch accounting, timing, or reward semantics in\n`ProduceBatchResult`.\n\nCheck this area when the PR changes any of these paths:\n\n- `RolloutState.status`, `finish_reason`, or status conversion.\n- Abort, filter, expire, retry, timeout, cancellation, or failure handling.\n- Producer, agent loop, judger, replay buffer, rollout worker, rollout controller, or backend pause\n cleanup logic.\n- Writers, readers, tests, or fake implementations that construct or consume `ProduceBatchResult`,\n including trainer-facing code outside `xtuner/v1/rl`.\n\nName the concrete field-level impact when any of these fields can change:\n\n- Batch status: `status`.\n- Returned groups: `rollout_states`.\n- Generation timing: `group_gen_count`, `group_gen_mean_s`, `group_gen_p50_s`, `group_gen_p99_s`,\n `group_gen_p99_p50_ratio`, `group_gen_pause_time_s`.\n- Replay-buffer leftovers: `leftover_init`, `leftover_completed`, `leftover_aborted`,\n `leftover_expired`, `leftover_failed`, `leftover_filtered`.\n- Reward accounting: `raw_rewards_sum`, `raw_rewards_count`.\n- Produced work counters: `produced_samples`, `produced_tokens`, `produce_time_s`.\n- Multi-task aggregation: `task_batch_sizes`, `task_results`.\n\nCommon impacts to call out explicitly:\n\n- A sample moving between `ABORTED`, `FAILED`, `FILTERED`, `EXPIRED`, and `COMPLETED` can change the\n corresponding `leftover_*` counts.\n- Pause, abort, timeout, and cancellation changes can inflate or deflate `group_gen_*` timing,\n especially `group_gen_pause_time_s`.\n- Reward or filter-path changes can change `raw_rewards_sum`, `raw_rewards_count`,\n `produced_samples`, and `produced_tokens`.\n\n## Ray Actor Concurrency Checklist\n\nReview whether high-volume rollout calls can starve pause, abort, health-check, restart, or shutdown\nRPCs.\n\nCheck this area when the PR changes any of these paths:\n\n- Ray actor methods decorated with `@ray.method(...)` or `@ray_method(...)`.\n- Decorators or wrappers stacked on Ray actor methods, especially wrappers using `functools.wraps`\n or exposing `__wrapped__`.\n- Actor construction through `ray.remote(...)`, including `concurrency_groups` and\n `max_concurrency`.\n- Generate, pause, continue, abort, health-check, restart, or shutdown methods.\n\nWhen a Ray method is combined with another decorator, keep the Ray method decorator closest to the\nfunction definition:\n\n```python\n@trace_rollout_endpoint(\"rollout.worker.generate\")\n@ray.method(concurrency_group=ROLLOUT_CONCURRENCY_GROUP_GENERATE)\nasync def generate(...):\n ...\n```\n\nDo not use the reverse order:\n\n```python\n@ray.method(concurrency_group=ROLLOUT_CONCURRENCY_GROUP_GENERATE)\n@trace_rollout_endpoint(\"rollout.worker.generate\")\nasync def generate(...):\n ...\n```\n\nThe current Ray actor-class processing unwraps decorated methods before reading Ray method metadata.\nIf `@ray.method(...)` is outside a wrapper, metadata such as `__ray_concurrency_group__` can remain\non the discarded wrapper, causing the method to fall back to the default concurrency group.\n\nReview the effective dispatch behavior, not only the visible decorator order:\n\n- After following `__wrapped__`, the callable inspected by Ray must retain the expected Ray method\n metadata.\n- Every named method concurrency group must be declared by the corresponding `ray.remote(...)`\n actor construction.\n- High-volume `generate` methods must not accidentally fall back to the default group.\n- Pause, abort, and other control RPCs must remain schedulable while generation is saturated.\n- Slow health or lifecycle methods must not occupy every slot needed by critical control RPCs.\n- Blocking work inside async actor methods must not block the actor event loop or concurrency lane.\n\n## RoutedExperts Checklist\n\nReview whether routed-experts ownership and cleanup remain correct.\n\nCheck this area when the PR changes any of these paths:\n\n- LMDeploy rollout response handling.\n- `return_routed_experts`, `routed_experts`, `RolloutState.extra_fields`, or object-ref plumbing.\n- Abort, cancellation, timeout, filter, retry, or failure paths before response handling completes.\n- Background tasks, replay-buffer storage, trainer batches, metrics, tests, or fake rollout\n responses that can retain routed-experts object refs.\n\nFor LMDeploy rollout, `rollout_worker` obtains routed-experts object refs from the LMDeploy shared\nstore. At that point, ownership moves from LMDeploy to XTuner. A review finding should call out both\nsides of the ownership boundary when relevant:\n\n- Leak before transfer: requests whose routed experts remain in LMDeploy because XTuner never\n obtains the object refs.\n- Leak after transfer: object refs that XTuner keeps alive too long through `RolloutState`, replay\n buffer, trainer batches, metrics, fake tests, or background tasks.\n","isInternal":false,"tokens":1329,"sizeBytes":6111},{"name":"SKILL.md","path":".claude/skills/add_hf_model/SKILL.md","rawUrl":"https://raw.githubusercontent.com/InternLM/xtuner/HEAD/.claude/skills/add_hf_model/SKILL.md","title":"add_hf_model","category":"anthropic-skill","format":"markdown","content":"---\nname: add_hf_model\ndescription: >\n Add support for a HuggingFace model in XTuner so it can be trained with the\n full set of parallelism strategies and optimization switches. The user can\n provide a HuggingFace Hub repo id, a local model directory, or a model family\n already supported by `transformers`. This skill walks through the full path:\n locating the reference implementation, classifying the model, splitting new\n code across the model / module / ops layers, implementing the XTuner model\n class and config, registering the entry point, validating bitwise numerical\n parity with HuggingFace, writing regression tests, and enabling the training\n optimizations (EP/SP, micro-batch, torch.compile, fp8, activation offload).\n---\n\n# Add a HuggingFace model to XTuner\n\nA complete handbook for porting any HuggingFace causal LM into XTuner with\ncorrect training behavior. Covers four buckets: **Dense LLM**, **MoE LLM**,\n**VLM / compose model**, and **`trust_remote_code` model**. Every step below is\ngrounded in concrete `file:line` references in this repository — read the cited\ncode before writing yours.\n\n> **Anti-spaghetti reminder** — XTuner separates *what the model is* (`*Config`),\n> *what it computes* (`Dense` / `MoE` / `BaseComposeModel` subclass), and *how its\n> weights map to HF safetensors* (`to_hf_key_list`, optional `safetensors_to_params`\n> / `param_to_safetensor`). Keep new code inside those three seams. **Never** add\n> `if model_name == \"...\"` branches to the base classes.\n\n---\n\n## 0. Locate the source code\n\nThe only goal of this step is to find the **HuggingFace reference implementation\nyou will port from**, and to note the `model_type` you will register under. Do\nnot pick attention/router/layer here — that happens naturally in §1 as you read\nthe code.\n\n1. If the user gives a **remote-code** model, the modeling code ships in the repo\n as `modeling_*.py` — read it directly.\n2. If the user gives **weights only**, confirm the target Python environment with\n the user, then find the matching implementation inside that env's\n `transformers` install. If no code exists for that `model_type`, stop and ask\n the user.\n3. Confirm the weights are **training-suitable** — at least bf16 — so training\n precision can be aligned later. Flag fp8/int4-only checkpoints to the user.\n4. Note two strings from the HF config — you will need both downstream:\n ```python\n from transformers import AutoConfig\n cfg = AutoConfig.from_pretrained(path, trust_remote_code=True)\n cfg.model_type # registration key (§4)\n cfg.architectures[0] # the modeling class you are mirroring\n ```\n\n---\n\n## 1. How to do the migration\n\nXTuner splits a model into three layers. You port by reusing what already exists\nat each layer and only adding what is genuinely new — so the work is \"read the HF\nmodeling code, then match each of its pieces to the XTuner component below\". The\nlocations are listed plainly; you do not need extra guidance to map your model's\nattention / router / layer onto the matching one.\n\n- **model layer** (`xtuner/v1/model`) — the model body.\n - Dense body: `xtuner/v1/model/dense/dense.py`\n - MoE body: `xtuner/v1/model/moe/moe.py`\n - Multi-modal (compose) bodies: `xtuner/v1/model/compose/` — reuse the language\n model and add only the ViT / projector pieces.\n- **module layer** (`xtuner/v1/module`):\n - Decoder layers: `DenseDecoderLayer`, `MoEDecoderLayer` —\n `xtuner/v1/module/decoder_layer`. Small changes add a switch on the existing\n layer; a genuinely novel architecture may add `{model_name}_decoder_layer`.\n - Attention configs: `MHAConfig` (`xtuner/v1/module/attention/mha.py`),\n `MLAConfig` (`.../mla.py`, DeepSeek-style latent),\n `GatedDeltaNetConfig` (`.../gated_deltanet.py`, linear).\n - Router configs (MoE): `GreedyRouterConfig`\n (`xtuner/v1/module/router/greedy.py`, softmax top-k — Qwen3 MoE, GptOss),\n `NoAuxRouterConfig` (`xtuner/v1/module/router/noaux_router.py`, grouped sigmoid\n aux-loss-free — DeepSeek V3).\n - Also: rope, rms_norm, etc.\n- **ops layer** (`xtuner/v1/ops`) — kernels such as attention and rms_norm.\n\n### Existing models to copy from\n\nPick the one whose attention + (router) match yours; the closer it is, the\nsmaller your diff.\n\n| Your model is… | Copy from |\n|--------------------------------------|--------------------------------------|\n| Dense LLM | `xtuner/v1/model/dense/qwen3.py` |\n| MoE LLM | `xtuner/v1/model/moe/qwen3.py` |\n| MoE with MLA + aux-loss-free router | `xtuner/v1/model/moe/deepseek_v3.py` |\n| MoE needing weight-layout transform | `xtuner/v1/model/moe/gpt_oss.py` |\n| Compose / VLM | `xtuner/v1/model/compose/qwen3_vl/` |\n\n\nAfter you have a plan, **write an integration design doc into `docs/design/model`**\nand confirm with the user that the decomposition is reasonable before\nimplementing.\n\n### File layout for the new model\n\n| Bucket | Model file | Public re-export | Test file |\n|----------------------|-------------------------------------------------------------|-----------------------------------------------|--------------------------------------------|\n| Dense LLM | `xtuner/v1/model/dense/.py` | `xtuner/v1/model/__init__.py` | `tests/model/test__dense.py` |\n| MoE LLM | `xtuner/v1/model/moe/.py` | `xtuner/v1/model/__init__.py` | `tests/model/test__moe.py` |\n| Compose / VLM | `xtuner/v1/model/compose//...` | `xtuner/v1/model/__init__.py` | `tests/model/test_.py` |\n\n---\n\n## 2. Implement the model class\n\nSubclass `Dense`, `MoE`, or `BaseComposeModel`. You typically only need to\noverride one method.\n\n### 2.1 `to_hf_key_list(self, key: str) -> list[str]` — **mandatory, abstract on `BaseModel`**\n\nDeclared at `xtuner/v1/model/base.py`. Translates an **XTuner-side** parameter\nname into one or more **HF-side** safetensors keys. Return-list cardinality:\n\n- **1 → 1** for plain weights (most params).\n- **1 → N** for fused MoE experts (one fused param explodes into N per-expert\n HF keys; see `xtuner/v1/model/moe/qwen3.py`).\n\nPatterns you almost always need (study `xtuner/v1/model/dense/qwen3.py`\nand `xtuner/v1/model/moe/qwen3.py`):\n\n```python\n# 1. Tied embeddings: redirect lm_head → embed_tokens before adding the model. prefix.\nif self.config.tie_word_embeddings and \"lm_head\" in key:\n key = key.replace(\"lm_head\", \"embed_tokens\")\n\n# 2. Add the \"model.\" prefix that HF wraps everything in.\nif \"layers\" in key or \"embed_tokens\" in key:\n key = \"model.\" + key\n\n# 3. HF MoE nests experts/gate under .mlp.; XTuner does not.\nif \"layers\" in key:\n key = re.sub(r\"layers\\.(\\d+)\\.(experts|gate)\", r\"layers.\\1.mlp.\\2\", key)\n\n# 4. Top-level norm.\nif key.startswith(\"norm.\"):\n return [key.replace(\"norm.\", \"model.norm.\")]\n\n# 5. MoE expert fusion: one fused param → N HF keys.\nif \"fused_w1w3.weight\" in key:\n out = []\n for i in range(self.config.n_routed_experts):\n out.append(key.replace(\"fused_w1w3.weight\", f\"{i}.gate_proj.weight\"))\n out.append(key.replace(\"fused_w1w3.weight\", f\"{i}.up_proj.weight\"))\n return out\n\n# 6. Model-specific buffers (e.g. FoPE: rotary_emb.sin_coef / cos_coef are persistent buffers).\nif key.startswith(\"rotary_emb.\"):\n return [key.replace(\"rotary_emb.\", \"model.rotary_emb.\")]\n```\n\nA prefix-region remap that differs by *deployment* (e.g. a text tower nested under a\nVLM as `model.language_model.`) does **not** belong here — put it in the config's\n`hf_key_mapping`. See §5.1.\n\n### 2.2 `safetensors_to_params` / `param_to_safetensor` — **optional, only for layout mismatches**\n\nIf the HF storage layout cannot be expressed as “same tensor, different name”,\nyou must transform the bytes. Canonical example: GptOss stores expert weights as\n`(num_experts, hidden_size, expert_dim * 2)` but XTuner fuses them as\n`(num_experts * 2 * expert_dim, hidden_size)` — see overrides at\n`xtuner/v1/model/moe/gpt_oss.py`.\n\nUse this hook only when transposing/reshaping is unavoidable. Renames go in\n`to_hf_key_list`; numerics go here.\n\n### 2.3 Do **not** override\n\n- `build_layers`, `build_embeddings`, `_init_weights` — the base handles them\n via the config. Override only if the new model adds a layer type the base\n cannot express (rare).\n- `forward` — same reasoning. If you find yourself touching it, re-evaluate the\n bucket choice; the problem is probably in the config (`attention`, `router`,\n `first_k_dense_replace`).\n\n---\n\n## 3. Implement the config\n\nTwo layers: a **base config** that reads HF and a **size-specific subclass** per\nreleased checkpoint.\n\n### 3.1 Base config — three required members\n\nMirror `Qwen3MoEConfig` (`xtuner/v1/model/moe/qwen3.py`) or\n`Qwen3DenseConfig` (`xtuner/v1/model/dense/qwen3.py`).\n\n1. **`build(self) -> `** — return `(self)`.\n\n2. **`@classmethod from_hf(cls, hf_path) -> Self`** — read the HF config and map\n its fields one-to-one onto your config. The field set is whatever the live\n config class declares (`TransformerConfig` in `xtuner/v1/model/base.py`,\n `MoEConfig` in `xtuner/v1/model/moe/moe.py`); read it, don't work from a\n hard-coded list. A few non-obvious traps:\n - Use `RopeParametersConfig.from_hf_config(hf_config)` from\n `xtuner/v1/module/rope/rope.py`. It already handles both the\n `rope_parameters` dict (HF ≥ 5.2.0) and the legacy `rope_scaling` dict\n (HF 4.57.x), plus YARN / FoPE / mrope special fields. **Do not** parse\n rope yourself.\n - For optional HF fields, use `getattr(hf_config, \"\", )`\n rather than `hf_config.` — older checkpoints drop fields.\n\n3. **`@property hf_config(self) -> | None`** — the inverse of\n `from_hf`: every field `from_hf` reads must be re-emitted so `save_hf`\n round-trips.\n\n**Built-in vs. `trust_remote_code` — the only axis that changes `from_hf` /\n`hf_config`:** a `trust_remote_code` model is structurally still Dense / MoE /\nVLM; the *only* difference is whether `transformers` ships a built-in\n`Config` class. Decide which case you are in and follow the matching column\n— this is the single place that contract is defined.\n\n| | Built-in config (e.g. Qwen3) | `trust_remote_code` (no built-in config) |\n|---|---|---|\n| `from_hf` reads via | `.from_pretrained(hf_path)` | `AutoConfig.from_pretrained(hf_path, trust_remote_code=True)`, then `assert hf_config.model_type == \"\"` |\n| `hf_config` returns | a populated `` re-emitting every field | `None` |\n| `save_hf` weight-map | written from `hf_config` | same |\n| `save_hf` side files | derived from `hf_config` | falls back to copying `config.json` / tokenizer / `*.py` from `self._hf_path` so the dir stays loadable with `trust_remote_code=True`; module-cache test in the matrix below |\n\nCompare `Qwen3MoEConfig.hf_config` (built-in) and `Qwen3MoEFoPEConfig.hf_config`\n(returns `None`), both in `xtuner/v1/model/moe/qwen3.py`. **Do not** invent a\nbuilt-in HF config class for a remote-code model — return `None`.\n\n**Invariants:**\n\n- `from_hf` ↔ `hf_config` must round-trip. The `test_save_hf` test (§7) is the\n enforcement mechanism — every key in the original index must appear in the\n saved index with byte-equal tensors.\n- The config must be **pickleable and side-effect free**. No env vars, no file\n I/O, no `torch.cuda.*` at construction time.\n\n### 3.2 Size-specific subclasses\n\nAdd one subclass per released checkpoint that hard-codes the published\ndimensions. Mirror `Qwen3MoE30BA3Config` / `Qwen3MoE235BA22Config` in\n`xtuner/v1/model/moe/qwen3.py`. Even if only one size ships, define at least one\n— it is the entry point for `get_model_config` and for tests.\n\n---\n\n## 4. Register the entry point\n\nOpen `xtuner/v1/model/__init__.py` — all four edits are short:\n\n1. **Import** — bring in the new config classes.\n2. **Alias** — add `\"\": ()` to `model_mapping`. Lookup\n normalizes case and `-`/`_`, so `\"my-model-7b\"` and `\"my_model_7b\"` are the\n same key.\n3. **Dispatch** — add `elif cfg.model_type == \"\":` to\n `get_model_config_from_hf` returning `.from_hf(model_path)`.\n The catch-all `raise ValueError(...)` must stay last.\n4. **`__all__`** — re-export the new public config names.\n\n**Never** key dispatch on parameter names, file paths, or architecture strings.\nOnly `model_type`.\n\n---\n\n## 5. Compose / VLM specifics\n\nCompose models live under `xtuner/v1/model/compose//` and follow a\nthree-tower pattern. See `xtuner/v1/model/compose/base.py` and\n`xtuner/v1/model/compose/qwen3_vl/qwen3_vl_config.py`.\n\n### 5.1 Config shape\n\n`BaseComposeConfig` (`compose/base.py`) carries three sub-configs:\n\n```python\nvision_config: XTunerBaseModelConfig # e.g. Qwen3VLVisionConfig\nprojector_config: XTunerBaseModelConfig # e.g. Qwen3VLProjectorConfig\ntext_config: XTunerBaseModelConfig # any Dense/MoEConfig subclass\nfreeze_vision: bool = False\nfreeze_projector: bool = False\nfreeze_language: bool = False\n```\n\nReuse existing text-tower configs where possible. `Qwen3VLDense*` delegates to\nthe Dense path via `xtuner/v1/model/dense/qwen3vl_text.py`. **Do not** copy-paste\nthe entire dense/MoE config — compose over them.\n\n**Reused text-tower key prefixes — use `hf_key_mapping`, not `to_hf_key_list`.** A\ntext tower reused inside a VLM keeps its *standalone* `to_hf_key_list` (it emits\n`model.<...>` as if it were a plain LLM). The compose checkpoint nests it one level\ndeeper (`model.language_model.<...>`). Do **not** teach the text tower's\n`to_hf_key_list` about that prefix — that leaks the compose context into a tower that\nmust also work standalone. Set the remap on the **text config** instead:\n`hf_key_mapping = {r\"^model\\.\": \"model.language_model.\"}` (applied in\n`_init_load_spec`). Keep *structural* renames in `to_hf_key_list`; the\ndeployment-dependent prefix region belongs in `hf_key_mapping`.\n\n### 5.2 Model shape\n\n`BaseComposeModel` (`compose/base.py`) builds three sub-models:\n`vision_tower`, `multi_modal_projector`, `language_model`.\n\n- **`from_hf(hf_path)`** — delegates to each sub-model with\n `strict=False` (vision keys are missing from the language-model state dict\n and vice versa) and unions the missing-keys sets.\n- **`save_hf(hf_dir)`** — saves each tower with a distinct prefix\n (`\"model-language\"`, `\"model-vision\"`, `\"model-projector\"`) and merges the\n three `weight_map`s into one `model.safetensors.index.json`.\n\n### 5.3 `hf_config` for VLMs\n\nMany VLMs ship without a stable HF top-level config class; `hf_config` returns\n`None` (`compose/qwen3_vl/qwen3_vl_config.py`). In that case, `save_hf`\nfalls back to copying source files from `self._hf_path` (see §3.1).\n\n### 5.4 Vision-side quirks to watch for\n\n- `attn_impl` selectable per checkpoint: `\"flash_attention\" | \"flex_attention\" |\n \"eager_attention\"`.\n- `deepstack_visual_indexes` (e.g. `[8, 16, 24]`) — auxiliary supervision depths\n on the vision tower; must round-trip.\n- Specialized rope: `rope_type=\"qwen3_vl\"`, `mrope_section=[24, 20, 20]` —\n delegated to `RopeParametersConfig.from_hf_config`.\n- The projector’s `torch.compile` path is enabled on torch ≥ 2.9.1 (see commit\n `f6d74efb`). Tests must validate both compiled and eager outputs match.\n\n---\n\n## 6. Baseline parity — no complex parallelism yet\n\nFirst make the model correct on the **simplest execution path**: single rank, no\nEP/SP, no `torch.compile`, no fp8, no activation offload. Align precision from\nthe inside out — ops → decoder layer → full model — reusing XTuner's own modules\nat every level. Only once this baseline holds do you add the parallel and\noptimization features (§8).\n\n1. **Reuse, don't reimplement.** Use the existing XTuner modules — MHA, MLA,\n GatedDeltaNet, GroupLinear, etc. Do not re-implement them with naive torch.\n\n2. **Route op differences through `XTUNER_HF_IMPL`.** This env var already exists\n (`xtuner/v1/utils/misc.py`) and selects HF-exact ops via per-op selectors:\n `xtuner/v1/ops/attn_imp.py::get_attn_impl_fn` (eager attention),\n `xtuner/v1/ops/rms_norm/__init__.py::get_rms_norm_fn` (native torch rms_norm),\n and `xtuner/v1/ops/gated_deltanet/__init__.py::{get_chunk_gated_delta_rule_fn,\n get_causal_conv1d_fn}` (return the canonical fla / causal_conv1d wrappers HF\n calls instead of XTuner's compile-friendly `torch.library.custom_op` wraps —\n same forward output, but the backward op graph now matches HF). Selectors\n read the env var **live** so a test can toggle it per model instance. If your\n model introduces a new op whose fast path is not bitwise against HF (typical\n for fla-backed linear-attention variants), add its HF-exact branch in the\n corresponding op selector the same way. **You may only patch at the ops\n level** — never take the shortcut of patching the entire\n `XTunerDecoderLayer.forward`.\n\n Pair `XTUNER_HF_IMPL` with `XTUNER_DETERMINISTIC=true`. The latter gates an\n autotune-pin block in `xtuner/v1/__init__.py` that monkey-patches\n `triton.autotune` to lock every kernel to its first config and disable the\n result cache. Without the pin, fla's `@triton.autotune` picks tiling /\n reduction order per kernel and the choice drifts between runs, producing\n 1 ULP per linear-attention layer that the chain rule amplifies to ~1.0 at\n the model boundary (§12). `tests/conftest.py` sets the env var and runs\n `import xtuner.v1` so the patch installs before fla is imported. The patch\n lives on the `xtuner.v1.*` import chain (not in the conftest) because\n `multiprocessing.spawn` children of `MultiProcessTestCase` don't go through\n pytest's conftest — they re-import the test class top-level (which pulls\n `xtuner.v1.model.*`) and the patch must run in those processes too.\n Production training doesn't set the env var, so the block is inert there.\n\n3. **Decoder-layer bitwise parity — forward AND backward, both required.** Build\n one XTuner `DecoderLayer` (one per distinct layer type — e.g. a linear and a\n full layer for a hybrid model) and the matching HF layer, set `XTUNER_HF_IMPL`\n and `XTUNER_DETERMINISTIC`, feed identical inputs, and align **bitwise** — not a\n tolerance. **Backward is not optional**: a `single layer → final norm → lm_head\n → CE loss → loss.backward()` test asserts the **layer output, the loss, and the\n input gradient `dL/dx`** all bitwise-equal to HF — test one full and one linear\n layer separately (reference: `test_decoder_layer_bitwise_parity`). A bitwise\n *forward* does not imply a bitwise backward — the forward values can round\n identically while the backward graph differs (see the fp32-softmax note in §12).\n This is the **primary** path, required at all scales, and the only feasible one\n for large models (it never loads the full model — see \"load only what you use\"\n below). When you hand-build a single layer's\n inputs, the two sides' `causal mask` / `seq_ctx` must match exactly — see §12.\n **Load only what you use** so the test fits any model size: build the XTuner\n tower on `meta` and materialize (`to_empty` + selective load) *only* the tested\n layer + `norm` + `lm_head` via `HFCheckpointLoader.load(key)` (it reads just the\n needed safetensors shard); build a *standalone* HF `DecoderLayer` + norm +\n lm_head and load the same keys. Never `from_pretrained` the whole model for a\n single-layer test. (`build_rotary_embedding` builds the rotary on CPU with real\n buffers even under meta, so `model.rotary_emb` is usable without materializing\n the layers.) For < 40B you may also read per-layer hidden states from one\n full-model forward (HF `output_hidden_states` + XTuner `return_hidden_states`);\n mind the off-by-one in §12.\n\n4. **Model config matches HF hyperparameters** — see the §3.1 built-in vs.\n remote-code contract for how `from_hf` / `hf_config` differ between the two.\n\n5. **Full-model forward parity for models < 40B**, bitwise against HF, on the\n single-rank path (FSDP on or off; no EP / compile / fp8 yet).\n\n6. **`save_hf` / `from_hf` correctness** (round-trip; see §7).\n\n7. **Loss-convergence trace + engine test.** Once model forward and decoder-layer\n forward/backward are aligned, record a loss-convergence trajectory and add an\n engine test case.\n\n**Commit the baseline — do not stop to ask.** Committing is part of this\nworkflow: invoking this skill authorizes the in-workflow commits. Once parity\nholds on the simple path and lint/tests pass, commit immediately so this correct\nbaseline is preserved before any parallel/optimization work begins (the §8\nsub-agents branch off this commit). Commit each logical step as you finish it\n(§10) rather than batching everything to the end or waiting for a separate\ngo-ahead.\n\nReport back to the user: whether bitwise parity was achieved, and the residual\nerror (decoder-layer level and model level) once XTuner's internal components are\nused.\n\n---\n\n## 7. Baseline tests — add these first\n\nBefore any parallel feature, add the most basic regression tests so the baseline\n(§6) is locked and the §8 work has a fixed reference to measure against. Add\n`tests/model/test__.py`, mirroring\n`tests/model/test_qwen3_moe.py` (MoE) or `tests/model/test_qwen3_dense.py`\n(Dense). All tests extend `xtuner._testing.DeterministicDDPTestCase` for\ndeterministic distributed setup.\n\nReference test cases:\n- decoder-layer bitwise parity + save_hf round-trip: `tests/model/test_qwen3_5_dense.py`\n (hybrid linear/full dense VLM; toggles `XTUNER_HF_IMPL` per case)\n- model forward & save & load parity: `tests/model/test_qwen3_moe.py`\n- engine training test: `tests/engine/test_moe_train_engine.py`\n\n### 7.1 Baseline test matrix\n\nAll single execution path — no EP / dispatcher / compile / fp8 here; those are\n§8 tests, written against this baseline.\n\n| Case | Dense | MoE | Compose | Notes |\n|-------------------------------------------------|:-----:|:---:|:-------:|------------------------------------------------------------------------------------|\n| Decoder-layer parity — output + loss + `dL/dx` | ✅ | ✅ | ✅ | **Required at all scales, bitwise.** Test one full and one linear layer *separately*. Standalone & memory-light: build the XTuner tower on `meta` and `materialize_xtuner_submodule` only {tested layer, `norm`, `lm_head`}; build a standalone HF `DecoderLayer`; load via `HFCheckpointLoader`. Under `XTUNER_HF_IMPL` assert the **layer output, loss, and `dL/dx` all bitwise** (forward AND backward at the layer level — a bitwise forward ≠ bitwise backward, see §12). Reference: `test_decoder_layer_bitwise_parity`. |\n| Whole-model forward + backward parity (by scale) | ✅(4B)| scale | ✅(4B) | Run the **top-level model** end-to-end vs HF, eager both sides (`XTUNER_HF_IMPL` + `XTUNER_DETERMINISTIC` + HF `eager`). For a **VLM** that's the compose model on an **image** prompt (vision + projector + text — subsumes \"VLM forward parity\"); for a plain LLM, the text/LM model. With both env switches on plus the autotune pin loaded (§6.2), **forward AND backward are bitwise**: assert logits + loss + `dL/d(pixel_values)` / `dL/d(inputs_embeds)` all `== 0.0`. **Scale**: small = fwd+bwd; medium = forward only; large (can't e2e-forward) = skip → integration. Reference: `test_vl_forward_parity`. |\n| Vision-tower bitwise parity | — | — | ✅ | **Compose/VLM only.** Compare HF's `visual` output (`pooler_output`, which is **post-merger**) bitwise (0.0), both eager. XTuner splits that boundary into `vision_tower` (patches → merged hidden) + `projector` (the merger MLP → text dim), so load **both** standalone to match it — don't build the whole VLM just for the vision path. Build them on the **real device, not `meta`**: the vision rotary is computed into a buffer at forward time, so `to_empty` would leave garbage (unlike the text rotary, which `build_rotary_embedding` builds real even under meta — §6.3). Vision attention is *non-causal* → `eager_attention(causal=False)`, which `XTUNER_HF_IMPL` selects. Reference: `test_vision_tower_bitwise_parity`. |\n| FSDP forward parity | ✅ | ✅ | ✅ | Looser tolerance (≈ 3e-2 is the established budget). |\n| `save_hf` round-trip (byte-equal tensors) | ✅ | ✅ | ✅ | See `tests/model/test_qwen3_moe.py`. |\n\n### 7.2 Parity bar by model scale\n\nTwo independent axes:\n\n**Per-layer (op) parity — required at _all_ scales, bitwise.** Test one full and\none linear decoder layer separately; for each assert layer **output + loss + `dL/dx`**\nbitwise vs HF under `XTUNER_HF_IMPL`. Keep it memory-light (XTuner `meta`-build +\nmaterialize only the tested layer + `norm` + `lm_head`; standalone HF layer; load\nvia `HFCheckpointLoader`) so it runs even for models too large to forward\nend-to-end. This is the bitwise guarantee for both forward and backward ops.\n\n**Whole-model (integration) parity — on the _top-level_ model, graded by what fits\none GPU.** Run the real model end-to-end: for a VLM that's the **compose model on an\nimage prompt** (one test covers vision + projector + text — don't add a separate\ntext-tower model test); for a plain LLM, the text/LM model.\n- **Small** (e.g. dense 4B): both forward (logits/loss) AND backward\n (`dL/d(pixel_values)` for a VLM, `dL/d(inputs_embeds)` for text) **bitwise**.\n Prerequisites: `XTUNER_HF_IMPL=true` so every op selector returns the\n HF-canonical callable (§6.2) AND `XTUNER_DETERMINISTIC=true` so Triton\n autotune is pinned (§6.2). With both on, even hybrid linear/full models match\n HF byte-for-byte through the whole compose chain. Don't check weight\n gradients — they accumulate across positions + tied `lm_head` and aren't the\n right granularity for parity.\n- **Medium**: forward only (logits/loss bitwise).\n- **Large** (cannot forward end-to-end on the test GPUs): skip the whole-model\n test; leave it to integration tests. The per-layer test still gives op-level\n forward+backward bitwise.\n\n### 7.3 Required test idioms\n\n`tests/model/test_qwen3_5_dense.py` is the reference template; extend\n`DeterministicDDPTestCase` (`xtuner._testing`), which provides the shared\nparity scaffolding so the test file only holds model-specific logic:\n\n- `with self.hf_impl():` — sets/restores `XTUNER_HF_IMPL=true`; build the XTuner\n model **inside** the block so its attention picks the eager op at construction.\n- `self.materialize_xtuner_submodule(model, submodule, prefix, loader)` — for a\n model built on `meta`, materialize *only* that submodule on CUDA and load its\n weights (so a single-layer test fits any model size).\n- `self.load_params_from_hf(module, key_for, loader)` /\n `self.xtuner_ckpt_key(model, name)` — selective per-param load by checkpoint key\n (`HFCheckpointLoader` reads only the needed shard); `xtuner_ckpt_key` applies\n `to_hf_key_list` + the config's `hf_key_mapping`.\n\n```python\n# Construct on meta, materialize on device — single-rank baseline, eager, no EP.\nwith torch.device(\"meta\"):\n cfg = get_model_config_from_hf(hf_model_path)\n cfg.compile_cfg = False # baseline tests run eager\n model = cfg.build()._to_device_dtype(dtype=torch.bfloat16, skip_buffers_dtype=True)\n\n# Optionally shard with FSDP (data-parallel only), then load.\nmodel.fully_shard(fsdp_config=FSDPConfig(cpu_offload=False))\nmodel.from_hf(hf_model_path)\n```\n\nOlder `patch_hf_rms_norm` / `patch_hf_rope` helpers also live in\n`xtuner._testing`. **Check they actually fit your model before using them** —\nthey were written for earlier models and match HF modules\nby class-name substring and attribute name (e.g. `variance_epsilon`). On a newer\nmodel they can silently mis-patch (wrong eps attribute, or replacing a\n`zero_centered` RMSNorm with a default one). Often the native XTuner ops already\nmatch HF bitwise (see §12), so no patching is needed; prefer `XTUNER_HF_IMPL`\nover these helpers for parity. The §8 parallel-feature tests reuse this same\nidiom with `cfg.dispatcher` / `cfg.ep_size` (and `FSDPConfig(ep_size=...)`) set.\n\n### 7.4 Checkpoint paths\n\nRead from env vars; **never** hard-code. Example:\n`QWEN3_MOE_PATH = os.environ[\"QWEN3_MOE_PATH\"]` at\n`tests/model/test_qwen3_moe.py`. Document new env vars in the PR body.\n\n### 7.5 Drop-in training config\n\nShip a runnable training config at `ci/config/.py` so a developer (and CI) can verify\nthe port end-to-end with one command: load the HF checkpoint, run a few real training steps,\nwatch the loss curve descend as expected. This is the smoke test that catches anything the\nunit tests miss — the full `from_hf` → `model.forward` → loss → backward → optimizer step →\nFSDP shard/reduce chain — and the file doubles as the example users copy when wiring the\nmodel into their own training pipeline. Mirror `ci/config/qwen3_moe_30BA3.py` (MoE) or\n`ci/config/qwen3_dense.py` (dense), keeping its structure: one `()` (from §3.2)\nfed into a `TrainerConfig` alongside `optim_cfg` / `lr_cfg` / `fsdp_cfg` / `dataset_cfg` /\n`dataloader_cfg` / `loss_cfg`. `load_from` and `tokenizer_path` read from an env var (§7.4 —\ntypically the same one as the parity test). Verify by running ~50 steps and confirming the\nloss drops monotonically into a plausible range for that model size; record the trajectory\nin the PR body alongside the §6 convergence trace.\n\n---\n\n## 8. Training optimizations\n\nOnly after the §6 baseline is committed and the §7 tests are green: verify each\nswitch runs correctly and keeps precision within budget **against that baseline**.\n\n1. EP and SP can be enabled normally.\n2. With `ep_size > 1`, `intra_layer_micro_batch` enables correctly and stays\n precise.\n3. `torch.compile` runs correctly.\n4. fp8 can be enabled.\n5. `XTUNER_ACTIVATION_OFFLOAD` can be enabled.\n\nEach switch needs a test that compares its output to the §6 baseline — e.g. the\ndispatcher × ep_size parity matrix (at minimum `{(None,1), (\"all2all\",4),\n(\"all2all\",8)}`, add `\"deepep\"` if supported), compiled-vs-eager, or fp8-vs-bf16\nwithin tolerance.\n\nThese items are well-suited to the multi-agent workflow in §11: the main agent\nanalyzes and creates tasks (with test cases), sub-agents develop in parallel, and\nthe main agent merges. Report the precision error introduced by each switch.\n\n---\n\n## 9. Reference cheat-sheet\n\n| Concept | Concrete location |\n|-------------------------|--------------------------------------------------------------------------------|\n| Dense base config | `xtuner/v1/model/base.py` (`TransformerConfig`) |\n| MoE base config | `xtuner/v1/model/moe/moe.py` (`MoEConfig`) |\n| Compose base config | `xtuner/v1/model/compose/base.py` (`BaseComposeConfig`) |\n| `from_hf` (model) | `xtuner/v1/model/base.py` (`from_hf`) |\n| `save_hf` (model) | `xtuner/v1/model/base.py` (`save_hf`) |\n| `to_hf_key_list` proto | `xtuner/v1/model/base.py` |\n| MoE layer split | `xtuner/v1/model/moe/moe.py` (`first_k_dense_replace`) |\n| RoPE auto-parse | `xtuner/v1/module/rope/rope.py` (`RopeParametersConfig.from_hf_config`)|\n| Dense reference | `xtuner/v1/model/dense/qwen3.py` |\n| MoE reference | `xtuner/v1/model/moe/qwen3.py` |\n| MoE w/ MLA + NoAux | `xtuner/v1/model/moe/deepseek_v3.py` |\n| MoE w/ layout transform | `xtuner/v1/model/moe/gpt_oss.py` |\n| Remote-code reference | `xtuner/v1/model/moe/qwen3.py` (`Qwen3MoEFoPEConfig`) |\n| Compose reference (VLM) | `xtuner/v1/model/compose/qwen3_vl/` |\n| Dense test reference | `tests/model/test_qwen3_dense.py` |\n| MoE test reference | `tests/model/test_qwen3_moe.py` |\n| Engine test reference | `tests/engine/test_moe_train_engine.py` |\n\n---\n\n## 10. Commit discipline\n\n0. **Commit as you go — this is authorized, do not pause for a separate\n go-ahead.** The user invoking this skill authorizes the commits this workflow\n produces. Finish a logical step → lint/test → commit, then move on. Do not\n leave the whole port uncommitted and end with \"should I commit?\"; that strands\n the work. (Hard-to-reverse or outward actions like `git push` / opening a PR\n still need explicit confirmation — committing locally does not.)\n1. **Follow the stacked-PR convention.** Plan upfront how many commits the model\n port needs — one per logical step (e.g. shared-path fixes; model class +\n config + registration; the `XTUNER_HF_IMPL` switch; baseline tests; then one\n per parallel/optimization feature). Order them by dependency (a shared-path fix\n the baseline relies on goes first). Land every later change as\n `git commit --fixup=` + `git rebase --autosquash` into the commit it\n belongs to. **Never grow the history with endless patch commits just to keep\n fixing things up.**\n2. **Every commit must pass lint:**\n ```bash\n pre-commit run --files $(find xtuner/v1)\n ```\n\n---\n\n## 11. Multi-agent workflow\n\nThis applies **only to the §8 parallel/optimization features** — and only once\nthe §6 baseline and §7 baseline tests are done and committed (§10). The baseline\nitself is built sequentially, not multi-agent; the §8 features are independent of\neach other, which is what makes them safe to develop in parallel.\n\n1. The **main agent** splits the §8 features into tasks and, per §8, writes each\n feature's comparison test against the committed baseline and commits the tests.\n2. **Sub-agents** branch off the baseline/test commit and develop one feature\n each (e.g. one switch / one tower) in parallel, making its test pass.\n3. **The two sides adjust each other** — a shared-interface change is made once by\n the main agent (it updates the test/baseline; sub-agents rebase), and a failing\n test pushes the sub-agent to fix its feature, not to weaken the test.\n4. The **main agent** merges, resolves conflicts, and reports the precision error\n each feature introduces relative to the baseline.\n\n---\n\n## 12. Parity debugging pitfalls\n\nHard-won notes from aligning hybrid (GatedDeltaNet + gated-MHA) models bitwise.\nThe meta-rule: **measure on real GPU and bisect to localize the divergence before\nconcluding anything** — most \"bugs\" here are measurement artifacts, not model\nerrors. A clean way to bisect is to feed one side's intermediate tensors into\nboth modules and compare step by step (norm → q/k/v → qk_norm → rope → attention\ncore → o_proj).\n\n- **Single-layer parity: the two sides' inputs must match exactly — especially\n the causal mask.** When you hand-build one layer's inputs, passing\n `attention_mask=None` to an HF attention/layer makes HF apply **no** causal\n mask while XTuner stays causal, producing a large *false* diff. Build a causal\n mask equivalent to XTuner's `seq_ctx`, or drive the layer through the model so\n the mask is constructed internally.\n\n- **`flash` ≠ `eager`, and HF eager upcasts softmax to fp32.** XTuner defaults to\n flash attention; HF reference eager does softmax in fp32. They are numerically\n close but not bitwise equal, so bitwise parity requires forcing eager on the\n XTuner side (that is exactly what `XTUNER_HF_IMPL` does). XTuner's\n `eager_attention` matches HF's `eager_attention_forward` bitwise.\n\n- **A bitwise forward does not imply a bitwise backward.** The eager softmax must\n run in **fp32** (`softmax(..., dtype=torch.float32).to(x.dtype)`, like HF), not\n bf16. With a bf16 softmax the forward logits can still round identically to HF\n (bitwise), but the *backward* differs (~1e-5 on `dL/dx`) because the gradient\n accumulates at bf16 precision. So always check backward bitwise separately\n (the `loss.backward()` test), not just forward — and align op dtypes (softmax,\n norms) in the backward-sensitive direction, fp32 where HF uses fp32.\n\n- **Whole-model backward is bitwise — but only with the full env stack.**\n Per-layer bitwise backward does not on its own imply whole-model bitwise\n backward; the missing pieces are `XTUNER_HF_IMPL=true` AND\n `XTUNER_DETERMINISTIC=true` AND the conftest must `import xtuner.v1` so the\n autotune-pin block in `xtuner/v1/__init__.py` runs before any test reaches\n fla. Without those, two distinct ULP-level sources compound through the LM\n chain rule and look indistinguishable from \"fundamental bf16 noise\":\n\n 1. **Triton autotune drift in fla.** fla's `@triton.autotune` picks\n tiling / num_warps / reduction-order per kernel. Across runs the choice\n can change, producing a ~1-ULP backward diff per linear-attention layer\n that the chain rule amplifies — a 32-layer LM stack turned a per-layer\n ~2e-3 abs diff at grad magnitude ~30 into a ~1.0 diff on the\n massive-activation channels of `dL/d(pixel_values)`. The pin (configs\n held to the first one, cache disabled) is in `xtuner/v1/__init__.py`\n under an `XTUNER_DETERMINISTIC` gate. It must run **before any module\n imports fla** — fla's `@triton.autotune` decorators evaluate at import\n time, so patching after the fact has no effect on kernels already\n constructed. That's why the patch is on the `xtuner.v1.*` import chain\n rather than in the conftest: `multiprocessing.spawn` children of\n `MultiProcessTestCase` re-import the test class top-level (which pulls\n `xtuner.v1.model.*`) without running conftest, so the conftest alone\n can't install the patch in those processes.\n 2. **`torch.library.custom_op` wraps for the linear-attention path**\n (`GatedDeltaNet`, `causal_conv1d`) produce a different backward autograd\n graph from the canonical fla / causal_conv1d call HF uses, even though\n the underlying kernel and forward output are identical. `XTUNER_HF_IMPL`\n routes through `get_chunk_gated_delta_rule_fn` / `get_causal_conv1d_fn`\n (§6.2) so the parity path matches HF byte-for-byte.\n\n Production train doesn't set either env var; the custom_op wraps + autotune\n still operate. These switches are **parity-test only**. If the whole-model\n backward is not bitwise but the per-layer backward is, suspect one of the\n two before suspecting depth-accumulated bf16 noise.\n\n- **`flash`(XTuner varlen) vs `flash_attention_2`(HF) is bitwise only for short,\n single-tile sequences.** They are the same kernel, so at a short seqlen (e.g.\n 32) the logits are bitwise (0.0); at longer sequences the tiling /\n accumulation order differs and they drift ~1e-2. So a short-seq text test can\n assert flash-vs-FA2 bitwise, but anything realistic-length (or a VLM whose\n image tokens make the sequence long) needs the **eager** path for\n seqlen-independent bitwise. Don't conclude \"flash matches FA2\" from one short\n input.\n\n- **Shared kernels are already bitwise; focus on what isn't.** GatedDeltaNet /\n linear attention go through the same `fla` / `causal_conv1d` kernels as HF, and\n native rms_norm / rope match HF bitwise — these come out at 0.0 with no work.\n Spend your effort on the path that actually differs (usually full attention).\n\n- **`output_hidden_states` off-by-one (full-forward route, < 40B only).** HF's\n tuple is `[emb, L0_out, …, L{n-2}_out, post_norm]`: index `i+1` is layer `i`'s\n pre-norm output only for `i ≤ n-2`; the **last** entry is post-norm, not the\n last layer's pre-norm output. Compare the last layer via `model.norm(...)` or\n you will see the final layer \"explode\".\n\n- **A big max-diff is not automatically a bug.** Late layers have *massive\n activations* (a few channels in the tens/hundreds); a tiny relative error there\n shows up as a large absolute max-diff. Look at the mean and the trend, not just\n the max. Conversely, feeding `randn` inputs to a single attention layer is\n ill-conditioned and misleads — use real (or model-produced) hidden states.\n\n- **A single-model failure is often a general-path bug.** E.g. `Dense.build_layers`\n not forwarding `rms_norm_type` to `DenseDecoderLayer` broke any `zero_centered`\n dense model, not just this one. Fix the shared invariant, not the symptom.\n\n- **Loading a standalone sub-tower uses `strict=False`.** A text tower loaded\n directly from a full VLM checkpoint will see vision (and deferred `mtp.*`) keys\n as unexpected; load with `strict=False` so they are ignored.\n\n- **Test the whole VLM, not just the text tower.** Text-tower parity leaves the\n vision tower and projector completely untested. A compose port needs a\n full-model forward-parity test with real image inputs (§7.1). The text path of\n the compose forward still runs a *dummy* vision forward to keep the tower in the\n autograd graph, so vision bugs surface even on text prompts.\n\n- **Vision attention is non-causal — give `eager_attention` a `causal=False`\n path.** XTuner's `eager_attention` defaulted to a causal block-diagonal mask;\n that is wrong for a vision tower (bidirectional within each image), which is why\n forcing vision to the causal eager op produced large diffs. The op now takes\n `causal=True|False` (False → non-causal block-diagonal mask), the vision module\n already calls it with `causal=False`, and the vision tower routes through\n `get_attn_impl_fn` so `XTUNER_HF_IMPL` selects eager. With that, the *whole* VLM\n is bitwise in one run, everything eager: `XTUNER_HF_IMPL` on the XTuner side\n (text causal-eager + vision non-causal-eager) and HF loaded with a single\n `attn_implementation=\"eager\"`. Prefer eager for parity — it is\n seqlen-independent, unlike flash/FA2 (see the flash seqlen bullet above).\n\n- **Vision pos-embed dtype when reusing the Qwen3-VL tower.** HF's\n `fast_pos_embed_interpolate` returns fp32 (bilinear weights are fp32) and HF's\n vision forward casts it (`pos_embeds.to(hidden_states.dtype)`); XTuner's vision\n forward adds it without a cast. If you patch in the HF method for parity, cast\n the result back to the param dtype, or you get an fp32/bf16 `LayerNorm` mismatch\n (`expected scalar type Float but found BFloat16`).\n\n- **Hybrid-model `torch.compile`: choose `fullgraph` per layer — verify, then fall\n back; don't gate *whether* to compile.** When a decoder layer's forward is wrapped\n into a higher-order op (HOP) — e.g. by an activation-checkpoint wrapper, which\n `torch.compile` lowers to a HOP whose body must be functional so it can be\n recomputed in backward — *and* that layer is a linear-attention layer\n (GatedDeltaNet), the layer mutates an outer-scope variable inside the HOP\n (GatedDeltaNet writes back into `seq_ctx`). Under `fullgraph=True` that raises\n `HigherOrderOperator ... Mutating a variable not in the current scope (SideEffects)`.\n This is **not** a blanket \"linear attention ⇒ `fullgraph=False`\" rule — whether it\n triggers depends on whether the layer is actually HOP-wrapped and whether that\n linear impl really writes outer state. So for such a HOP + linear combination,\n **first test whether `fullgraph=True` runs**; keep it if it does (it compiles more\n of the layer), and only **fall back to `fullgraph=False`** if it doesn't —\n `fullgraph=False` survives by graph-breaking at the mutation and running that\n side-effecting write in eager. Make this a **per-layer** choice in\n `Dense.fully_shard` (decide each layer's `fullgraph`), not a decision about whether\n a layer is compiled at all.\n\n","frontmatter":{"name":"add_hf_model","description":"Add support for a HuggingFace model in XTuner so it can be trained with the full set of parallelism strategies and optimization switches. The user can provide a HuggingFace Hub repo id, a local model directory, or a model family already supported by `transformers`. This skill walks through the full path: locating the reference implementation, classifying the model, splitting new code across the model / module / ops layers, implementing the XTuner model class and config, registering the entry point, validating bitwise numerical parity with HuggingFace, writing regression tests, and enabling the training optimizations (EP/SP, micro-batch, torch.compile, fp8, activation offload).\n"},"isInternal":false,"tokens":11578,"sizeBytes":44555},{"name":"openai.yaml","path":".claude/skills/add-chat-template/agents/openai.yaml","rawUrl":"https://raw.githubusercontent.com/InternLM/xtuner/HEAD/.claude/skills/add-chat-template/agents/openai.yaml","title":"Agents Skill","category":"anthropic-skill","format":"yaml","content":"interface:\n display_name: \"Add Chat Template\"\n short_description: \"Add and verify XTuner chat templates\"\n default_prompt: \"Use $add-chat-template to add or audit XTuner chat-template rendering, loss masks, and stop-token behavior for this official Hugging Face model.\"\n","isInternal":false,"tokens":59,"sizeBytes":272},{"name":"SKILL.md","path":".claude/skills/add-chat-template/SKILL.md","rawUrl":"https://raw.githubusercontent.com/InternLM/xtuner/HEAD/.claude/skills/add-chat-template/SKILL.md","title":"add-chat-template","category":"anthropic-skill","format":"markdown","content":"---\nname: add-chat-template\ndescription: Add or audit an XTuner chat-template implementation and its loss mask from an official Hugging Face model repository or local model directory. Use when supporting a new model under xtuner/v1/data_proto/messages, checking an existing implementation against tokenizer or processor apply_chat_template, deciding which assistant output and stop/EOS tokens receive loss, adding regression tests, or validating that exported training behavior matches vLLM and SGLang inference.\n---\n\n# Add Chat Template\n\n## Goal\n\nTreat the official Hugging Face tokenizer, or processor for a multimodal model,\nas the rendering oracle. Produce the same conversation token sequence in\nXTuner, supervise model-generated output by default, and supervise the\ncontext-appropriate token that ends each assistant generation. Prove the\nbehavior through XTuner's public tokenize API and, when available, the target\nvLLM and SGLang versions.\n\nKeep the implementation small and model-local. Do not infer a template or stop\ncontract from another model in the family.\n\n## 1. Establish the reference\n\n1. Record the official HF repo or local directory, immutable revision, tokenizer\n class, Transformers version, and whether `trust_remote_code=True` is required.\n2. Inspect the actual artifacts used by inference:\n `tokenizer_config.json`, `chat_template.jinja` or `chat_template`,\n `special_tokens_map.json`, `generation_config.json`, and `config.json`.\n Inspect `processor_config.json` and remote processor/template code when the\n official multimodal path uses `AutoProcessor.apply_chat_template`.\n3. Run the bundled inventory against the same revision:\n\n ```bash\n python .claude/skills/add-chat-template/scripts/audit_hf_chat_template.py \\\n --trust-remote-code\n ```\n\n Omit `--trust-remote-code` unless the official model requires it. For\n model-specific branches such as tools, thinking, or multimodal inputs, pass a\n JSON case file with `--cases `; see the script's `--help` output.\n The script inventories the tokenizer side; for a processor-owned multimodal\n template, run the corresponding `AutoProcessor` renderer separately.\n4. Render every supported branch with the same `tokenizer.apply_chat_template`\n or `processor.apply_chat_template` entry point used by the official example.\n For multimodal models, do not assume tokenizer-only rendering reproduces\n media placeholders or processor expansion. Record all required template\n kwargs and their defaults, including\n `add_generation_prompt`, `continue_final_message`, tool definitions,\n thinking/reasoning flags, named templates, and multimodal options. Do not\n copy rendered strings from a model card when the executable tokenizer is\n available.\n5. When the selected official template contains Jinja `{% generation %}`\n regions, call its `apply_chat_template` with `tokenize=True`,\n `return_dict=True`, and\n `return_assistant_tokens_mask=True`. Use its assistant mask as an additional\n oracle, then separately audit stop boundaries: a next-role stop token can sit\n outside the official generation region while still belonging to the previous\n assistant for SFT.\n6. Pin the reference revision in tests or CI configuration. A moving HF branch\n is not a stable oracle.\n\n## 2. Audit before implementing\n\nSearch these integration points first:\n\n- `xtuner/v1/data_proto/templates/__init__.py`\n- `xtuner/v1/data_proto/messages/`\n- `xtuner/v1/data_proto/messages/__init__.py`\n- `xtuner/v1/datasets/sft_tokenize_fn/openai.py`\n- multimodal tokenize functions when applicable\n- `tests/chat_template/` and `tests/datasets/`\n\nIf the model already exists, run its public tokenize path before changing it:\n\n```python\ntokenize_fn = OpenaiTokenizeFunctionConfig(chat_template=\"\").build(tokenizer)\nresult = tokenize_fn({\"messages\": messages, \"tools\": tools})\n```\n\nCompare the result with the official renderer and inspect existing tests. Find a\nconcrete failing message sequence before fixing a mismatch. Do not rewrite a\nworking implementation because its structure looks unusual.\n\nUse `HybridChatTemplate` plus `ChatMessages` only when fixed role wrappers fully\nexpress the official template. Add a dedicated\n`xtuner/v1/data_proto/messages/_chat.py` when rendering depends on\nmessage history, tools, reasoning, content types, or the next role. Follow\n`glm52_chat.py` and `qwen35_chat.py` as integration examples, not as rendering\nspecifications for another model.\n\n## 3. Write the stop contract first\n\nInventory three distinct sources; do not collapse them into one `eos_token`:\n\n1. **Configured generation stops**: `tokenizer.eos_token_id`, `config.json`,\n `generation_config.json`, including integer lists and stop strings.\n2. **Template boundaries**: the exact token or token sequence following an\n assistant when the next item is user, tool/observation, system/developer, or\n the end of the sample.\n3. **Engine stops**: the IDs and strings actually installed by the target vLLM\n and SGLang versions after loading the exported checkpoint.\n\nCreate an explicit table before writing the mask:\n\n| Transition | Official serialized boundary | Token IDs | Engine mechanism | Loss when assistant loss is true |\n|---|---|---|---|---|\n| assistant → user | exact role/EOT sequence | exact IDs | EOS ID, stop ID/string, or parser | yes for the generated stop target |\n| assistant tool call → tool | exact observation boundary | exact IDs | EOS ID, stop ID/string, or parser | yes for the generated stop target |\n| final assistant → end | exact EOT/EOS, or documented training-only EOS | exact IDs | EOS ID or stop ID/string | yes |\n\nVerify each token by both encoding the exact boundary and converting its ID back\nto a token. Reject unknown-token conversions. Handle a multi-token stop as a\nsequence rather than assuming every stop is one special token.\n\n### Role tokens can belong to the preceding assistant\n\nSome official templates have no dedicated per-turn EOT. The next role token is\nthen the token the model must generate to stop the previous assistant. In that\ncase, keep the official serialized order and assign loss only on that boundary\ntoken or sequence to the preceding assistant:\n\n```text\nassistant answer user text\n ^ assistant loss ^ masked\n```\n\nFor a GLM-like protocol this can mean supervising the user boundary after a\nnormal answer, the observation boundary after a tool call, and the configured\nterminal EOS only when no following role supplies a boundary.\n\nDo not append a generic EOS after every assistant when the official inference\ntemplate places a role/observation boundary there. That creates training history\nthat vLLM and SGLang do not render. Conversely, when the official full-message\nrenderer omits the final token that the model must generate to terminate,\nappend exactly that one terminal token for SFT and document this single\ntraining-only suffix.\n\nConfigured EOS IDs are alternatives, not a sequence to append together.\nSupervise the actual context-specific stop token present in each training path.\n\n## 4. Implement rendering and loss together\n\nPrefer one cohesive renderer that returns rendered text plus a character- or\ntoken-level loss mask. Keep context-dependent boundary ownership beside the\nrendering branch that emits the boundary; avoid a collection of shallow helper\nfunctions.\n\nApply these defaults:\n\n- Treat `assistant` messages with no `loss` field as `loss=True`.\n- Mask system/developer/user/tool inputs and role scaffolding unless the stop\n contract proves that a boundary is generated by the preceding assistant.\n- Supervise all model-generated assistant components: visible content,\n reasoning that remains in the rendered sample, reasoning closing syntax,\n tool calls, and the appropriate stop target.\n- Treat `loss=False` as masking the entire assistant output and its stop target.\n- Mask `add_generation_prompt` completely; it is prompt scaffolding, not a\n demonstrated model output.\n- Preserve explicit model behavior such as clearing historical reasoning only\n when the official template does so for the chosen kwargs.\n\nIf character offsets are used, verify special-token and Unicode boundaries with\nthe real tokenizer. A token overlapping a loss span needs a deliberate,\ntested rule. Keep a slow, independent token-level oracle for regression tests;\ndo not call the production renderer from both sides of the same parity test.\n\nMaintain exact rendering parity before any documented training suffix:\n\n```mermaid\nflowchart LR\n A[\"Official HF messages\"] --> B[\"tokenizer.apply_chat_template\"]\n A --> C[\"XTuner public tokenize API\"]\n B --> D[\"Exact text and token IDs\"]\n C --> D\n D --> E[\"Optional one terminal SFT stop\"]\n E --> F[\"Loss-mask contract\"]\n```\n\nDo not use decoded-text equality as the sole check: decoding can hide a token-ID\nmismatch.\n\n## 5. Register the smallest integration\n\nFor a dedicated message implementation, update only the required seams:\n\n1. Export the message class from `xtuner/v1/data_proto/messages/__init__.py`.\n2. Add the public name and serving stop metadata to `CHAT_TEMPLATE_MAP`.\n3. Dispatch that name in `OpenaiTokenizeFunction`; update multimodal dispatch\n only if the model uses it.\n4. Update typed CLI/config literals only where they restrict the new name.\n\nKeep `CHAT_TEMPLATE_MAP.stop_words` consistent with the discovered contract,\nbut do not treat it as proof of engine behavior. vLLM and SGLang load their\nruntime stops from the exported HF artifacts and request/engine configuration.\n\n## 6. Add good regression tests\n\nUse the real official tokenizer/processor and XTuner's public tokenize\nfunction. Mock only unavailable external services. Cover every supported\ntemplate branch with the smallest useful matrix:\n\n- user → assistant, with `loss` omitted;\n- assistant → user in a multi-turn conversation;\n- assistant tool call → tool/observation → assistant;\n- final assistant;\n- `loss=False` followed by another role and at sample end;\n- system/developer input;\n- `add_generation_prompt=True`;\n- reasoning enabled/disabled and historical reasoning behavior;\n- tools and multimodal content when supported;\n- repeated assistant text, Unicode, and adjacent special tokens to expose\n faulty substring or offset alignment.\n\nAssert public behavior:\n\n1. Official tokenizer/processor rendered text equals XTuner rendered text,\n except for the one explicitly documented final SFT suffix if required.\n2. Official token IDs equal XTuner `input_ids` before that suffix.\n3. `len(input_ids) == len(labels)` and every label is either the matching input\n ID or `IGNORE_INDEX`.\n4. Assistant output receives loss by default; `loss=False` removes all of it.\n5. Each transition's exact stop token IDs receive or do not receive loss as the\n stop table specifies.\n6. User/tool content and generation prompts remain masked.\n7. The fast implementation matches an independent slow oracle.\n\nTest token positions and IDs directly. A substring assertion can supplement but\nmust not replace the loss assertion. Avoid snapshots that pass after both the\nrenderer and expected string are changed to the same incorrect value.\n\n## 7. Validate vLLM and SGLang\n\nResolve exact target versions. Prefer the user's production versions; otherwise\ninspect the current installed versions and latest stable official source. For\neach engine:\n\n1. Load the same exported tokenizer, `config.json`, and\n `generation_config.json`.\n2. Compare the engine's chat-render/tokenize output with the official\n tokenizer/processor renderer for every stop-table transition.\n3. Inspect or log the final stop token IDs/strings installed in the generation\n request. Distinguish tokenizer EOS, model generation-config EOS, request stop\n strings, and model-specific parsers.\n4. Run generation smoke tests for a normal final answer and a tool call. Use a\n sufficiently high output limit and assert the engine reports a stop rather\n than a length limit, with no role leakage.\n\nObtain the repository GPU lock before a local GPU server or generation test. If\nthe exact runtime cannot be executed, audit the exact tagged engine source and\nreport the result as a static source audit, not runtime compatibility.\n\nSwitching from Chat Completions to Responses does not remove this requirement:\nfor ordinary HF models both interfaces ultimately render model messages through\nthe chat template.\n\n## 8. Completion checklist\n\nFinish only when the report includes:\n\n- official HF repo, revision, and Transformers version;\n- implementation path and whether it is generic or dedicated;\n- rendering-parity matrix;\n- stop-contract table with token strings, IDs, and loss ownership;\n- tests proving default assistant loss and `loss=False`;\n- vLLM/SGLang versions and runtime or exact-tag source validation;\n- every intentional difference from the official renderer, normally at most a\n final training stop suffix.\n\nIf auditing an existing implementation, state which requirements already had\ntests, which were missing, and the concrete mismatches found. Do not claim the\nimplementation is correct merely because a test file exists.\n","frontmatter":{"name":"add-chat-template","description":"Add or audit an XTuner chat-template implementation and its loss mask from an official Hugging Face model repository or local model directory. Use when supporting a new model under xtuner/v1/data_proto/messages, checking an existing implementation against tokenizer or processor apply_chat_template, deciding which assistant output and stop/EOS tokens receive loss, adding regression tests, or validating that exported training behavior matches vLLM and SGLang inference."},"isInternal":false,"tokens":2749,"sizeBytes":13199},{"name":"openai.yaml","path":".claude/skills/check-hf-config-save/agents/openai.yaml","rawUrl":"https://raw.githubusercontent.com/InternLM/xtuner/HEAD/.claude/skills/check-hf-config-save/agents/openai.yaml","title":"Agents Skill","category":"anthropic-skill","format":"yaml","content":"interface:\n display_name: \"Check HF Config Save\"\n short_description: \"Implement and validate exported HF configs\"\n default_prompt: \"Use $check-hf-config-save to implement any missing official Hugging Face config export and validate it across Transformers and inference engines.\"\n","isInternal":false,"tokens":55,"sizeBytes":282},{"name":"SKILL.md","path":".claude/skills/check-hf-config-save/SKILL.md","rawUrl":"https://raw.githubusercontent.com/InternLM/xtuner/HEAD/.claude/skills/check-hf-config-save/SKILL.md","title":"check-hf-config-save","category":"anthropic-skill","format":"markdown","content":"---\nname: check-hf-config-save\ndescription: Implement a missing XTuner Hugging Face config export from the official HF model when possible, then validate it against the installed Transformers round-trip and versioned inference-engine field contracts. Use when adding a model, when hf_config is missing or returns None, when changing from_hf/hf_config/save_hf, when reviewing config.json differences, or when debugging vLLM/SGLang checkpoint-load or inference failures. Also use for requests named check_hf_config_save.\n---\n\n# Check HF Config Save\n\n## Overview\n\nFirst ensure that a model with an official built-in Transformers config has an\ninverse `from_hf <-> hf_config` mapping. If that export is missing, implement it\nfrom the official HF model before using `xtuner._testing.check_hf_config_save`\nto test the real public `from_hf -> save_hf` path. Separate changes forced by\nthe installed Transformers version from fields dropped by XTuner, then protect\nfields that exact inference engine versions use for model construction or\nweight loading.\n\n## Workflow\n\n### 1. Fix the scope and version matrix\n\n1. Identify the source HF model directory, XTuner config class, export path, and\n any engine/version named by the user or failure log.\n2. Record the executable environment versions for Python, Transformers, and any\n installed engines. Use the requested environment; otherwise follow the repo's\n default environment instructions.\n3. A user-specified or production-log version takes precedence. If a component\n version is unspecified, look up the latest stable release from its official\n PyPI project/API or official release page and inspect the matching source tag.\n By default audit both vLLM and SGLang.\n4. List every checked version in the result. Distinguish an installed runtime\n test from a static audit of an exact source tag.\n\nUse a table with at least these columns:\n\n| Component | Version | How selected | Validation |\n|---|---:|---|---|\n| Transformers | exact version | active environment | executable round-trip |\n| vLLM | exact version | user/log/latest official | runtime or exact-tag source audit |\n| SGLang | exact version | user/log/latest official | runtime or exact-tag source audit |\n\nDo not write `latest` without resolving it to an exact version and source.\n\n### 2. Implement a missing HF config export\n\nExercise the supported model's public conversion path first:\n\n```python\nconfig = get_model_config_from_hf(SOURCE_HF_DIR)\nexported_config = config.hf_config\n```\n\n`exported_config is None`, or the corresponding public `config.save_hf(...)`\nraising the base missing-`hf_config` `NotImplementedError`, means the config-only\nexport is not implemented. Classify the official model before changing code:\n\n```mermaid\nflowchart TD\n A[\"Official HF model and exact revision\"] --> B{\"Built-in official Transformers config?\"}\n B -->|Yes| C{\"XTuner hf_config implemented?\"}\n C -->|No| D[\"Implement inverse from_hf and hf_config mapping\"]\n C -->|Yes| E[\"Continue round-trip validation\"]\n D --> E\n B -->|No; trust_remote_code only| F[\"Keep hf_config as None and validate source-config copying\"]\n```\n\nFor a built-in official Transformers config:\n\n1. Read the official checkpoint's raw `config.json` and resolve its exact\n `model_type`, `architectures`, repository revision, and official config\n implementation. Load it with the selected Transformers version and record\n the concrete `PretrainedConfig` subclass. Do not use a third-party model\n implementation as the reference.\n2. Implement or complete `from_hf` with that official class. Map every\n architecture value needed by XTuner, use `getattr` for genuinely optional\n versioned fields, and use `RopeParametersConfig.from_hf_config` for RoPE.\n3. Implement `hf_config` by constructing the same official config class from\n the XTuner config's current values. It must be the inverse of `from_hf`, not\n a cached copy of the source object. Re-emit every field consumed by\n `from_hf`, plus raw compatibility fields required by vLLM or SGLang even when\n Transformers treats them as optional or legacy.\n4. Preserve the official `model_type` and architecture name. Do not substitute\n a generic `PretrainedConfig`, duplicate the official config class in XTuner,\n or return a hand-written dictionary.\n5. Verify through public behavior that `config.hf_config` has the expected\n official type and that `config.save_hf(...)` succeeds, then continue with\n the helper below.\n\nIf the official repository only works through `trust_remote_code` and the\nselected Transformers versions have no built-in config class, do not invent an\n`hf_config` implementation. Keep `hf_config=None`, retain the source `_hf_path`,\nand test the model's public HF save path that copies the original config,\ntokenizer, and required remote-code files. If the entire model or dispatch is\nnot yet supported by XTuner, follow `$add_hf_model`; this skill only fills a\nmissing exporter for an otherwise supported model.\n\n### 3. Establish the Transformers reference\n\nCompare three raw JSON states:\n\n```mermaid\nflowchart LR\n A[\"Source config.json\"] --> B[\"Current Transformers load + save\"]\n A --> C[\"XTuner from_hf + save_hf\"]\n B --> D[\"Expected serialized reference\"]\n C --> E[\"XTuner export\"]\n D --> F[\"Public helper comparison\"]\n E --> F\n```\n\n- Read the source `config.json` before `AutoConfig` normalization.\n- Load and save it directly with the active Transformers version. This is the\n serialized reference and captures forced defaults or `__post_init__` changes.\n- Build through XTuner's public `get_model_config_from_hf`/`from_hf` API and call\n the public `save_hf` API.\n- Compare the Transformers reference with the XTuner export. Do not use direct\n source-versus-export equality as the primary assertion: it misclassifies\n Transformers normalization as an XTuner bug.\n\nInspect the exact Transformers config implementation, including generated or\nmodular source and `__post_init__`, for every source-to-reference difference.\nTypical examples are derived `head_dim`, generated `layer_types`, new defaults,\nand serializer metadata, but never assume these examples cover a new model.\n\n### 4. Audit inference-engine dependencies\n\nFor every changed, missing, newly defaulted, or architecture-selecting field:\n\n1. Search the exact engine tag, not an arbitrary installed or main-branch copy.\n2. Follow model registration, config access, module construction, and the weight\n loader. Search aliases, `getattr` defaults, direct attribute access, and tensor\n name/shape conditions.\n3. Classify the use as one of:\n - module/parameter registration;\n - checkpoint key or tensor-shape selection;\n - layer topology or MoE routing;\n - attention/RoPE behavior;\n - ignored or default-compatible.\n4. Encode every value-sensitive dependency as `HFConfigFieldDependency`, with\n exact engine version, JSON pointer, expected value, reason, and an official\n source permalink.\n\nA field can be HF-equivalent yet engine-critical. For example, an engine may\nregister a checkpoint parameter only when a legacy routing field has one exact\nvalue; omitting that field then becomes a real weight-loader failure.\n\nStatic source inspection proves the dependency but not end-to-end engine\ncompatibility. Run an engine checkpoint-load smoke test when the exact runtime\nand required hardware are available. Otherwise report `exact-tag source audit`\nand do not claim runtime success. Follow the repo GPU-lock instructions before\nany local GPU run.\n\n### 5. Add the model regression test\n\nDelete narrow hand-written assertions that duplicate this contract, then add a\nmodel test on the real public conversion path:\n\n```python\nimport transformers\n\nfrom xtuner._testing import HFConfigFieldDependency, check_hf_config_save\nfrom xtuner.v1.model import get_model_config_from_hf\n\n\ndef test_save_hf_matches_transformers_and_engine_contracts():\n config = get_model_config_from_hf(SOURCE_HF_DIR)\n assert isinstance(config.hf_config, OFFICIAL_HF_CONFIG_CLASS)\n report = check_hf_config_save(\n config,\n SOURCE_HF_DIR,\n engine_dependencies=(\n HFConfigFieldDependency(\n engine=\"vllm\",\n version=\"\",\n path=\"/\",\n expected=\"\",\n reason=\"\",\n source=\"\",\n ),\n ),\n )\n\n assert report.transformers_version == transformers.__version__\n assert report.checked_engine_versions == (\"vllm==\",)\n```\n\nThe helper performs two independent checks:\n\n- XTuner export matches the active Transformers direct round-trip.\n- Exported values satisfy the declared inference-engine contracts, even when\n Transformers itself does not consume those fields.\n\nUse `allowed_export_differences={\"/json/pointer\": \"specific reason\"}` only for\nan intentional XTuner difference that is not already an engine dependency.\nEvery exception needs a non-empty model-level reason.\n\n### 6. Prove the regression and run the matrix\n\n1. Run the helper's own behavior tests.\n2. Run the generated model test in the project's pinned Transformers version.\n3. Run it in each user-requested/current upgraded Transformers environment.\n4. If the old broken export is available, pass its `config.json` through the\n same helper and show that the expected missing/extra paths fail. This is the\n minimal proof that the test catches the original bug.\n5. Run formatting and the smallest relevant model test suite.\n\nReport:\n\n- whether `hf_config` already existed, was implemented from which official\n config class/revision, or correctly remained `None` for remote code;\n- source-to-Transformers normalization paths;\n- Transformers-reference-to-XTuner paths (normally none, apart from documented\n allowed contracts);\n- engine dependency, expected value, exact version, and effect;\n- which checks were executable and which were source-only.\n\n## Guardrails\n\n- Compare raw JSON, not only `AutoConfig` attributes; unknown compatibility\n fields may disappear from a newer config class while engines still read them.\n- Exercise public APIs and real conversion behavior. Do not mock XTuner model\n internals.\n- Do not replace the helper with a blanket list of expected keys.\n- Do not silently refresh an engine version in a test. Re-audit its exact source\n before updating the version and permalink.\n- HF semantic equivalence is not evidence of vLLM/SGLang compatibility.\n- Do not set `hf_config=None` for a model with a built-in official config merely\n to bypass config reconstruction or round-trip failures.\n- Preserve unrelated worktree changes and keep the implementation model-agnostic.\n\n## Completion criteria\n\nThe work is complete only when the official config export path exists where it\nshould, the public round-trip matches Transformers, every audited engine\ncontract is preserved, and the report identifies the exact official model\nrevision and component versions used.\n","frontmatter":{"name":"check-hf-config-save","description":"Implement a missing XTuner Hugging Face config export from the official HF model when possible, then validate it against the installed Transformers round-trip and versioned inference-engine field contracts. Use when adding a model, when hf_config is missing or returns None, when changing from_hf/hf_config/save_hf, when reviewing config.json differences, or when debugging vLLM/SGLang checkpoint-load or inference failures. Also use for requests named check_hf_config_save."},"isInternal":false,"tokens":2363,"sizeBytes":11069},{"name":"SKILL.md","path":".claude/skills/model_normalize/SKILL.md","rawUrl":"https://raw.githubusercontent.com/InternLM/xtuner/HEAD/.claude/skills/model_normalize/SKILL.md","title":"model_normalize","category":"anthropic-skill","format":"markdown","content":"---\nname: model_normalize\ndescription: Normalize an HF model checkpoint — quantize bf16/fp16 weights to per-block FP8, and/or repack engine-named safetensors into HF-standard ~4GB shards.\n---\n\n# model_normalize\n\nTwo complementary capabilities for getting a training-engine checkpoint into a\nformat that HF / vLLM / SGLang can consume:\n\n1. **FP8 quantization** of large linears, in two modes:\n - **Reference-guided** — point at an existing FP8 checkpoint of the same\n architecture; the tool reads its index and reproduces the exact same\n quantization decisions on the source.\n - **Heuristic** — apply a built-in rule set that quantizes only the large\n matrix multiplications (attention, MLP, MoE experts) and keeps norms,\n routers, biases, embeddings, and vision tower in their original dtype.\n2. **Standard repack** — re-shard arbitrary safetensors directories into the\n HF-standard ``model-{i:05d}-of-{n:05d}.safetensors`` layout with ~4GB\n shards and a fresh ``model.safetensors.index.json``.\n\nWhen ``to-fp8`` detects non-standard source shard names, it automatically\nruns the repack step on the quantized output, so the user-visible result is\nalways a clean HF-conformant directory.\n\n## Layout\n\n```\n.claude/skills/model_normalize/\n├── SKILL.md # this file\n├── model_normalize.py # top-level CLI (to-fp8 / repack)\n├── hf_to_fp8.py # FP8 conversion library + standalone CLI\n├── heuristics.py # codified default quantization rule set\n├── repack_hf.py # 4GB-shard repacker library + CLI\n└── convert_interns2_to_fp8.sh # example invocation for InternS2\n```\n\n## Usage\n\n### Mode 1 — Reference-guided FP8 conversion\n\n```bash\npython model_normalize.py to-fp8 \\\n --source /path/to/bf16-model \\\n --output /path/to/fp8-model \\\n --reference /path/to/fp8-reference\n```\n\nBehavior:\n- Loads the reference's ``model.safetensors.index.json``.\n- For each ``_scale_inv`` entry in the reference index, marks ````\n as an FP8-quantized tensor.\n- Applies that exact set to the source as a literal name match (no regex\n surprise; works even when names don't follow any pattern).\n\n### Mode 2 — Heuristic FP8 conversion (no reference)\n\n```bash\npython model_normalize.py to-fp8 \\\n --source /path/to/bf16-model \\\n --output /path/to/fp8-model\n```\n\nThe rule set is defined in ``heuristics.py``. Concretely, **quantize iff** the\ntensor name matches one of:\n\n| Group | Pattern (anchored) |\n| -------------------------------- | ------------------------------------------------------------------------------------------------------------------- |\n| Self-attention projections | ``[…]layers..self_attn.{q,k,v,o}_proj.weight`` |\n| Dense MLP | ``[…]layers..mlp.{gate,up,down}_proj.weight`` |\n| MoE per-expert (unfused) | ``[…]layers..mlp.experts..{gate,up,down}_proj.weight`` |\n| MoE fused-expert (3D tensors) | ``[…]layers..mlp.experts.{gate_up_proj, down_proj}`` |\n| MoE shared expert | ``[…]layers..mlp.shared_expert.{gate,up,down}_proj.weight`` |\n| Linear-attn wide projections | ``[…]layers..linear_attn.{in_proj_qkv, in_proj_z, out_proj}.weight`` |\n\nThe ``[…]`` prefix allows ``model.`` and ``model.language_model.``; the same\npatterns repeat under ``mtp.layers..`` for multi-token-prediction blocks.\n\n**Everything else is kept in the original dtype**, in particular:\n- all norms (``*norm*``, ``*_layernorm.*``)\n- MoE routers (``mlp.gate.weight``, ``mlp.shared_expert_gate.weight``)\n- embeddings (``embed_tokens``, ``lm_head``, ``pos_embed``, ``patch_embed``, ``mtp.fc``)\n- vision tower (``model.visual.*``)\n- all ``*.bias``\n- ``linear_attn`` control-flow tensors (``A_log``, ``conv1d.weight``, ``dt_bias``, ``in_proj_a/b.weight``, ``norm.weight``)\n\nThis rule set was validated against the InternS2 / Qwen3-MoE FP8 references:\nit reproduces their quantization decisions on every tensor (no false\npositives or false negatives).\n\n### Mode 3 — Standalone repack (no quantization)\n\n```bash\npython model_normalize.py repack \\\n --source /path/to/messy-shards \\\n --output /path/to/standard-shards \\\n --shard-size-gb 4\n```\n\nGreedy bin-packs tensors into ``model-{i:05d}-of-{n:05d}.safetensors`` shards\nof at most ``--shard-size-gb`` GiB, writes a fresh\n``model.safetensors.index.json`` (including ``metadata.total_size``), and\ncopies all non-safetensors files (config, tokenizer, etc.) through.\n\n## When to use which mode\n\n| Situation | Mode |\n| ---------------------------------------------------------------------------- | ------------------ |\n| You already have an FP8 reference for this architecture. | Reference-guided |\n| New architecture / first-time conversion; standard naming. | Heuristic |\n| Quantized model has engine-specific shard names; need HF-conformant output. | (auto, see below) |\n| You only need to standardize shard naming; weights are already FP8 / bf16. | ``repack`` |\n\nThe ``to-fp8`` subcommand auto-detects non-standard source shard names (anything\nthat isn't ``model.safetensors`` or ``model-NNNNN-of-NNNNN.safetensors``) and\nchains the repack step. Pass ``--no-repack`` to keep the source naming.\n\n## Quantization details\n\n- Block size **128 × 128**, dtype ``float8_e4m3fn`` with saturating cast.\n- Per-block scale stored as ``_scale_inv`` (``float32``) in the same\n shard as the parent weight. Layout matches what vLLM / SGLang load for\n per-block FP8.\n- 3D MoE-expert tensors (shape ``(E, D0, D1)``) are quantized per-expert; the\n per-expert scales are stacked along a leading dim.\n- ``config.json`` receives a ``quantization_config`` block with\n ``quant_method=fp8``, ``weight_block_size=[128, 128]``,\n ``scale_fmt=ue8m0``, and an explicit ``modules_to_not_convert`` list\n derived from the actual quantization decisions.\n\n## Post-conversion handoff (mandatory for the agent)\n\nAfter **any** successful run of `to-fp8` or `repack`, the agent must ask the\nuser — via `AskUserQuestion` — where to **place** the produced\n`*.safetensors` and `model.safetensors.index.json` files. The conversion\noutput is treated as a staging area; the user decides the final home.\n\nOffer at least these options:\n1. **Keep in place** — leave the files at the `--output` path as-is.\n2. **Copy to a new directory** — let the user specify a target; copy the\n shards + index there. Always **copy** (`cp`), never move — the\n `--output` staging directory must remain intact unless the user\n explicitly asks to delete it as a separate step.\n3. **Copy into an existing model directory** — copy shards/index into a\n user-specified directory while preserving its other files (tokenizer,\n config, etc.).\n\nIf the user picks (2) or (3), execute the copy with `cp` (confirming the\ndestination path back to the user before any overwrite). Do not infer the\ndestination from earlier turns — the user may have a fresh target in mind\neach time. Cleaning up the staging directory afterwards is a separate,\nuser-initiated step — never delete it as part of the handoff.\n\n## Example: InternS2 Preview\n\n```bash\npython model_normalize.py to-fp8 \\\n --source /mnt/shared-storage-user/puyudelivery/user/yehaochen/models/InternS2PreviewCandidate5 \\\n --output /tmp/InternS2-FP8 \\\n --reference /mnt/shared-storage-user/puyudelivery/user/yehaochen/models/InternS2PreviewCandidate5-FP8\n```\n\nThe source uses engine-specific names like\n``model-language-0001-fused-save_rank0.safetensors``; the output will be\nrepacked into the standard ``model-00001-of-NNNNN.safetensors`` layout.\n","frontmatter":{"name":"model_normalize","description":"Normalize an HF model checkpoint — quantize bf16/fp16 weights to per-block FP8, and/or repack engine-named safetensors into HF-standard ~4GB shards."},"isInternal":false,"tokens":2006,"sizeBytes":8287},{"name":"SKILL.md","path":".claude/skills/model_patch/SKILL.md","rawUrl":"https://raw.githubusercontent.com/InternLM/xtuner/HEAD/.claude/skills/model_patch/SKILL.md","title":"model_patch","category":"anthropic-skill","format":"markdown","content":"---\nname: model_patch\ndescription: Merge a base HF model dir with an extra HF model dir by index diff — take tensors only present in extra, append them to a new output dir in HF-standard shard layout. Base wins on overlap.\n---\n\n# model_patch\n\nProduce ``out = base ∪ (extra \\ base)`` at the **tensor level**, by diffing the\ntwo ``model.safetensors.index.json`` files. Useful when ``extra`` is a sibling\ncheckpoint (different training run, fine-tune, or fp8 conversion) that holds a\nfew tensors the base is missing — e.g. an updated head, a newly-added MTP\nblock, or auxiliary projections — and you want a self-contained merged dir\nwithout re-running the whole conversion pipeline.\n\nOverlap rule: **base wins**. Any tensor present in both indexes is taken from\n``base``; the corresponding entry in ``extra`` is ignored.\n\n## Layout\n\n```\n.claude/skills/model_patch/\n├── SKILL.md # this file\n└── model_patch.py # CLI + library\n```\n\n## Usage\n\n```bash\npython model_patch.py \\\n --base /path/to/base-model \\\n --extra /path/to/extra-model \\\n --out /path/to/merged-model\n```\n\n## What it writes\n\n- ``model-{i:05d}-of-{N:05d}.safetensors`` shards (total ``N`` = base shards +\n newly-packed shards holding the extra-only tensors)\n- a fresh ``model.safetensors.index.json``\n- **every non-tensor file from ``base``** — ``config.json``, tokenizer files,\n modeling/configuration ``.py``, ``generation_config.json``, README, etc. —\n copied verbatim so ``out`` is loadable on its own. Subdirectories are\n copied recursively. Existing entries in ``out`` are overwritten.\n\n``extra``'s aux files are deliberately **not** copied. The skill's overlap\nrule is \"base wins\" at the tensor level, and that extends to configs: extra\noften carries engine-specific edits or modality keys that don't match the\ntensor topology we keep. If you need fields from ``extra``'s ``config.json``\n(e.g. a new modality), merge them into ``out/config.json`` as a separate,\nexplicit step.\n\n## How shards are produced\n\n- **Base shards**: re-emitted under the new ``-of-N`` total. By default the\n bytes are fully copied so ``out`` is independent of ``base``. Pass\n ``--hardlink`` to use ``os.link`` instead (fast, but the two trees share\n inodes — editing one mutates the other; deleting one keeps the other\n intact since hardlinks are symmetric).\n- **Extra-only tensors**: greedily bin-packed into new shards of at most\n ``--shard-size-gb`` GiB (default 4), appended after the base shards.\n\nSo a base with 12 shards plus 5 GB of extra-only tensors yields:\n``model-00001-of-00014.safetensors`` … ``model-00012-of-00014.safetensors``\n(copies of the base shards) and\n``model-00013-of-00014.safetensors`` / ``model-00014-of-00014.safetensors``\n(freshly written, containing only the extra tensors).\n\n## When to use this skill\n\n| Situation | Use this? |\n| -------------------------------------------------------------------------------------- | --------- |\n| ``extra`` contains a few tensors ``base`` is missing; everything else is identical. | Yes |\n| You need to merge two checkpoints whose overlapping tensors **differ** in value. | No — this skill silently keeps base; you probably want a manual decision per key. |\n| ``extra`` is a full quantization of ``base`` and you want the quantized weights. | No — use ``model_normalize`` with ``--reference extra`` instead. |\n| You want a single ``model.safetensors`` instead of standard shards. | No — repack downstream. |\n\n## Required usage protocol (mandatory for the agent)\n\nBefore invoking ``model_patch.py``:\n\n1. Confirm the three paths (``--base``, ``--extra``, ``--out``) with the user\n via ``AskUserQuestion``. Never infer them from earlier conversation.\n2. Show the user the **diff summary** before writing, by reading the two\n ``model.safetensors.index.json`` files and reporting:\n - number of tensors in base\n - number of tensors in extra\n - number of extra-only keys that will actually be added\n - number of overlapping keys that will be dropped from extra\n3. Ask the user to confirm those numbers before launching the run.\n\nAfter a successful run, **do not** ask the user where the output should go —\n``--out`` is already the final destination the user named. Just report what\nlanded there (shard count, index path, total size, and that base's aux files\nwere mirrored over). Unlike ``model_normalize``, ``model_patch`` has no\nstaging-area concept: there is no temp dir, and the caller already made the\nplacement decision when they passed ``--out``.\n\nIf the user wanted any of ``extra``'s aux files (e.g. a ``config.json`` that\ndocuments a new modality), call that out — those are not copied by the\nskill and must be merged in by hand.\n","frontmatter":{"name":"model_patch","description":"Merge a base HF model dir with an extra HF model dir by index diff — take tensors only present in extra, append them to a new output dir in HF-standard shard layout. Base wins on overlap."},"isInternal":false,"tokens":1169,"sizeBytes":4841},{"name":"SKILL.md","path":".claude/skills/sphinx-debug/SKILL.md","rawUrl":"https://raw.githubusercontent.com/InternLM/xtuner/HEAD/.claude/skills/sphinx-debug/SKILL.md","title":"sphinx-debug","category":"anthropic-skill","format":"markdown","content":"---\nname: sphinx-debug\ndescription: >\n Use when `make html` or `sphinx-build` fails with\n `Extension error (sphinx.ext.autosummary)`,\n `ImportExceptionGroup`, or other Sphinx autosummary / autodoc\n import failures. Provides automated pdb diagnosis for\n `autodoc_mock_imports` issues.\n---\n\n# Sphinx Debug\n\n## Quick Diagnostic Command\n\nRun the build with maximum verbosity and pdb on exception:\n\n```bash\nmake html SPHINXOPTS=\"-vv -T -P\"\n```\n\nFlags:\n- `-vv`: Verbose autodoc mock logging\n- `-T`: Full traceback\n- `-P`: Drop into pdb on exception\n\n## For agents: automate pdb via monkey-patch\n\nIf you are an agent and cannot interact with an interactive pdb session,\n**do not redesign a subprocess/pty capture system**. Instead, monkey-patch\n`pdb.post_mortem` so it runs scripted commands and prints the output\ndirectly to stdout.\n\nRun this one-shot script from the docs directory (`docs/en` or `docs/zh_cn`):\n\n```bash\ncd docs/en # or docs/zh_cn\npython -c \"\nimport pdb, sys, io, traceback\n\n# Hijack pdb.post_mortem to auto-run diagnostic commands\n_original_post_mortem = pdb.post_mortem\n\ndef scripted_post_mortem(tb=None):\n out = io.StringIO()\n p = pdb.Pdb(stdout=out, stdin=io.StringIO(''))\n p.use_rawinput = False\n p.reset()\n if tb is None:\n tb = sys.exc_info()[2]\n p.setup(None, tb)\n\n old_stderr = sys.stderr\n sys.stderr = out\n\n # List all inner exceptions\n p.onecmd('for i, exc in enumerate(exceptions): print(f\\\"[{i}] {type(exc).__name__}: {exc}\\\")')\n # Print full traceback of the first TypeError (the real root cause)\n p.onecmd('type_errs = [exc for exc in exceptions if type(exc).__name__ == \\\"TypeError\\\"]')\n p.onecmd('if type_errs: import traceback; traceback.print_exception(type(type_errs[0]), type_errs[0], type_errs[0].__traceback__)')\n p.do_quit('')\n\n sys.stderr = old_stderr\n print('=== PDB AUTOMATED OUTPUT ===')\n print(out.getvalue())\n print('=== END OUTPUT ===')\n\npdb.post_mortem = scripted_post_mortem\n\nfrom sphinx.cmd.build import build_main\nbuild_main(['-b', 'html', '-vv', '-T', '-P', '.', '_build/html'])\n\"\n```\n\nThis runs the real Sphinx build with `-P`, but when pdb drops in it\nautomatically executes your commands and dumps the results to stdout.\nYou read the output exactly as if you had typed the pdb commands yourself.\n\n## What to look for in the automated output\n\nSphinx autosummary raises `ImportExceptionGroup` when it cannot resolve a module.\nThe **top-level message is often misleading** (e.g. `no module named xtuner.v1.ray.dataflow`).\nThe real root cause is hidden inside the grouped exceptions.\n\n### Common root-cause patterns\n\n1. **TypeError about `__version__`**\n ```\n TypeError: expected string or bytes-like object, got '__version__'\n ```\n This happens when a mocked module (e.g. `torch`) is imported for real by an\n intermediate library (e.g. `fla`) that calls `packaging.version.parse()` on\n the mocked `__version__` object.\n\n **Fix**: Add the intermediate library to `autodoc_mock_imports` in `conf.py`.\n To identify the exact library, read the traceback: the frame just above\n `packaging/version.py` is the culprit (e.g. `fla/utils.py`).\n\n ```python\n autodoc_mock_imports = [\n ...\n \"fla\", # or whatever library triggers the real import\n ]\n ```\n\n2. **Missing-comma string-concatenation bug**\n\n If `autodoc_mock_imports` contains an entry like `\"scipytorchvision\"`,\n a comma is missing between two string literals in `conf.py`:\n\n ```python\n # WRONG\n autodoc_mock_imports = [\n ...\n \"scipy\"\n \"torchvision\",\n ...\n ]\n\n # CORRECT\n autodoc_mock_imports = [\n ...\n \"scipy\",\n \"torchvision\",\n ...\n ]\n ```\n\n3. **AttributeError after the TypeErrors**\n\n ```\n AttributeError: module 'xtuner.v1.ray' has no attribute 'dataflow'\n ```\n This is usually a *secondary* failure caused by the earlier `TypeError`s.\n Fix the `TypeError` first, then rebuild.\n\n## Step-by-step workflow\n\n1. Run the automated pdb script (see \"For agents\" section above).\n2. Read the `=== PDB AUTOMATED OUTPUT ===` block.\n3. If you see `TypeError: expected string or bytes-like object, got '__version__'`,\n read the traceback to find the intermediate library name.\n4. Add the identified library to `autodoc_mock_imports` in **both**\n `docs/en/conf.py` and `docs/zh_cn/conf.py`.\n5. Also eyeball `autodoc_mock_imports` for missing commas while you are there.\n6. Re-run `make html` (without `-P`) to verify the fix.\n","frontmatter":{"name":"sphinx-debug","description":"Use when `make html` or `sphinx-build` fails with `Extension error (sphinx.ext.autosummary)`, `ImportExceptionGroup`, or other Sphinx autosummary / autodoc import failures. Provides automated pdb diagnosis for `autodoc_mock_imports` issues.\n"},"isInternal":false,"tokens":1186,"sizeBytes":4499},{"name":"SKILL.md","path":".claude/skills/xtuner-custom-judger/SKILL.md","rawUrl":"https://raw.githubusercontent.com/InternLM/xtuner/HEAD/.claude/skills/xtuner-custom-judger/SKILL.md","title":"xtuner-custom-judger","category":"anthropic-skill","format":"markdown","content":"---\nname: xtuner-custom-judger\ndescription: Use when creating, modifying, or reviewing XTuner RL judgers under xtuner/v1/rl/judger or configs that build custom judgers. Enforces the preferred Judger payload contract and the BaseJudger/ComposedJudger constraints.\n---\n\n# XTuner Custom Judger\n\nUse this skill whenever you implement or review an XTuner RL judger.\n\n## Preferred Pattern\n\nPrefer subclassing `Judger`, not `BaseJudger`.\n\nFor custom scoring logic, override these methods as needed:\n\n```python\nclass MyJudger(Judger):\n def preprocess(self, rollout_state: RolloutState) -> JudgerPayload:\n ...\n\n async def judge_payload(self, payload: JudgerPayloadBatch) -> JudgerOutputBatch:\n ...\n\n def postprocess(self, rollout_state: RolloutState, output: JudgerOutput) -> RolloutState:\n ...\n```\n\nDo not override `Judger.judge()` just to read extra fields from `RolloutState`.\nPut the field extraction in `preprocess()` instead.\n\n## Why This Contract Exists\n\n`preprocess -> judge_payload -> postprocess` keeps the scoring payload small and\ncomposable:\n\n- `preprocess()` extracts only the fields required for scoring.\n- `judge_payload()` can run locally, in a Ray actor, or behind a judger pool.\n- `postprocess()` writes the output back to the original `RolloutState`.\n\nThis avoids `deepcopy(RolloutState)` in composed and remote judging. Deep-copying\nrollout states can add serialization overhead and can be risky for large fields,\nRay object refs, tensors, numpy arrays, or externally owned objects.\n\n## BaseJudger Rule\n\nSubclass `BaseJudger` only when the implementation must own the full\n`judge()` / `batch_judge()` flow and cannot reasonably be expressed as a\npayload contract.\n\nBaseJudger-only judgers cannot be used as `ComposedJudger` branches.\n`ComposedJudger` intentionally calls branch `preprocess()` and\n`judge_payload()` rather than arbitrary `BaseJudger.judge()` implementations.\nBecause it does not deep-copy `RolloutState`, supporting arbitrary\nBaseJudger-only branches would allow multiple branches to mutate the same\nrollout state concurrently and overwrite `reward` or other fields.\n\n## ComposedJudger Checklist\n\nWhen adding a branch intended for `ComposedJudger`:\n\n1. Ensure the class inherits `Judger`.\n2. Ensure branch-specific fields are copied into the payload in `preprocess()`.\n3. Ensure `judge_payload()` returns a `JudgerOutput` for one payload, or a\n `list[JudgerOutput]` for a batch payload.\n4. Ensure `postprocess()` writes `rollout_state.reward`.\n5. For multi-branch composed judging, ensure `merge_fn` creates the final\n reward shape, including `reward[\"score\"]` when training expects it.\n\n## Tests\n\nFor custom judger changes, add focused tests for:\n\n- Single-sample `judge()`.\n- `batch_judge()` if batch support is implemented.\n- `ComposedJudger` routing if the judger is intended as a composed branch.\n- Rejection or explicit non-support if the implementation directly subclasses\n `BaseJudger`.\n","frontmatter":{"name":"xtuner-custom-judger","description":"Use when creating, modifying, or reviewing XTuner RL judgers under xtuner/v1/rl/judger or configs that build custom judgers. Enforces the preferred Judger payload contract and the BaseJudger/ComposedJudger constraints."},"isInternal":false,"tokens":686,"sizeBytes":2958},{"name":"SKILL.md","path":".claude/skills/xtuner-sync-supported-models/SKILL.md","rawUrl":"https://raw.githubusercontent.com/InternLM/xtuner/HEAD/.claude/skills/xtuner-sync-supported-models/SKILL.md","title":"xtuner-sync-supported-models","category":"anthropic-skill","format":"markdown","content":"---\nname: xtuner-sync-supported-models\ndescription: Synchronize xtuner's supported model documentation (docs/en/pretrain_sft/advanced_tutorial/model.md and docs/zh_cn/pretrain_sft/advanced_tutorial/model.md) with the actual Config classes defined under xtuner/v1/model/. Use when (1) new TransformerConfig, MoEConfig, or BaseComposeConfig subclasses are added, removed, or renamed in xtuner/v1/model/, (2) existing model configs change their inheritance hierarchy, scale, or HuggingFace counterpart, or (3) a code review or user request points out that model.md is out of sync with the codebase.\n---\n\n# Update XTuner Supported Model Docs\n\nKeep the English and Chinese `model.md` files synchronized with the actual Config classes in `xtuner/v1/model/`.\n\n## Scan the Codebase\n\nRun the bundled scan script from the xtuner project root to discover all Config classes and their inheritance:\n\n```bash\npython3 .agents/skills/xtuner-sync-supported-models/scripts/scan_model_configs.py\n```\n\nThe script outputs JSON with two keys:\n- `configs`: list of every `*Config` class under `xtuner/v1/model/` with its parent classes and file path\n- `children`: parent-to-children mapping for the hierarchy tree\n\n## What to Update\n\nCompare the script output against the two files:\n- `docs/en/pretrain_sft/advanced_tutorial/model.md`\n- `docs/zh_cn/pretrain_sft/advanced_tutorial/model.md`\n\nBoth files share the same structure and must stay in sync:\n\n1. **Base Config Classes** — configs that directly inherit from `TransformerConfig` (or `MoEConfig`) and provide a `from_hf` classmethod for loading HuggingFace weights\n2. **Concrete Model Configs** — fixed-scale subclasses of the base configs above\n3. **Compose Models** — multimodal configs that inherit from `BaseComposeConfig`\n4. **Inheritance Hierarchy** — a text tree showing the full `XTunerBaseModelConfig` hierarchy\n\n### Rules for the Base Config table\n\nInclude these direct descendants of `TransformerConfig`/`MoEConfig`:\n- `Qwen2DenseConfig`\n- `Qwen3DenseConfig`\n- `DeepSeekV3Config`\n- `GptOssConfig`\n- `Qwen3MoEConfig`\n\nExclude from the base table:\n- `MoEConfig` — it is an intermediate base class, not a usable model family\n- `Qwen3_5_VLTextMoEConfig` — it is an intermediate base with only one concrete child; its child `Qwen3_5_VLTextMoE35BA3BConfig` belongs under the MoE concrete table\n\n### Rules for the Concrete Model table\n\nInclude every concrete subclass that has fixed parameter defaults. For each row note:\n- `Config Class`\n- `Base Class / Family`\n- `Architecture Type`: `Dense`, `MoE`, `Dense (VL backbone)`, `MoE (VL backbone)`\n- `Scale / Notes`: parameter count or total/activated size; for VL backbones note \"for multimodal\"\n\n`DeepSeekV3Config` appears here even though it has no separate base entry (it is both base and concrete).\n\n### Rules for the Compose Models section\n\nInclude three sub-tables:\n1. **Compose Base Config Classes** — `Qwen3VLBaseConfig`, `InternVLBaseConfig`, `InternS1BaseConfig`\n - `Qwen3VLBaseConfig`: VL model based on Qwen3 text backbone\n - `InternVLBaseConfig`: VL model based on InternViT + Qwen3\n - `InternS1BaseConfig`: Science multimodal model based on InternViT + Qwen3\n2. **Concrete Compose Model Configs** — every subclass of the above bases; for each row note the wrapped `Text Config` and scale\n\n### Rules for the Inheritance Hierarchy tree\n\nRebuild the tree from `XTunerBaseModelConfig` with two top-level branches:\n\n```text\nXTunerBaseModelConfig\n├── TransformerConfig\n│ ├── Dense Models\n│ │ ├── Qwen2DenseConfig\n│ │ │ └── Qwen2Dense7BConfig\n│ │ └── Qwen3DenseConfig\n│ │ ├── Qwen3Dense8BConfig\n│ │ │ └── Qwen3VLTextDense8BConfig\n│ │ ├── Qwen3Dense4BConfig\n│ │ │ └── Qwen3VLTextDense4BConfig\n│ │ └── Qwen3Dense0P6BConfig\n│ └── MoE Models (via MoEConfig)\n│ ├── DeepSeekV3Config\n│ ├── GptOssConfig\n│ │ ├── GptOss21BA3P6Config\n│ │ └── GptOss117BA5P8Config\n│ ├── Qwen3MoEConfig\n│ │ ├── Qwen3MoE30BA3Config\n│ │ │ └── Qwen3VLTextMoE30BA3Config\n│ │ ├── Qwen3MoE235BA22Config\n│ │ │ └── Qwen3VLTextMoE235BA22Config\n│ │ └── Qwen3MoEFoPEConfig\n│ └── Qwen3_5_VLTextMoEConfig\n│ └── Qwen3_5_VLTextMoE35BA3BConfig\n└── BaseComposeConfig\n ├── Qwen3VLBaseConfig\n │ ├── Qwen3VLMoE30BA3Config\n │ ├── Qwen3VLMoE235BA22Config\n │ ├── Qwen3VLDense4BConfig\n │ ├── Qwen3VLDense8BConfig\n │ └── Qwen3_5_BaseConfig\n │ └── Qwen3_5_VLMoE35BA3Config\n ├── InternVLBaseConfig\n │ ├── InternVL3P5Dense8BConfig\n │ ├── InternVL3P5MoE30BA3Config\n │ └── InternVL3P5Dense1BConfig\n └── InternS1BaseConfig\n ├── InternS1Config\n └── InternS1MiniConfig\n```\n\nWhen new configs are added, insert them into the appropriate branch following the same indentation style.\n\n## Translation Notes\n\nKeep the Chinese `model.md` (`docs/zh_cn/...`) structurally identical to the English one. Translate:\n- Section headings\n- Table header cells\n- Description cells (e.g., \"Image / Video + Text\" → \"图像/视频 + 文本\")\n- Scale descriptions (e.g., \"~7B parameters\" → \"约 7B 参数\", \"FoPE variant\" → \"FoPE 变体\")\n\nDo **not** translate Config class names, file paths, or code identifiers.\n","frontmatter":{"name":"xtuner-sync-supported-models","description":"Synchronize xtuner's supported model documentation (docs/en/pretrain_sft/advanced_tutorial/model.md and docs/zh_cn/pretrain_sft/advanced_tutorial/model.md) with the actual Config classes defined under xtuner/v1/model/. Use when (1) new TransformerConfig, MoEConfig, or BaseComposeConfig subclasses are added, removed, or renamed in xtuner/v1/model/, (2) existing model configs change their inheritance hierarchy, scale, or HuggingFace counterpart, or (3) a code review or user request points out that model.md is out of sync with the codebase."},"isInternal":false,"tokens":1533,"sizeBytes":5608},{"name":"blackbox-agent-trace.md","path":".claude/skills/xtuner-trace/references/blackbox-agent-trace.md","rawUrl":"https://raw.githubusercontent.com/InternLM/xtuner/HEAD/.claude/skills/xtuner-trace/references/blackbox-agent-trace.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Blackbox Agent Trace Reference\n\nThis reference captures the XTuner black-box agent trace reconstruction workflow. It is intended for implementation and review work, not as user-facing docs.\n\nRead this reference when `xtuner-trace/SKILL.md` routes an agent trace request here. The design background lives in `docs/design/blackbox_agent_trace_reconstruction.md`.\n\n## Contents\n\n- [Decision Flow](#decision-flow)\n- [Path A: Trajectory-Backed Reconstruction](#path-a-trajectory-backed-reconstruction)\n- [Path B: SessionServer-Only Fallback](#path-b-sessionserver-only-fallback)\n- [`record_synthetic_span(...)` Usage](#record_synthetic_span-usage)\n- [Reconstructed Span Shape](#reconstructed-span-shape)\n- [Viewer And Jaeger Requirements](#viewer-and-jaeger-requirements)\n- [Example: mini-SWE-agent Trajectory Conversion](#example-mini-swe-agent-trajectory-conversion)\n- [Verification](#verification)\n- [Common Failure Modes](#common-failure-modes)\n\n## Decision Flow\n\nWhen adding trace support for a black-box agent, classify the available evidence before deciding where to instrument.\n\n1. Confirm this is a black-box agent path: XTuner exposes an OpenAI-compatible inference endpoint to the agent through `SessionServer`; the agent runs its native loop and calls that endpoint for LLM generation. Tool execution, environment interaction, and observation handling happen inside the agent runtime and are not directly instrumented by XTuner.\n2. Confirm whether the agent can save a trajectory for each run.\n3. If a trajectory exists, check whether it contains tool-call metadata such as tool name, action/command, status, return code, error information, observation, or timestamp.\n4. Use Path A when the trajectory contains tool-call metadata. The trajectory supplies tool semantics, and `session_server.request` spans supply the LLM request timeline.\n5. Use Path B only when no useful trajectory exists. This can recover request gaps, but cannot recover reliable tool names, return codes, errors, or observations.\n\nDo not require the trajectory to contain complete LLM generate timing before using Path A. `SessionServer` remains the stable source for LLM request start/end times. If the trajectory also records LLM calls, use them as an optional consistency check.\n\n## Path A: Trajectory-Backed Reconstruction\n\nUse this path when the agent can save a trajectory with tool-call metadata.\n\n### Data Contract\n\nThe `session_server.request` spans must provide:\n\n- `xtuner.session_id` or `xtuner.rollout_id`\n- span start time\n- span end time\n\nThe trajectory should provide best-effort tool metadata:\n\n- tool event order, id, or enough context to infer order\n- tool name, action, or command\n- status, return code, or exception information\n- observation or output summary\n- tool start/end timestamps, or a single tool-related timestamp with documented semantics, if the agent records one\n\nDo not define a universal trajectory schema in trace core. Agent trajectory formats are agent-specific; keep parsing inside that agent's recipe or trajectory-to-trace adapter until multiple agents prove a shared model is worth extracting.\n\n### Where To Add Trace\n\nAdd trace at the three black-box observation boundaries. These instructions must be enough to re-create the instrumentation even when no example implementation exists in the checkout.\n\n#### 1. SessionServer Request Trace\n\nFirst check whether the `SessionServer` HTTP request handler already emits `session_server.request` spans. If not, add this instrumentation to the handler that receives the agent's OpenAI-compatible request.\n\nImplementation requirements:\n\n- Locate the outer request handler, such as an aiohttp/FastAPI handler named like `_handle_request(request)`.\n- Read enough request headers/body to determine:\n - `session_id`\n - optional rollout id\n - whether the request is streaming\n - optional trace carrier from standard headers or a request-body carrier field\n- Wrap the full inference request lifecycle in:\n\n```python\nwith trace_span(\n \"session_server.request\",\n attributes={\n \"xtuner.stage\": \"llm_generate\",\n \"xtuner.session_id\": session_id,\n \"xtuner.rollout_id\": rollout_id or session_id,\n \"session.stream\": is_streaming,\n },\n parent_carrier=parent_carrier,\n):\n ...\n```\n\n- Keep the outer `session_server.request` span around prepare, backend request, response read, response post-processing, and return-to-agent.\n- Use `set_trace_attributes(...)` before exiting the span to record request-level status or error metadata when available.\n- If adding child spans, use these names:\n - `session_server.prepare_request`\n - `session_server.send_request`\n - `session_server.read_response`\n - `session_server.record_response`\n- Put `xtuner.session_id` on every child span too. This makes later trajectory-to-trace adapters able to filter a single agent run reliably.\n\nDo not parse agent trajectory or infer tool calls inside `SessionServer`. Its black-box responsibility is to provide the LLM request timeline.\n\n#### 2. Agent Run Boundary Trace\n\nIn the black-box recipe or launcher, find the call site that invokes the agent's native run loop, such as `agent.run(...)`.\n\nWrap only the outer run boundary:\n\n```python\nwith trace_span(\n \".run\",\n attributes={\n \"xtuner.stage\": \"agent.run\",\n \"xtuner.session_id\": session_id,\n \"xtuner.rollout_id\": rollout_id or session_id,\n \"agent.name\": agent_name,\n },\n):\n agent.run(...)\n```\n\nThis span is a lifecycle boundary. Do not wrap the agent's internal model, environment, or tool methods as the default black-box trace path. LLM/tool detail should come from `SessionServer` spans plus trajectory reconstruction.\n\n#### 3. Trajectory-To-Trace Reconstruction\n\nAfter the native agent run finishes, call the trajectory-to-trace adapter before `close_trace()`. Prefer a `finally` block so partial trajectories can still be converted after failures:\n\n```python\nreturn_code = 1\ntry:\n with trace_span(\".run\", attributes=agent_run_attributes):\n return_code = agent.run(...)\nfinally:\n emit_reconstructed_trace(\n trace_jsonl=trace_jsonl_path,\n trajectory=trajectory_path,\n session_id=session_id,\n )\n close_trace()\nreturn return_code\n```\n\nThe adapter must read the trajectory, read same-session `session_server.request` spans, derive tool timing from trajectory timestamps when possible, use request gaps only for correlation or inferred fallback, and emit reconstructed synthetic spans with `record_synthetic_span(...)`.\n\n### Adapter Responsibilities\n\nWrite a trajectory-to-trace adapter for the specific agent. The adapter should:\n\n1. Load the agent trajectory.\n2. Extract ordered tool events and their metadata.\n3. Load `session_server.request` spans from the same `traces.jsonl`.\n4. Filter request spans by `session_id`.\n5. Sort request spans by start time and assign local indexes.\n6. Extract tool timestamps from the trajectory and interpret their semantics:\n - If the trajectory provides reliable tool start/end times, use them as the `reconstructed_tool_call.N` span boundaries and set `xtuner.inferred=false`.\n - If the trajectory provides only one timestamp, record what it means if known, such as tool start, tool end, observation write, or trajectory write. Use it to correlate the tool event with the surrounding `session_server.request` window.\n - If the trajectory timestamp semantics are unknown or incomplete, use request-gap boundaries only as an inferred fallback and set `xtuner.inferred=true`.\n7. Use `session_server.request` spans as the LLM timeline and correlation windows, not as the default source of trajectory-backed tool timing.\n8. When using request-gap fallback for a bounded tool event, use:\n\n```text\ntool_i.start = request_i.end\ntool_i.end = request_{i+1}.start\n```\n\n9. Emit `reconstructed_agent_run`, `reconstructed_llm_call.N`, and bounded `reconstructed_tool_call.N` spans with `record_synthetic_span(...)`.\n10. Treat final tool events without a next LLM request as unbounded unless the trajectory has trustworthy start/end times.\n\nWhen request-gap fallback is used, the interval is not pure tool runtime. It can include LLM-response parsing, user confirmation time, tool execution, observation formatting, trajectory write time, and loop overhead before the next LLM request.\n\n## Path B: SessionServer-Only Fallback\n\nUse this path only when the agent does not save a useful trajectory.\n\n### Where To Add Trace\n\nAdd trace to the `SessionServer` request path so every LLM request produces a reliable timeline. At minimum, `session_server.request` spans must include:\n\n- `xtuner.session_id`\n- `xtuner.rollout_id`, falling back to the session id when there is no rollout id\n- request start time\n- request end time\n\nSupporting spans such as `session_server.prepare_request`, `session_server.send_request`, `session_server.read_response`, and `session_server.record_response` may remain useful for debugging latency, but the fallback reconstruction only requires the outer `session_server.request` start/end times.\n\nDo not put tool inference inside `SessionServer`. `SessionServer` should not parse trajectories, infer tool names, track return codes, or maintain complex per-session reconstruction state.\n\n### Fallback Reconstruction\n\nAfter the agent run, a fallback adapter may infer intervals from adjacent request spans:\n\n```text\ninferred_tool_call_i.start = request_i.end\ninferred_tool_call_i.end = request_{i+1}.start\n```\n\nThe fallback can emit synthetic spans such as `inferred_tool_call.N` with:\n\n```text\nxtuner.stage=tool_call\nxtuner.synthetic=true\nxtuner.inferred=true\nxtuner.session_id=\nreconstruction.source=session_server_only\nreconstruction.method=session_server_gap\n```\n\nDo not claim unavailable metadata in this mode. SessionServer-only fallback cannot reliably recover tool name, command, return code, stderr/stdout, agent exception, skipped action, or observation content.\n\nDo not infer the final tool event without a next LLM request. The interval has no reliable end time.\n\n## `record_synthetic_span(...)` Usage\n\nUse `record_synthetic_span(...)` to write reconstructed historical intervals back into the active XTuner trace runtime.\n\n```python\nrecord_synthetic_span(\n name: str,\n *,\n start_time_unix_ns: int,\n end_time_unix_ns: int,\n attributes: Mapping[str, Any] | None = None,\n parent_carrier: Mapping[str, str] | None = None,\n status: str = \"completed\",\n error_message: str | None = None,\n) -> dict[str, str] | None\n```\n\nRules:\n\n- Call it before `close_trace()`.\n- Use absolute Unix nanoseconds, not monotonic timestamps.\n- Ensure `end_time_unix_ns > start_time_unix_ns`.\n- Let the API default `xtuner.synthetic=true`.\n- Set `xtuner.inferred=true` only when the time boundary is inferred, such as from `session_server.request` gaps.\n- Use `status=\"error\"` and `error_message=...` for real errors.\n- Use `parent_carrier` to attach reconstructed children under one synthetic root trace.\n- Treat `None` return as trace-disabled no-op.\n\nTypical root and child pattern:\n\n```python\nroot_ids = record_synthetic_span(\n \"reconstructed_agent_run\",\n start_time_unix_ns=first_request.start_time_unix_ns,\n end_time_unix_ns=last_request.end_time_unix_ns,\n attributes={\n \"xtuner.stage\": \"agent.run\",\n \"xtuner.session_id\": session_id,\n \"xtuner.inferred\": False,\n \"reconstruction.method\": \"trajectory_session_timeline\",\n },\n)\nroot_carrier = None\nif root_ids:\n root_carrier = {\"traceparent\": f\"00-{root_ids['trace_id']}-{root_ids['span_id']}-01\"}\n\nrecord_synthetic_span(\n \"reconstructed_tool_call.1\",\n start_time_unix_ns=request_1.end_time_unix_ns,\n end_time_unix_ns=request_2.start_time_unix_ns,\n attributes={\n \"xtuner.stage\": \"tool_call\",\n \"xtuner.session_id\": session_id,\n \"xtuner.inferred\": True,\n \"agent.tool_call_index\": 1,\n \"agent.tool.name\": \"bash\",\n \"reconstruction.source\": \"agent_trajectory\",\n \"reconstruction.method\": \"session_server_gap\",\n },\n parent_carrier=root_carrier,\n)\n```\n\nDo not call OpenTelemetry SDK directly from the adapter. Use XTuner trace APIs.\n\n## Reconstructed Span Shape\n\nTrajectory-backed reconstruction should generate one synthetic root trace:\n\n```text\nreconstructed_agent_run\n|-- reconstructed_llm_call.1\n|-- reconstructed_tool_call.1\n|-- reconstructed_llm_call.2\n|-- reconstructed_tool_call.2\n...\n```\n\nUse these naming rules:\n\n- `reconstructed_agent_run` for the synthetic root.\n- `reconstructed_llm_call.N` for LLM request intervals copied from `session_server.request`.\n- `reconstructed_tool_call.N` for trajectory-backed tool metadata.\n- `inferred_tool_call.N` only for SessionServer-only fallback without trajectory metadata.\n\nRecommended root attributes:\n\n```text\nxtuner.stage=agent.run\nxtuner.synthetic=true\nxtuner.inferred=false\nxtuner.session_id=\nxtuner.rollout_id=\nagent.name=\nagent.llm_call_count=\nagent.tool_call_count=\nreconstruction.source=\nreconstruction.method=trajectory_session_timeline\n```\n\nRecommended LLM attributes:\n\n```text\nxtuner.stage=llm_generate\nxtuner.synthetic=true\nxtuner.inferred=false\nagent.llm_call_index=N\nreconstruction.source=session_server_request\nreconstruction.method=session_server_request_copy\nreconstruction.original_trace_id=\nreconstruction.original_span_id=\nreconstruction.original_operation=session_server.request\n```\n\nRecommended tool attributes:\n\n```text\nxtuner.stage=tool_call\nxtuner.synthetic=true\nxtuner.inferred=true\nagent.tool_call_index=N\nagent.tool.name=\nagent.tool.status=\nagent.tool.returncode=\nagent.tool.command_preview=\nreconstruction.source=\nreconstruction.method=session_server_gap\nreconstruction.timestamp_check=\n```\n\nDo not call trajectory-backed spans `inferred_tool_call.N`. Use `reconstructed_tool_call.N` and put uncertainty in attributes.\n\n## Viewer And Jaeger Requirements\n\nXTuner viewer and Jaeger must consume the same reconstructed spans. Do not implement one inference path in the viewer and another inference path for Jaeger.\n\nViewer expectations:\n\n- Show one sample per agent run/session after live and completed records are merged.\n- Prefer the trace containing `reconstructed_agent_run` as the primary sample trace.\n- Keep raw `session_server.request` trace ids available for audit.\n- Display the reconstructed path:\n\n```text\nreconstructed_agent_run -> reconstructed_llm_call.1 -> reconstructed_tool_call.1 -> ...\n```\n\nJaeger expectations:\n\n- The viewer's Jaeger link should open the trace containing `reconstructed_agent_run`.\n- Opening the first raw `session_server.request` trace is a bug for reconstructed black-box samples.\n- A healthy reconstructed trace contains `1 + llm_count + bounded_tool_count` reconstructed spans.","isInternal":false,"tokens":3238,"sizeBytes":15044},{"name":"trace-patterns.md","path":".claude/skills/xtuner-trace/references/trace-patterns.md","rawUrl":"https://raw.githubusercontent.com/InternLM/xtuner/HEAD/.claude/skills/xtuner-trace/references/trace-patterns.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# XTuner Trace Patterns\n\nThese patterns describe the retained trace surface in this branch: runtime, basic API, viewer, and local setup tooling.\n\n## Basic API\n\nUse the public facade:\n\n```python\nfrom xtuner.v1.rl.trace import (\n inject_trace_context,\n set_trace_attributes,\n trace_event,\n trace_function,\n trace_span,\n)\n```\n\n### Local Span\n\n```python\nwith trace_span(\"phase.name\", attributes={\"xtuner.stage\": \"phase\"}):\n ...\n set_trace_attributes({\"phase.count\": 3})\n```\n\n### Decorated Function\n\n```python\n@trace_function(\"phase.load\")\ndef load_item(path: str) -> object:\n ...\n```\n\n### Parent Carrier\n\n```python\ncarrier = inject_trace_context()\n\nwith trace_span(\"child.phase\", parent_carrier=carrier):\n ...\n```\n\n`parent_carrier` is a basic W3C context carrier. It is not tied to `RolloutState`, Ray, aiohttp, or any XTuner business object.\n\n## End-to-End Trace Wiring\n\nUse this pattern when adding trace support to an XTuner training run.\n\n### Launch Script\n\nEnable trace bootstrap from the launcher, guarded by `XTUNER_TRACE_ENABLED`:\n\n```bash\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\nREPO_ROOT=\"$(cd \"${SCRIPT_DIR}/../../..\" && pwd)\"\nif [ \"${XTUNER_TRACE_ENABLED:-0}\" = \"1\" ]; then\n source \"${REPO_ROOT}/recipe/trace/setup_trace.sh\"\nfi\n```\n\nKeep training stdout/stderr in the main training log so runtime trace messages\nare visible:\n\n```bash\npython xtuner/v1/train/cli/rl.py \\\n --config \"$CONFIG_PATH\" \\\n --num-workers \"$XTUNER_RL_NUM_WORKERS\" \\\n 2>&1 | tee -a \"${WORK_DIR}/training_log_${current_time}.txt\"\n```\n\n### Training Config\n\nAdd `TraceConfig` to the training config and pass it to the trainer config:\n\n```python\nimport os\n\nfrom xtuner.v1.rl.trace import TraceConfig\n\n\ntrace_config = TraceConfig(\n enabled=os.environ.get(\"XTUNER_TRACE_ENABLED\") == \"1\",\n service_name=\"xtuner-agent-rollout\",\n xtuner_viewer_enabled=True,\n xtuner_viewer_host=\"0.0.0.0\",\n xtuner_viewer_port=18080,\n xtuner_viewer_jaeger_query_url=\"http://127.0.0.1:16686\",\n enable_rollout_trace=True,\n)\n\ntrainer = RLColocateTrainerConfig(\n ...,\n trace_config=trace_config,\n)\n```\n\nWhen `output_dir` is omitted in an RL trainer config, the trainer places trace\nartifacts under the current experiment directory, for example\n`work_dirs/name/20260713071356/otel/`.\n\n### Stage Spans\n\nBefore adding spans, ask the user which stages they want to observe and which\nmetrics each stage should expose.\n\nPut fields known when entering the stage, especially fields used for grouping\nor filtering, in `trace_span(..., attributes=...)`. Put results known only\nafter or during execution, such as status, duration, token counts, errors, and\nreward values, in `set_trace_attributes(...)`. This lets the viewer reconstruct\nthe call chain and identify latency hotspots or abnormal stages.\n\nInfer stage:\n\n```python\nimport time\n\n\nstarted_at = time.monotonic()\nattributes = {\n \"xtuner.stage\": \"infer\",\n \"xtuner.task_name\": task_name,\n \"xtuner.sample_id\": sample_id,\n \"rollout.backend\": backend,\n}\n\nwith trace_span(\"agent.infer\", attributes=attributes):\n result = await run_infer(...)\n set_trace_attributes(\n {\n \"xtuner.status\": \"ok\" if result.ok else \"error\",\n \"prompt.tokens\": result.prompt_tokens,\n \"completion.tokens\": result.completion_tokens,\n \"stage.duration_ms\": int((time.monotonic() - started_at) * 1000),\n }\n )\n```\n\nResource acquire stage:\n\n```python\nwith trace_span(\n \"sandbox.acquire\",\n attributes={\n \"xtuner.stage\": \"acquire\",\n \"xtuner.sample_id\": sample_id,\n \"sandbox.name\": sandbox_name,\n },\n):\n client = await pool.get(sandbox_name)\n set_trace_attributes(\n {\n \"sandbox.env_id\": env_id,\n \"sandbox.image\": image,\n \"sandbox.reused\": reused,\n }\n )\n```\n\nValidation or reward stage:\n\n```python\nwith trace_span(\n \"sample.validate\",\n attributes={\n \"xtuner.stage\": \"validate\",\n \"xtuner.sample_id\": sample_id,\n \"validator.name\": validator_name,\n },\n):\n record = await validate(...)\n set_trace_attributes(\n {\n \"xtuner.status\": \"ok\" if record.ok else \"error\",\n \"reward.score\": record.score,\n \"reward.passed\": record.passed,\n }\n )\n\n if record.error is not None:\n set_trace_attributes(\n {\n \"error\": True,\n \"error.type\": record.error.type,\n \"error.message\": record.error.message,\n }\n )\n```\n\nDo not record prompts, responses, full configs, secrets, raw headers, stack\ntraces, or large payloads.\n\n### Cross-Boundary Propagation\n\nWhen a traced operation crosses a process, actor, or HTTP boundary, keep the\npropagation helper in the business or transport module that owns that boundary.\nDo not put Ray, HTTP, rollout, agent, or judger helpers back into\n`xtuner/v1/rl/trace`.\n\n#### Ray Remote Calls\n\nFor Ray calls, inject the current context into a plain carrier and temporarily\nattach that carrier to a serializable domain object or explicit call argument.\n\n```python\nfrom collections.abc import Mapping\nfrom contextlib import contextmanager\nfrom typing import Any\n\nfrom xtuner.v1.rl.trace import inject_trace_context, trace_span\n\n\n_TRACE_CARRIER_FIELD = \"_xtuner_trace_carrier\"\n\n\n@contextmanager\ndef attach_trace_carrier_temporarily(target: Any, carrier: Mapping[str, str]):\n if not carrier:\n yield\n return\n\n extra_fields = getattr(target, \"extra_fields\", None)\n if extra_fields is None:\n extra_fields = {}\n target.extra_fields = extra_fields\n had_previous = _TRACE_CARRIER_FIELD in extra_fields\n previous = extra_fields.get(_TRACE_CARRIER_FIELD)\n extra_fields[_TRACE_CARRIER_FIELD] = dict(carrier)\n try:\n yield\n finally:\n if had_previous:\n extra_fields[_TRACE_CARRIER_FIELD] = previous\n else:\n extra_fields.pop(_TRACE_CARRIER_FIELD, None)\n\n\ndef call_ray_remote_with_trace(remote_method, *args, trace_target: Any, **kwargs):\n carrier: dict[str, str] = {}\n inject_trace_context(carrier)\n with attach_trace_carrier_temporarily(trace_target, carrier):\n return remote_method.remote(*args, **kwargs)\n\n\ndef pop_trace_parent_carrier(target: Any) -> dict[str, str] | None:\n extra_fields = getattr(target, \"extra_fields\", None)\n if not isinstance(extra_fields, dict):\n return None\n carrier = extra_fields.pop(_TRACE_CARRIER_FIELD, None)\n if not isinstance(carrier, Mapping):\n return None\n return {str(key): str(value) for key, value in carrier.items()}\n```\n\nOn the receiver side, attach the carrier to the new span:\n\n```python\nparent_carrier = pop_trace_parent_carrier(trace_target)\n\nwith trace_span(\"worker.stage\", attributes=attributes, parent_carrier=parent_carrier):\n ...\n```\n\nKeep the helper domain-specific. Validate that exactly one trace target is used\nwhen the transport needs one target object; reject collections if restoring the\ncarrier to the right child span would be ambiguous.\n\n#### HTTP Calls\n\nFor HTTP calls, prefer W3C headers. If a third-party client or proxy may drop\ncustom headers, also put a copied carrier into the JSON body under an internal\nfield and remove it before forwarding the payload downstream.\n\n```python\nfrom collections.abc import Mapping\nfrom typing import Any\n\nfrom xtuner.v1.rl.trace import inject_trace_context, trace_span\n\n\n_TRACE_CARRIER_FIELD = \"_xtuner_trace_carrier\"\n\n\ndef inject_trace_carrier_into_json_body(kwargs: dict[str, Any], carrier: Mapping[str, str]) -> None:\n if not carrier:\n return\n payload = kwargs.get(\"json\")\n if not isinstance(payload, dict):\n return\n payload = dict(payload)\n payload.setdefault(_TRACE_CARRIER_FIELD, dict(carrier))\n kwargs[\"json\"] = payload\n\n\ndef extract_trace_carrier_from_mapping(payload: Mapping[str, Any] | None) -> dict[str, str] | None:\n if not isinstance(payload, Mapping):\n return None\n carrier = payload.get(_TRACE_CARRIER_FIELD)\n if not isinstance(carrier, Mapping):\n return None\n return {str(key): str(value) for key, value in carrier.items()}\n\n\ndef remove_trace_carrier_from_mapping(payload: dict[str, Any]) -> None:\n payload.pop(_TRACE_CARRIER_FIELD, None)\n\n\ndef extract_http_parent_carrier(headers: Mapping[str, Any], payload: Mapping[str, Any] | None) -> dict[str, str] | None:\n header_carrier = {\n str(key): str(value)\n for key, value in headers.items()\n if str(key).lower() in {\"traceparent\", \"tracestate\", \"baggage\"}\n }\n return header_carrier or extract_trace_carrier_from_mapping(payload)\n```\n\nCaller:\n\n```python\nheaders = dict(headers or {})\nrequest_kwargs = {\"json\": payload}\ncarrier: dict[str, str] = {}\ninject_trace_context(carrier)\nheaders.update(carrier)\ninject_trace_carrier_into_json_body(request_kwargs, carrier)\n\nwith trace_span(\"http.client.request\", attributes={\"http.method\": \"POST\", \"http.url\": url}):\n response = await client.post(url, headers=headers, **request_kwargs)\n```\n\nReceiver:\n\n```python\nheaders = dict(request.headers)\npayload = await request.json()\nparent_carrier = extract_http_parent_carrier(headers, payload)\n\nwith trace_span(\"http.server.request\", attributes=attributes, parent_carrier=parent_carrier):\n remove_trace_carrier_from_mapping(payload)\n ...\n```\n\nDo not record raw headers or full payloads as span attributes. Only record\nderived fields such as method, route, status code, stage, IDs, and timing.\n\n### Viewer Output\n\nWhen `xtuner_viewer_enabled=True`, the trace runtime logs the trace JSONL path,\nthe XTuner viewer URL, and the equivalent manual viewer command. The viewer\ndefaults to port `18080`; `xtuner_viewer_port=0` is still available when a run\nshould pick a free port automatically. The viewer process inherits the training\nstdout/stderr; keep those runtime log lines visible in the main training log and\nreport them to the user after the run.\n\nUseful reference points from the full trace implementation:\n\n- `examples/v1/scripts/run_rl_run.sh`\n- `recipe/trace/setup_trace.sh`\n- `examples/v1/config/agentic_rl_qwen3p5vl_mtp_ep_code.py`\n\n## Runtime\n\nConfigure tracing explicitly:\n\n```python\nfrom xtuner.v1.rl.trace import TraceConfig, close_trace, configure_trace\n\nruntime = configure_trace(\n TraceConfig(\n enabled=True,\n output_dir=\"work_dirs/example/otel\",\n service_name=\"xtuner\",\n xtuner_viewer_enabled=True,\n )\n)\n\ntry:\n ...\nfinally:\n close_trace()\n```\n\nThe runtime owns OTel collector setup, trace JSONL output, and optional lightweight XTuner viewer startup.\n\n## Viewer\n\nThe viewer reads Jaeger-style traces or OTel JSONL converted into Jaeger-style payloads. It should not depend on hard-coded rollout/agent span registries.\n\nStage display rules:\n\n1. Use `xtuner.stage` when present.\n2. Else use `stage` when present.\n3. Else use `stage.name` when present.\n4. Else fall back to the raw span name.\n\nUseful stable attributes:\n\n- IDs: `xtuner.rollout_id`, `xtuner.group_id`, `xtuner.session_id`, `xtuner.task_name`\n- Status: `xtuner.status`\n- Stage: `xtuner.stage`, `stage`, `stage.name`\n- Counts/timing: `prompt.tokens`, `completion.tokens`, `http.status_code`\n- Errors: `error`, `error.message`, `exception.type`\n\nAvoid recording prompts, responses, full configs, secrets, raw headers, or large payloads.\n\n## Local Setup\n\n`recipe/trace/setup_trace.sh` and the rest of `recipe/trace/` are local helper tooling for installing OTel collector binaries, starting the local Jaeger dependency, and clearing stale `recipe.trace.viewer.server` processes on `XTUNER_TRACE_VIEWER_PORT` or the default viewer port `18080`. Keep these as setup assets; do not use them to add automatic trace behavior to training configs or launch scripts unless that integration is explicitly requested.\n\n## Rollout Starter Preset\n\nThe built-in rollout starter trace is opt-in:\n\n```python\ntrace_config = TraceConfig(\n enabled=True,\n xtuner_viewer_enabled=True,\n enable_rollout_trace=True,\n)\n```\n\nIts rollout-specific helpers live in `xtuner/v1/rl/trace/rollout_api.py`:\n\n- `trace_rollout_endpoint(...)`\n- `trace_rollout_remote(...)`\n\nUse this only for the starter rollout chain. For custom instrumentation, ask\nwhich stages and metrics the user wants before adding spans.\n\n## Removed Legacy Semantics\n\nDo not use or recreate these removed surfaces in this branch:\n\n- `trace_remote`\n- `traced_rollout_endpoint`\n- `traced_agent_item_endpoint`\n- `traced_judger_endpoint`\n- `xtuner/v1/rl/trace/trace_utils.py`\n- `xtuner/v1/rl/trace/context_propagation.py`\n- fixed `TRACE_SPAN_*` registries\n- rollout/agent/judger attribute builders\n","isInternal":false,"tokens":2965,"sizeBytes":12705},{"name":"SKILL.md","path":".claude/skills/xtuner-trace/SKILL.md","rawUrl":"https://raw.githubusercontent.com/InternLM/xtuner/HEAD/.claude/skills/xtuner-trace/SKILL.md","title":"xtuner-trace","category":"anthropic-skill","format":"markdown","content":"---\nname: xtuner-trace\ndescription: Use when the user wants to instrument, inspect, or debug XTuner traces to reconstruct a sample/request execution path, follow cross-process call chains, or identify latency hotspots, bottleneck stages, and abnormal paths from span data.\n---\n\n# XTuner Trace\n\nUse this skill for the current trace layer in this repository: OpenTelemetry runtime setup, the basic public API, local setup scripts, and the trace viewer.\n\n## Current Boundary\n\nKeep this package infrastructure-only:\n\n- Runtime/configuration: `xtuner/v1/rl/trace/runtime.py`\n- OTel SDK adapter: `xtuner/v1/rl/trace/otel_utils.py`\n- Public facade: `xtuner/v1/rl/trace/__init__.py`\n- Basic API: `xtuner/v1/rl/trace/api.py`\n- Rollout starter preset: `xtuner/v1/rl/trace/rollout_api.py`\n- Local trace tooling and viewer: `recipe/trace/`\n\nDo not add rollout, agent, judger, Ray remote, HTTP proxy, reward, status, or session-server business semantics to `api.py`, `runtime.py`, or `otel_utils.py`. Keep rollout-specific starter behavior inside `rollout_api.py` and gated by `TraceConfig.enable_rollout_trace`.\n\n## Viewer Choice\n\nChoose the viewer by the observation question:\n\n- Use `recipe/trace/viewer` when the user wants rollout-oriented inspection, such as comparing samples within a training step by stage duration, status, reward/filter metadata, or recorded sample call chain.\n- Use Jaeger when the question is not sample/step-centric, such as inspecting raw spans, service boundaries, cross-process causality, backend/client calls, collector/export behavior, or custom instrumentation that does not emit the rollout/sample attributes expected by the recipe viewer.\n\nThe recipe viewer reads `traces.jsonl` and adds XTuner rollout/sample aggregation. Jaeger remains the raw trace drill-down surface for the same traced run.\n\n## Black-Box Agent Trace Routing\n\nWhen the trace request involves an agent, first ask whether this is a black-box\nagent trace. In this mode, XTuner exposes an OpenAI-compatible inference endpoint\nto the agent through `SessionServer`; the agent runs its native loop and calls\nthat endpoint for LLM generation. Tool execution, environment interaction, and\nobservation handling happen inside the agent runtime and are not directly\ninstrumented by XTuner.\n\nIf the answer is yes, or the request mentions trajectory reconstruction,\n`SessionServer` request gaps, `record_synthetic_span(...)`,\n`reconstructed_agent_run`, `reconstructed_tool_call.N`, XTuner viewer plus\nJaeger display of recovered tool calls, read\n[references/blackbox-agent-trace.md](references/blackbox-agent-trace.md) before\nplanning or editing.\n\n## Basic API\n\nThese are the interfaces defined in `xtuner/v1/rl/trace/api.py` and re-exported\nfrom `xtuner.v1.rl.trace`:\n\n- `trace_span(name, attributes=None, parent_carrier=None)`\n- `trace_function(name=None, attributes=None)`\n- `trace_event(name, attributes=None)`\n- `set_trace_attributes(attributes)`\n- `inject_trace_context(carrier=None)`\n\n## Runtime API\n\nUse these `xtuner/v1/rl/trace/runtime.py` interfaces for explicit trace\nruntime setup:\n\n- `TraceConfig`\n- `configure_trace(...)`\n- `close_trace()`\n\n## Rollout Starter Preset\n\n`TraceConfig(enable_rollout_trace=True)` enables the built-in rollout starter\ntrace. Its helpers live in `xtuner/v1/rl/trace/rollout_api.py`:\n\n- `trace_rollout_endpoint(...)`\n- `trace_rollout_remote(...)`\n\nKeep this layer narrow: it may depend on `RolloutState` and Ray call boundaries,\nbut it must not leak those dependencies into the basic API.\n\nRollout endpoint `xtuner.span_name_path` is a sample call chain, not just the\ncurrent process context stack. `trace_rollout_endpoint(...)` should append the\nendpoint span name to an internal trace-only chain carried in\n`RolloutState.extra_fields`, pass that state through remote rollout boundaries,\nand clean the internal field at the outermost endpoint. For example, after a\nsample generates and then judges, `judger.run` should record a path like\n`single_turn_agent_loop.run -> rollout.controller.generate -> rollout.worker.generate -> judger.run`\neven though the local active context at judge time may only be\n`single_turn_agent_loop.run -> judger.run`. Keep this behavior in\n`rollout_api.py` and viewer tests; do not add rollout call-chain semantics to\nthe basic trace API.\n\n## Add Trace Workflow\n\nWhen adding trace instrumentation to an XTuner run:\n\n1. Ask for or locate the launch script and training config before editing.\n2. In the launch script, source `recipe/trace/setup_trace.sh` when\n `XTUNER_TRACE_ENABLED=1`.\n3. In the training config, add `TraceConfig` and set `enabled=True`; set\n `xtuner_viewer_enabled=True` when the user needs interactive sample/step inspection. Set\n `enable_rollout_trace=True` only when the user wants the built-in rollout\n starter trace.\n4. Before adding any `trace_span(...)` instrumentation, you must ask the user which\n stages they want to observe and which metrics each stage should expose.\n Do not infer default stages unless the user explicitly asks you to choose.\n5. Add `trace_span(...)` only around the user-confirmed observed stages. Put fields known at span\n start in initial attributes, and update runtime or final fields with\n `set_trace_attributes(...)`.\n6. For cross-process or request boundaries, inject a carrier with\n `inject_trace_context(...)` and pass it to downstream\n `trace_span(..., parent_carrier=carrier)`.\n7. Keep transport-specific propagation at the caller boundary; do not move\n rollout, agent, judger, Ray, or HTTP semantics into the basic trace package.\n8. Ensure the main training log includes the trace output path, XTuner viewer URL,\n and equivalent manual viewer command when the XTuner viewer is enabled.\n\n## Guardrails\n\n- Do not recreate `trace_utils.py`, `context_propagation.py`, span-name registries, or business attribute builders under `xtuner/v1/rl/trace`.\n- Do not import `RolloutState`, agent item classes, judgers, rollout workers, Ray actors, aiohttp clients, or trainer configs from the basic trace package.\n- Do not call OpenTelemetry SDK directly from business code; use the basic API only when trace instrumentation is explicitly requested.\n- Do not record prompts, responses, full configs, secrets, raw headers, stack traces, or large payloads as attributes.\n- Viewer stage grouping must come from span attributes such as `xtuner.stage`, `stage`, or `stage.name`; otherwise fall back to the raw span name.\n\nFor concrete patterns, read [references/trace-patterns.md](references/trace-patterns.md) before editing trace code.\n\n## Verification\n\nUse focused checks:\n\n- `PYTHONPATH=. python -m compileall -q xtuner/v1/rl/trace recipe/trace`\n- `PYTHONPATH=. python -m unittest discover -s tests/rl -p 'test_trace*.py' -v` when the trace tests exist in the worktree.\n- `git diff --check`\n","frontmatter":{"name":"xtuner-trace","description":"Use when the user wants to instrument, inspect, or debug XTuner traces to reconstruct a sample/request execution path, follow cross-process call chains, or identify latency hotspots, bottleneck stages, and abnormal paths from span data."},"isInternal":false,"tokens":1547,"sizeBytes":6811},{"name":"README.md","path":"xtuner/tools/llm_calc/README.md","rawUrl":"https://raw.githubusercontent.com/InternLM/xtuner/HEAD/xtuner/tools/llm_calc/README.md","title":"llm_calc Documentation","category":"plugin-manifest","format":"markdown","content":"### Summary\n\nThis PR introduces a comprehensive GPU memory calculator for LLM training, which helps users estimate per-GPU memory consumption before actually running training jobs. This tool is essential for capacity planning and optimizing parallel configurations.\n\n### Features\n\n- **Memory Breakdown Analysis**: Calculates detailed memory usage including:\n - Model parameters (embedding, attention, MLP, head)\n - Gradients\n - Optimizer states (Adam momentum & variance)\n - Master parameters and gradients (for mixed-precision training)\n - Intermediate activations\n\n- **Parallel Strategy Support**:\n - Data Parallelism with ZeRO Stage 1(for Megatron)/3 (FSDP for xtuenr)\n - Tensor Parallelism (TP)\n - Pipeline Parallelism (PP) and Virtual Pipeline Parallelism (VPP) for Megatron\n - Pipeline parallelism memory accounts for flying micro-batches during 1F1B schedule\n - Expert Parallelism (EP) for MoE models\n\n- **Model Architecture Support**:\n - Dense models (e.g., Qwen3-8B)\n - MoE models (e.g., Qwen3-235B-A22B, DeepSeek-V3-671B)\n - Multi-Head Attention (MHA)\n - Multi-Latent Attention (MLA) used in DeepSeek\n\n- **Training Optimizations**:\n - Activation checkpointing (recomputation)\n - Flash Attention\n - Chunked loss computation\n\n### Usage\n\n```bash\npython xtuner/tools/llm_calc/llm_calculator.py -c [-o ]\n```\n\n### Example Configurations\n\n| Config File | Model | Scale | TrainFramework |\n|------------|-------|-------|-------|\n| `xtuner_qwen_dense_8b_1node.yaml` | Qwen3-8B (Dense) | 1 node | xtuner |\n| `xtuner_qwen_moe_30b_1node.yaml` | Qwen3-30B-A3B (MoE) | 1 node | xtuner |\n| `xtuner_qwen_moe_235b_64node.yaml` | Qwen3-235B-A22B (MoE) | 64 nodes | xtuner |\n| `xtuner_deepseekv3_671b_128node.yaml` | DeepSeek-V3-671B (MoE) | 128 nodes | xtuner |\n| `megatron_qwen_moe_235b_64node.yaml` | Qwen3-235B-A22B (MoE) | 64 nodes | Megatron |\n\n### Output Example\n\nThe tool outputs:\n1. Total model parameters breakdown\n2. Per-GPU memory consumption for different training phases (forward/backward)\n3. Detailed memory breakdown by component (params, grads, optimizer states, activations)\n4. Optional Excel export for further analysis\n \n```\n# python llm_calculator.py -c xtuner_qwen_dense_8b_1node.yaml\n----------------------------------------------setting:----------------------------------------------\nmodel:Qwen3-8B zero_stage:3 tp:1 pp:1 dp:8 ep:1 vpp:1 nodes:1 mbs:1 gbs:8 L:32768 recompute:True flash_attn:True capacity:1.0 LN:36\n----------------------------------------------------------------------------------------------------\nTotal_params_num: 7.628208160400391 B, embed_params_num: 0.57958984375 B, attn_params_num: 1.4063873291015625 B, mlp_params_num: 5.0626373291015625 B, head_params_num: 0.5795936584472656 B\nMax_mem for microbatch=1: 26.58784294128418 GiB. micro1_forward_last: 26.58784294128418 GiB, micro1_backward_last: 21.386390686035156 GiB\nMax_mem for microbatch>1: 30.401947021484375 GiB. micro1_forward_last: 26.58784294128418 GiB, micro1_backward_last: 21.386390686035156 GiB, micro2_forward_last: 30.401947021484375 GiB\n---------------------------------------------------------------------memory parts---------------------------------------------------------------------\n params grads opt_states master_params master_grads acts\nname\nembed 1.159180 1.159180 0.579590 0.289795 0.289795 0.000061\nhead 1.159187 1.159187 0.579594 0.289797 0.289797 0.625000\nperlayer_attn 0.078133 0.078133 0.039066 0.019533 0.019533 1.751953\nperlayer_mlp 0.281258 0.281258 0.140629 0.070314 0.070314 2.500000\nall_attn 0.000000 0.000000 1.406387 0.703194 0.703194 0.000000\nall_mlp 0.000000 0.000000 5.062637 2.531319 2.531319 0.000000\nin_layernorm 0.000000 0.000000 0.000000 0.000000 0.000000 0.250000\n\n------------------------------------------------------micro1_forward_last: 26.58784294128418 GiB------------------------------------------------------\n [Static OS&Param] embed, all_attn, all_mlp, head: master_params, opt_states = 11.442312240600586 GiB\n [Allgathered Last Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Head] head: params, acts = 1.7841873168945312 GiB\n [Recompute Checkpoint] in_layernorm.acts * (LN - 1) = 8.75 GiB\n\n-----------------------------------------------------micro1_backward_last: 21.386390686035156 GiB-----------------------------------------------------\n [Static OS&Param] embed, all_attn, all_mlp, head: master_params, opt_states = 11.442312240600586 GiB\n [Static Grad] embed, all_attn, all_mlp, head: master_grads = 3.8141040802001953 GiB\n [Allgathered First Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Embed] embed: params, acts = 1.15924072265625 GiB\n [Previous Layer Grad ReduceScatering] perlayer_attn, perlayer_mlp: grads = 0.3593902587890625 GiB\n [Recompute Checkpoint] None = 0 GiB\n\n-----------------------------------------------------micro2_forward_last: 30.401947021484375 GiB------------------------------------------------------\n [Static OS&Param] embed, all_attn, all_mlp, head: master_params, opt_states = 11.442312240600586 GiB\n [Static Grad] embed, all_attn, all_mlp, head: master_grads = 3.8141040802001953 GiB\n [Allgathered Last Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Head] head: params, acts = 1.7841873168945312 GiB\n [Recompute Checkpoint] in_layernorm.acts * (LN - 1) = 8.75 GiB\n```\n\n### Technical Notes for Xtuner FSDP (with recompute) Memory Breakdown of the above example\n\n1. At initilization, each GPU holds sharded master params and optimizer states (usually both in fp32).\n```\n [Static OS&Param] embed, all_attn, all_mlp, head: master_params, opt_states = 11.442312240600586 GiB\n```\n\n2. At the end of the first micro batch forward: \n2.1 The whole last decoder layer and lm head are gathered in each GPU (in bf16).\n```\n [Allgathered Last Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Head] head: params, acts = 1.7841873168945312 GiB\n```\n2.2 The checkpoint of all decoder layers also exist in each GPU\n```\n [Recompute Checkpoint] in_layernorm.acts * (LN - 1) = 8.75 GiB\n```\n\n3. At the end of the first micro batch backward\n3.1 Each GPU holds master params' graident now\n```\n [Static Grad] embed, all_attn, all_mlp, head: master_grads = 3.8141040802001953 GiB\n```\n3.2 The whole first decoder layer and embedding layer are gathered in each GPU\n```\n [Allgathered First Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Embed] embed: params, acts = 1.15924072265625 GiB\n```\n3.3 FSDP does ReduceScatter of previous layer's gradients at the same time (to overlap computation and communication)\n```\n [Previous Layer Grad ReduceScatering] perlayer_attn, perlayer_mlp: grads = 0.3593902587890625 GiB\n```\n3.4 The checkpoints for recomputing have been released, along with the forward process\n```\n [Recompute Checkpoint] None = 0 GiB\n```\n\n4. At the end of the second micro batch forward, the only difference with the first micro batch is that each GPU also holds master gradients now.\n```\n [Static Grad] embed, all_attn, all_mlp, head: master_grads = 3.8141040802001953 GiB\n```\n\nThe maximum memory usually occurs in these cases, but maybe different when under different configs such as `chunk_loss_size`.\n","isInternal":false,"tokens":2272,"sizeBytes":7516}],"systemPromptSnippet":"\n\n../.claude/CLAUDE.md\n\n\n# XTuner Programming Standards\n\n## Python\n\n### Development Standards\n\n1. When adding methods or functions to a module or class, declare **public interfaces** as early as possible and place **private interfaces** toward the end.\n2. Python **>= 3.10** is required. Use modern syntax accordingly (e.g., `X | Y` union types).\n3. Use `{}` instead of `dict()` for creating dictionaries. Use `[]` instead of `list()` for creating lists.\n4. Use **double quotes** (`\"`) for strings instead of single quotes (`'`).\n5. Use `TypedDict` with `{}` syntax when a parameter requires a typed dictionary.\n6. Prefer `pathlib.Path` over `os.path` for filesystem operations.\n7. All new code must include **type hints** for function signatures (parameters and return types).\n8. Maximum line length is **119** characters (configured in `pyproject.toml`).\n\n### Docstring Standards\n\n1. Use **Google Style** docstrings.\n2. For classes, include `__init__` documentation directly under the **class docstring**; do not repeat it under the `__init__` method.\n3. Parameter types **must** be declared in docstrings — include the type hint in parentheses immediately following the parameter name.\n4. The **return type** must be explicitly declared in the docstring.\n5. Do **not** declare raised exceptions in the docstring.\n6. Do **not** declare Attributes in the class docstring.\n7. If the return value is `None`, omit the `Returns` section.\n8. Only provide docstrings for **public methods**. Private methods do not require docstrings.\n\n### Linting, Formatting & Type Checking\n\n- **Ruff** for linting and formatting, **mypy** for type checking — see `pyproject.toml` for configuration.\n- `mypy` is strict for `xtuner/v1/*` (`ignore_missing_imports = false`).\n- Pre-commit hooks apply **only** to `xtuner/v1/` and `autotest/` — run `pre-commit run --all-files` before submitting a PR.\n- Docstrings are wrapped at **119** characters by `docformatter`.\n\n### Error Handling\n\n1. Never use bare `except:` — always catch specific exception types.\n2. Avoid silencing exceptions with `pass` unless there is a documented reason.\n\n### Testing\n\n1. Every bug fix PR **must** include a regression test that reproduces the original bug.\n2. Test functions must be named `test_` and grouped in classes named `Test`.\n3. Use `pytest` as the test framework. Prefer `pytest.raises` for exception assertions.\n4. Tests should be self-contained — avoid reliance on external services or GPUs unless explicitly marked with `@pytest.mark.gpu`.\n\n---\n\n## Git\n\n### Branch Naming\n\n- Bug fixes: `fix/`\n- Features: `feat/`\n- Refactors: `refactor/`\n\n### Commit Message Standards\n\n1. Follow the style of previous commit messages to maintain a **consistent** project history.\n2. Use the format: `[Tag] Short description` (e.g., `[Fix] Fix OOM in MoE expert routing`).\n3. Common tags: `[Fix]`, `[Feature]`, `[Refactor]`, `[CI]`, `[Docs]`, `[Test]`.\n4. **Never** include metadata or watermarks unrelated to the code itself, such as \"generated by AI\" or \"Co-Authored-By: Claude.\"\n\n---\n\n## Pull Request Standards\n\n### Bug Fix PRs\n\n1. **Title**: `[Fix] `\n2. **Body** must include:\n - **Root Cause**: A clear explanation of why the bug occurred.\n - **Fix**: What was changed and why this is the correct fix.\n - **Test Plan**: How the fix was verified (unit test, manual reproduction, etc.).\n3. Keep the diff **minimal** — fix only the bug. Do not refactor surrounding code, add unrelated improvements, or change formatting of untouched lines.\n4. If the fix touches performance-critical paths (data loading, forward/backward pass, communication), include **before/after** benchmarks or justify why benchmarks are unnecessary.\n\n### General PR Rules\n\n1. One logical change per PR. Do not mix bug fixes with features or refactors.\n2. All CI checks must pass before requesting review.\n3. Resolve all review comments before merging — do not dismiss without explanation.\n\n---\n\n## Code Review Standards\n\n### What to Check\n\n1. **Correctness**: Does the code do what it claims? Are edge cases handled?\n2. **Concurrency & Distributed**: Check for race conditions, deadlocks, incorrect `dist.barrier()` placement, and mismatched collective operations across ranks.\n3. **Memory**: Watch for GPU memory leaks — tensors not freed, caches growing unbounded, missing `torch.no_grad()` in inference paths.\n4. **Numerical Stability**: Flag potential issues like `log(0)`, division by zero, `float32`/`float16` overflow in loss computation or gradient scaling.\n5. **Performance**: Unnecessary CPU-GPU synchronization (`item()`, `.cpu()` in hot paths), redundant data copies, inefficient collective communication patterns.\n6. **API Contracts**: Do public interface changes maintain backward compatibility? Are deprecation warnings added for breaking changes?\n7. **Resource Cleanup**: Are file handles, NCCL communicators, and CUDA streams properly cleaned up?\n\n### Area-Specific Review Rules\n\nIf a PR changes code under `xtuner/v1/rl`, read `.claude/rules/rl_review.md` before commenting or\nposting the final review summary.\n\n### Review Output Format\n\nWhen performing code reviews, use the following structure:\n\n```\n## Summary\n<1-2 sentence overview of the change>\n\n## Issues\n### Critical\n- [file:line] Description of the issue and suggested fix\n\n### Warning\n- [file:line] Description of the concern\n\n### Nit\n- [file:line] Minor style or readability suggestion\n\n## Verdict\nAPPROVE / REQUEST_CHANGES / COMMENT\n```\n\n### Review Principles\n\n1. **Be specific** — always reference the exact file and line number.\n2. **Explain why** — don't just say \"this is wrong\"; explain the failure mode.\n3. **Suggest a fix** — provide a concrete code suggestion when possible.\n4. **Distinguish severity** — separate critical bugs from style nits.\n5. Do not flag issues in code that is **outside the scope** of the PR diff.\n\n\n\n## Rules\n\nPlease refer to the docs in `.claude/rules` for the development guidelines.\n\n\n---\npaths:\n - \"xtuner/v1/datasets/**/*\"\n---\n\n# XTuner v1 Datasets Architecture Rules\n\n## Core Data Contract (DataItem Protocol)\n\nEvery dataset layer must produce/consume `DataItem`:\n```python\n{\n \"input_ids\": list[int], # tokenized token IDs\n \"labels\": list[int], # same length; first token masked to -100\n \"num_tokens\": int, # len(input_ids); used for sampler and packing\n}\n```\n\n- `labels[0] = -100` always (pretrain: first token not predicted)\n- `HardPackDataset.__getitem__` returns `list[DataItem]` (one per source sample in pack)\n- All other datasets return a single `DataItem`\n\n## Tokenization Functions\n\n**Standard pretrain**: `PretrainTokenizeFunction`\n- Accepts: `{\"content\": \"text\"}` or `{\"messages\": [{\"role\": \"pretrain\", \"content\": \"text\"}]}`\n- Hash: `\"{tokenizer_hash}_{source_hash}_{add_bos_token}_{add_eos_token}\"`\n\n**Long text pretrain**: `LongTextPretrainTokenizeFunction`\n- Two-phase: cache phase returns chunk metadata, runtime phase tokenizes specific char ranges\n- Parameters: `chunk_size`, `tokenizer_chunk_chars`, `overlap_chars`, `min_chunk_tokens`\n- BOS only at `char_start==0`, EOS only at document end\n- Hash includes all chunking parameters\n\n## JsonlDataset Architecture\n\n- **Shared memory**: entire file loaded once at init\n- **Offset table**: `np.ndarray` for O(1) line access\n- **Chunked mode**: activated when `chunks.npy` exists in cache\n - Expands `sampled` to per-chunk entries\n - `__getitem__` calls `tokenize_fn(data, char_start=cs, char_end=ce)`\n - `max_length` filter skipped (would incorrectly discard valid long-text samples)\n - `__len__` returns chunk count (not line count)\n\n## Cache Directory Layout\n\n```\ncache_dir/\n .xpuyu-cache-meta.json\n {file_xxhash}/\n offsets.npy\n {tokenize_fn_hash}/\n num_tokens.npy # always present\n chunks.npy # only for LongText mode: (total_chunks, 4)\n # [line_idx, char_start, char_end, num_tokens]\n```\n\nCache key: `(file xxhash, tokenizer hash, tokenize_fn source hash)`\n- Tag-based lookup via `cache_tag` on `DatasetConfig`\n- Source hash auto-invalidates on code change\n\n## Packing Strategies\n\n**HardPackDataset**:\n- Static pack_infos computed at init (deterministic from seed)\n- Returns `list[DataItem]` per pack\n- `global_pack=False`: per-dataset packing (default)\n- `global_pack=True`: concat all datasets then pack\n\n**PresetPackDataset**:\n- User-provided pack config (NPY directory + `paths.json`)\n- Validates structure and token ranges via vectorized numpy\n- Strategies for length mismatch: error/padding (short), error/truncate (long)\n- No state (deterministic from config file)\n\n## Samplers\n\n**LengthGroupedSampler**:\n- Megabatch logic: group similar lengths, sort DESC, shuffle chunks\n- State: `{epoch, step, world_size, num_samples, total_size}`\n- `step` = `consumed_samples % total_size`\n\n**PresetSampler**:\n- User-provided order via `sampler_config_path` (`.npy` path only, mmap read)\n- Round-down to `global_batch_size * world_size` multiple (truncate tail; error if length too short)\n- Per-rank slice: `effective[local_rank::world_size]` on the truncated mmap-backed view (`self.global_order`)\n- `pack_level=\"preset\"` in `DataloaderConfig`; requires `pack_config_path` and `sampler_config_path`\n- State dict compatible with existing resume logic\n\n## Resume Contract\n\n```python\n# Save\nstate = dataloader.get_state_dict(consumed_samples)\n# → {\"sampler\": {...}, \"dataset\": {}}\n\n# Restore\ndataloader.load_state_dict(state)\n```\n\n- `consumed_samples` = total globally-seen samples (across all ranks)\n- Sampler converts to per-epoch `step` by modulo\n- Packing datasets have no state (deterministic)\n\n## Multi-Dataset Support\n\n- `DatasetConfigList` = list of `{dataset, tokenize_fn}` pairs\n- `build_datasets()` returns `list[JsonlDataset]` (1 file → 1 dataset)\n- `sample_ratio` applied per-DatasetConfig via `sampled[]` repetition\n\n## Key Implementation Files\n\n| File | Purpose |\n|------|---------|\n| `config.py` | Pipeline orchestration via `DataloaderConfig.build()` |\n| `pt_tokenize_fn/text.py` | `PretrainTokenizeFunction` |\n| `pt_tokenize_fn/long_text.py` | `LongTextPretrainTokenizeFunction` |\n| `jsonl.py` | `JsonlDataset` with shared memory + offset table |\n| `packing.py` | `HardPackDataset`, `ExpandSoftPackDataset` |\n| `sampler.py` | `LengthGroupedSampler`, `ParallelSampler` |\n| `preset_pack.py` | `PresetPackDataset` |\n| `preset_sampler.py` | `PresetSampler` |\n| `dataloader.py` | `Dataloader` wrapper |\n| `data_item.py` | `DataItem` schema |\n\n## Design Principles\n\n1. **Protocol-based**: All layers communicate via `DataItem` contract\n2. **Cache-aware**: Tokenization cached with automatic invalidation\n3. **Deterministic**: Packing and sampling reproducible from seed/config\n4. **Resumable**: State dict pattern for checkpoint/resume\n5. **Composable**: Tokenize → Dataset → Pack → Sample → Collate pipeline\n\n\n# RL Review Guidelines\n\nUse these rules whenever a PR changes code under `xtuner/v1/rl`. They extend the general review\nstandards in `.claude/CLAUDE.md`; they do not replace them.\n\n## Required Review Output\n\nEvery review for an RL change must state the ProduceBatchResult impact near the top-level summary:\n\n```text\nProduceBatchResult impact: \n```\n\nAlso include routed-experts impact when the PR touches routed-experts logic, rollout response\nhandling, `RolloutState.extra_fields`, object references, or memory ownership around rollout outputs:\n\n```text\nRoutedExperts impact: \n```\n\nAlso include Ray concurrency impact when the PR touches Ray actor methods, actor construction,\nmethod decorators or wrappers, concurrency-group definitions, or rollout control paths:\n\n```text\nRay concurrency impact: \n```\n\nDo not leave these impacts implicit in the finding text. If a finding is about rollout status,\npause/abort/timeout behavior, response handling, producer aggregation, routed-experts ownership, or\nRay actor concurrency, repeat the relevant impact line inside that finding so the downstream effect\nis visible at the point of review.\n\n## ProduceBatchResult Checklist\n\nReview whether the PR can change trainer-visible batch accounting, timing, or reward semantics in\n`ProduceBatchResult`.\n\nCheck this area when the PR changes any of these paths:\n\n- `RolloutState.status`, `finish_reason`, or status conversion.\n- Abort, filter, expire, retry, timeout, cancellation, or failure handling.\n- Producer, agent loop, judger, replay buffer, rollout worker, rollout controller, or backend pause\n cleanup logic.\n- Writers, readers, tests, or fake implementations that construct or consume `ProduceBatchResult`,\n including trainer-facing code outside `xtuner/v1/rl`.\n\nName the concrete field-level impact when any of these fields can change:\n\n- Batch status: `status`.\n- Returned groups: `rollout_states`.\n- Generation timing: `group_gen_count`, `group_gen_mean_s`, `group_gen_p50_s`, `group_gen_p99_s`,\n `group_gen_p99_p50_ratio`, `group_gen_pause_time_s`.\n- Replay-buffer leftovers: `leftover_init`, `leftover_completed`, `leftover_aborted`,\n `leftover_expired`, `leftover_failed`, `leftover_filtered`.\n- Reward accounting: `raw_rewards_sum`, `raw_rewards_count`.\n- Produced work counters: `produced_samples`, `produced_tokens`, `produce_time_s`.\n- Multi-task aggregation: `task_batch_sizes`, `task_results`.\n\nCommon impacts to call out explicitly:\n\n- A sample moving between `ABORTED`, `FAILED`, `FILTERED`, `EXPIRED`, and `COMPLETED` can change the\n corresponding `leftover_*` counts.\n- Pause, abort, timeout, and cancellation changes can inflate or deflate `group_gen_*` timing,\n especially `group_gen_pause_time_s`.\n- Reward or filter-path changes can change `raw_rewards_sum`, `raw_rewards_count`,\n `produced_samples`, and `produced_tokens`.\n\n## Ray Actor Concurrency Checklist\n\nReview whether high-volume rollout calls can starve pause, abort, health-check, restart, or shutdown\nRPCs.\n\nCheck this area when the PR changes any of these paths:\n\n- Ray actor methods decorated with `@ray.method(...)` or `@ray_method(...)`.\n- Decorators or wrappers stacked on Ray actor methods, especially wrappers using `functools.wraps`\n or exposing `__wrapped__`.\n- Actor construction through `ray.remote(...)`, including `concurrency_groups` and\n `max_concurrency`.\n- Generate, pause, continue, abort, health-check, restart, or shutdown methods.\n\nWhen a Ray method is combined with another decorator, keep the Ray method decorator closest to the\nfunction definition:\n\n```python\n@trace_rollout_endpoint(\"rollout.worker.generate\")\n@ray.method(concurrency_group=ROLLOUT_CONCURRENCY_GROUP_GENERATE)\nasync def generate(...):\n ...\n```\n\nDo not use the reverse order:\n\n```python\n@ray.method(concurrency_group=ROLLOUT_CONCURRENCY_GROUP_GENERATE)\n@trace_rollout_endpoint(\"rollout.worker.generate\")\nasync def generate(...):\n ...\n```\n\nThe current Ray actor-class processing unwraps decorated methods before reading Ray method metadata.\nIf `@ray.method(...)` is outside a wrapper, metadata such as `__ray_concurrency_group__` can remain\non the discarded wrapper, causing the method to fall back to the default concurrency group.\n\nReview the effective dispatch behavior, not only the visible decorator order:\n\n- After following `__wrapped__`, the callable inspected by Ray must retain the expected Ray method\n metadata.\n- Every named method concurrency group must be declared by the corresponding `ray.remote(...)`\n actor construction.\n- High-volume `generate` methods must not accidentally fall back to the default group.\n- Pause, abort, and other control RPCs must remain schedulable while generation is saturated.\n- Slow health or lifecycle methods must not occupy every slot needed by critical control RPCs.\n- Blocking work inside async actor methods must not block the actor event loop or concurrency lane.\n\n## RoutedExperts Checklist\n\nReview whether routed-experts ownership and cleanup remain correct.\n\nCheck this area when the PR changes any of these paths:\n\n- LMDeploy rollout response handling.\n- `return_routed_experts`, `routed_experts`, `RolloutState.extra_fields`, or object-ref plumbing.\n- Abort, cancellation, timeout, filter, retry, or failure paths before response handling completes.\n- Background tasks, replay-buffer storage, trainer batches, metrics, tests, or fake rollout\n responses that can retain routed-experts object refs.\n\nFor LMDeploy rollout, `rollout_worker` obtains routed-experts object refs from the LMDeploy shared\nstore. At that point, ownership moves from LMDeploy to XTuner. A review finding should call out both\nsides of the ownership boundary when relevant:\n\n- Leak before transfer: requests whose routed experts remain in LMDeploy because XTuner never\n obtains the object refs.\n- Leak after transfer: object refs that XTuner keeps alive too long through `RolloutState`, replay\n buffer, trainer batches, metrics, fake tests, or background tasks.\n\n\n---\nname: add_hf_model\ndescription: >\n Add support for a HuggingFace model in XTuner so it can be trained with the\n full set of parallelism strategies and optimization switches. The user can\n provide a HuggingFace Hub repo id, a local model directory, or a model family\n already supported by `transformers`. This skill walks through the full path:\n locating the reference implementation, classifying the model, splitting new\n code across the model / module / ops layers, implementing the XTuner model\n class and config, registering the entry point, validating bitwise numerical\n parity with HuggingFace, writing regression tests, and enabling the training\n optimizations (EP/SP, micro-batch, torch.compile, fp8, activation offload).\n---\n\n# Add a HuggingFace model to XTuner\n\nA complete handbook for porting any HuggingFace causal LM into XTuner with\ncorrect training behavior. Covers four buckets: **Dense LLM**, **MoE LLM**,\n**VLM / compose model**, and **`trust_remote_code` model**. Every step below is\ngrounded in concrete `file:line` references in this repository — read the cited\ncode before writing yours.\n\n> **Anti-spaghetti reminder** — XTuner separates *what the model is* (`*Config`),\n> *what it computes* (`Dense` / `MoE` / `BaseComposeModel` subclass), and *how its\n> weights map to HF safetensors* (`to_hf_key_list`, optional `safetensors_to_params`\n> / `param_to_safetensor`). Keep new code inside those three seams. **Never** add\n> `if model_name == \"...\"` branches to the base classes.\n\n---\n\n## 0. Locate the source code\n\nThe only goal of this step is to find the **HuggingFace reference implementation\nyou will port from**, and to note the `model_type` you will register under. Do\nnot pick attention/router/layer here — that happens naturally in §1 as you read\nthe code.\n\n1. If the user gives a **remote-code** model, the modeling code ships in the repo\n as `modeling_*.py` — read it directly.\n2. If the user gives **weights only**, confirm the target Python environment with\n the user, then find the matching implementation inside that env's\n `transformers` install. If no code exists for that `model_type`, stop and ask\n the user.\n3. Confirm the weights are **training-suitable** — at least bf16 — so training\n precision can be aligned later. Flag fp8/int4-only checkpoints to the user.\n4. Note two strings from the HF config — you will need both downstream:\n ```python\n from transformers import AutoConfig\n cfg = AutoConfig.from_pretrained(path, trust_remote_code=True)\n cfg.model_type # registration key (§4)\n cfg.architectures[0] # the modeling class you are mirroring\n ```\n\n---\n\n## 1. How to do the migration\n\nXTuner splits a model into three layers. You port by reusing what already exists\nat each layer and only adding what is genuinely new — so the work is \"read the HF\nmodeling code, then match each of its pieces to the XTuner component below\". The\nlocations are listed plainly; you do not need extra guidance to map your model's\nattention / router / layer onto the matching one.\n\n- **model layer** (`xtuner/v1/model`) — the model body.\n - Dense body: `xtuner/v1/model/dense/dense.py`\n - MoE body: `xtuner/v1/model/moe/moe.py`\n - Multi-modal (compose) bodies: `xtuner/v1/model/compose/` — reuse the language\n model and add only the ViT / projector pieces.\n- **module layer** (`xtuner/v1/module`):\n - Decoder layers: `DenseDecoderLayer`, `MoEDecoderLayer` —\n `xtuner/v1/module/decoder_layer`. Small changes add a switch on the existing\n layer; a genuinely novel architecture may add `{model_name}_decoder_layer`.\n - Attention configs: `MHAConfig` (`xtuner/v1/module/attention/mha.py`),\n `MLAConfig` (`.../mla.py`, DeepSeek-style latent),\n `GatedDeltaNetConfig` (`.../gated_deltanet.py`, linear).\n - Router configs (MoE): `GreedyRouterConfig`\n (`xtuner/v1/module/router/greedy.py`, softmax top-k — Qwen3 MoE, GptOss),\n `NoAuxRouterConfig` (`xtuner/v1/module/router/noaux_router.py`, grouped sigmoid\n aux-loss-free — DeepSeek V3).\n - Also: rope, rms_norm, etc.\n- **ops layer** (`xtuner/v1/ops`) — kernels such as attention and rms_norm.\n\n### Existing models to copy from\n\nPick the one whose attention + (router) match yours; the closer it is, the\nsmaller your diff.\n\n| Your model is… | Copy from |\n|--------------------------------------|--------------------------------------|\n| Dense LLM | `xtuner/v1/model/dense/qwen3.py` |\n| MoE LLM | `xtuner/v1/model/moe/qwen3.py` |\n| MoE with MLA + aux-loss-free router | `xtuner/v1/model/moe/deepseek_v3.py` |\n| MoE needing weight-layout transform | `xtuner/v1/model/moe/gpt_oss.py` |\n| Compose / VLM | `xtuner/v1/model/compose/qwen3_vl/` |\n\n\nAfter you have a plan, **write an integration design doc into `docs/design/model`**\nand confirm with the user that the decomposition is reasonable before\nimplementing.\n\n### File layout for the new model\n\n| Bucket | Model file | Public re-export | Test file |\n|----------------------|-------------------------------------------------------------|-----------------------------------------------|--------------------------------------------|\n| Dense LLM | `xtuner/v1/model/dense/.py` | `xtuner/v1/model/__init__.py` | `tests/model/test__dense.py` |\n| MoE LLM | `xtuner/v1/model/moe/.py` | `xtuner/v1/model/__init__.py` | `tests/model/test__moe.py` |\n| Compose / VLM | `xtuner/v1/model/compose//...` | `xtuner/v1/model/__init__.py` | `tests/model/test_.py` |\n\n---\n\n## 2. Implement the model class\n\nSubclass `Dense`, `MoE`, or `BaseComposeModel`. You typically only need to\noverride one method.\n\n### 2.1 `to_hf_key_list(self, key: str) -> list[str]` — **mandatory, abstract on `BaseModel`**\n\nDeclared at `xtuner/v1/model/base.py`. Translates an **XTuner-side** parameter\nname into one or more **HF-side** safetensors keys. Return-list cardinality:\n\n- **1 → 1** for plain weights (most params).\n- **1 → N** for fused MoE experts (one fused param explodes into N per-expert\n HF keys; see `xtuner/v1/model/moe/qwen3.py`).\n\nPatterns you almost always need (study `xtuner/v1/model/dense/qwen3.py`\nand `xtuner/v1/model/moe/qwen3.py`):\n\n```python\n# 1. Tied embeddings: redirect lm_head → embed_tokens before adding the model. prefix.\nif self.config.tie_word_embeddings and \"lm_head\" in key:\n key = key.replace(\"lm_head\", \"embed_tokens\")\n\n# 2. Add the \"model.\" prefix that HF wraps everything in.\nif \"layers\" in key or \"embed_tokens\" in key:\n key = \"model.\" + key\n\n# 3. HF MoE nests experts/gate under .mlp.; XTuner does not.\nif \"layers\" in key:\n key = re.sub(r\"layers\\.(\\d+)\\.(experts|gate)\", r\"layers.\\1.mlp.\\2\", key)\n\n# 4. Top-level norm.\nif key.startswith(\"norm.\"):\n return [key.replace(\"norm.\", \"model.norm.\")]\n\n# 5. MoE expert fusion: one fused param → N HF keys.\nif \"fused_w1w3.weight\" in key:\n out = []\n for i in range(self.config.n_routed_experts):\n out.append(key.replace(\"fused_w1w3.weight\", f\"{i}.gate_proj.weight\"))\n out.append(key.replace(\"fused_w1w3.weight\", f\"{i}.up_proj.weight\"))\n return out\n\n# 6. Model-specific buffers (e.g. FoPE: rotary_emb.sin_coef / cos_coef are persistent buffers).\nif key.startswith(\"rotary_emb.\"):\n return [key.replace(\"rotary_emb.\", \"model.rotary_emb.\")]\n```\n\nA prefix-region remap that differs by *deployment* (e.g. a text tower nested under a\nVLM as `model.language_model.`) does **not** belong here — put it in the config's\n`hf_key_mapping`. See §5.1.\n\n### 2.2 `safetensors_to_params` / `param_to_safetensor` — **optional, only for layout mismatches**\n\nIf the HF storage layout cannot be expressed as “same tensor, different name”,\nyou must transform the bytes. Canonical example: GptOss stores expert weights as\n`(num_experts, hidden_size, expert_dim * 2)` but XTuner fuses them as\n`(num_experts * 2 * expert_dim, hidden_size)` — see overrides at\n`xtuner/v1/model/moe/gpt_oss.py`.\n\nUse this hook only when transposing/reshaping is unavoidable. Renames go in\n`to_hf_key_list`; numerics go here.\n\n### 2.3 Do **not** override\n\n- `build_layers`, `build_embeddings`, `_init_weights` — the base handles them\n via the config. Override only if the new model adds a layer type the base\n cannot express (rare).\n- `forward` — same reasoning. If you find yourself touching it, re-evaluate the\n bucket choice; the problem is probably in the config (`attention`, `router`,\n `first_k_dense_replace`).\n\n---\n\n## 3. Implement the config\n\nTwo layers: a **base config** that reads HF and a **size-specific subclass** per\nreleased checkpoint.\n\n### 3.1 Base config — three required members\n\nMirror `Qwen3MoEConfig` (`xtuner/v1/model/moe/qwen3.py`) or\n`Qwen3DenseConfig` (`xtuner/v1/model/dense/qwen3.py`).\n\n1. **`build(self) -> `** — return `(self)`.\n\n2. **`@classmethod from_hf(cls, hf_path) -> Self`** — read the HF config and map\n its fields one-to-one onto your config. The field set is whatever the live\n config class declares (`TransformerConfig` in `xtuner/v1/model/base.py`,\n `MoEConfig` in `xtuner/v1/model/moe/moe.py`); read it, don't work from a\n hard-coded list. A few non-obvious traps:\n - Use `RopeParametersConfig.from_hf_config(hf_config)` from\n `xtuner/v1/module/rope/rope.py`. It already handles both the\n `rope_parameters` dict (HF ≥ 5.2.0) and the legacy `rope_scaling` dict\n (HF 4.57.x), plus YARN / FoPE / mrope special fields. **Do not** parse\n rope yourself.\n - For optional HF fields, use `getattr(hf_config, \"\", )`\n rather than `hf_config.` — older checkpoints drop fields.\n\n3. **`@property hf_config(self) -> | None`** — the inverse of\n `from_hf`: every field `from_hf` reads must be re-emitted so `save_hf`\n round-trips.\n\n**Built-in vs. `trust_remote_code` — the only axis that changes `from_hf` /\n`hf_config`:** a `trust_remote_code` model is structurally still Dense / MoE /\nVLM; the *only* difference is whether `transformers` ships a built-in\n`Config` class. Decide which case you are in and follow the matching column\n— this is the single place that contract is defined.\n\n| | Built-in config (e.g. Qwen3) | `trust_remote_code` (no built-in config) |\n|---|---|---|\n| `from_hf` reads via | `.from_pretrained(hf_path)` | `AutoConfig.from_pretrained(hf_path, trust_remote_code=True)`, then `assert hf_config.model_type == \"\"` |\n| `hf_config` returns | a populated `` re-emitting every field | `None` |\n| `save_hf` weight-map | written from `hf_config` | same |\n| `save_hf` side files | derived from `hf_config` | falls back to copying `config.json` / tokenizer / `*.py` from `self._hf_path` so the dir stays loadable with `trust_remote_code=True`; module-cache test in the matrix below |\n\nCompare `Qwen3MoEConfig.hf_config` (built-in) and `Qwen3MoEFoPEConfig.hf_config`\n(returns `None`), both in `xtuner/v1/model/moe/qwen3.py`. **Do not** invent a\nbuilt-in HF config class for a remote-code model — return `None`.\n\n**Invariants:**\n\n- `from_hf` ↔ `hf_config` must round-trip. The `test_save_hf` test (§7) is the\n enforcement mechanism — every key in the original index must appear in the\n saved index with byte-equal tensors.\n- The config must be **pickleable and side-effect free**. No env vars, no file\n I/O, no `torch.cuda.*` at construction time.\n\n### 3.2 Size-specific subclasses\n\nAdd one subclass per released checkpoint that hard-codes the published\ndimensions. Mirror `Qwen3MoE30BA3Config` / `Qwen3MoE235BA22Config` in\n`xtuner/v1/model/moe/qwen3.py`. Even if only one size ships, define at least one\n— it is the entry point for `get_model_config` and for tests.\n\n---\n\n## 4. Register the entry point\n\nOpen `xtuner/v1/model/__init__.py` — all four edits are short:\n\n1. **Import** — bring in the new config classes.\n2. **Alias** — add `\"\": ()` to `model_mapping`. Lookup\n normalizes case and `-`/`_`, so `\"my-model-7b\"` and `\"my_model_7b\"` are the\n same key.\n3. **Dispatch** — add `elif cfg.model_type == \"\":` to\n `get_model_config_from_hf` returning `.from_hf(model_path)`.\n The catch-all `raise ValueError(...)` must stay last.\n4. **`__all__`** — re-export the new public config names.\n\n**Never** key dispatch on parameter names, file paths, or architecture strings.\nOnly `model_type`.\n\n---\n\n## 5. Compose / VLM specifics\n\nCompose models live under `xtuner/v1/model/compose//` and follow a\nthree-tower pattern. See `xtuner/v1/model/compose/base.py` and\n`xtuner/v1/model/compose/qwen3_vl/qwen3_vl_config.py`.\n\n### 5.1 Config shape\n\n`BaseComposeConfig` (`compose/base.py`) carries three sub-configs:\n\n```python\nvision_config: XTunerBaseModelConfig # e.g. Qwen3VLVisionConfig\nprojector_config: XTunerBaseModelConfig # e.g. Qwen3VLProjectorConfig\ntext_config: XTunerBaseModelConfig # any Dense/MoEConfig subclass\nfreeze_vision: bool = False\nfreeze_projector: bool = False\nfreeze_language: bool = False\n```\n\nReuse existing text-tower configs where possible. `Qwen3VLDense*` delegates to\nthe Dense path via `xtuner/v1/model/dense/qwen3vl_text.py`. **Do not** copy-paste\nthe entire dense/MoE config — compose over them.\n\n**Reused text-tower key prefixes — use `hf_key_mapping`, not `to_hf_key_list`.** A\ntext tower reused inside a VLM keeps its *standalone* `to_hf_key_list` (it emits\n`model.<...>` as if it were a plain LLM). The compose checkpoint nests it one level\ndeeper (`model.language_model.<...>`). Do **not** teach the text tower's\n`to_hf_key_list` about that prefix — that leaks the compose context into a tower that\nmust also work standalone. Set the remap on the **text config** instead:\n`hf_key_mapping = {r\"^model\\.\": \"model.language_model.\"}` (applied in\n`_init_load_spec`). Keep *structural* renames in `to_hf_key_list`; the\ndeployment-dependent prefix region belongs in `hf_key_mapping`.\n\n### 5.2 Model shape\n\n`BaseComposeModel` (`compose/base.py`) builds three sub-models:\n`vision_tower`, `multi_modal_projector`, `language_model`.\n\n- **`from_hf(hf_path)`** — delegates to each sub-model with\n `strict=False` (vision keys are missing from the language-model state dict\n and vice versa) and unions the missing-keys sets.\n- **`save_hf(hf_dir)`** — saves each tower with a distinct prefix\n (`\"model-language\"`, `\"model-vision\"`, `\"model-projector\"`) and merges the\n three `weight_map`s into one `model.safetensors.index.json`.\n\n### 5.3 `hf_config` for VLMs\n\nMany VLMs ship without a stable HF top-level config class; `hf_config` returns\n`None` (`compose/qwen3_vl/qwen3_vl_config.py`). In that case, `save_hf`\nfalls back to copying source files from `self._hf_path` (see §3.1).\n\n### 5.4 Vision-side quirks to watch for\n\n- `attn_impl` selectable per checkpoint: `\"flash_attention\" | \"flex_attention\" |\n \"eager_attention\"`.\n- `deepstack_visual_indexes` (e.g. `[8, 16, 24]`) — auxiliary supervision depths\n on the vision tower; must round-trip.\n- Specialized rope: `rope_type=\"qwen3_vl\"`, `mrope_section=[24, 20, 20]` —\n delegated to `RopeParametersConfig.from_hf_config`.\n- The projector’s `torch.compile` path is enabled on torch ≥ 2.9.1 (see commit\n `f6d74efb`). Tests must validate both compiled and eager outputs match.\n\n---\n\n## 6. Baseline parity — no complex parallelism yet\n\nFirst make the model correct on the **simplest execution path**: single rank, no\nEP/SP, no `torch.compile`, no fp8, no activation offload. Align precision from\nthe inside out — ops → decoder layer → full model — reusing XTuner's own modules\nat every level. Only once this baseline holds do you add the parallel and\noptimization features (§8).\n\n1. **Reuse, don't reimplement.** Use the existing XTuner modules — MHA, MLA,\n GatedDeltaNet, GroupLinear, etc. Do not re-implement them with naive torch.\n\n2. **Route op differences through `XTUNER_HF_IMPL`.** This env var already exists\n (`xtuner/v1/utils/misc.py`) and selects HF-exact ops via per-op selectors:\n `xtuner/v1/ops/attn_imp.py::get_attn_impl_fn` (eager attention),\n `xtuner/v1/ops/rms_norm/__init__.py::get_rms_norm_fn` (native torch rms_norm),\n and `xtuner/v1/ops/gated_deltanet/__init__.py::{get_chunk_gated_delta_rule_fn,\n get_causal_conv1d_fn}` (return the canonical fla / causal_conv1d wrappers HF\n calls instead of XTuner's compile-friendly `torch.library.custom_op` wraps —\n same forward output, but the backward op graph now matches HF). Selectors\n read the env var **live** so a test can toggle it per model instance. If your\n model introduces a new op whose fast path is not bitwise against HF (typical\n for fla-backed linear-attention variants), add its HF-exact branch in the\n corresponding op selector the same way. **You may only patch at the ops\n level** — never take the shortcut of patching the entire\n `XTunerDecoderLayer.forward`.\n\n Pair `XTUNER_HF_IMPL` with `XTUNER_DETERMINISTIC=true`. The latter gates an\n autotune-pin block in `xtuner/v1/__init__.py` that monkey-patches\n `triton.autotune` to lock every kernel to its first config and disable the\n result cache. Without the pin, fla's `@triton.autotune` picks tiling /\n reduction order per kernel and the choice drifts between runs, producing\n 1 ULP per linear-attention layer that the chain rule amplifies to ~1.0 at\n the model boundary (§12). `tests/conftest.py` sets the env var and runs\n `import xtuner.v1` so the patch installs before fla is imported. The patch\n lives on the `xtuner.v1.*` import chain (not in the conftest) because\n `multiprocessing.spawn` children of `MultiProcessTestCase` don't go through\n pytest's conftest — they re-import the test class top-level (which pulls\n `xtuner.v1.model.*`) and the patch must run in those processes too.\n Production training doesn't set the env var, so the block is inert there.\n\n3. **Decoder-layer bitwise parity — forward AND backward, both required.** Build\n one XTuner `DecoderLayer` (one per distinct layer type — e.g. a linear and a\n full layer for a hybrid model) and the matching HF layer, set `XTUNER_HF_IMPL`\n and `XTUNER_DETERMINISTIC`, feed identical inputs, and align **bitwise** — not a\n tolerance. **Backward is not optional**: a `single layer → final norm → lm_head\n → CE loss → loss.backward()` test asserts the **layer output, the loss, and the\n input gradient `dL/dx`** all bitwise-equal to HF — test one full and one linear\n layer separately (reference: `test_decoder_layer_bitwise_parity`). A bitwise\n *forward* does not imply a bitwise backward — the forward values can round\n identically while the backward graph differs (see the fp32-softmax note in §12).\n This is the **primary** path, required at all scales, and the only feasible one\n for large models (it never loads the full model — see \"load only what you use\"\n below). When you hand-build a single layer's\n inputs, the two sides' `causal mask` / `seq_ctx` must match exactly — see §12.\n **Load only what you use** so the test fits any model size: build the XTuner\n tower on `meta` and materialize (`to_empty` + selective load) *only* the tested\n layer + `norm` + `lm_head` via `HFCheckpointLoader.load(key)` (it reads just the\n needed safetensors shard); build a *standalone* HF `DecoderLayer` + norm +\n lm_head and load the same keys. Never `from_pretrained` the whole model for a\n single-layer test. (`build_rotary_embedding` builds the rotary on CPU with real\n buffers even under meta, so `model.rotary_emb` is usable without materializing\n the layers.) For < 40B you may also read per-layer hidden states from one\n full-model forward (HF `output_hidden_states` + XTuner `return_hidden_states`);\n mind the off-by-one in §12.\n\n4. **Model config matches HF hyperparameters** — see the §3.1 built-in vs.\n remote-code contract for how `from_hf` / `hf_config` differ between the two.\n\n5. **Full-model forward parity for models < 40B**, bitwise against HF, on the\n single-rank path (FSDP on or off; no EP / compile / fp8 yet).\n\n6. **`save_hf` / `from_hf` correctness** (round-trip; see §7).\n\n7. **Loss-convergence trace + engine test.** Once model forward and decoder-layer\n forward/backward are aligned, record a loss-convergence trajectory and add an\n engine test case.\n\n**Commit the baseline — do not stop to ask.** Committing is part of this\nworkflow: invoking this skill authorizes the in-workflow commits. Once parity\nholds on the simple path and lint/tests pass, commit immediately so this correct\nbaseline is preserved before any parallel/optimization work begins (the §8\nsub-agents branch off this commit). Commit each logical step as you finish it\n(§10) rather than batching everything to the end or waiting for a separate\ngo-ahead.\n\nReport back to the user: whether bitwise parity was achieved, and the residual\nerror (decoder-layer level and model level) once XTuner's internal components are\nused.\n\n---\n\n## 7. Baseline tests — add these first\n\nBefore any parallel feature, add the most basic regression tests so the baseline\n(§6) is locked and the §8 work has a fixed reference to measure against. Add\n`tests/model/test__.py`, mirroring\n`tests/model/test_qwen3_moe.py` (MoE) or `tests/model/test_qwen3_dense.py`\n(Dense). All tests extend `xtuner._testing.DeterministicDDPTestCase` for\ndeterministic distributed setup.\n\nReference test cases:\n- decoder-layer bitwise parity + save_hf round-trip: `tests/model/test_qwen3_5_dense.py`\n (hybrid linear/full dense VLM; toggles `XTUNER_HF_IMPL` per case)\n- model forward & save & load parity: `tests/model/test_qwen3_moe.py`\n- engine training test: `tests/engine/test_moe_train_engine.py`\n\n### 7.1 Baseline test matrix\n\nAll single execution path — no EP / dispatcher / compile / fp8 here; those are\n§8 tests, written against this baseline.\n\n| Case | Dense | MoE | Compose | Notes |\n|-------------------------------------------------|:-----:|:---:|:-------:|------------------------------------------------------------------------------------|\n| Decoder-layer parity — output + loss + `dL/dx` | ✅ | ✅ | ✅ | **Required at all scales, bitwise.** Test one full and one linear layer *separately*. Standalone & memory-light: build the XTuner tower on `meta` and `materialize_xtuner_submodule` only {tested layer, `norm`, `lm_head`}; build a standalone HF `DecoderLayer`; load via `HFCheckpointLoader`. Under `XTUNER_HF_IMPL` assert the **layer output, loss, and `dL/dx` all bitwise** (forward AND backward at the layer level — a bitwise forward ≠ bitwise backward, see §12). Reference: `test_decoder_layer_bitwise_parity`. |\n| Whole-model forward + backward parity (by scale) | ✅(4B)| scale | ✅(4B) | Run the **top-level model** end-to-end vs HF, eager both sides (`XTUNER_HF_IMPL` + `XTUNER_DETERMINISTIC` + HF `eager`). For a **VLM** that's the compose model on an **image** prompt (vision + projector + text — subsumes \"VLM forward parity\"); for a plain LLM, the text/LM model. With both env switches on plus the autotune pin loaded (§6.2), **forward AND backward are bitwise**: assert logits + loss + `dL/d(pixel_values)` / `dL/d(inputs_embeds)` all `== 0.0`. **Scale**: small = fwd+bwd; medium = forward only; large (can't e2e-forward) = skip → integration. Reference: `test_vl_forward_parity`. |\n| Vision-tower bitwise parity | — | — | ✅ | **Compose/VLM only.** Compare HF's `visual` output (`pooler_output`, which is **post-merger**) bitwise (0.0), both eager. XTuner splits that boundary into `vision_tower` (patches → merged hidden) + `projector` (the merger MLP → text dim), so load **both** standalone to match it — don't build the whole VLM just for the vision path. Build them on the **real device, not `meta`**: the vision rotary is computed into a buffer at forward time, so `to_empty` would leave garbage (unlike the text rotary, which `build_rotary_embedding` builds real even under meta — §6.3). Vision attention is *non-causal* → `eager_attention(causal=False)`, which `XTUNER_HF_IMPL` selects. Reference: `test_vision_tower_bitwise_parity`. |\n| FSDP forward parity | ✅ | ✅ | ✅ | Looser tolerance (≈ 3e-2 is the established budget). |\n| `save_hf` round-trip (byte-equal tensors) | ✅ | ✅ | ✅ | See `tests/model/test_qwen3_moe.py`. |\n\n### 7.2 Parity bar by model scale\n\nTwo independent axes:\n\n**Per-layer (op) parity — required at _all_ scales, bitwise.** Test one full and\none linear decoder layer separately; for each assert layer **output + loss + `dL/dx`**\nbitwise vs HF under `XTUNER_HF_IMPL`. Keep it memory-light (XTuner `meta`-build +\nmaterialize only the tested layer + `norm` + `lm_head`; standalone HF layer; load\nvia `HFCheckpointLoader`) so it runs even for models too large to forward\nend-to-end. This is the bitwise guarantee for both forward and backward ops.\n\n**Whole-model (integration) parity — on the _top-level_ model, graded by what fits\none GPU.** Run the real model end-to-end: for a VLM that's the **compose model on an\nimage prompt** (one test covers vision + projector + text — don't add a separate\ntext-tower model test); for a plain LLM, the text/LM model.\n- **Small** (e.g. dense 4B): both forward (logits/loss) AND backward\n (`dL/d(pixel_values)` for a VLM, `dL/d(inputs_embeds)` for text) **bitwise**.\n Prerequisites: `XTUNER_HF_IMPL=true` so every op selector returns the\n HF-canonical callable (§6.2) AND `XTUNER_DETERMINISTIC=true` so Triton\n autotune is pinned (§6.2). With both on, even hybrid linear/full models match\n HF byte-for-byte through the whole compose chain. Don't check weight\n gradients — they accumulate across positions + tied `lm_head` and aren't the\n right granularity for parity.\n- **Medium**: forward only (logits/loss bitwise).\n- **Large** (cannot forward end-to-end on the test GPUs): skip the whole-model\n test; leave it to integration tests. The per-layer test still gives op-level\n forward+backward bitwise.\n\n### 7.3 Required test idioms\n\n`tests/model/test_qwen3_5_dense.py` is the reference template; extend\n`DeterministicDDPTestCase` (`xtuner._testing`), which provides the shared\nparity scaffolding so the test file only holds model-specific logic:\n\n- `with self.hf_impl():` — sets/restores `XTUNER_HF_IMPL=true`; build the XTuner\n model **inside** the block so its attention picks the eager op at construction.\n- `self.materialize_xtuner_submodule(model, submodule, prefix, loader)` — for a\n model built on `meta`, materialize *only* that submodule on CUDA and load its\n weights (so a single-layer test fits any model size).\n- `self.load_params_from_hf(module, key_for, loader)` /\n `self.xtuner_ckpt_key(model, name)` — selective per-param load by checkpoint key\n (`HFCheckpointLoader` reads only the needed shard); `xtuner_ckpt_key` applies\n `to_hf_key_list` + the config's `hf_key_mapping`.\n\n```python\n# Construct on meta, materialize on device — single-rank baseline, eager, no EP.\nwith torch.device(\"meta\"):\n cfg = get_model_config_from_hf(hf_model_path)\n cfg.compile_cfg = False # baseline tests run eager\n model = cfg.build()._to_device_dtype(dtype=torch.bfloat16, skip_buffers_dtype=True)\n\n# Optionally shard with FSDP (data-parallel only), then load.\nmodel.fully_shard(fsdp_config=FSDPConfig(cpu_offload=False))\nmodel.from_hf(hf_model_path)\n```\n\nOlder `patch_hf_rms_norm` / `patch_hf_rope` helpers also live in\n`xtuner._testing`. **Check they actually fit your model before using them** —\nthey were written for earlier models and match HF modules\nby class-name substring and attribute name (e.g. `variance_epsilon`). On a newer\nmodel they can silently mis-patch (wrong eps attribute, or replacing a\n`zero_centered` RMSNorm with a default one). Often the native XTuner ops already\nmatch HF bitwise (see §12), so no patching is needed; prefer `XTUNER_HF_IMPL`\nover these helpers for parity. The §8 parallel-feature tests reuse this same\nidiom with `cfg.dispatcher` / `cfg.ep_size` (and `FSDPConfig(ep_size=...)`) set.\n\n### 7.4 Checkpoint paths\n\nRead from env vars; **never** hard-code. Example:\n`QWEN3_MOE_PATH = os.environ[\"QWEN3_MOE_PATH\"]` at\n`tests/model/test_qwen3_moe.py`. Document new env vars in the PR body.\n\n### 7.5 Drop-in training config\n\nShip a runnable training config at `ci/config/.py` so a developer (and CI) can verify\nthe port end-to-end with one command: load the HF checkpoint, run a few real training steps,\nwatch the loss curve descend as expected. This is the smoke test that catches anything the\nunit tests miss — the full `from_hf` → `model.forward` → loss → backward → optimizer step →\nFSDP shard/reduce chain — and the file doubles as the example users copy when wiring the\nmodel into their own training pipeline. Mirror `ci/config/qwen3_moe_30BA3.py` (MoE) or\n`ci/config/qwen3_dense.py` (dense), keeping its structure: one `()` (from §3.2)\nfed into a `TrainerConfig` alongside `optim_cfg` / `lr_cfg` / `fsdp_cfg` / `dataset_cfg` /\n`dataloader_cfg` / `loss_cfg`. `load_from` and `tokenizer_path` read from an env var (§7.4 —\ntypically the same one as the parity test). Verify by running ~50 steps and confirming the\nloss drops monotonically into a plausible range for that model size; record the trajectory\nin the PR body alongside the §6 convergence trace.\n\n---\n\n## 8. Training optimizations\n\nOnly after the §6 baseline is committed and the §7 tests are green: verify each\nswitch runs correctly and keeps precision within budget **against that baseline**.\n\n1. EP and SP can be enabled normally.\n2. With `ep_size > 1`, `intra_layer_micro_batch` enables correctly and stays\n precise.\n3. `torch.compile` runs correctly.\n4. fp8 can be enabled.\n5. `XTUNER_ACTIVATION_OFFLOAD` can be enabled.\n\nEach switch needs a test that compares its output to the §6 baseline — e.g. the\ndispatcher × ep_size parity matrix (at minimum `{(None,1), (\"all2all\",4),\n(\"all2all\",8)}`, add `\"deepep\"` if supported), compiled-vs-eager, or fp8-vs-bf16\nwithin tolerance.\n\nThese items are well-suited to the multi-agent workflow in §11: the main agent\nanalyzes and creates tasks (with test cases), sub-agents develop in parallel, and\nthe main agent merges. Report the precision error introduced by each switch.\n\n---\n\n## 9. Reference cheat-sheet\n\n| Concept | Concrete location |\n|-------------------------|--------------------------------------------------------------------------------|\n| Dense base config | `xtuner/v1/model/base.py` (`TransformerConfig`) |\n| MoE base config | `xtuner/v1/model/moe/moe.py` (`MoEConfig`) |\n| Compose base config | `xtuner/v1/model/compose/base.py` (`BaseComposeConfig`) |\n| `from_hf` (model) | `xtuner/v1/model/base.py` (`from_hf`) |\n| `save_hf` (model) | `xtuner/v1/model/base.py` (`save_hf`) |\n| `to_hf_key_list` proto | `xtuner/v1/model/base.py` |\n| MoE layer split | `xtuner/v1/model/moe/moe.py` (`first_k_dense_replace`) |\n| RoPE auto-parse | `xtuner/v1/module/rope/rope.py` (`RopeParametersConfig.from_hf_config`)|\n| Dense reference | `xtuner/v1/model/dense/qwen3.py` |\n| MoE reference | `xtuner/v1/model/moe/qwen3.py` |\n| MoE w/ MLA + NoAux | `xtuner/v1/model/moe/deepseek_v3.py` |\n| MoE w/ layout transform | `xtuner/v1/model/moe/gpt_oss.py` |\n| Remote-code reference | `xtuner/v1/model/moe/qwen3.py` (`Qwen3MoEFoPEConfig`) |\n| Compose reference (VLM) | `xtuner/v1/model/compose/qwen3_vl/` |\n| Dense test reference | `tests/model/test_qwen3_dense.py` |\n| MoE test reference | `tests/model/test_qwen3_moe.py` |\n| Engine test reference | `tests/engine/test_moe_train_engine.py` |\n\n---\n\n## 10. Commit discipline\n\n0. **Commit as you go — this is authorized, do not pause for a separate\n go-ahead.** The user invoking this skill authorizes the commits this workflow\n produces. Finish a logical step → lint/test → commit, then move on. Do not\n leave the whole port uncommitted and end with \"should I commit?\"; that strands\n the work. (Hard-to-reverse or outward actions like `git push` / opening a PR\n still need explicit confirmation — committing locally does not.)\n1. **Follow the stacked-PR convention.** Plan upfront how many commits the model\n port needs — one per logical step (e.g. shared-path fixes; model class +\n config + registration; the `XTUNER_HF_IMPL` switch; baseline tests; then one\n per parallel/optimization feature). Order them by dependency (a shared-path fix\n the baseline relies on goes first). Land every later change as\n `git commit --fixup=` + `git rebase --autosquash` into the commit it\n belongs to. **Never grow the history with endless patch commits just to keep\n fixing things up.**\n2. **Every commit must pass lint:**\n ```bash\n pre-commit run --files $(find xtuner/v1)\n ```\n\n---\n\n## 11. Multi-agent workflow\n\nThis applies **only to the §8 parallel/optimization features** — and only once\nthe §6 baseline and §7 baseline tests are done and committed (§10). The baseline\nitself is built sequentially, not multi-agent; the §8 features are independent of\neach other, which is what makes them safe to develop in parallel.\n\n1. The **main agent** splits the §8 features into tasks and, per §8, writes each\n feature's comparison test against the committed baseline and commits the tests.\n2. **Sub-agents** branch off the baseline/test commit and develop one feature\n each (e.g. one switch / one tower) in parallel, making its test pass.\n3. **The two sides adjust each other** — a shared-interface change is made once by\n the main agent (it updates the test/baseline; sub-agents rebase), and a failing\n test pushes the sub-agent to fix its feature, not to weaken the test.\n4. The **main agent** merges, resolves conflicts, and reports the precision error\n each feature introduces relative to the baseline.\n\n---\n\n## 12. Parity debugging pitfalls\n\nHard-won notes from aligning hybrid (GatedDeltaNet + gated-MHA) models bitwise.\nThe meta-rule: **measure on real GPU and bisect to localize the divergence before\nconcluding anything** — most \"bugs\" here are measurement artifacts, not model\nerrors. A clean way to bisect is to feed one side's intermediate tensors into\nboth modules and compare step by step (norm → q/k/v → qk_norm → rope → attention\ncore → o_proj).\n\n- **Single-layer parity: the two sides' inputs must match exactly — especially\n the causal mask.** When you hand-build one layer's inputs, passing\n `attention_mask=None` to an HF attention/layer makes HF apply **no** causal\n mask while XTuner stays causal, producing a large *false* diff. Build a causal\n mask equivalent to XTuner's `seq_ctx`, or drive the layer through the model so\n the mask is constructed internally.\n\n- **`flash` ≠ `eager`, and HF eager upcasts softmax to fp32.** XTuner defaults to\n flash attention; HF reference eager does softmax in fp32. They are numerically\n close but not bitwise equal, so bitwise parity requires forcing eager on the\n XTuner side (that is exactly what `XTUNER_HF_IMPL` does). XTuner's\n `eager_attention` matches HF's `eager_attention_forward` bitwise.\n\n- **A bitwise forward does not imply a bitwise backward.** The eager softmax must\n run in **fp32** (`softmax(..., dtype=torch.float32).to(x.dtype)`, like HF), not\n bf16. With a bf16 softmax the forward logits can still round identically to HF\n (bitwise), but the *backward* differs (~1e-5 on `dL/dx`) because the gradient\n accumulates at bf16 precision. So always check backward bitwise separately\n (the `loss.backward()` test), not just forward — and align op dtypes (softmax,\n norms) in the backward-sensitive direction, fp32 where HF uses fp32.\n\n- **Whole-model backward is bitwise — but only with the full env stack.**\n Per-layer bitwise backward does not on its own imply whole-model bitwise\n backward; the missing pieces are `XTUNER_HF_IMPL=true` AND\n `XTUNER_DETERMINISTIC=true` AND the conftest must `import xtuner.v1` so the\n autotune-pin block in `xtuner/v1/__init__.py` runs before any test reaches\n fla. Without those, two distinct ULP-level sources compound through the LM\n chain rule and look indistinguishable from \"fundamental bf16 noise\":\n\n 1. **Triton autotune drift in fla.** fla's `@triton.autotune` picks\n tiling / num_warps / reduction-order per kernel. Across runs the choice\n can change, producing a ~1-ULP backward diff per linear-attention layer\n that the chain rule amplifies — a 32-layer LM stack turned a per-layer\n ~2e-3 abs diff at grad magnitude ~30 into a ~1.0 diff on the\n massive-activation channels of `dL/d(pixel_values)`. The pin (configs\n held to the first one, cache disabled) is in `xtuner/v1/__init__.py`\n under an `XTUNER_DETERMINISTIC` gate. It must run **before any module\n imports fla** — fla's `@triton.autotune` decorators evaluate at import\n time, so patching after the fact has no effect on kernels already\n constructed. That's why the patch is on the `xtuner.v1.*` import chain\n rather than in the conftest: `multiprocessing.spawn` children of\n `MultiProcessTestCase` re-import the test class top-level (which pulls\n `xtuner.v1.model.*`) without running conftest, so the conftest alone\n can't install the patch in those processes.\n 2. **`torch.library.custom_op` wraps for the linear-attention path**\n (`GatedDeltaNet`, `causal_conv1d`) produce a different backward autograd\n graph from the canonical fla / causal_conv1d call HF uses, even though\n the underlying kernel and forward output are identical. `XTUNER_HF_IMPL`\n routes through `get_chunk_gated_delta_rule_fn` / `get_causal_conv1d_fn`\n (§6.2) so the parity path matches HF byte-for-byte.\n\n Production train doesn't set either env var; the custom_op wraps + autotune\n still operate. These switches are **parity-test only**. If the whole-model\n backward is not bitwise but the per-layer backward is, suspect one of the\n two before suspecting depth-accumulated bf16 noise.\n\n- **`flash`(XTuner varlen) vs `flash_attention_2`(HF) is bitwise only for short,\n single-tile sequences.** They are the same kernel, so at a short seqlen (e.g.\n 32) the logits are bitwise (0.0); at longer sequences the tiling /\n accumulation order differs and they drift ~1e-2. So a short-seq text test can\n assert flash-vs-FA2 bitwise, but anything realistic-length (or a VLM whose\n image tokens make the sequence long) needs the **eager** path for\n seqlen-independent bitwise. Don't conclude \"flash matches FA2\" from one short\n input.\n\n- **Shared kernels are already bitwise; focus on what isn't.** GatedDeltaNet /\n linear attention go through the same `fla` / `causal_conv1d` kernels as HF, and\n native rms_norm / rope match HF bitwise — these come out at 0.0 with no work.\n Spend your effort on the path that actually differs (usually full attention).\n\n- **`output_hidden_states` off-by-one (full-forward route, < 40B only).** HF's\n tuple is `[emb, L0_out, …, L{n-2}_out, post_norm]`: index `i+1` is layer `i`'s\n pre-norm output only for `i ≤ n-2`; the **last** entry is post-norm, not the\n last layer's pre-norm output. Compare the last layer via `model.norm(...)` or\n you will see the final layer \"explode\".\n\n- **A big max-diff is not automatically a bug.** Late layers have *massive\n activations* (a few channels in the tens/hundreds); a tiny relative error there\n shows up as a large absolute max-diff. Look at the mean and the trend, not just\n the max. Conversely, feeding `randn` inputs to a single attention layer is\n ill-conditioned and misleads — use real (or model-produced) hidden states.\n\n- **A single-model failure is often a general-path bug.** E.g. `Dense.build_layers`\n not forwarding `rms_norm_type` to `DenseDecoderLayer` broke any `zero_centered`\n dense model, not just this one. Fix the shared invariant, not the symptom.\n\n- **Loading a standalone sub-tower uses `strict=False`.** A text tower loaded\n directly from a full VLM checkpoint will see vision (and deferred `mtp.*`) keys\n as unexpected; load with `strict=False` so they are ignored.\n\n- **Test the whole VLM, not just the text tower.** Text-tower parity leaves the\n vision tower and projector completely untested. A compose port needs a\n full-model forward-parity test with real image inputs (§7.1). The text path of\n the compose forward still runs a *dummy* vision forward to keep the tower in the\n autograd graph, so vision bugs surface even on text prompts.\n\n- **Vision attention is non-causal — give `eager_attention` a `causal=False`\n path.** XTuner's `eager_attention` defaulted to a causal block-diagonal mask;\n that is wrong for a vision tower (bidirectional within each image), which is why\n forcing vision to the causal eager op produced large diffs. The op now takes\n `causal=True|False` (False → non-causal block-diagonal mask), the vision module\n already calls it with `causal=False`, and the vision tower routes through\n `get_attn_impl_fn` so `XTUNER_HF_IMPL` selects eager. With that, the *whole* VLM\n is bitwise in one run, everything eager: `XTUNER_HF_IMPL` on the XTuner side\n (text causal-eager + vision non-causal-eager) and HF loaded with a single\n `attn_implementation=\"eager\"`. Prefer eager for parity — it is\n seqlen-independent, unlike flash/FA2 (see the flash seqlen bullet above).\n\n- **Vision pos-embed dtype when reusing the Qwen3-VL tower.** HF's\n `fast_pos_embed_interpolate` returns fp32 (bilinear weights are fp32) and HF's\n vision forward casts it (`pos_embeds.to(hidden_states.dtype)`); XTuner's vision\n forward adds it without a cast. If you patch in the HF method for parity, cast\n the result back to the param dtype, or you get an fp32/bf16 `LayerNorm` mismatch\n (`expected scalar type Float but found BFloat16`).\n\n- **Hybrid-model `torch.compile`: choose `fullgraph` per layer — verify, then fall\n back; don't gate *whether* to compile.** When a decoder layer's forward is wrapped\n into a higher-order op (HOP) — e.g. by an activation-checkpoint wrapper, which\n `torch.compile` lowers to a HOP whose body must be functional so it can be\n recomputed in backward — *and* that layer is a linear-attention layer\n (GatedDeltaNet), the layer mutates an outer-scope variable inside the HOP\n (GatedDeltaNet writes back into `seq_ctx`). Under `fullgraph=True` that raises\n `HigherOrderOperator ... Mutating a variable not in the current scope (SideEffects)`.\n This is **not** a blanket \"linear attention ⇒ `fullgraph=False`\" rule — whether it\n triggers depends on whether the layer is actually HOP-wrapped and whether that\n linear impl really writes outer state. So for such a HOP + linear combination,\n **first test whether `fullgraph=True` runs**; keep it if it does (it compiles more\n of the layer), and only **fall back to `fullgraph=False`** if it doesn't —\n `fullgraph=False` survives by graph-breaking at the mutation and running that\n side-effecting write in eager. Make this a **per-layer** choice in\n `Dense.fully_shard` (decide each layer's `fullgraph`), not a decision about whether\n a layer is compiled at all.\n\n\ninterface:\n display_name: \"Add Chat Template\"\n short_description: \"Add and verify XTuner chat templates\"\n default_prompt: \"Use $add-chat-template to add or audit XTuner chat-template rendering, loss masks, and stop-token behavior for this official Hugging Face model.\"\n\n\n---\nname: add-chat-template\ndescription: Add or audit an XTuner chat-template implementation and its loss mask from an official Hugging Face model repository or local model directory. Use when supporting a new model under xtuner/v1/data_proto/messages, checking an existing implementation against tokenizer or processor apply_chat_template, deciding which assistant output and stop/EOS tokens receive loss, adding regression tests, or validating that exported training behavior matches vLLM and SGLang inference.\n---\n\n# Add Chat Template\n\n## Goal\n\nTreat the official Hugging Face tokenizer, or processor for a multimodal model,\nas the rendering oracle. Produce the same conversation token sequence in\nXTuner, supervise model-generated output by default, and supervise the\ncontext-appropriate token that ends each assistant generation. Prove the\nbehavior through XTuner's public tokenize API and, when available, the target\nvLLM and SGLang versions.\n\nKeep the implementation small and model-local. Do not infer a template or stop\ncontract from another model in the family.\n\n## 1. Establish the reference\n\n1. Record the official HF repo or local directory, immutable revision, tokenizer\n class, Transformers version, and whether `trust_remote_code=True` is required.\n2. Inspect the actual artifacts used by inference:\n `tokenizer_config.json`, `chat_template.jinja` or `chat_template`,\n `special_tokens_map.json`, `generation_config.json`, and `config.json`.\n Inspect `processor_config.json` and remote processor/template code when the\n official multimodal path uses `AutoProcessor.apply_chat_template`.\n3. Run the bundled inventory against the same revision:\n\n ```bash\n python .claude/skills/add-chat-template/scripts/audit_hf_chat_template.py \\\n --trust-remote-code\n ```\n\n Omit `--trust-remote-code` unless the official model requires it. For\n model-specific branches such as tools, thinking, or multimodal inputs, pass a\n JSON case file with `--cases `; see the script's `--help` output.\n The script inventories the tokenizer side; for a processor-owned multimodal\n template, run the corresponding `AutoProcessor` renderer separately.\n4. Render every supported branch with the same `tokenizer.apply_chat_template`\n or `processor.apply_chat_template` entry point used by the official example.\n For multimodal models, do not assume tokenizer-only rendering reproduces\n media placeholders or processor expansion. Record all required template\n kwargs and their defaults, including\n `add_generation_prompt`, `continue_final_message`, tool definitions,\n thinking/reasoning flags, named templates, and multimodal options. Do not\n copy rendered strings from a model card when the executable tokenizer is\n available.\n5. When the selected official template contains Jinja `{% generation %}`\n regions, call its `apply_chat_template` with `tokenize=True`,\n `return_dict=True`, and\n `return_assistant_tokens_mask=True`. Use its assistant mask as an additional\n oracle, then separately audit stop boundaries: a next-role stop token can sit\n outside the official generation region while still belonging to the previous\n assistant for SFT.\n6. Pin the reference revision in tests or CI configuration. A moving HF branch\n is not a stable oracle.\n\n## 2. Audit before implementing\n\nSearch these integration points first:\n\n- `xtuner/v1/data_proto/templates/__init__.py`\n- `xtuner/v1/data_proto/messages/`\n- `xtuner/v1/data_proto/messages/__init__.py`\n- `xtuner/v1/datasets/sft_tokenize_fn/openai.py`\n- multimodal tokenize functions when applicable\n- `tests/chat_template/` and `tests/datasets/`\n\nIf the model already exists, run its public tokenize path before changing it:\n\n```python\ntokenize_fn = OpenaiTokenizeFunctionConfig(chat_template=\"\").build(tokenizer)\nresult = tokenize_fn({\"messages\": messages, \"tools\": tools})\n```\n\nCompare the result with the official renderer and inspect existing tests. Find a\nconcrete failing message sequence before fixing a mismatch. Do not rewrite a\nworking implementation because its structure looks unusual.\n\nUse `HybridChatTemplate` plus `ChatMessages` only when fixed role wrappers fully\nexpress the official template. Add a dedicated\n`xtuner/v1/data_proto/messages/_chat.py` when rendering depends on\nmessage history, tools, reasoning, content types, or the next role. Follow\n`glm52_chat.py` and `qwen35_chat.py` as integration examples, not as rendering\nspecifications for another model.\n\n## 3. Write the stop contract first\n\nInventory three distinct sources; do not collapse them into one `eos_token`:\n\n1. **Configured generation stops**: `tokenizer.eos_token_id`, `config.json`,\n `generation_config.json`, including integer lists and stop strings.\n2. **Template boundaries**: the exact token or token sequence following an\n assistant when the next item is user, tool/observation, system/developer, or\n the end of the sample.\n3. **Engine stops**: the IDs and strings actually installed by the target vLLM\n and SGLang versions after loading the exported checkpoint.\n\nCreate an explicit table before writing the mask:\n\n| Transition | Official serialized boundary | Token IDs | Engine mechanism | Loss when assistant loss is true |\n|---|---|---|---|---|\n| assistant → user | exact role/EOT sequence | exact IDs | EOS ID, stop ID/string, or parser | yes for the generated stop target |\n| assistant tool call → tool | exact observation boundary | exact IDs | EOS ID, stop ID/string, or parser | yes for the generated stop target |\n| final assistant → end | exact EOT/EOS, or documented training-only EOS | exact IDs | EOS ID or stop ID/string | yes |\n\nVerify each token by both encoding the exact boundary and converting its ID back\nto a token. Reject unknown-token conversions. Handle a multi-token stop as a\nsequence rather than assuming every stop is one special token.\n\n### Role tokens can belong to the preceding assistant\n\nSome official templates have no dedicated per-turn EOT. The next role token is\nthen the token the model must generate to stop the previous assistant. In that\ncase, keep the official serialized order and assign loss only on that boundary\ntoken or sequence to the preceding assistant:\n\n```text\nassistant answer user text\n ^ assistant loss ^ masked\n```\n\nFor a GLM-like protocol this can mean supervising the user boundary after a\nnormal answer, the observation boundary after a tool call, and the configured\nterminal EOS only when no following role supplies a boundary.\n\nDo not append a generic EOS after every assistant when the official inference\ntemplate places a role/observation boundary there. That creates training history\nthat vLLM and SGLang do not render. Conversely, when the official full-message\nrenderer omits the final token that the model must generate to terminate,\nappend exactly that one terminal token for SFT and document this single\ntraining-only suffix.\n\nConfigured EOS IDs are alternatives, not a sequence to append together.\nSupervise the actual context-specific stop token present in each training path.\n\n## 4. Implement rendering and loss together\n\nPrefer one cohesive renderer that returns rendered text plus a character- or\ntoken-level loss mask. Keep context-dependent boundary ownership beside the\nrendering branch that emits the boundary; avoid a collection of shallow helper\nfunctions.\n\nApply these defaults:\n\n- Treat `assistant` messages with no `loss` field as `loss=True`.\n- Mask system/developer/user/tool inputs and role scaffolding unless the stop\n contract proves that a boundary is generated by the preceding assistant.\n- Supervise all model-generated assistant components: visible content,\n reasoning that remains in the rendered sample, reasoning closing syntax,\n tool calls, and the appropriate stop target.\n- Treat `loss=False` as masking the entire assistant output and its stop target.\n- Mask `add_generation_prompt` completely; it is prompt scaffolding, not a\n demonstrated model output.\n- Preserve explicit model behavior such as clearing historical reasoning only\n when the official template does so for the chosen kwargs.\n\nIf character offsets are used, verify special-token and Unicode boundaries with\nthe real tokenizer. A token overlapping a loss span needs a deliberate,\ntested rule. Keep a slow, independent token-level oracle for regression tests;\ndo not call the production renderer from both sides of the same parity test.\n\nMaintain exact rendering parity before any documented training suffix:\n\n```mermaid\nflowchart LR\n A[\"Official HF messages\"] --> B[\"tokenizer.apply_chat_template\"]\n A --> C[\"XTuner public tokenize API\"]\n B --> D[\"Exact text and token IDs\"]\n C --> D\n D --> E[\"Optional one terminal SFT stop\"]\n E --> F[\"Loss-mask contract\"]\n```\n\nDo not use decoded-text equality as the sole check: decoding can hide a token-ID\nmismatch.\n\n## 5. Register the smallest integration\n\nFor a dedicated message implementation, update only the required seams:\n\n1. Export the message class from `xtuner/v1/data_proto/messages/__init__.py`.\n2. Add the public name and serving stop metadata to `CHAT_TEMPLATE_MAP`.\n3. Dispatch that name in `OpenaiTokenizeFunction`; update multimodal dispatch\n only if the model uses it.\n4. Update typed CLI/config literals only where they restrict the new name.\n\nKeep `CHAT_TEMPLATE_MAP.stop_words` consistent with the discovered contract,\nbut do not treat it as proof of engine behavior. vLLM and SGLang load their\nruntime stops from the exported HF artifacts and request/engine configuration.\n\n## 6. Add good regression tests\n\nUse the real official tokenizer/processor and XTuner's public tokenize\nfunction. Mock only unavailable external services. Cover every supported\ntemplate branch with the smallest useful matrix:\n\n- user → assistant, with `loss` omitted;\n- assistant → user in a multi-turn conversation;\n- assistant tool call → tool/observation → assistant;\n- final assistant;\n- `loss=False` followed by another role and at sample end;\n- system/developer input;\n- `add_generation_prompt=True`;\n- reasoning enabled/disabled and historical reasoning behavior;\n- tools and multimodal content when supported;\n- repeated assistant text, Unicode, and adjacent special tokens to expose\n faulty substring or offset alignment.\n\nAssert public behavior:\n\n1. Official tokenizer/processor rendered text equals XTuner rendered text,\n except for the one explicitly documented final SFT suffix if required.\n2. Official token IDs equal XTuner `input_ids` before that suffix.\n3. `len(input_ids) == len(labels)` and every label is either the matching input\n ID or `IGNORE_INDEX`.\n4. Assistant output receives loss by default; `loss=False` removes all of it.\n5. Each transition's exact stop token IDs receive or do not receive loss as the\n stop table specifies.\n6. User/tool content and generation prompts remain masked.\n7. The fast implementation matches an independent slow oracle.\n\nTest token positions and IDs directly. A substring assertion can supplement but\nmust not replace the loss assertion. Avoid snapshots that pass after both the\nrenderer and expected string are changed to the same incorrect value.\n\n## 7. Validate vLLM and SGLang\n\nResolve exact target versions. Prefer the user's production versions; otherwise\ninspect the current installed versions and latest stable official source. For\neach engine:\n\n1. Load the same exported tokenizer, `config.json`, and\n `generation_config.json`.\n2. Compare the engine's chat-render/tokenize output with the official\n tokenizer/processor renderer for every stop-table transition.\n3. Inspect or log the final stop token IDs/strings installed in the generation\n request. Distinguish tokenizer EOS, model generation-config EOS, request stop\n strings, and model-specific parsers.\n4. Run generation smoke tests for a normal final answer and a tool call. Use a\n sufficiently high output limit and assert the engine reports a stop rather\n than a length limit, with no role leakage.\n\nObtain the repository GPU lock before a local GPU server or generation test. If\nthe exact runtime cannot be executed, audit the exact tagged engine source and\nreport the result as a static source audit, not runtime compatibility.\n\nSwitching from Chat Completions to Responses does not remove this requirement:\nfor ordinary HF models both interfaces ultimately render model messages through\nthe chat template.\n\n## 8. Completion checklist\n\nFinish only when the report includes:\n\n- official HF repo, revision, and Transformers version;\n- implementation path and whether it is generic or dedicated;\n- rendering-parity matrix;\n- stop-contract table with token strings, IDs, and loss ownership;\n- tests proving default assistant loss and `loss=False`;\n- vLLM/SGLang versions and runtime or exact-tag source validation;\n- every intentional difference from the official renderer, normally at most a\n final training stop suffix.\n\nIf auditing an existing implementation, state which requirements already had\ntests, which were missing, and the concrete mismatches found. Do not claim the\nimplementation is correct merely because a test file exists.\n\n\ninterface:\n display_name: \"Check HF Config Save\"\n short_description: \"Implement and validate exported HF configs\"\n default_prompt: \"Use $check-hf-config-save to implement any missing official Hugging Face config export and validate it across Transformers and inference engines.\"\n\n\n---\nname: check-hf-config-save\ndescription: Implement a missing XTuner Hugging Face config export from the official HF model when possible, then validate it against the installed Transformers round-trip and versioned inference-engine field contracts. Use when adding a model, when hf_config is missing or returns None, when changing from_hf/hf_config/save_hf, when reviewing config.json differences, or when debugging vLLM/SGLang checkpoint-load or inference failures. Also use for requests named check_hf_config_save.\n---\n\n# Check HF Config Save\n\n## Overview\n\nFirst ensure that a model with an official built-in Transformers config has an\ninverse `from_hf <-> hf_config` mapping. If that export is missing, implement it\nfrom the official HF model before using `xtuner._testing.check_hf_config_save`\nto test the real public `from_hf -> save_hf` path. Separate changes forced by\nthe installed Transformers version from fields dropped by XTuner, then protect\nfields that exact inference engine versions use for model construction or\nweight loading.\n\n## Workflow\n\n### 1. Fix the scope and version matrix\n\n1. Identify the source HF model directory, XTuner config class, export path, and\n any engine/version named by the user or failure log.\n2. Record the executable environment versions for Python, Transformers, and any\n installed engines. Use the requested environment; otherwise follow the repo's\n default environment instructions.\n3. A user-specified or production-log version takes precedence. If a component\n version is unspecified, look up the latest stable release from its official\n PyPI project/API or official release page and inspect the matching source tag.\n By default audit both vLLM and SGLang.\n4. List every checked version in the result. Distinguish an installed runtime\n test from a static audit of an exact source tag.\n\nUse a table with at least these columns:\n\n| Component | Version | How selected | Validation |\n|---|---:|---|---|\n| Transformers | exact version | active environment | executable round-trip |\n| vLLM | exact version | user/log/latest official | runtime or exact-tag source audit |\n| SGLang | exact version | user/log/latest official | runtime or exact-tag source audit |\n\nDo not write `latest` without resolving it to an exact version and source.\n\n### 2. Implement a missing HF config export\n\nExercise the supported model's public conversion path first:\n\n```python\nconfig = get_model_config_from_hf(SOURCE_HF_DIR)\nexported_config = config.hf_config\n```\n\n`exported_config is None`, or the corresponding public `config.save_hf(...)`\nraising the base missing-`hf_config` `NotImplementedError`, means the config-only\nexport is not implemented. Classify the official model before changing code:\n\n```mermaid\nflowchart TD\n A[\"Official HF model and exact revision\"] --> B{\"Built-in official Transformers config?\"}\n B -->|Yes| C{\"XTuner hf_config implemented?\"}\n C -->|No| D[\"Implement inverse from_hf and hf_config mapping\"]\n C -->|Yes| E[\"Continue round-trip validation\"]\n D --> E\n B -->|No; trust_remote_code only| F[\"Keep hf_config as None and validate source-config copying\"]\n```\n\nFor a built-in official Transformers config:\n\n1. Read the official checkpoint's raw `config.json` and resolve its exact\n `model_type`, `architectures`, repository revision, and official config\n implementation. Load it with the selected Transformers version and record\n the concrete `PretrainedConfig` subclass. Do not use a third-party model\n implementation as the reference.\n2. Implement or complete `from_hf` with that official class. Map every\n architecture value needed by XTuner, use `getattr` for genuinely optional\n versioned fields, and use `RopeParametersConfig.from_hf_config` for RoPE.\n3. Implement `hf_config` by constructing the same official config class from\n the XTuner config's current values. It must be the inverse of `from_hf`, not\n a cached copy of the source object. Re-emit every field consumed by\n `from_hf`, plus raw compatibility fields required by vLLM or SGLang even when\n Transformers treats them as optional or legacy.\n4. Preserve the official `model_type` and architecture name. Do not substitute\n a generic `PretrainedConfig`, duplicate the official config class in XTuner,\n or return a hand-written dictionary.\n5. Verify through public behavior that `config.hf_config` has the expected\n official type and that `config.save_hf(...)` succeeds, then continue with\n the helper below.\n\nIf the official repository only works through `trust_remote_code` and the\nselected Transformers versions have no built-in config class, do not invent an\n`hf_config` implementation. Keep `hf_config=None`, retain the source `_hf_path`,\nand test the model's public HF save path that copies the original config,\ntokenizer, and required remote-code files. If the entire model or dispatch is\nnot yet supported by XTuner, follow `$add_hf_model`; this skill only fills a\nmissing exporter for an otherwise supported model.\n\n### 3. Establish the Transformers reference\n\nCompare three raw JSON states:\n\n```mermaid\nflowchart LR\n A[\"Source config.json\"] --> B[\"Current Transformers load + save\"]\n A --> C[\"XTuner from_hf + save_hf\"]\n B --> D[\"Expected serialized reference\"]\n C --> E[\"XTuner export\"]\n D --> F[\"Public helper comparison\"]\n E --> F\n```\n\n- Read the source `config.json` before `AutoConfig` normalization.\n- Load and save it directly with the active Transformers version. This is the\n serialized reference and captures forced defaults or `__post_init__` changes.\n- Build through XTuner's public `get_model_config_from_hf`/`from_hf` API and call\n the public `save_hf` API.\n- Compare the Transformers reference with the XTuner export. Do not use direct\n source-versus-export equality as the primary assertion: it misclassifies\n Transformers normalization as an XTuner bug.\n\nInspect the exact Transformers config implementation, including generated or\nmodular source and `__post_init__`, for every source-to-reference difference.\nTypical examples are derived `head_dim`, generated `layer_types`, new defaults,\nand serializer metadata, but never assume these examples cover a new model.\n\n### 4. Audit inference-engine dependencies\n\nFor every changed, missing, newly defaulted, or architecture-selecting field:\n\n1. Search the exact engine tag, not an arbitrary installed or main-branch copy.\n2. Follow model registration, config access, module construction, and the weight\n loader. Search aliases, `getattr` defaults, direct attribute access, and tensor\n name/shape conditions.\n3. Classify the use as one of:\n - module/parameter registration;\n - checkpoint key or tensor-shape selection;\n - layer topology or MoE routing;\n - attention/RoPE behavior;\n - ignored or default-compatible.\n4. Encode every value-sensitive dependency as `HFConfigFieldDependency`, with\n exact engine version, JSON pointer, expected value, reason, and an official\n source permalink.\n\nA field can be HF-equivalent yet engine-critical. For example, an engine may\nregister a checkpoint parameter only when a legacy routing field has one exact\nvalue; omitting that field then becomes a real weight-loader failure.\n\nStatic source inspection proves the dependency but not end-to-end engine\ncompatibility. Run an engine checkpoint-load smoke test when the exact runtime\nand required hardware are available. Otherwise report `exact-tag source audit`\nand do not claim runtime success. Follow the repo GPU-lock instructions before\nany local GPU run.\n\n### 5. Add the model regression test\n\nDelete narrow hand-written assertions that duplicate this contract, then add a\nmodel test on the real public conversion path:\n\n```python\nimport transformers\n\nfrom xtuner._testing import HFConfigFieldDependency, check_hf_config_save\nfrom xtuner.v1.model import get_model_config_from_hf\n\n\ndef test_save_hf_matches_transformers_and_engine_contracts():\n config = get_model_config_from_hf(SOURCE_HF_DIR)\n assert isinstance(config.hf_config, OFFICIAL_HF_CONFIG_CLASS)\n report = check_hf_config_save(\n config,\n SOURCE_HF_DIR,\n engine_dependencies=(\n HFConfigFieldDependency(\n engine=\"vllm\",\n version=\"\",\n path=\"/\",\n expected=\"\",\n reason=\"\",\n source=\"\",\n ),\n ),\n )\n\n assert report.transformers_version == transformers.__version__\n assert report.checked_engine_versions == (\"vllm==\",)\n```\n\nThe helper performs two independent checks:\n\n- XTuner export matches the active Transformers direct round-trip.\n- Exported values satisfy the declared inference-engine contracts, even when\n Transformers itself does not consume those fields.\n\nUse `allowed_export_differences={\"/json/pointer\": \"specific reason\"}` only for\nan intentional XTuner difference that is not already an engine dependency.\nEvery exception needs a non-empty model-level reason.\n\n### 6. Prove the regression and run the matrix\n\n1. Run the helper's own behavior tests.\n2. Run the generated model test in the project's pinned Transformers version.\n3. Run it in each user-requested/current upgraded Transformers environment.\n4. If the old broken export is available, pass its `config.json` through the\n same helper and show that the expected missing/extra paths fail. This is the\n minimal proof that the test catches the original bug.\n5. Run formatting and the smallest relevant model test suite.\n\nReport:\n\n- whether `hf_config` already existed, was implemented from which official\n config class/revision, or correctly remained `None` for remote code;\n- source-to-Transformers normalization paths;\n- Transformers-reference-to-XTuner paths (normally none, apart from documented\n allowed contracts);\n- engine dependency, expected value, exact version, and effect;\n- which checks were executable and which were source-only.\n\n## Guardrails\n\n- Compare raw JSON, not only `AutoConfig` attributes; unknown compatibility\n fields may disappear from a newer config class while engines still read them.\n- Exercise public APIs and real conversion behavior. Do not mock XTuner model\n internals.\n- Do not replace the helper with a blanket list of expected keys.\n- Do not silently refresh an engine version in a test. Re-audit its exact source\n before updating the version and permalink.\n- HF semantic equivalence is not evidence of vLLM/SGLang compatibility.\n- Do not set `hf_config=None` for a model with a built-in official config merely\n to bypass config reconstruction or round-trip failures.\n- Preserve unrelated worktree changes and keep the implementation model-agnostic.\n\n## Completion criteria\n\nThe work is complete only when the official config export path exists where it\nshould, the public round-trip matches Transformers, every audited engine\ncontract is preserved, and the report identifies the exact official model\nrevision and component versions used.\n\n\n---\nname: model_normalize\ndescription: Normalize an HF model checkpoint — quantize bf16/fp16 weights to per-block FP8, and/or repack engine-named safetensors into HF-standard ~4GB shards.\n---\n\n# model_normalize\n\nTwo complementary capabilities for getting a training-engine checkpoint into a\nformat that HF / vLLM / SGLang can consume:\n\n1. **FP8 quantization** of large linears, in two modes:\n - **Reference-guided** — point at an existing FP8 checkpoint of the same\n architecture; the tool reads its index and reproduces the exact same\n quantization decisions on the source.\n - **Heuristic** — apply a built-in rule set that quantizes only the large\n matrix multiplications (attention, MLP, MoE experts) and keeps norms,\n routers, biases, embeddings, and vision tower in their original dtype.\n2. **Standard repack** — re-shard arbitrary safetensors directories into the\n HF-standard ``model-{i:05d}-of-{n:05d}.safetensors`` layout with ~4GB\n shards and a fresh ``model.safetensors.index.json``.\n\nWhen ``to-fp8`` detects non-standard source shard names, it automatically\nruns the repack step on the quantized output, so the user-visible result is\nalways a clean HF-conformant directory.\n\n## Layout\n\n```\n.claude/skills/model_normalize/\n├── SKILL.md # this file\n├── model_normalize.py # top-level CLI (to-fp8 / repack)\n├── hf_to_fp8.py # FP8 conversion library + standalone CLI\n├── heuristics.py # codified default quantization rule set\n├── repack_hf.py # 4GB-shard repacker library + CLI\n└── convert_interns2_to_fp8.sh # example invocation for InternS2\n```\n\n## Usage\n\n### Mode 1 — Reference-guided FP8 conversion\n\n```bash\npython model_normalize.py to-fp8 \\\n --source /path/to/bf16-model \\\n --output /path/to/fp8-model \\\n --reference /path/to/fp8-reference\n```\n\nBehavior:\n- Loads the reference's ``model.safetensors.index.json``.\n- For each ``_scale_inv`` entry in the reference index, marks ````\n as an FP8-quantized tensor.\n- Applies that exact set to the source as a literal name match (no regex\n surprise; works even when names don't follow any pattern).\n\n### Mode 2 — Heuristic FP8 conversion (no reference)\n\n```bash\npython model_normalize.py to-fp8 \\\n --source /path/to/bf16-model \\\n --output /path/to/fp8-model\n```\n\nThe rule set is defined in ``heuristics.py``. Concretely, **quantize iff** the\ntensor name matches one of:\n\n| Group | Pattern (anchored) |\n| -------------------------------- | ------------------------------------------------------------------------------------------------------------------- |\n| Self-attention projections | ``[…]layers..self_attn.{q,k,v,o}_proj.weight`` |\n| Dense MLP | ``[…]layers..mlp.{gate,up,down}_proj.weight`` |\n| MoE per-expert (unfused) | ``[…]layers..mlp.experts..{gate,up,down}_proj.weight`` |\n| MoE fused-expert (3D tensors) | ``[…]layers..mlp.experts.{gate_up_proj, down_proj}`` |\n| MoE shared expert | ``[…]layers..mlp.shared_expert.{gate,up,down}_proj.weight`` |\n| Linear-attn wide projections | ``[…]layers..linear_attn.{in_proj_qkv, in_proj_z, out_proj}.weight`` |\n\nThe ``[…]`` prefix allows ``model.`` and ``model.language_model.``; the same\npatterns repeat under ``mtp.layers..`` for multi-token-prediction blocks.\n\n**Everything else is kept in the original dtype**, in particular:\n- all norms (``*norm*``, ``*_layernorm.*``)\n- MoE routers (``mlp.gate.weight``, ``mlp.shared_expert_gate.weight``)\n- embeddings (``embed_tokens``, ``lm_head``, ``pos_embed``, ``patch_embed``, ``mtp.fc``)\n- vision tower (``model.visual.*``)\n- all ``*.bias``\n- ``linear_attn`` control-flow tensors (``A_log``, ``conv1d.weight``, ``dt_bias``, ``in_proj_a/b.weight``, ``norm.weight``)\n\nThis rule set was validated against the InternS2 / Qwen3-MoE FP8 references:\nit reproduces their quantization decisions on every tensor (no false\npositives or false negatives).\n\n### Mode 3 — Standalone repack (no quantization)\n\n```bash\npython model_normalize.py repack \\\n --source /path/to/messy-shards \\\n --output /path/to/standard-shards \\\n --shard-size-gb 4\n```\n\nGreedy bin-packs tensors into ``model-{i:05d}-of-{n:05d}.safetensors`` shards\nof at most ``--shard-size-gb`` GiB, writes a fresh\n``model.safetensors.index.json`` (including ``metadata.total_size``), and\ncopies all non-safetensors files (config, tokenizer, etc.) through.\n\n## When to use which mode\n\n| Situation | Mode |\n| ---------------------------------------------------------------------------- | ------------------ |\n| You already have an FP8 reference for this architecture. | Reference-guided |\n| New architecture / first-time conversion; standard naming. | Heuristic |\n| Quantized model has engine-specific shard names; need HF-conformant output. | (auto, see below) |\n| You only need to standardize shard naming; weights are already FP8 / bf16. | ``repack`` |\n\nThe ``to-fp8`` subcommand auto-detects non-standard source shard names (anything\nthat isn't ``model.safetensors`` or ``model-NNNNN-of-NNNNN.safetensors``) and\nchains the repack step. Pass ``--no-repack`` to keep the source naming.\n\n## Quantization details\n\n- Block size **128 × 128**, dtype ``float8_e4m3fn`` with saturating cast.\n- Per-block scale stored as ``_scale_inv`` (``float32``) in the same\n shard as the parent weight. Layout matches what vLLM / SGLang load for\n per-block FP8.\n- 3D MoE-expert tensors (shape ``(E, D0, D1)``) are quantized per-expert; the\n per-expert scales are stacked along a leading dim.\n- ``config.json`` receives a ``quantization_config`` block with\n ``quant_method=fp8``, ``weight_block_size=[128, 128]``,\n ``scale_fmt=ue8m0``, and an explicit ``modules_to_not_convert`` list\n derived from the actual quantization decisions.\n\n## Post-conversion handoff (mandatory for the agent)\n\nAfter **any** successful run of `to-fp8` or `repack`, the agent must ask the\nuser — via `AskUserQuestion` — where to **place** the produced\n`*.safetensors` and `model.safetensors.index.json` files. The conversion\noutput is treated as a staging area; the user decides the final home.\n\nOffer at least these options:\n1. **Keep in place** — leave the files at the `--output` path as-is.\n2. **Copy to a new directory** — let the user specify a target; copy the\n shards + index there. Always **copy** (`cp`), never move — the\n `--output` staging directory must remain intact unless the user\n explicitly asks to delete it as a separate step.\n3. **Copy into an existing model directory** — copy shards/index into a\n user-specified directory while preserving its other files (tokenizer,\n config, etc.).\n\nIf the user picks (2) or (3), execute the copy with `cp` (confirming the\ndestination path back to the user before any overwrite). Do not infer the\ndestination from earlier turns — the user may have a fresh target in mind\neach time. Cleaning up the staging directory afterwards is a separate,\nuser-initiated step — never delete it as part of the handoff.\n\n## Example: InternS2 Preview\n\n```bash\npython model_normalize.py to-fp8 \\\n --source /mnt/shared-storage-user/puyudelivery/user/yehaochen/models/InternS2PreviewCandidate5 \\\n --output /tmp/InternS2-FP8 \\\n --reference /mnt/shared-storage-user/puyudelivery/user/yehaochen/models/InternS2PreviewCandidate5-FP8\n```\n\nThe source uses engine-specific names like\n``model-language-0001-fused-save_rank0.safetensors``; the output will be\nrepacked into the standard ``model-00001-of-NNNNN.safetensors`` layout.\n\n\n---\nname: model_patch\ndescription: Merge a base HF model dir with an extra HF model dir by index diff — take tensors only present in extra, append them to a new output dir in HF-standard shard layout. Base wins on overlap.\n---\n\n# model_patch\n\nProduce ``out = base ∪ (extra \\ base)`` at the **tensor level**, by diffing the\ntwo ``model.safetensors.index.json`` files. Useful when ``extra`` is a sibling\ncheckpoint (different training run, fine-tune, or fp8 conversion) that holds a\nfew tensors the base is missing — e.g. an updated head, a newly-added MTP\nblock, or auxiliary projections — and you want a self-contained merged dir\nwithout re-running the whole conversion pipeline.\n\nOverlap rule: **base wins**. Any tensor present in both indexes is taken from\n``base``; the corresponding entry in ``extra`` is ignored.\n\n## Layout\n\n```\n.claude/skills/model_patch/\n├── SKILL.md # this file\n└── model_patch.py # CLI + library\n```\n\n## Usage\n\n```bash\npython model_patch.py \\\n --base /path/to/base-model \\\n --extra /path/to/extra-model \\\n --out /path/to/merged-model\n```\n\n## What it writes\n\n- ``model-{i:05d}-of-{N:05d}.safetensors`` shards (total ``N`` = base shards +\n newly-packed shards holding the extra-only tensors)\n- a fresh ``model.safetensors.index.json``\n- **every non-tensor file from ``base``** — ``config.json``, tokenizer files,\n modeling/configuration ``.py``, ``generation_config.json``, README, etc. —\n copied verbatim so ``out`` is loadable on its own. Subdirectories are\n copied recursively. Existing entries in ``out`` are overwritten.\n\n``extra``'s aux files are deliberately **not** copied. The skill's overlap\nrule is \"base wins\" at the tensor level, and that extends to configs: extra\noften carries engine-specific edits or modality keys that don't match the\ntensor topology we keep. If you need fields from ``extra``'s ``config.json``\n(e.g. a new modality), merge them into ``out/config.json`` as a separate,\nexplicit step.\n\n## How shards are produced\n\n- **Base shards**: re-emitted under the new ``-of-N`` total. By default the\n bytes are fully copied so ``out`` is independent of ``base``. Pass\n ``--hardlink`` to use ``os.link`` instead (fast, but the two trees share\n inodes — editing one mutates the other; deleting one keeps the other\n intact since hardlinks are symmetric).\n- **Extra-only tensors**: greedily bin-packed into new shards of at most\n ``--shard-size-gb`` GiB (default 4), appended after the base shards.\n\nSo a base with 12 shards plus 5 GB of extra-only tensors yields:\n``model-00001-of-00014.safetensors`` … ``model-00012-of-00014.safetensors``\n(copies of the base shards) and\n``model-00013-of-00014.safetensors`` / ``model-00014-of-00014.safetensors``\n(freshly written, containing only the extra tensors).\n\n## When to use this skill\n\n| Situation | Use this? |\n| -------------------------------------------------------------------------------------- | --------- |\n| ``extra`` contains a few tensors ``base`` is missing; everything else is identical. | Yes |\n| You need to merge two checkpoints whose overlapping tensors **differ** in value. | No — this skill silently keeps base; you probably want a manual decision per key. |\n| ``extra`` is a full quantization of ``base`` and you want the quantized weights. | No — use ``model_normalize`` with ``--reference extra`` instead. |\n| You want a single ``model.safetensors`` instead of standard shards. | No — repack downstream. |\n\n## Required usage protocol (mandatory for the agent)\n\nBefore invoking ``model_patch.py``:\n\n1. Confirm the three paths (``--base``, ``--extra``, ``--out``) with the user\n via ``AskUserQuestion``. Never infer them from earlier conversation.\n2. Show the user the **diff summary** before writing, by reading the two\n ``model.safetensors.index.json`` files and reporting:\n - number of tensors in base\n - number of tensors in extra\n - number of extra-only keys that will actually be added\n - number of overlapping keys that will be dropped from extra\n3. Ask the user to confirm those numbers before launching the run.\n\nAfter a successful run, **do not** ask the user where the output should go —\n``--out`` is already the final destination the user named. Just report what\nlanded there (shard count, index path, total size, and that base's aux files\nwere mirrored over). Unlike ``model_normalize``, ``model_patch`` has no\nstaging-area concept: there is no temp dir, and the caller already made the\nplacement decision when they passed ``--out``.\n\nIf the user wanted any of ``extra``'s aux files (e.g. a ``config.json`` that\ndocuments a new modality), call that out — those are not copied by the\nskill and must be merged in by hand.\n\n\n---\nname: sphinx-debug\ndescription: >\n Use when `make html` or `sphinx-build` fails with\n `Extension error (sphinx.ext.autosummary)`,\n `ImportExceptionGroup`, or other Sphinx autosummary / autodoc\n import failures. Provides automated pdb diagnosis for\n `autodoc_mock_imports` issues.\n---\n\n# Sphinx Debug\n\n## Quick Diagnostic Command\n\nRun the build with maximum verbosity and pdb on exception:\n\n```bash\nmake html SPHINXOPTS=\"-vv -T -P\"\n```\n\nFlags:\n- `-vv`: Verbose autodoc mock logging\n- `-T`: Full traceback\n- `-P`: Drop into pdb on exception\n\n## For agents: automate pdb via monkey-patch\n\nIf you are an agent and cannot interact with an interactive pdb session,\n**do not redesign a subprocess/pty capture system**. Instead, monkey-patch\n`pdb.post_mortem` so it runs scripted commands and prints the output\ndirectly to stdout.\n\nRun this one-shot script from the docs directory (`docs/en` or `docs/zh_cn`):\n\n```bash\ncd docs/en # or docs/zh_cn\npython -c \"\nimport pdb, sys, io, traceback\n\n# Hijack pdb.post_mortem to auto-run diagnostic commands\n_original_post_mortem = pdb.post_mortem\n\ndef scripted_post_mortem(tb=None):\n out = io.StringIO()\n p = pdb.Pdb(stdout=out, stdin=io.StringIO(''))\n p.use_rawinput = False\n p.reset()\n if tb is None:\n tb = sys.exc_info()[2]\n p.setup(None, tb)\n\n old_stderr = sys.stderr\n sys.stderr = out\n\n # List all inner exceptions\n p.onecmd('for i, exc in enumerate(exceptions): print(f\\\"[{i}] {type(exc).__name__}: {exc}\\\")')\n # Print full traceback of the first TypeError (the real root cause)\n p.onecmd('type_errs = [exc for exc in exceptions if type(exc).__name__ == \\\"TypeError\\\"]')\n p.onecmd('if type_errs: import traceback; traceback.print_exception(type(type_errs[0]), type_errs[0], type_errs[0].__traceback__)')\n p.do_quit('')\n\n sys.stderr = old_stderr\n print('=== PDB AUTOMATED OUTPUT ===')\n print(out.getvalue())\n print('=== END OUTPUT ===')\n\npdb.post_mortem = scripted_post_mortem\n\nfrom sphinx.cmd.build import build_main\nbuild_main(['-b', 'html', '-vv', '-T', '-P', '.', '_build/html'])\n\"\n```\n\nThis runs the real Sphinx build with `-P`, but when pdb drops in it\nautomatically executes your commands and dumps the results to stdout.\nYou read the output exactly as if you had typed the pdb commands yourself.\n\n## What to look for in the automated output\n\nSphinx autosummary raises `ImportExceptionGroup` when it cannot resolve a module.\nThe **top-level message is often misleading** (e.g. `no module named xtuner.v1.ray.dataflow`).\nThe real root cause is hidden inside the grouped exceptions.\n\n### Common root-cause patterns\n\n1. **TypeError about `__version__`**\n ```\n TypeError: expected string or bytes-like object, got '__version__'\n ```\n This happens when a mocked module (e.g. `torch`) is imported for real by an\n intermediate library (e.g. `fla`) that calls `packaging.version.parse()` on\n the mocked `__version__` object.\n\n **Fix**: Add the intermediate library to `autodoc_mock_imports` in `conf.py`.\n To identify the exact library, read the traceback: the frame just above\n `packaging/version.py` is the culprit (e.g. `fla/utils.py`).\n\n ```python\n autodoc_mock_imports = [\n ...\n \"fla\", # or whatever library triggers the real import\n ]\n ```\n\n2. **Missing-comma string-concatenation bug**\n\n If `autodoc_mock_imports` contains an entry like `\"scipytorchvision\"`,\n a comma is missing between two string literals in `conf.py`:\n\n ```python\n # WRONG\n autodoc_mock_imports = [\n ...\n \"scipy\"\n \"torchvision\",\n ...\n ]\n\n # CORRECT\n autodoc_mock_imports = [\n ...\n \"scipy\",\n \"torchvision\",\n ...\n ]\n ```\n\n3. **AttributeError after the TypeErrors**\n\n ```\n AttributeError: module 'xtuner.v1.ray' has no attribute 'dataflow'\n ```\n This is usually a *secondary* failure caused by the earlier `TypeError`s.\n Fix the `TypeError` first, then rebuild.\n\n## Step-by-step workflow\n\n1. Run the automated pdb script (see \"For agents\" section above).\n2. Read the `=== PDB AUTOMATED OUTPUT ===` block.\n3. If you see `TypeError: expected string or bytes-like object, got '__version__'`,\n read the traceback to find the intermediate library name.\n4. Add the identified library to `autodoc_mock_imports` in **both**\n `docs/en/conf.py` and `docs/zh_cn/conf.py`.\n5. Also eyeball `autodoc_mock_imports` for missing commas while you are there.\n6. Re-run `make html` (without `-P`) to verify the fix.\n\n\n---\nname: xtuner-custom-judger\ndescription: Use when creating, modifying, or reviewing XTuner RL judgers under xtuner/v1/rl/judger or configs that build custom judgers. Enforces the preferred Judger payload contract and the BaseJudger/ComposedJudger constraints.\n---\n\n# XTuner Custom Judger\n\nUse this skill whenever you implement or review an XTuner RL judger.\n\n## Preferred Pattern\n\nPrefer subclassing `Judger`, not `BaseJudger`.\n\nFor custom scoring logic, override these methods as needed:\n\n```python\nclass MyJudger(Judger):\n def preprocess(self, rollout_state: RolloutState) -> JudgerPayload:\n ...\n\n async def judge_payload(self, payload: JudgerPayloadBatch) -> JudgerOutputBatch:\n ...\n\n def postprocess(self, rollout_state: RolloutState, output: JudgerOutput) -> RolloutState:\n ...\n```\n\nDo not override `Judger.judge()` just to read extra fields from `RolloutState`.\nPut the field extraction in `preprocess()` instead.\n\n## Why This Contract Exists\n\n`preprocess -> judge_payload -> postprocess` keeps the scoring payload small and\ncomposable:\n\n- `preprocess()` extracts only the fields required for scoring.\n- `judge_payload()` can run locally, in a Ray actor, or behind a judger pool.\n- `postprocess()` writes the output back to the original `RolloutState`.\n\nThis avoids `deepcopy(RolloutState)` in composed and remote judging. Deep-copying\nrollout states can add serialization overhead and can be risky for large fields,\nRay object refs, tensors, numpy arrays, or externally owned objects.\n\n## BaseJudger Rule\n\nSubclass `BaseJudger` only when the implementation must own the full\n`judge()` / `batch_judge()` flow and cannot reasonably be expressed as a\npayload contract.\n\nBaseJudger-only judgers cannot be used as `ComposedJudger` branches.\n`ComposedJudger` intentionally calls branch `preprocess()` and\n`judge_payload()` rather than arbitrary `BaseJudger.judge()` implementations.\nBecause it does not deep-copy `RolloutState`, supporting arbitrary\nBaseJudger-only branches would allow multiple branches to mutate the same\nrollout state concurrently and overwrite `reward` or other fields.\n\n## ComposedJudger Checklist\n\nWhen adding a branch intended for `ComposedJudger`:\n\n1. Ensure the class inherits `Judger`.\n2. Ensure branch-specific fields are copied into the payload in `preprocess()`.\n3. Ensure `judge_payload()` returns a `JudgerOutput` for one payload, or a\n `list[JudgerOutput]` for a batch payload.\n4. Ensure `postprocess()` writes `rollout_state.reward`.\n5. For multi-branch composed judging, ensure `merge_fn` creates the final\n reward shape, including `reward[\"score\"]` when training expects it.\n\n## Tests\n\nFor custom judger changes, add focused tests for:\n\n- Single-sample `judge()`.\n- `batch_judge()` if batch support is implemented.\n- `ComposedJudger` routing if the judger is intended as a composed branch.\n- Rejection or explicit non-support if the implementation directly subclasses\n `BaseJudger`.\n\n\n---\nname: xtuner-sync-supported-models\ndescription: Synchronize xtuner's supported model documentation (docs/en/pretrain_sft/advanced_tutorial/model.md and docs/zh_cn/pretrain_sft/advanced_tutorial/model.md) with the actual Config classes defined under xtuner/v1/model/. Use when (1) new TransformerConfig, MoEConfig, or BaseComposeConfig subclasses are added, removed, or renamed in xtuner/v1/model/, (2) existing model configs change their inheritance hierarchy, scale, or HuggingFace counterpart, or (3) a code review or user request points out that model.md is out of sync with the codebase.\n---\n\n# Update XTuner Supported Model Docs\n\nKeep the English and Chinese `model.md` files synchronized with the actual Config classes in `xtuner/v1/model/`.\n\n## Scan the Codebase\n\nRun the bundled scan script from the xtuner project root to discover all Config classes and their inheritance:\n\n```bash\npython3 .agents/skills/xtuner-sync-supported-models/scripts/scan_model_configs.py\n```\n\nThe script outputs JSON with two keys:\n- `configs`: list of every `*Config` class under `xtuner/v1/model/` with its parent classes and file path\n- `children`: parent-to-children mapping for the hierarchy tree\n\n## What to Update\n\nCompare the script output against the two files:\n- `docs/en/pretrain_sft/advanced_tutorial/model.md`\n- `docs/zh_cn/pretrain_sft/advanced_tutorial/model.md`\n\nBoth files share the same structure and must stay in sync:\n\n1. **Base Config Classes** — configs that directly inherit from `TransformerConfig` (or `MoEConfig`) and provide a `from_hf` classmethod for loading HuggingFace weights\n2. **Concrete Model Configs** — fixed-scale subclasses of the base configs above\n3. **Compose Models** — multimodal configs that inherit from `BaseComposeConfig`\n4. **Inheritance Hierarchy** — a text tree showing the full `XTunerBaseModelConfig` hierarchy\n\n### Rules for the Base Config table\n\nInclude these direct descendants of `TransformerConfig`/`MoEConfig`:\n- `Qwen2DenseConfig`\n- `Qwen3DenseConfig`\n- `DeepSeekV3Config`\n- `GptOssConfig`\n- `Qwen3MoEConfig`\n\nExclude from the base table:\n- `MoEConfig` — it is an intermediate base class, not a usable model family\n- `Qwen3_5_VLTextMoEConfig` — it is an intermediate base with only one concrete child; its child `Qwen3_5_VLTextMoE35BA3BConfig` belongs under the MoE concrete table\n\n### Rules for the Concrete Model table\n\nInclude every concrete subclass that has fixed parameter defaults. For each row note:\n- `Config Class`\n- `Base Class / Family`\n- `Architecture Type`: `Dense`, `MoE`, `Dense (VL backbone)`, `MoE (VL backbone)`\n- `Scale / Notes`: parameter count or total/activated size; for VL backbones note \"for multimodal\"\n\n`DeepSeekV3Config` appears here even though it has no separate base entry (it is both base and concrete).\n\n### Rules for the Compose Models section\n\nInclude three sub-tables:\n1. **Compose Base Config Classes** — `Qwen3VLBaseConfig`, `InternVLBaseConfig`, `InternS1BaseConfig`\n - `Qwen3VLBaseConfig`: VL model based on Qwen3 text backbone\n - `InternVLBaseConfig`: VL model based on InternViT + Qwen3\n - `InternS1BaseConfig`: Science multimodal model based on InternViT + Qwen3\n2. **Concrete Compose Model Configs** — every subclass of the above bases; for each row note the wrapped `Text Config` and scale\n\n### Rules for the Inheritance Hierarchy tree\n\nRebuild the tree from `XTunerBaseModelConfig` with two top-level branches:\n\n```text\nXTunerBaseModelConfig\n├── TransformerConfig\n│ ├── Dense Models\n│ │ ├── Qwen2DenseConfig\n│ │ │ └── Qwen2Dense7BConfig\n│ │ └── Qwen3DenseConfig\n│ │ ├── Qwen3Dense8BConfig\n│ │ │ └── Qwen3VLTextDense8BConfig\n│ │ ├── Qwen3Dense4BConfig\n│ │ │ └── Qwen3VLTextDense4BConfig\n│ │ └── Qwen3Dense0P6BConfig\n│ └── MoE Models (via MoEConfig)\n│ ├── DeepSeekV3Config\n│ ├── GptOssConfig\n│ │ ├── GptOss21BA3P6Config\n│ │ └── GptOss117BA5P8Config\n│ ├── Qwen3MoEConfig\n│ │ ├── Qwen3MoE30BA3Config\n│ │ │ └── Qwen3VLTextMoE30BA3Config\n│ │ ├── Qwen3MoE235BA22Config\n│ │ │ └── Qwen3VLTextMoE235BA22Config\n│ │ └── Qwen3MoEFoPEConfig\n│ └── Qwen3_5_VLTextMoEConfig\n│ └── Qwen3_5_VLTextMoE35BA3BConfig\n└── BaseComposeConfig\n ├── Qwen3VLBaseConfig\n │ ├── Qwen3VLMoE30BA3Config\n │ ├── Qwen3VLMoE235BA22Config\n │ ├── Qwen3VLDense4BConfig\n │ ├── Qwen3VLDense8BConfig\n │ └── Qwen3_5_BaseConfig\n │ └── Qwen3_5_VLMoE35BA3Config\n ├── InternVLBaseConfig\n │ ├── InternVL3P5Dense8BConfig\n │ ├── InternVL3P5MoE30BA3Config\n │ └── InternVL3P5Dense1BConfig\n └── InternS1BaseConfig\n ├── InternS1Config\n └── InternS1MiniConfig\n```\n\nWhen new configs are added, insert them into the appropriate branch following the same indentation style.\n\n## Translation Notes\n\nKeep the Chinese `model.md` (`docs/zh_cn/...`) structurally identical to the English one. Translate:\n- Section headings\n- Table header cells\n- Description cells (e.g., \"Image / Video + Text\" → \"图像/视频 + 文本\")\n- Scale descriptions (e.g., \"~7B parameters\" → \"约 7B 参数\", \"FoPE variant\" → \"FoPE 变体\")\n\nDo **not** translate Config class names, file paths, or code identifiers.\n\n\n# Blackbox Agent Trace Reference\n\nThis reference captures the XTuner black-box agent trace reconstruction workflow. It is intended for implementation and review work, not as user-facing docs.\n\nRead this reference when `xtuner-trace/SKILL.md` routes an agent trace request here. The design background lives in `docs/design/blackbox_agent_trace_reconstruction.md`.\n\n## Contents\n\n- [Decision Flow](#decision-flow)\n- [Path A: Trajectory-Backed Reconstruction](#path-a-trajectory-backed-reconstruction)\n- [Path B: SessionServer-Only Fallback](#path-b-sessionserver-only-fallback)\n- [`record_synthetic_span(...)` Usage](#record_synthetic_span-usage)\n- [Reconstructed Span Shape](#reconstructed-span-shape)\n- [Viewer And Jaeger Requirements](#viewer-and-jaeger-requirements)\n- [Example: mini-SWE-agent Trajectory Conversion](#example-mini-swe-agent-trajectory-conversion)\n- [Verification](#verification)\n- [Common Failure Modes](#common-failure-modes)\n\n## Decision Flow\n\nWhen adding trace support for a black-box agent, classify the available evidence before deciding where to instrument.\n\n1. Confirm this is a black-box agent path: XTuner exposes an OpenAI-compatible inference endpoint to the agent through `SessionServer`; the agent runs its native loop and calls that endpoint for LLM generation. Tool execution, environment interaction, and observation handling happen inside the agent runtime and are not directly instrumented by XTuner.\n2. Confirm whether the agent can save a trajectory for each run.\n3. If a trajectory exists, check whether it contains tool-call metadata such as tool name, action/command, status, return code, error information, observation, or timestamp.\n4. Use Path A when the trajectory contains tool-call metadata. The trajectory supplies tool semantics, and `session_server.request` spans supply the LLM request timeline.\n5. Use Path B only when no useful trajectory exists. This can recover request gaps, but cannot recover reliable tool names, return codes, errors, or observations.\n\nDo not require the trajectory to contain complete LLM generate timing before using Path A. `SessionServer` remains the stable source for LLM request start/end times. If the trajectory also records LLM calls, use them as an optional consistency check.\n\n## Path A: Trajectory-Backed Reconstruction\n\nUse this path when the agent can save a trajectory with tool-call metadata.\n\n### Data Contract\n\nThe `session_server.request` spans must provide:\n\n- `xtuner.session_id` or `xtuner.rollout_id`\n- span start time\n- span end time\n\nThe trajectory should provide best-effort tool metadata:\n\n- tool event order, id, or enough context to infer order\n- tool name, action, or command\n- status, return code, or exception information\n- observation or output summary\n- tool start/end timestamps, or a single tool-related timestamp with documented semantics, if the agent records one\n\nDo not define a universal trajectory schema in trace core. Agent trajectory formats are agent-specific; keep parsing inside that agent's recipe or trajectory-to-trace adapter until multiple agents prove a shared model is worth extracting.\n\n### Where To Add Trace\n\nAdd trace at the three black-box observation boundaries. These instructions must be enough to re-create the instrumentation even when no example implementation exists in the checkout.\n\n#### 1. SessionServer Request Trace\n\nFirst check whether the `SessionServer` HTTP request handler already emits `session_server.request` spans. If not, add this instrumentation to the handler that receives the agent's OpenAI-compatible request.\n\nImplementation requirements:\n\n- Locate the outer request handler, such as an aiohttp/FastAPI handler named like `_handle_request(request)`.\n- Read enough request headers/body to determine:\n - `session_id`\n - optional rollout id\n - whether the request is streaming\n - optional trace carrier from standard headers or a request-body carrier field\n- Wrap the full inference request lifecycle in:\n\n```python\nwith trace_span(\n \"session_server.request\",\n attributes={\n \"xtuner.stage\": \"llm_generate\",\n \"xtuner.session_id\": session_id,\n \"xtuner.rollout_id\": rollout_id or session_id,\n \"session.stream\": is_streaming,\n },\n parent_carrier=parent_carrier,\n):\n ...\n```\n\n- Keep the outer `session_server.request` span around prepare, backend request, response read, response post-processing, and return-to-agent.\n- Use `set_trace_attributes(...)` before exiting the span to record request-level status or error metadata when available.\n- If adding child spans, use these names:\n - `session_server.prepare_request`\n - `session_server.send_request`\n - `session_server.read_response`\n - `session_server.record_response`\n- Put `xtuner.session_id` on every child span too. This makes later trajectory-to-trace adapters able to filter a single agent run reliably.\n\nDo not parse agent trajectory or infer tool calls inside `SessionServer`. Its black-box responsibility is to provide the LLM request timeline.\n\n#### 2. Agent Run Boundary Trace\n\nIn the black-box recipe or launcher, find the call site that invokes the agent's native run loop, such as `agent.run(...)`.\n\nWrap only the outer run boundary:\n\n```python\nwith trace_span(\n \".run\",\n attributes={\n \"xtuner.stage\": \"agent.run\",\n \"xtuner.session_id\": session_id,\n \"xtuner.rollout_id\": rollout_id or session_id,\n \"agent.name\": agent_name,\n },\n):\n agent.run(...)\n```\n\nThis span is a lifecycle boundary. Do not wrap the agent's internal model, environment, or tool methods as the default black-box trace path. LLM/tool detail should come from `SessionServer` spans plus trajectory reconstruction.\n\n#### 3. Trajectory-To-Trace Reconstruction\n\nAfter the native agent run finishes, call the trajectory-to-trace adapter before `close_trace()`. Prefer a `finally` block so partial trajectories can still be converted after failures:\n\n```python\nreturn_code = 1\ntry:\n with trace_span(\".run\", attributes=agent_run_attributes):\n return_code = agent.run(...)\nfinally:\n emit_reconstructed_trace(\n trace_jsonl=trace_jsonl_path,\n trajectory=trajectory_path,\n session_id=session_id,\n )\n close_trace()\nreturn return_code\n```\n\nThe adapter must read the trajectory, read same-session `session_server.request` spans, derive tool timing from trajectory timestamps when possible, use request gaps only for correlation or inferred fallback, and emit reconstructed synthetic spans with `record_synthetic_span(...)`.\n\n### Adapter Responsibilities\n\nWrite a trajectory-to-trace adapter for the specific agent. The adapter should:\n\n1. Load the agent trajectory.\n2. Extract ordered tool events and their metadata.\n3. Load `session_server.request` spans from the same `traces.jsonl`.\n4. Filter request spans by `session_id`.\n5. Sort request spans by start time and assign local indexes.\n6. Extract tool timestamps from the trajectory and interpret their semantics:\n - If the trajectory provides reliable tool start/end times, use them as the `reconstructed_tool_call.N` span boundaries and set `xtuner.inferred=false`.\n - If the trajectory provides only one timestamp, record what it means if known, such as tool start, tool end, observation write, or trajectory write. Use it to correlate the tool event with the surrounding `session_server.request` window.\n - If the trajectory timestamp semantics are unknown or incomplete, use request-gap boundaries only as an inferred fallback and set `xtuner.inferred=true`.\n7. Use `session_server.request` spans as the LLM timeline and correlation windows, not as the default source of trajectory-backed tool timing.\n8. When using request-gap fallback for a bounded tool event, use:\n\n```text\ntool_i.start = request_i.end\ntool_i.end = request_{i+1}.start\n```\n\n9. Emit `reconstructed_agent_run`, `reconstructed_llm_call.N`, and bounded `reconstructed_tool_call.N` spans with `record_synthetic_span(...)`.\n10. Treat final tool events without a next LLM request as unbounded unless the trajectory has trustworthy start/end times.\n\nWhen request-gap fallback is used, the interval is not pure tool runtime. It can include LLM-response parsing, user confirmation time, tool execution, observation formatting, trajectory write time, and loop overhead before the next LLM request.\n\n## Path B: SessionServer-Only Fallback\n\nUse this path only when the agent does not save a useful trajectory.\n\n### Where To Add Trace\n\nAdd trace to the `SessionServer` request path so every LLM request produces a reliable timeline. At minimum, `session_server.request` spans must include:\n\n- `xtuner.session_id`\n- `xtuner.rollout_id`, falling back to the session id when there is no rollout id\n- request start time\n- request end time\n\nSupporting spans such as `session_server.prepare_request`, `session_server.send_request`, `session_server.read_response`, and `session_server.record_response` may remain useful for debugging latency, but the fallback reconstruction only requires the outer `session_server.request` start/end times.\n\nDo not put tool inference inside `SessionServer`. `SessionServer` should not parse trajectories, infer tool names, track return codes, or maintain complex per-session reconstruction state.\n\n### Fallback Reconstruction\n\nAfter the agent run, a fallback adapter may infer intervals from adjacent request spans:\n\n```text\ninferred_tool_call_i.start = request_i.end\ninferred_tool_call_i.end = request_{i+1}.start\n```\n\nThe fallback can emit synthetic spans such as `inferred_tool_call.N` with:\n\n```text\nxtuner.stage=tool_call\nxtuner.synthetic=true\nxtuner.inferred=true\nxtuner.session_id=\nreconstruction.source=session_server_only\nreconstruction.method=session_server_gap\n```\n\nDo not claim unavailable metadata in this mode. SessionServer-only fallback cannot reliably recover tool name, command, return code, stderr/stdout, agent exception, skipped action, or observation content.\n\nDo not infer the final tool event without a next LLM request. The interval has no reliable end time.\n\n## `record_synthetic_span(...)` Usage\n\nUse `record_synthetic_span(...)` to write reconstructed historical intervals back into the active XTuner trace runtime.\n\n```python\nrecord_synthetic_span(\n name: str,\n *,\n start_time_unix_ns: int,\n end_time_unix_ns: int,\n attributes: Mapping[str, Any] | None = None,\n parent_carrier: Mapping[str, str] | None = None,\n status: str = \"completed\",\n error_message: str | None = None,\n) -> dict[str, str] | None\n```\n\nRules:\n\n- Call it before `close_trace()`.\n- Use absolute Unix nanoseconds, not monotonic timestamps.\n- Ensure `end_time_unix_ns > start_time_unix_ns`.\n- Let the API default `xtuner.synthetic=true`.\n- Set `xtuner.inferred=true` only when the time boundary is inferred, such as from `session_server.request` gaps.\n- Use `status=\"error\"` and `error_message=...` for real errors.\n- Use `parent_carrier` to attach reconstructed children under one synthetic root trace.\n- Treat `None` return as trace-disabled no-op.\n\nTypical root and child pattern:\n\n```python\nroot_ids = record_synthetic_span(\n \"reconstructed_agent_run\",\n start_time_unix_ns=first_request.start_time_unix_ns,\n end_time_unix_ns=last_request.end_time_unix_ns,\n attributes={\n \"xtuner.stage\": \"agent.run\",\n \"xtuner.session_id\": session_id,\n \"xtuner.inferred\": False,\n \"reconstruction.method\": \"trajectory_session_timeline\",\n },\n)\nroot_carrier = None\nif root_ids:\n root_carrier = {\"traceparent\": f\"00-{root_ids['trace_id']}-{root_ids['span_id']}-01\"}\n\nrecord_synthetic_span(\n \"reconstructed_tool_call.1\",\n start_time_unix_ns=request_1.end_time_unix_ns,\n end_time_unix_ns=request_2.start_time_unix_ns,\n attributes={\n \"xtuner.stage\": \"tool_call\",\n \"xtuner.session_id\": session_id,\n \"xtuner.inferred\": True,\n \"agent.tool_call_index\": 1,\n \"agent.tool.name\": \"bash\",\n \"reconstruction.source\": \"agent_trajectory\",\n \"reconstruction.method\": \"session_server_gap\",\n },\n parent_carrier=root_carrier,\n)\n```\n\nDo not call OpenTelemetry SDK directly from the adapter. Use XTuner trace APIs.\n\n## Reconstructed Span Shape\n\nTrajectory-backed reconstruction should generate one synthetic root trace:\n\n```text\nreconstructed_agent_run\n|-- reconstructed_llm_call.1\n|-- reconstructed_tool_call.1\n|-- reconstructed_llm_call.2\n|-- reconstructed_tool_call.2\n...\n```\n\nUse these naming rules:\n\n- `reconstructed_agent_run` for the synthetic root.\n- `reconstructed_llm_call.N` for LLM request intervals copied from `session_server.request`.\n- `reconstructed_tool_call.N` for trajectory-backed tool metadata.\n- `inferred_tool_call.N` only for SessionServer-only fallback without trajectory metadata.\n\nRecommended root attributes:\n\n```text\nxtuner.stage=agent.run\nxtuner.synthetic=true\nxtuner.inferred=false\nxtuner.session_id=\nxtuner.rollout_id=\nagent.name=\nagent.llm_call_count=\nagent.tool_call_count=\nreconstruction.source=\nreconstruction.method=trajectory_session_timeline\n```\n\nRecommended LLM attributes:\n\n```text\nxtuner.stage=llm_generate\nxtuner.synthetic=true\nxtuner.inferred=false\nagent.llm_call_index=N\nreconstruction.source=session_server_request\nreconstruction.method=session_server_request_copy\nreconstruction.original_trace_id=\nreconstruction.original_span_id=\nreconstruction.original_operation=session_server.request\n```\n\nRecommended tool attributes:\n\n```text\nxtuner.stage=tool_call\nxtuner.synthetic=true\nxtuner.inferred=true\nagent.tool_call_index=N\nagent.tool.name=\nagent.tool.status=\nagent.tool.returncode=\nagent.tool.command_preview=\nreconstruction.source=\nreconstruction.method=session_server_gap\nreconstruction.timestamp_check=\n```\n\nDo not call trajectory-backed spans `inferred_tool_call.N`. Use `reconstructed_tool_call.N` and put uncertainty in attributes.\n\n## Viewer And Jaeger Requirements\n\nXTuner viewer and Jaeger must consume the same reconstructed spans. Do not implement one inference path in the viewer and another inference path for Jaeger.\n\nViewer expectations:\n\n- Show one sample per agent run/session after live and completed records are merged.\n- Prefer the trace containing `reconstructed_agent_run` as the primary sample trace.\n- Keep raw `session_server.request` trace ids available for audit.\n- Display the reconstructed path:\n\n```text\nreconstructed_agent_run -> reconstructed_llm_call.1 -> reconstructed_tool_call.1 -> ...\n```\n\nJaeger expectations:\n\n- The viewer's Jaeger link should open the trace containing `reconstructed_agent_run`.\n- Opening the first raw `session_server.request` trace is a bug for reconstructed black-box samples.\n- A healthy reconstructed trace contains `1 + llm_count + bounded_tool_count` reconstructed spans.\n\n\n# XTuner Trace Patterns\n\nThese patterns describe the retained trace surface in this branch: runtime, basic API, viewer, and local setup tooling.\n\n## Basic API\n\nUse the public facade:\n\n```python\nfrom xtuner.v1.rl.trace import (\n inject_trace_context,\n set_trace_attributes,\n trace_event,\n trace_function,\n trace_span,\n)\n```\n\n### Local Span\n\n```python\nwith trace_span(\"phase.name\", attributes={\"xtuner.stage\": \"phase\"}):\n ...\n set_trace_attributes({\"phase.count\": 3})\n```\n\n### Decorated Function\n\n```python\n@trace_function(\"phase.load\")\ndef load_item(path: str) -> object:\n ...\n```\n\n### Parent Carrier\n\n```python\ncarrier = inject_trace_context()\n\nwith trace_span(\"child.phase\", parent_carrier=carrier):\n ...\n```\n\n`parent_carrier` is a basic W3C context carrier. It is not tied to `RolloutState`, Ray, aiohttp, or any XTuner business object.\n\n## End-to-End Trace Wiring\n\nUse this pattern when adding trace support to an XTuner training run.\n\n### Launch Script\n\nEnable trace bootstrap from the launcher, guarded by `XTUNER_TRACE_ENABLED`:\n\n```bash\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\nREPO_ROOT=\"$(cd \"${SCRIPT_DIR}/../../..\" && pwd)\"\nif [ \"${XTUNER_TRACE_ENABLED:-0}\" = \"1\" ]; then\n source \"${REPO_ROOT}/recipe/trace/setup_trace.sh\"\nfi\n```\n\nKeep training stdout/stderr in the main training log so runtime trace messages\nare visible:\n\n```bash\npython xtuner/v1/train/cli/rl.py \\\n --config \"$CONFIG_PATH\" \\\n --num-workers \"$XTUNER_RL_NUM_WORKERS\" \\\n 2>&1 | tee -a \"${WORK_DIR}/training_log_${current_time}.txt\"\n```\n\n### Training Config\n\nAdd `TraceConfig` to the training config and pass it to the trainer config:\n\n```python\nimport os\n\nfrom xtuner.v1.rl.trace import TraceConfig\n\n\ntrace_config = TraceConfig(\n enabled=os.environ.get(\"XTUNER_TRACE_ENABLED\") == \"1\",\n service_name=\"xtuner-agent-rollout\",\n xtuner_viewer_enabled=True,\n xtuner_viewer_host=\"0.0.0.0\",\n xtuner_viewer_port=18080,\n xtuner_viewer_jaeger_query_url=\"http://127.0.0.1:16686\",\n enable_rollout_trace=True,\n)\n\ntrainer = RLColocateTrainerConfig(\n ...,\n trace_config=trace_config,\n)\n```\n\nWhen `output_dir` is omitted in an RL trainer config, the trainer places trace\nartifacts under the current experiment directory, for example\n`work_dirs/name/20260713071356/otel/`.\n\n### Stage Spans\n\nBefore adding spans, ask the user which stages they want to observe and which\nmetrics each stage should expose.\n\nPut fields known when entering the stage, especially fields used for grouping\nor filtering, in `trace_span(..., attributes=...)`. Put results known only\nafter or during execution, such as status, duration, token counts, errors, and\nreward values, in `set_trace_attributes(...)`. This lets the viewer reconstruct\nthe call chain and identify latency hotspots or abnormal stages.\n\nInfer stage:\n\n```python\nimport time\n\n\nstarted_at = time.monotonic()\nattributes = {\n \"xtuner.stage\": \"infer\",\n \"xtuner.task_name\": task_name,\n \"xtuner.sample_id\": sample_id,\n \"rollout.backend\": backend,\n}\n\nwith trace_span(\"agent.infer\", attributes=attributes):\n result = await run_infer(...)\n set_trace_attributes(\n {\n \"xtuner.status\": \"ok\" if result.ok else \"error\",\n \"prompt.tokens\": result.prompt_tokens,\n \"completion.tokens\": result.completion_tokens,\n \"stage.duration_ms\": int((time.monotonic() - started_at) * 1000),\n }\n )\n```\n\nResource acquire stage:\n\n```python\nwith trace_span(\n \"sandbox.acquire\",\n attributes={\n \"xtuner.stage\": \"acquire\",\n \"xtuner.sample_id\": sample_id,\n \"sandbox.name\": sandbox_name,\n },\n):\n client = await pool.get(sandbox_name)\n set_trace_attributes(\n {\n \"sandbox.env_id\": env_id,\n \"sandbox.image\": image,\n \"sandbox.reused\": reused,\n }\n )\n```\n\nValidation or reward stage:\n\n```python\nwith trace_span(\n \"sample.validate\",\n attributes={\n \"xtuner.stage\": \"validate\",\n \"xtuner.sample_id\": sample_id,\n \"validator.name\": validator_name,\n },\n):\n record = await validate(...)\n set_trace_attributes(\n {\n \"xtuner.status\": \"ok\" if record.ok else \"error\",\n \"reward.score\": record.score,\n \"reward.passed\": record.passed,\n }\n )\n\n if record.error is not None:\n set_trace_attributes(\n {\n \"error\": True,\n \"error.type\": record.error.type,\n \"error.message\": record.error.message,\n }\n )\n```\n\nDo not record prompts, responses, full configs, secrets, raw headers, stack\ntraces, or large payloads.\n\n### Cross-Boundary Propagation\n\nWhen a traced operation crosses a process, actor, or HTTP boundary, keep the\npropagation helper in the business or transport module that owns that boundary.\nDo not put Ray, HTTP, rollout, agent, or judger helpers back into\n`xtuner/v1/rl/trace`.\n\n#### Ray Remote Calls\n\nFor Ray calls, inject the current context into a plain carrier and temporarily\nattach that carrier to a serializable domain object or explicit call argument.\n\n```python\nfrom collections.abc import Mapping\nfrom contextlib import contextmanager\nfrom typing import Any\n\nfrom xtuner.v1.rl.trace import inject_trace_context, trace_span\n\n\n_TRACE_CARRIER_FIELD = \"_xtuner_trace_carrier\"\n\n\n@contextmanager\ndef attach_trace_carrier_temporarily(target: Any, carrier: Mapping[str, str]):\n if not carrier:\n yield\n return\n\n extra_fields = getattr(target, \"extra_fields\", None)\n if extra_fields is None:\n extra_fields = {}\n target.extra_fields = extra_fields\n had_previous = _TRACE_CARRIER_FIELD in extra_fields\n previous = extra_fields.get(_TRACE_CARRIER_FIELD)\n extra_fields[_TRACE_CARRIER_FIELD] = dict(carrier)\n try:\n yield\n finally:\n if had_previous:\n extra_fields[_TRACE_CARRIER_FIELD] = previous\n else:\n extra_fields.pop(_TRACE_CARRIER_FIELD, None)\n\n\ndef call_ray_remote_with_trace(remote_method, *args, trace_target: Any, **kwargs):\n carrier: dict[str, str] = {}\n inject_trace_context(carrier)\n with attach_trace_carrier_temporarily(trace_target, carrier):\n return remote_method.remote(*args, **kwargs)\n\n\ndef pop_trace_parent_carrier(target: Any) -> dict[str, str] | None:\n extra_fields = getattr(target, \"extra_fields\", None)\n if not isinstance(extra_fields, dict):\n return None\n carrier = extra_fields.pop(_TRACE_CARRIER_FIELD, None)\n if not isinstance(carrier, Mapping):\n return None\n return {str(key): str(value) for key, value in carrier.items()}\n```\n\nOn the receiver side, attach the carrier to the new span:\n\n```python\nparent_carrier = pop_trace_parent_carrier(trace_target)\n\nwith trace_span(\"worker.stage\", attributes=attributes, parent_carrier=parent_carrier):\n ...\n```\n\nKeep the helper domain-specific. Validate that exactly one trace target is used\nwhen the transport needs one target object; reject collections if restoring the\ncarrier to the right child span would be ambiguous.\n\n#### HTTP Calls\n\nFor HTTP calls, prefer W3C headers. If a third-party client or proxy may drop\ncustom headers, also put a copied carrier into the JSON body under an internal\nfield and remove it before forwarding the payload downstream.\n\n```python\nfrom collections.abc import Mapping\nfrom typing import Any\n\nfrom xtuner.v1.rl.trace import inject_trace_context, trace_span\n\n\n_TRACE_CARRIER_FIELD = \"_xtuner_trace_carrier\"\n\n\ndef inject_trace_carrier_into_json_body(kwargs: dict[str, Any], carrier: Mapping[str, str]) -> None:\n if not carrier:\n return\n payload = kwargs.get(\"json\")\n if not isinstance(payload, dict):\n return\n payload = dict(payload)\n payload.setdefault(_TRACE_CARRIER_FIELD, dict(carrier))\n kwargs[\"json\"] = payload\n\n\ndef extract_trace_carrier_from_mapping(payload: Mapping[str, Any] | None) -> dict[str, str] | None:\n if not isinstance(payload, Mapping):\n return None\n carrier = payload.get(_TRACE_CARRIER_FIELD)\n if not isinstance(carrier, Mapping):\n return None\n return {str(key): str(value) for key, value in carrier.items()}\n\n\ndef remove_trace_carrier_from_mapping(payload: dict[str, Any]) -> None:\n payload.pop(_TRACE_CARRIER_FIELD, None)\n\n\ndef extract_http_parent_carrier(headers: Mapping[str, Any], payload: Mapping[str, Any] | None) -> dict[str, str] | None:\n header_carrier = {\n str(key): str(value)\n for key, value in headers.items()\n if str(key).lower() in {\"traceparent\", \"tracestate\", \"baggage\"}\n }\n return header_carrier or extract_trace_carrier_from_mapping(payload)\n```\n\nCaller:\n\n```python\nheaders = dict(headers or {})\nrequest_kwargs = {\"json\": payload}\ncarrier: dict[str, str] = {}\ninject_trace_context(carrier)\nheaders.update(carrier)\ninject_trace_carrier_into_json_body(request_kwargs, carrier)\n\nwith trace_span(\"http.client.request\", attributes={\"http.method\": \"POST\", \"http.url\": url}):\n response = await client.post(url, headers=headers, **request_kwargs)\n```\n\nReceiver:\n\n```python\nheaders = dict(request.headers)\npayload = await request.json()\nparent_carrier = extract_http_parent_carrier(headers, payload)\n\nwith trace_span(\"http.server.request\", attributes=attributes, parent_carrier=parent_carrier):\n remove_trace_carrier_from_mapping(payload)\n ...\n```\n\nDo not record raw headers or full payloads as span attributes. Only record\nderived fields such as method, route, status code, stage, IDs, and timing.\n\n### Viewer Output\n\nWhen `xtuner_viewer_enabled=True`, the trace runtime logs the trace JSONL path,\nthe XTuner viewer URL, and the equivalent manual viewer command. The viewer\ndefaults to port `18080`; `xtuner_viewer_port=0` is still available when a run\nshould pick a free port automatically. The viewer process inherits the training\nstdout/stderr; keep those runtime log lines visible in the main training log and\nreport them to the user after the run.\n\nUseful reference points from the full trace implementation:\n\n- `examples/v1/scripts/run_rl_run.sh`\n- `recipe/trace/setup_trace.sh`\n- `examples/v1/config/agentic_rl_qwen3p5vl_mtp_ep_code.py`\n\n## Runtime\n\nConfigure tracing explicitly:\n\n```python\nfrom xtuner.v1.rl.trace import TraceConfig, close_trace, configure_trace\n\nruntime = configure_trace(\n TraceConfig(\n enabled=True,\n output_dir=\"work_dirs/example/otel\",\n service_name=\"xtuner\",\n xtuner_viewer_enabled=True,\n )\n)\n\ntry:\n ...\nfinally:\n close_trace()\n```\n\nThe runtime owns OTel collector setup, trace JSONL output, and optional lightweight XTuner viewer startup.\n\n## Viewer\n\nThe viewer reads Jaeger-style traces or OTel JSONL converted into Jaeger-style payloads. It should not depend on hard-coded rollout/agent span registries.\n\nStage display rules:\n\n1. Use `xtuner.stage` when present.\n2. Else use `stage` when present.\n3. Else use `stage.name` when present.\n4. Else fall back to the raw span name.\n\nUseful stable attributes:\n\n- IDs: `xtuner.rollout_id`, `xtuner.group_id`, `xtuner.session_id`, `xtuner.task_name`\n- Status: `xtuner.status`\n- Stage: `xtuner.stage`, `stage`, `stage.name`\n- Counts/timing: `prompt.tokens`, `completion.tokens`, `http.status_code`\n- Errors: `error`, `error.message`, `exception.type`\n\nAvoid recording prompts, responses, full configs, secrets, raw headers, or large payloads.\n\n## Local Setup\n\n`recipe/trace/setup_trace.sh` and the rest of `recipe/trace/` are local helper tooling for installing OTel collector binaries, starting the local Jaeger dependency, and clearing stale `recipe.trace.viewer.server` processes on `XTUNER_TRACE_VIEWER_PORT` or the default viewer port `18080`. Keep these as setup assets; do not use them to add automatic trace behavior to training configs or launch scripts unless that integration is explicitly requested.\n\n## Rollout Starter Preset\n\nThe built-in rollout starter trace is opt-in:\n\n```python\ntrace_config = TraceConfig(\n enabled=True,\n xtuner_viewer_enabled=True,\n enable_rollout_trace=True,\n)\n```\n\nIts rollout-specific helpers live in `xtuner/v1/rl/trace/rollout_api.py`:\n\n- `trace_rollout_endpoint(...)`\n- `trace_rollout_remote(...)`\n\nUse this only for the starter rollout chain. For custom instrumentation, ask\nwhich stages and metrics the user wants before adding spans.\n\n## Removed Legacy Semantics\n\nDo not use or recreate these removed surfaces in this branch:\n\n- `trace_remote`\n- `traced_rollout_endpoint`\n- `traced_agent_item_endpoint`\n- `traced_judger_endpoint`\n- `xtuner/v1/rl/trace/trace_utils.py`\n- `xtuner/v1/rl/trace/context_propagation.py`\n- fixed `TRACE_SPAN_*` registries\n- rollout/agent/judger attribute builders\n\n\n---\nname: xtuner-trace\ndescription: Use when the user wants to instrument, inspect, or debug XTuner traces to reconstruct a sample/request execution path, follow cross-process call chains, or identify latency hotspots, bottleneck stages, and abnormal paths from span data.\n---\n\n# XTuner Trace\n\nUse this skill for the current trace layer in this repository: OpenTelemetry runtime setup, the basic public API, local setup scripts, and the trace viewer.\n\n## Current Boundary\n\nKeep this package infrastructure-only:\n\n- Runtime/configuration: `xtuner/v1/rl/trace/runtime.py`\n- OTel SDK adapter: `xtuner/v1/rl/trace/otel_utils.py`\n- Public facade: `xtuner/v1/rl/trace/__init__.py`\n- Basic API: `xtuner/v1/rl/trace/api.py`\n- Rollout starter preset: `xtuner/v1/rl/trace/rollout_api.py`\n- Local trace tooling and viewer: `recipe/trace/`\n\nDo not add rollout, agent, judger, Ray remote, HTTP proxy, reward, status, or session-server business semantics to `api.py`, `runtime.py`, or `otel_utils.py`. Keep rollout-specific starter behavior inside `rollout_api.py` and gated by `TraceConfig.enable_rollout_trace`.\n\n## Viewer Choice\n\nChoose the viewer by the observation question:\n\n- Use `recipe/trace/viewer` when the user wants rollout-oriented inspection, such as comparing samples within a training step by stage duration, status, reward/filter metadata, or recorded sample call chain.\n- Use Jaeger when the question is not sample/step-centric, such as inspecting raw spans, service boundaries, cross-process causality, backend/client calls, collector/export behavior, or custom instrumentation that does not emit the rollout/sample attributes expected by the recipe viewer.\n\nThe recipe viewer reads `traces.jsonl` and adds XTuner rollout/sample aggregation. Jaeger remains the raw trace drill-down surface for the same traced run.\n\n## Black-Box Agent Trace Routing\n\nWhen the trace request involves an agent, first ask whether this is a black-box\nagent trace. In this mode, XTuner exposes an OpenAI-compatible inference endpoint\nto the agent through `SessionServer`; the agent runs its native loop and calls\nthat endpoint for LLM generation. Tool execution, environment interaction, and\nobservation handling happen inside the agent runtime and are not directly\ninstrumented by XTuner.\n\nIf the answer is yes, or the request mentions trajectory reconstruction,\n`SessionServer` request gaps, `record_synthetic_span(...)`,\n`reconstructed_agent_run`, `reconstructed_tool_call.N`, XTuner viewer plus\nJaeger display of recovered tool calls, read\n[references/blackbox-agent-trace.md](references/blackbox-agent-trace.md) before\nplanning or editing.\n\n## Basic API\n\nThese are the interfaces defined in `xtuner/v1/rl/trace/api.py` and re-exported\nfrom `xtuner.v1.rl.trace`:\n\n- `trace_span(name, attributes=None, parent_carrier=None)`\n- `trace_function(name=None, attributes=None)`\n- `trace_event(name, attributes=None)`\n- `set_trace_attributes(attributes)`\n- `inject_trace_context(carrier=None)`\n\n## Runtime API\n\nUse these `xtuner/v1/rl/trace/runtime.py` interfaces for explicit trace\nruntime setup:\n\n- `TraceConfig`\n- `configure_trace(...)`\n- `close_trace()`\n\n## Rollout Starter Preset\n\n`TraceConfig(enable_rollout_trace=True)` enables the built-in rollout starter\ntrace. Its helpers live in `xtuner/v1/rl/trace/rollout_api.py`:\n\n- `trace_rollout_endpoint(...)`\n- `trace_rollout_remote(...)`\n\nKeep this layer narrow: it may depend on `RolloutState` and Ray call boundaries,\nbut it must not leak those dependencies into the basic API.\n\nRollout endpoint `xtuner.span_name_path` is a sample call chain, not just the\ncurrent process context stack. `trace_rollout_endpoint(...)` should append the\nendpoint span name to an internal trace-only chain carried in\n`RolloutState.extra_fields`, pass that state through remote rollout boundaries,\nand clean the internal field at the outermost endpoint. For example, after a\nsample generates and then judges, `judger.run` should record a path like\n`single_turn_agent_loop.run -> rollout.controller.generate -> rollout.worker.generate -> judger.run`\neven though the local active context at judge time may only be\n`single_turn_agent_loop.run -> judger.run`. Keep this behavior in\n`rollout_api.py` and viewer tests; do not add rollout call-chain semantics to\nthe basic trace API.\n\n## Add Trace Workflow\n\nWhen adding trace instrumentation to an XTuner run:\n\n1. Ask for or locate the launch script and training config before editing.\n2. In the launch script, source `recipe/trace/setup_trace.sh` when\n `XTUNER_TRACE_ENABLED=1`.\n3. In the training config, add `TraceConfig` and set `enabled=True`; set\n `xtuner_viewer_enabled=True` when the user needs interactive sample/step inspection. Set\n `enable_rollout_trace=True` only when the user wants the built-in rollout\n starter trace.\n4. Before adding any `trace_span(...)` instrumentation, you must ask the user which\n stages they want to observe and which metrics each stage should expose.\n Do not infer default stages unless the user explicitly asks you to choose.\n5. Add `trace_span(...)` only around the user-confirmed observed stages. Put fields known at span\n start in initial attributes, and update runtime or final fields with\n `set_trace_attributes(...)`.\n6. For cross-process or request boundaries, inject a carrier with\n `inject_trace_context(...)` and pass it to downstream\n `trace_span(..., parent_carrier=carrier)`.\n7. Keep transport-specific propagation at the caller boundary; do not move\n rollout, agent, judger, Ray, or HTTP semantics into the basic trace package.\n8. Ensure the main training log includes the trace output path, XTuner viewer URL,\n and equivalent manual viewer command when the XTuner viewer is enabled.\n\n## Guardrails\n\n- Do not recreate `trace_utils.py`, `context_propagation.py`, span-name registries, or business attribute builders under `xtuner/v1/rl/trace`.\n- Do not import `RolloutState`, agent item classes, judgers, rollout workers, Ray actors, aiohttp clients, or trainer configs from the basic trace package.\n- Do not call OpenTelemetry SDK directly from business code; use the basic API only when trace instrumentation is explicitly requested.\n- Do not record prompts, responses, full configs, secrets, raw headers, stack traces, or large payloads as attributes.\n- Viewer stage grouping must come from span attributes such as `xtuner.stage`, `stage`, or `stage.name`; otherwise fall back to the raw span name.\n\nFor concrete patterns, read [references/trace-patterns.md](references/trace-patterns.md) before editing trace code.\n\n## Verification\n\nUse focused checks:\n\n- `PYTHONPATH=. python -m compileall -q xtuner/v1/rl/trace recipe/trace`\n- `PYTHONPATH=. python -m unittest discover -s tests/rl -p 'test_trace*.py' -v` when the trace tests exist in the worktree.\n- `git diff --check`\n\n\n### Summary\n\nThis PR introduces a comprehensive GPU memory calculator for LLM training, which helps users estimate per-GPU memory consumption before actually running training jobs. This tool is essential for capacity planning and optimizing parallel configurations.\n\n### Features\n\n- **Memory Breakdown Analysis**: Calculates detailed memory usage including:\n - Model parameters (embedding, attention, MLP, head)\n - Gradients\n - Optimizer states (Adam momentum & variance)\n - Master parameters and gradients (for mixed-precision training)\n - Intermediate activations\n\n- **Parallel Strategy Support**:\n - Data Parallelism with ZeRO Stage 1(for Megatron)/3 (FSDP for xtuenr)\n - Tensor Parallelism (TP)\n - Pipeline Parallelism (PP) and Virtual Pipeline Parallelism (VPP) for Megatron\n - Pipeline parallelism memory accounts for flying micro-batches during 1F1B schedule\n - Expert Parallelism (EP) for MoE models\n\n- **Model Architecture Support**:\n - Dense models (e.g., Qwen3-8B)\n - MoE models (e.g., Qwen3-235B-A22B, DeepSeek-V3-671B)\n - Multi-Head Attention (MHA)\n - Multi-Latent Attention (MLA) used in DeepSeek\n\n- **Training Optimizations**:\n - Activation checkpointing (recomputation)\n - Flash Attention\n - Chunked loss computation\n\n### Usage\n\n```bash\npython xtuner/tools/llm_calc/llm_calculator.py -c [-o ]\n```\n\n### Example Configurations\n\n| Config File | Model | Scale | TrainFramework |\n|------------|-------|-------|-------|\n| `xtuner_qwen_dense_8b_1node.yaml` | Qwen3-8B (Dense) | 1 node | xtuner |\n| `xtuner_qwen_moe_30b_1node.yaml` | Qwen3-30B-A3B (MoE) | 1 node | xtuner |\n| `xtuner_qwen_moe_235b_64node.yaml` | Qwen3-235B-A22B (MoE) | 64 nodes | xtuner |\n| `xtuner_deepseekv3_671b_128node.yaml` | DeepSeek-V3-671B (MoE) | 128 nodes | xtuner |\n| `megatron_qwen_moe_235b_64node.yaml` | Qwen3-235B-A22B (MoE) | 64 nodes | Megatron |\n\n### Output Example\n\nThe tool outputs:\n1. Total model parameters breakdown\n2. Per-GPU memory consumption for different training phases (forward/backward)\n3. Detailed memory breakdown by component (params, grads, optimizer states, activations)\n4. Optional Excel export for further analysis\n \n```\n# python llm_calculator.py -c xtuner_qwen_dense_8b_1node.yaml\n----------------------------------------------setting:----------------------------------------------\nmodel:Qwen3-8B zero_stage:3 tp:1 pp:1 dp:8 ep:1 vpp:1 nodes:1 mbs:1 gbs:8 L:32768 recompute:True flash_attn:True capacity:1.0 LN:36\n----------------------------------------------------------------------------------------------------\nTotal_params_num: 7.628208160400391 B, embed_params_num: 0.57958984375 B, attn_params_num: 1.4063873291015625 B, mlp_params_num: 5.0626373291015625 B, head_params_num: 0.5795936584472656 B\nMax_mem for microbatch=1: 26.58784294128418 GiB. micro1_forward_last: 26.58784294128418 GiB, micro1_backward_last: 21.386390686035156 GiB\nMax_mem for microbatch>1: 30.401947021484375 GiB. micro1_forward_last: 26.58784294128418 GiB, micro1_backward_last: 21.386390686035156 GiB, micro2_forward_last: 30.401947021484375 GiB\n---------------------------------------------------------------------memory parts---------------------------------------------------------------------\n params grads opt_states master_params master_grads acts\nname\nembed 1.159180 1.159180 0.579590 0.289795 0.289795 0.000061\nhead 1.159187 1.159187 0.579594 0.289797 0.289797 0.625000\nperlayer_attn 0.078133 0.078133 0.039066 0.019533 0.019533 1.751953\nperlayer_mlp 0.281258 0.281258 0.140629 0.070314 0.070314 2.500000\nall_attn 0.000000 0.000000 1.406387 0.703194 0.703194 0.000000\nall_mlp 0.000000 0.000000 5.062637 2.531319 2.531319 0.000000\nin_layernorm 0.000000 0.000000 0.000000 0.000000 0.000000 0.250000\n\n------------------------------------------------------micro1_forward_last: 26.58784294128418 GiB------------------------------------------------------\n [Static OS&Param] embed, all_attn, all_mlp, head: master_params, opt_states = 11.442312240600586 GiB\n [Allgathered Last Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Head] head: params, acts = 1.7841873168945312 GiB\n [Recompute Checkpoint] in_layernorm.acts * (LN - 1) = 8.75 GiB\n\n-----------------------------------------------------micro1_backward_last: 21.386390686035156 GiB-----------------------------------------------------\n [Static OS&Param] embed, all_attn, all_mlp, head: master_params, opt_states = 11.442312240600586 GiB\n [Static Grad] embed, all_attn, all_mlp, head: master_grads = 3.8141040802001953 GiB\n [Allgathered First Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Embed] embed: params, acts = 1.15924072265625 GiB\n [Previous Layer Grad ReduceScatering] perlayer_attn, perlayer_mlp: grads = 0.3593902587890625 GiB\n [Recompute Checkpoint] None = 0 GiB\n\n-----------------------------------------------------micro2_forward_last: 30.401947021484375 GiB------------------------------------------------------\n [Static OS&Param] embed, all_attn, all_mlp, head: master_params, opt_states = 11.442312240600586 GiB\n [Static Grad] embed, all_attn, all_mlp, head: master_grads = 3.8141040802001953 GiB\n [Allgathered Last Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Head] head: params, acts = 1.7841873168945312 GiB\n [Recompute Checkpoint] in_layernorm.acts * (LN - 1) = 8.75 GiB\n```\n\n### Technical Notes for Xtuner FSDP (with recompute) Memory Breakdown of the above example\n\n1. At initilization, each GPU holds sharded master params and optimizer states (usually both in fp32).\n```\n [Static OS&Param] embed, all_attn, all_mlp, head: master_params, opt_states = 11.442312240600586 GiB\n```\n\n2. At the end of the first micro batch forward: \n2.1 The whole last decoder layer and lm head are gathered in each GPU (in bf16).\n```\n [Allgathered Last Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Head] head: params, acts = 1.7841873168945312 GiB\n```\n2.2 The checkpoint of all decoder layers also exist in each GPU\n```\n [Recompute Checkpoint] in_layernorm.acts * (LN - 1) = 8.75 GiB\n```\n\n3. At the end of the first micro batch backward\n3.1 Each GPU holds master params' graident now\n```\n [Static Grad] embed, all_attn, all_mlp, head: master_grads = 3.8141040802001953 GiB\n```\n3.2 The whole first decoder layer and embedding layer are gathered in each GPU\n```\n [Allgathered First Layer] perlayer_attn, perlayer_mlp: params, acts = 4.6113433837890625 GiB\n [Allgathered Embed] embed: params, acts = 1.15924072265625 GiB\n```\n3.3 FSDP does ReduceScatter of previous layer's gradients at the same time (to overlap computation and communication)\n```\n [Previous Layer Grad ReduceScatering] perlayer_attn, perlayer_mlp: grads = 0.3593902587890625 GiB\n```\n3.4 The checkpoints for recomputing have been released, along with the forward process\n```\n [Recompute Checkpoint] None = 0 GiB\n```\n\n4. At the end of the second micro batch forward, the only difference with the first micro batch is that each GPU also holds master gradients now.\n```\n [Static Grad] embed, all_attn, all_mlp, head: master_grads = 3.8141040802001953 GiB\n```\n\nThe maximum memory usually occurs in these cases, but maybe different when under different configs such as `chunk_loss_size`.\n"}