### README # slime Documentation We recommend new contributors start from writing documentation, which helps you quickly understand slime codebase. Most documentation files are located under the `docs/` folder. ## Docs Workflow ### Install Dependency ```bash apt-get update && apt-get install -y pandoc parallel retry pip install -r requirements.txt ``` ### Update Documentation You can update the documentation in the en and zh folders by adding Markdown or Jupyter Notebook files to the appropriate subdirectories. If you create new files, make sure to update index.rst (or any other relevant .rst files) accordingly. ## Build and Render ```bash # build english version bash ./build.sh en bash ./serve.sh en # build chinese version bash ./build.sh zh bash ./serve.sh zh ``` You can then visit `http://localhost:8000` to view the documentation. --- ### En/Get Started/Agent # Agentic RL Training Roadmap slime is not limited to single-turn RL. Its main advantage for agentic training is the combination of high-performance training, SGLang rollout serving, and pluggable data-generation interfaces. This makes it suitable for multi-turn tool use, sandbox interaction, subagent branches, context compaction, and test-based rewards. This page is a roadmap: use it to decide which docs and examples to read when plugging an agent workflow into slime. ## Where To Start | Goal | Recommended entry point | | :--- | :--- | | Run a custom agent loop, tool calls, RAG, browser/terminal/sandbox interaction for each sample | [`--custom-generate-function-path`](customization.md#2-custom-generate-function---custom-generate-function-path), [writing a custom generation function](quick_start.md#writing-custom-generation-function) | | Implement verifier rewards, test-based rewards, environment success checks, or an external reward service | [`--custom-rm-path`](customization.md#3-reward-model---custom-rm-path), [writing a custom reward function](quick_start.md#writing-custom-reward-function) | | Return multiple training samples from one prompt, such as subagent, multi-agent, or context-compaction segments | [fan-out return from custom generate](customization.md#returning-multiple-training-samples-for-one-prompt), [`examples/multi_agent`](../_examples_synced/multi_agent/README.md) | | Avoid blocking training on long-tail agent rollouts | [`examples/fully_async`](../_examples_synced/fully_async/README.md) | | Study a full end-to-end agent example with sandboxing, real code edits, and test-based grading | [`examples/coding_agent_rl`](../_examples_synced/coding_agent_rl/README.md) | | Improve SGLang serving throughput for multi-turn agents | [PD Disaggregation](../advanced/pd-disaggregation.md), [SGLang Config](../advanced/sglang-config.md) | | Enable SGLang optimization flags, router policies, or multi-model serving | [How to Use SGLang](usage.md#how-to-use-sglang), [SGLang Config](../advanced/sglang-config.md), [Speculative Decoding](../advanced/speculative-decoding.md), [Low Precision Training](../advanced/low-precision.md) | ## Recommended Integration Pattern Most agentic RL tasks should start with `--custom-generate-function-path`. This function converts one agent execution into slime-trainable `Sample` objects: fill `tokens`, `response_length`, `loss_mask`, and `status`, then either fill `reward` directly or let `--custom-rm-path` compute it. The agent workflow itself may speak in strings, chat messages, tool calls, environment observations, or framework-specific events. The training target, however, should stay token based. Preserve the model-sampled token ids and use `loss_mask` to separate trainable model output from prompt, template, tool-observation, or environment text. If one prompt rollout corresponds to one training sample, return a single `Sample`. If one rollout splits into multiple trainable segments, such as subagent trajectories, main-agent continuations, or pre/post-compaction segments, return `list[Sample]` and set the same `rollout_id` on all sibling samples. slime then keeps those samples together for train-step splitting and loss aggregation instead of counting them as independent rollouts. Reach for `--rollout-function-path` only when you need to replace the whole rollout orchestration. Common reasons include custom data-source scheduling, cross-rollout background queues, fully asynchronous generation, or workflows that cannot fit the default `sglang_rollout` prompt-by-sample structure. ## Agent Runtime Adapters slime includes protocol adapters for existing agent runtimes: - `slime.agent.adapters.AnthropicAdapter`: Anthropic Messages API, used by Claude Code style agents. - `slime.agent.adapters.OpenAIAdapter`: OpenAI Chat Completions and Responses APIs, used by OpenAI SDK / OpenAI Agents SDK style clients. Adapters are a convenience layer, not a separate agent framework. Their contract is message history in, sampled tokens out: they render the chat template, call SGLang with `input_ids` and `return_logprob=True`, and export the returned token ids/logprobs as trainable trajectory segments. They avoid re-tokenizing response text to recover the training target. Instantiate the protocol-specific adapter in your custom generate function, run its `app` with aiohttp, then manage each rollout through the adapter instance: ```python from slime.agent.adapters import AnthropicAdapter adapter = AnthropicAdapter( tokenizer=tokenizer, sglang_url=sglang_url, tool_parser=tool_parser, reasoning_parser=reasoning_parser, ) adapter.open_session(session_id, sampling_defaults=sampling_params) # Agent client sends requests to adapter.app. segments = await adapter.finish_session(session_id) ``` For multi-turn agents, use a stable `session_id`. The adapters pass it as `X-SMG-Routing-Key` so SGLang can route one session to the same worker and reuse prefix cache. ## Agent Serving And Performance Agentic rollouts tend to depend more heavily on serving configuration than ordinary single-turn generation: contexts are longer, requests are multi-turn, latency has a heavier tail, and the workflow may need actor, reference, reward, or tool-side models at the same time. - Regular SGLang server arguments are passed as `--sglang-*`. For example, SGLang's `--context-length` becomes `--sglang-context-length`, and `--mem-fraction-static` becomes `--sglang-mem-fraction-static`. - Router arguments are passed as `--router-*`. For multi-turn agents, consider `--router-policy consistent_hashing` so requests for the same `sample.session_id` go to the same worker and improve prefix-cache hit rate. See [Session-Affinity Routing for Multi-Turn Agents](../advanced/sglang-config.md#session-affinity-routing-for-multi-turn-agents). - Use `--sglang-config` for more complex topologies: PD disaggregation, multi-model serving, heterogeneous server groups, and per-group SGLang overrides. - For multi-turn or agentic RL, evaluate PD disaggregation. Prefill and decode have different workload shapes, and separating them makes it easier to scale each resource independently. - For rollout-throughput optimization, also see [Speculative Decoding](../advanced/speculative-decoding.md) and [Low Precision Training](../advanced/low-precision.md). ## Reference Example The full coding-agent example is [`examples/coding_agent_rl`](../_examples_synced/coding_agent_rl/README.md). It shows an end-to-end agent RL setup that is close to a real software-engineering workflow: each sample boots an isolated sandbox, the agent uses tools to edit code, the rollout captures a `git diff`, and a clean sandbox runs the tests to produce the reward. This example also demonstrates agent fan-out training. Its middleware splits one trajectory into `subagent`, `wipe` (the chain frozen before compaction), and `final` segments. `generate()` returns `list[Sample]`, and all segments share the same `rollout_id`. For smaller starting points, see [`examples/search-r1`](../_examples_synced/search-r1/README.md) for multi-turn tool use, [`examples/retool`](../_examples_synced/retool/README.md) for tool-augmented generation, and [`examples/multi_agent`](../_examples_synced/multi_agent/README.md) for the multi-agent pattern. --- ### En/Get Started/Customization # Customization Guide slime provides extensive customization capabilities through function path arguments. These allow you to inject custom logic at various stages of the training and rollout pipeline without modifying the core codebase. ## Overview of Customization Interfaces Below is a summary of all available customization interfaces and their purposes. | Interface Argument | Purpose | | :--- | :--- | | [`--rollout-function-path`](#1-rollout-function---rollout-function-path) | Override the entire rollout generation logic. | | [`--custom-generate-function-path`](#2-custom-generate-function---custom-generate-function-path) | Override only the generation step (e.g., for RAG or tool use). | | [`--custom-rm-path`](#3-reward-model---custom-rm-path) | Implement custom reward computation logic. | | [`--dynamic-sampling-filter-path`](#4-dynamic-sampling-filter---dynamic-sampling-filter-path) | Filter samples during dynamic sampling (e.g., DAPO). | | [`--buffer-filter-path`](#5-buffer-filter---buffer-filter-path) | Filter samples in the rollout buffer before training. | | [`--rollout-sample-filter-path`](#6-rollout-sample-filter---rollout-sample-filter-path) | Determine if individual samples participate in loss calculation. | | [`--rollout-all-samples-process-path`](#7-rollout-all-samples-process---rollout-all-samples-process-path) | Process all samples (including filtered ones) after rollout. | | [`--rollout-data-postprocess-path`](#8-rollout-data-postprocess---rollout-data-postprocess-path) | Post-process rollout data after log probs are computed. | | [`--custom-loss-function-path`](#9-custom-loss-function---custom-loss-function-path) | Implement custom training loss computation. | | [`--custom-tis-function-path`](#10-custom-tisrs-function---custom-tis-function-path) | Implement custom importance sampling for off-policy correction. | | [`--custom-pg-loss-reducer-function-path`](#11-custom-pg-loss-reducer---custom-pg-loss-reducer-function-path) | Customize pg_loss reduction (e.g., for Dr.GRPO). | | [`--custom-reward-post-process-path`](#12-reward-post-processing---custom-reward-post-process-path) | Custom post-processing of rewards before advantage computation. | | [`--custom-convert-samples-to-train-data-path`](#13-samples-to-train-data-conversion---custom-convert-samples-to-train-data-path) | Override the conversion of samples to training data format. | | [`--custom-rollout-log-function-path`](#14-logging-functions) | Custom logging for training rollouts. | | [`--custom-eval-rollout-log-function-path`](#14-logging-functions) | Custom logging for evaluation rollouts. | | [`--data-source-path`](#15-data-source---data-source-path) | Override the data source for rollout prompts. | | [`--eval-function-path`](#16-evaluation-function---eval-function-path) | Override the rollout function specifically for evaluation. | | [`--custom-megatron-init-path`](#17-megatron-hooks) | Custom initialization after Megatron setup. | | [`--custom-megatron-before-log-prob-hook-path`](#17-megatron-hooks) | Custom logic before log probability computation. | | [`--custom-megatron-before-train-step-hook-path`](#17-megatron-hooks) | Custom logic before each training step. | ## Agentic workflows through customization interfaces Agentic workflows — multi-turn tool use, sandbox interaction, environment feedback, verifier/test-based rewards — are an important class of data generation workflows. They plug into slime through the existing customization interfaces; slime does not require a separate agent framework. For most agentic use cases, **start with `--custom-generate-function-path` plus `--custom-rm-path`**, and only override the full rollout function when the default rollout loop is insufficient. | If you need to … | Use | | :--- | :--- | | Run a custom agent loop, tool calls, RAG, sandbox execution, browser/terminal interaction, or multi-turn generation for each sample, while reusing slime's default rollout loop | [`--custom-generate-function-path`](#2-custom-generate-function---custom-generate-function-path) | | Compute verifier rewards, test-based rewards, environment success checks, rule-based rewards, or call an external reward service | [`--custom-rm-path`](#3-reward-model---custom-rm-path) | | Replace the entire rollout orchestration (only when per-sample customization is not enough) | [`--rollout-function-path`](#1-rollout-function---rollout-function-path) | | Control task sampling, buffering, requeueing, or custom prompt/task sources | [`--data-source-path`](#15-data-source---data-source-path) | | Attach custom loss masks, metadata, or convert agentic outputs into training data | [`--rollout-data-postprocess-path`](#8-rollout-data-postprocess---rollout-data-postprocess-path), [`--custom-convert-samples-to-train-data-path`](#13-samples-to-train-data-conversion---custom-convert-samples-to-train-data-path) | | Debug long-running custom generation, verifier calls, tool calls, or sandbox steps | trace utilities in [`slime.utils.trace_utils`](../developer_guide/trace.md) | A native example of this pattern is [`examples/search-r1`](../../../examples/search-r1/), which adds search-augmented multi-turn generation via `--custom-generate-function-path` while keeping slime's default `sglang_rollout` outer loop. See also [`examples/multi_agent`](../../../examples/multi_agent/README.md) for a `--rollout-function-path`-based multi-agent pattern and [`examples/fully_async`](../../../examples/fully_async/README.md) for long-tail agentic generation. ## Detailed Interface Reference ### 1. Rollout Function (`--rollout-function-path`) **Default**: `slime.rollout.sglang_rollout.generate_rollout` **Purpose**: Override the entire rollout generation logic. **Signature**: ```python def generate_rollout(args, rollout_id, data_source, evaluation=False) -> RolloutFnTrainOutput | RolloutFnEvalOutput ``` **Use Cases**: - Implementing complex multi-turn conversations - Adding custom sampling strategies - Integrating external tools or APIs during generation **Example**: See [examples/multi_agent/rollout_with_multi_agents.py](../../../examples/multi_agent/rollout_with_multi_agents.py) --- ### 2. Custom Generate Function (`--custom-generate-function-path`) **Default**: `None` (uses built-in generate function) **Purpose**: Override only the generation step within the default rollout function. **Signature**: ```python async def custom_generate(args, sample: Sample, sampling_params: dict) -> Sample | list[Sample] ``` **Use Cases**: - Implementing tool-calling or function-calling capabilities - Adding retrieval-augmented generation (RAG) - Multi-turn conversation handling #### Returning multiple training samples for one prompt In agentic settings such as subagents, multi-agent execution, or context compaction, one prompt rollout can naturally split into multiple trainable segments. For example, a subagent trajectory and the main-agent continuation may both need to be trained, or the context before and after compaction may be represented as separate segments. You do not need to replace the whole rollout function for this. A `custom_generate` function may return `list[Sample]`. The key contract is that sibling samples produced by the same rollout must share the same `rollout_id`, so slime keeps them together for train-step splitting and loss aggregation instead of counting them as independent rollouts. ```python import copy from slime.utils.types import Sample async def custom_generate(args, sample: Sample, sampling_params: dict) -> list[Sample]: segments = await run_agent_and_split_segments(args, sample, sampling_params) rollout_id = sample.rollout_id if sample.rollout_id is not None else sample.index samples: list[Sample] = [] for segment in segments: s = copy.copy(sample) s.tokens = segment.tokens s.response = segment.response s.response_length = segment.response_length s.loss_mask = segment.loss_mask s.reward = segment.reward s.status = Sample.Status.COMPLETED s.rollout_id = rollout_id samples.append(s) return samples ``` If one full trajectory has a single total reward but is split into `K` training segments, a common pattern is to distribute that reward across the segments, for example by assigning `reward / K` to each segment, so the same rollout reward is not amplified. **Example**: See [examples/search-r1/generate_with_search.py](../../../examples/search-r1/generate_with_search.py) --- ### 3. Reward Model (`--custom-rm-path`) **Default**: `None` (uses built-in reward models based on `--rm-type`) **Purpose**: Implement custom reward computation logic. **Signature** (single sample mode): ```python async def custom_rm(args, sample: Sample) -> float ``` **Signature** (batch mode, when `--group-rm` is enabled): ```python async def batched_custom_rm(args, samples: list[Sample]) -> list[float] ``` **Use Cases**: - Custom rule-based rewards - Integration with external reward model services - Multi-dimensional reward signals **Built-in Options** (`--rm-type`): - `math`: Mathematical answer verification - `dapo`: DAPO-style scoring - `deepscaler`: DeepScaler rule-based reward - `f1`: F1 score computation - `gpqa`: GPQA reward computation - `ifbench`: IFBench reward computation - `remote_rm`: Remote reward model service (requires `--rm-url`) --- ### 4. Dynamic Sampling Filter (`--dynamic-sampling-filter-path`) **Default**: `None` **Purpose**: Filter samples during dynamic sampling (e.g., DAPO-style filtering). **Signature**: ```python def filter_function(args, samples: list[Sample], **kwargs) -> DynamicFilterOutput ``` **Return Type**: ```python @dataclass class DynamicFilterOutput: keep: bool # Whether to keep this sample group reason: str | None # Reason for filtering (for logging) ``` **Use Cases**: - Filtering out samples where all responses have the same reward - Implementing curriculum learning strategies - Quality-based sample selection **Example**: `slime.rollout.filter_hub.dynamic_sampling_filters.check_reward_nonzero_std` --- ### 5. Buffer Filter (`--buffer-filter-path`) **Default**: `None` **Purpose**: Filter samples in the rollout buffer before training. **Signature**: ```python def buffer_filter(args, rollout_id, buffer: list[list[Sample]], num_samples: int) -> list[list[Sample]] ``` **Use Cases**: - Removing low-quality samples before training - Implementing priority-based sample selection - Balancing sample distributions --- ### 6. Rollout Sample Filter (`--rollout-sample-filter-path`) **Default**: `None` **Purpose**: Determine whether individual samples participate in loss calculation. **Signature**: ```python def filter_function(args, samples: list[Sample]) -> None ``` **Note**: This function should directly modify the `remove_sample` attribute of each `Sample` object. **Use Cases**: - Filtering samples based on response quality - Implementing selective training strategies --- ### 7. Rollout All Samples Process (`--rollout-all-samples-process-path`) **Default**: `None` **Purpose**: Process all samples (including filtered ones) after rollout. **Signature**: ```python def process_function(args, samples: list[list[Sample]], data_source) -> None ``` **Use Cases**: - Logging and analysis of all generated samples - Computing statistics across filtered and kept samples --- ### 8. Rollout Data Postprocess (`--rollout-data-postprocess-path`) **Default**: `None` **Purpose**: Post-process rollout data after log probabilities are computed. **Signature**: ```python def postprocess_function(args, samples: list[list[Sample]]) -> None ``` **Use Cases**: - Updating loss masks based on computed values - Adding additional metadata to samples --- ### 9. Custom Loss Function (`--custom-loss-function-path`) **Default**: `None` (requires `--loss-type custom_loss`) **Purpose**: Implement custom training loss computation. **Use Cases**: - Novel RL objectives - Multi-objective optimization - Custom regularization terms --- ### 10. Custom TIS/RS Function (`--custom-tis-function-path`) **Default**: `None` **Purpose**: Implement custom importance sampling for off-policy correction. **Use Cases**: - Custom importance sampling ratio computation - Advanced off-policy correction methods **Example**: `examples/train_infer_mismatch_helper/mis.py:compute_mis_weights_with_cp` --- ### 11. Custom pg_loss Reducer (`--custom-pg-loss-reducer-function-path`) **Default**: `None` **Purpose**: Customize the reduction of pg_loss while other metrics (pg_clipfrac, ppo_kl, entropy_loss, etc.) still use the default sum_of_sample_mean. **Signature**: ```python def get_pg_loss_reducer( total_lengths: list[int], response_lengths: list[int], loss_masks: list[torch.Tensor], calculate_per_token_loss: bool = False, ) -> Callable[[torch.Tensor], torch.Tensor] ``` **Use Cases**: - Dr.GRPO: Divide by a constant instead of effective token count - Custom loss normalization strategies --- ### 12. Reward Post-Processing (`--custom-reward-post-process-path`) **Default**: `None` (uses default GRPO normalization) **Purpose**: Custom post-processing of rewards before advantage computation. **Use Cases**: - Custom reward normalization strategies - Reward shaping --- ### 13. Samples to Train Data Conversion (`--custom-convert-samples-to-train-data-path`) **Default**: `None` (uses built-in conversion logic) **Purpose**: Override the conversion of samples to training data format. **Signature**: ```python def convert_samples_to_train_data( args, samples: list[Sample] | list[list[Sample]], ) -> dict ``` **Return Type**: ```python dict: { "tokens": list[list[int]], # Token IDs for each sample "response_lengths": list[int], # Response lengths "rewards": list[float], # Normalized rewards "raw_reward": list[float], # Raw rewards "truncated": list[int], # Truncation flags (0 or 1) "sample_indices": list[int], # Sample indices "loss_masks": list[list[int]], # Loss masks for each sample # Optional fields: "round_number": list[int], # Round numbers (for rollout buffer) "rollout_log_probs": list, # Log probs (for off-policy correction) "rollout_routed_experts": list, # Routed experts (for MoE) "metadata": list, # Train metadata "multimodal_train_inputs": list, # Multimodal tensors (for VLM) "teacher_log_probs": list, # Teacher log probs (for distillation) } ``` **Use Cases**: - Handling `list[list[Sample]]` inputs - Custom data format requirements for training --- ### 14. Logging Functions #### Training Rollout Logging (`--custom-rollout-log-function-path`) **Signature**: ```python def log_rollout_data(rollout_id, args, samples, rollout_extra_metrics, rollout_time) -> bool ``` **Return**: `True` to skip default logging, `False` to continue with default logging. #### Evaluation Rollout Logging (`--custom-eval-rollout-log-function-path`) **Signature**: ```python def log_eval_rollout_data(rollout_id, args, data, extra_metrics) -> bool ``` **Return**: `True` to skip default logging, `False` to continue with default logging. --- ### 15. Data Source (`--data-source-path`) **Default**: `slime.rollout.data_source.RolloutDataSourceWithBuffer` **Purpose**: Override the data source for rollout prompts. **Base Class**: `slime.rollout.data_source.DataSource` **Required Methods**: ```python class CustomDataSource(DataSource): def get_samples(self, num_samples: int) -> list[list[Sample]]: """Return num_samples samples""" def add_samples(self, samples: list[list[Sample]]): """Add samples back to the data source""" def save(self, rollout_id): """Save state for checkpointing""" def load(self, rollout_id=None): """Load state from checkpoint""" def __len__(self): """Length of the data source. May change when samples are added/fetched.""" ``` --- ### 16. Evaluation Function (`--eval-function-path`) **Default**: Same as `--rollout-function-path` **Purpose**: Override the rollout function specifically for evaluation. **Use Cases**: - Different sampling parameters for evaluation - Evaluation-specific logic --- ### 17. Megatron Hooks #### Megatron Initialization (`--custom-megatron-init-path`) **Signature**: ```python def custom_init(args) -> None ``` **Purpose**: Custom initialization after Megatron setup. #### Before Log Prob Hook (`--custom-megatron-before-log-prob-hook-path`) **Signature**: ```python def custom_hook(args, model, store_prefix) -> None ``` **Purpose**: Custom logic before log probability computation. #### Before Train Step Hook (`--custom-megatron-before-train-step-hook-path`) **Signature**: ```python def custom_hook(args, rollout_id, step_id, model, optimizer, opt_param_scheduler) -> None ``` **Purpose**: Custom logic before each training step. --- ### 18. MoE Routing Replay Stabilize MoE RL training by recording and replaying expert routing decisions to ensure consistency. | Argument | Description | | --- | --- | | `--use-routing-replay` | Forward-backward routing consistency in training. ([arXiv:2507.18071](https://arxiv.org/abs/2507.18071)) | | `--use-rollout-routing-replay` | R3: Replay routing from rollout during training. Supported by slime's default `sglang_router` path. ([arXiv:2510.11370](https://arxiv.org/abs/2510.11370)) | --- ### 19. Disk Weight-Sync Post-Write Hook (`--custom-update-weight-post-write-path`) **Signature**: ```python def hook(args, version_dir: str, rollout_engines) -> None ``` **Purpose**: Called on each trainer rank after a disk weight sync's files are written (`--update-weight-transport disk`, full or delta mode), before the engines read them. Use it to publish the writes on a non-POSIX shared filesystem — e.g. upload pending writes to the backing object store — where another host cannot see the files without an explicit sync. The hook is called on every rank and must gate itself (e.g. once per container). The read-side counterpart runs inside the inference engine, on every host it spans, and is therefore an sglang server argument rather than a slime hook: pass `--sglang-custom-pull-weights-pre-read-hook ` with signature `hook(source_dir: str, target_version: int)` — called before `/pull_weights` reads the published weights (e.g. refresh the mount's view). See [Delta Weight Sync](../advanced/delta-weight-sync.md) for the full mechanism. ## Testing Custom Function Paths slime also provides CPU-only contract tests for customization interfaces. These tests resolve components through import-path strings, so they can validate both built-in hooks and user-defined implementations passed through the same CLI arguments used by training. The tests live under `tests/plugin_contracts/` and are grouped by hook shape: - `tests/plugin_contracts/test_plugin_rollout_contracts.py` Covers `--rollout-function-path` - `tests/plugin_contracts/test_plugin_generate_contracts.py` Covers `--custom-generate-function-path` - `tests/plugin_contracts/test_plugin_path_loading_contracts.py` Covers `--eval-function-path`, `--custom-rm-path`, `--dynamic-sampling-filter-path`, `--buffer-filter-path`, `--data-source-path`, `--rollout-sample-filter-path`, and `--rollout-all-samples-process-path` - `tests/plugin_contracts/test_plugin_runtime_hook_contracts.py` Covers `--custom-rollout-log-function-path`, `--custom-eval-rollout-log-function-path`, `--custom-reward-post-process-path`, `--custom-convert-samples-to-train-data-path`, and `--rollout-data-postprocess-path` Run all customization contract tests locally: ```bash python -m pytest \ tests/plugin_contracts/test_plugin_rollout_contracts.py \ tests/plugin_contracts/test_plugin_generate_contracts.py \ tests/plugin_contracts/test_plugin_path_loading_contracts.py \ tests/plugin_contracts/test_plugin_runtime_hook_contracts.py ``` Each test file can also be executed directly with `python tests/plugin_contracts/.py`, which keeps them compatible with `run-ci-changed`. A dedicated `run-ci-cpu-unittest` CI label is also available. Adding it to a PR triggers the CPU-only unit-test job, which runs the contract tests plus other lightweight unit tests in parallel (no GPU required). For user-defined implementations, you can either export environment variables such as `SLIME_CONTRACT_ROLLOUT_FUNCTION_PATH` and `SLIME_CONTRACT_CUSTOM_RM_PATH`, or pass overrides directly when running a test file, for example: ```bash python tests/plugin_contracts/test_plugin_rollout_contracts.py \ --rollout-function-path my_project.custom_rollout.generate_rollout ``` To validate your own custom implementation, replace the plugin paths used in these tests with your module path and keep the same assertions on signatures, return structure, and side effects. --- ### En/Get Started/Qa # FAQ 1. **Why do I see garbled text during training?** This situation generally occurs because Megatron is not loaded correctly. Please check if there is a corresponding checkpoint in the directory specified by `--load` or `--ref-load`. Note that Megatron can only load a directory that contains a `latest_checkpointed_iteration.txt` file. If you need to specify a particular iteration, you can refer to the current Megatron usage instructions. Generally, you can specify the step number using `--ckpt-step`. 2. **Why is my task stuck on the Ray submission page?** Please check whether your task is set up for co-located training and inference or decoupled training and inference. If it's **co-located** (training and inference share the same GPUs), please check: * Whether the `--colocate` parameter is set to enable co-located mode. * Whether the total number of GPUs for the current task is greater than or equal to `actor_num_nodes * actor_num_gpus_per_node`. If it's **decoupled**, please check: * Whether the total number of GPUs for the current task is greater than or equal to `actor_num_nodes * actor_num_gpus_per_node + rollout_num_gpus`. 3. **Why did I encounter an Out-of-Memory (OOM) error during training? What is `max_tokens_per_gpu` for?** OOM errors often happen because `max_tokens_per_gpu` is set too high. This parameter defines the maximum number of tokens that can be processed on each GPU during training. If you are concerned about OOM, you can initially set this value to `rollout_max_response_len / cp_size` and then increase it later to improve training efficiency. Note that `--max-tokens-per-gpu` is only active when `--use-dynamic-batch-size` is enabled. If you still experience OOM with a small `max_tokens_per_gpu`, check if the data generated in a single pass is too long. You may need to enable context parallelism (CP) with `--context-parallel-size`. If you are using custom data generation, check if the total length of multi-turn generations is much longer than expected. 4. **During multi-node training, what should I do if the `transformers` library reports it cannot find a model?** This usually happens when multiple processes try to read local files simultaneously using methods like `AutoConfig.from_pretrained` or `AutoModelForCausalLM.from_pretrained`, causing file system write conflicts. You can mitigate this issue by setting the `--model-name` argument. 5. **How do I resume training?** Simply set the `--load` directory to your `--save` directory. 6. **How is the batch size calculated?** A single rollout uses `rollout_batch_size` prompts. For each prompt, `n_samples_per_prompt` samples are generated. Therefore, one rollout contains a total of `rollout_batch_size * n_samples_per_prompt` data entries. You can use `--num-steps-per-rollout` to determine how many steps to run per rollout. This is equivalent to setting the `global_batch_size` to `rollout_batch_size * n_samples_per_prompt // num_steps_per_rollout`. 7. **Does slime perform data packing / variable-length (varlen) processing?** Yes. Data packing refers to the process of concatenating samples of varying lengths during training to improve GPU utilization. slime performs this operation by default. 8. **What should I do if the sglang component shows a `Max retries exceeded with url: /get_model_info (Caused by NewConnectionError)` error?** This issue primarily stems from port conflicts caused by multiple sglang servers running on a single machine. We are currently working with the sglang team to resolve this. A temporary workaround is to minimize the number of sglang servers on a single machine, for example, by setting `tp=8`. 9. **My gradient norm is very high and the training crashes. What should I do?** First, ensure that your data and model are compatible. For example, if your data already uses a chat template, check if this template matches the one used by the original model. If the data is correct, please refer to our [Debug Guide](../developer_guide/debug.md) for a more in-depth analysis. 10. **My sglang generation takes an extremely long time, GPU power is maxed out, and there's no output for a long while. Why?** Please verify that the model corresponding to `--hf-checkpoint` has its stop tokens configured correctly. If not, you can set them using the `--rollout-stop` or `--rollout-stop-token-ids` arguments. 11. **Sglang shows an `an illegal memory access was encountered` error.** According to [SGLang documentation](https://docs.sglang.io/references/faq.html), this could be an OOM error. Consider reducing the value of `--sglang-mem-fraction-static`. 12. **A `JSONDecodeError` occurs related to torch compile/inductor.** This is generally an issue with the torch compiler's cache read/write operations. You can try adding `"TORCHINDUCTOR_FORCE_DISABLE_CACHES": "1"` to the `env_vars` in your Ray configuration. 13. **Gradient becomes NaN or Inf during training.** You can try setting the `--no-check-for-nan-in-loss-and-grad` flag to skip the corresponding training steps. --- ### En/Get Started/Quick Start # Quick Start This document will guide you through setting up the environment and getting started with slime within one hour, covering environment configuration, data preparation, training startup, and key code analysis and modifications. ## Basic Environment Setup Since slime may contain temporary patches for sglang/megatron, to avoid potential environment configuration issues, we strongly recommend **users to use our latest Docker image**, which comes pre-configured with all dependencies. ### Hardware Support **slime** supports multiple NVIDIA GPU hardware platforms: - **B200 Series**: Fully supported with identical setup steps as H-series GPUs - **H-Series (H100/H200)**: Official support with comprehensive CI testing and stable performance **Important Notes**: - Latest Docker images are compatible with both B-series and H-series GPUs without additional configuration - Megatron backend on H-series GPUs has CI protection, thoroughly validated, recommended for production environments - B-series basic functionality is stable and suitable for development/testing, but currently lacks CI protection - Both hardware platforms use identical installation and startup procedures - For scenarios where Docker is not convenient, please refer to [build_conda.sh](https://github.com/THUDM/slime/blob/main/build_conda.sh); - For AMD support, please refer to [AMD Usage Tutorial](../platform_support/amd_tutorial.md). ### Pull and Start Docker Container Please execute the following commands to pull the latest image and start an interactive container: ```shell # Pull the latest image docker pull slimerl/slime:latest # Start the container docker run --rm --gpus all --ipc=host --shm-size=16g \ --ulimit memlock=-1 --ulimit stack=67108864 \ -it slimerl/slime:latest /bin/bash ``` ### Install slime slime is already installed in the docker image. To update to the latest version, please execute the following command: ```bash # Path can be adjusted according to actual situation cd /root/slime git pull pip install -e . --no-deps ``` ## Model and Dataset Download You can download required models and datasets from platforms like Hugging Face, ModelScope, etc. Here are the commands to download example resources using `huggingface_hub`: ```bash # Download model weights (GLM-Z1-9B) hf download zai-org/GLM-Z1-9B-0414 --local-dir /root/GLM-Z1-9B-0414 # Download training dataset (dapo-math-17k) hf download --repo-type dataset zhuzilin/dapo-math-17k \ --local-dir /root/dapo-math-17k # Download evaluation dataset (aime-2024) hf download --repo-type dataset zhuzilin/aime-2024 \ --local-dir /root/aime-2024 ``` ## Model Weight Conversion ### Convert from Hugging Face Format to Megatron Format When using Megatron as the training backend, you need to first convert Hugging Face format model weights to Megatron `torch_dist` format. First, load the configuration file of the target model. The `slime/scripts/models` directory contains configuration files for supported models. You need to `source` the corresponding model script to load the configuration parameters into the current environment. Here we use GLM4-9B model as an example, and it's similar for Qwen3-4B, Qwen3.5, Qwen3.6, GLM-4.7-Flash, Qwen3-30B-A3B, etc. ```bash cd /root/slime source scripts/models/glm4-9B.sh ``` Next, run the conversion script. Please note the following parameters: - `--hf-checkpoint`: Specify the path of the downloaded Hugging Face model weights. - `--save`: Specify the save path for the converted `torch_dist` format weights. ```bash PYTHONPATH=/root/Megatron-LM python tools/convert_hf_to_torch_dist.py \ ${MODEL_ARGS[@]} \ --hf-checkpoint /root/GLM-Z1-9B-0414 \ --save /root/GLM-Z1-9B-0414_torch_dist ``` For larger models, you can use `torchrun` to start the conversion script to convert with multi-gpus or even multi-nodes. Note: When converting the kimi-k2 model weights, you need to open config.json in the model path and change "model_type": "kimi_k2" to "model_type": "deepseek_v3". ### Convert from Megatron Format to Hugging Face Format You can use the following script to convert the saved Megatron checkpoints back to Hugging Face format: ```bash PYTHONPATH=/root/Megatron-LM python tools/convert_torch_dist_to_hf.py \ --input-dir /path/to/torch_dist_ckpt/iter_xxx/ \ --output-dir /root/GLM-Z1-9B-0414-iter_xxx \ --origin-hf-dir /root/GLM-Z1-9B-0414 ``` Note that as Megatron will do padding to embedding for better performance, it may happen that the converted embedding is not correct. In that case, please manually set `--vocab-size` during convertion. ## Training Script and Parameter Overview After completing the above preparation work, you can run the training script. ```bash cd /root/slime bash scripts/run-glm4-9B.sh ``` We still use the run-glm4-9B.sh script as an example to briefly analyze the main parameters. ### MODEL_ARGS: Model Configuration Parameters ```bash SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)" source "${SCRIPT_DIR}/models/glm4-9B.sh" ``` This part loads model configuration from the `scripts/models/glm4-9B.sh` file through the `source` command. These configurations are all hyperparameters required by Megatron. Since Megatron cannot directly read model configuration from checkpoints, it needs to be manually specified. We provide configuration examples for some commonly used models in the `scripts/models/` directory. > ⚠️ **Note**: > Please make sure to check whether the parameters in the model configuration file (such as `--rotary-base`) completely match the model you are currently using. Different versions of the same model structure may use different configuration values. If you need to modify, you can directly override after `source`, for example: > ```bash > source "${SCRIPT_DIR}/models/glm4-9B.sh" > MODEL_ARGS+=(--rotary-base 10000) > ``` ### CKPT_ARGS: Checkpoint and Path Parameters ```bash CKPT_ARGS=( # To load tokenizer and other information, won't actually use model weight parameters from hf path --hf-checkpoint /root/GLM-Z1-9B-0414 # Reference Model's Megatron format checkpoint --ref-load /root/GLM-Z1-9B-0414_torch_dist # Actor model loading path. Should typically match --save for checkpoint resumption # If empty or doesn't contain a valid checkpoint, loads from --ref-load instead --load /root/GLM-Z1-9B-0414_slime/ # Model save path during training --save /root/GLM-Z1-9B-0414_slime/ # Model save interval (steps) --save-interval 20 ) ``` ### ROLLOUT_ARGS: Data Generation (Rollout) Parameters The entire training process can be viewed as a closed loop of **"Data Sampling → Weight Update"**. **Phase One: Data Sampling (Rollout)** - `--rollout-batch-size`: Defines the **number of Prompts** for each round of sampling - `--n-samples-per-prompt`: Defines the **number of responses** generated for each Prompt (used for GRPO-like algorithms) > The product of the two determines the **total number of samples generated in a single round of sampling**. **Phase Two: Model Training (Training)** - `--global-batch-size`: Defines the **sample size required to execute one parameter update (optimizer.step)** - `--num-steps-per-rollout`: Defines **how many parameter updates to execute** using the current sampled data (we default to 1, using on-policy training) > The product of the two determines the **total number of samples consumed in a single round of training**. > ⚠️ The **parameter update** here refers to the optimizer.step() in the training phase, which is different from the weight synchronization (Weight Sync) initiated by the training engine to the inference engine. In this process, the "output" and "consumption" of each round must be equal, following this constraint: **`(rollout-batch-size × n-samples-per-prompt) = (global-batch-size × num-steps-per-rollout)`** - In slime, if `--num-steps-per-rollout` is set, `--global-batch-size` will be automatically set if not set, and if set, it will be validated using the above formula. **Training Process Count Control** - `--num-rollout`: Controls the **total number of execution rounds** of the entire **"sampling→training"** loop. ```bash ROLLOUT_ARGS=( # Prompt dataset, JSONL format --prompt-data /root/dapo-math-17k/dapo-math-17k.jsonl --input-key prompt --label-key label # If the `input_key` of Prompt is in OpenAI message format, apply Chat Template --apply-chat-template # Whether to shuffle data in Rollout phase --rollout-shuffle # Reward Model type. slime has built-in multiple types, also supports custom through --custom-rm-path --rm-type deepscaler # These five parameters control the relationship between rollout and train --num-rollout 3000 --rollout-batch-size 16 --n-samples-per-prompt 8 --num-steps-per-rollout 1 --global-batch-size 128 # Rollout sampling parameters --rollout-max-response-len 8192 --rollout-temperature 1 # Load balancing for data collected in rollout phase. It ensures that the computational workload allocated to each training process (DP rank) is roughly equal, which may be beneficial for training speed --balance-data ) ``` ### EVAL_ARGS: Evaluation Parameters The evaluation process inherits most of the Rollout parameters, but you can override them with the following parameters to implement evaluation strategies different from training. ```bash EVAL_ARGS=( # Evaluation interval (number of Rollouts) --eval-interval 5 # Prompt dataset for evaluation --eval-prompt-data aime /root/aime-2024/aime-2024.jsonl # Number of samples per evaluation Prompt --n-samples-per-eval-prompt 16 # Maximum response length during evaluation --eval-max-response-len 16384 # Sampling parameters during evaluation --eval-top-p 1 ) ``` ### PERF_ARGS: Performance and Parallelism Parameters This part mainly contains Megatron's parallel configuration. `--use-dynamic-batch-size` and `--max-tokens-per-gpu` are slime-specific optimizations. - `--max-tokens-per-gpu`: Maximum number of tokens processed per GPU. After enabling dynamic batching (`use_dynamic_batch_size`), the system will intelligently pack samples of varying lengths so that the total token count of each micro-batch approaches this limit, thereby improving training efficiency. If a single sample length exceeds this value, it will form an independent batch. In context parallel (CP) mode, `N` CP cards share the total length of `N * max_tokens_per_gpu`. - `--use-dynamic-batch-size`: Enable dynamic batching. At this time, `--micro-batch-size` will be ignored. > 💡 **Tip**: > slime always trains models through data packing methods and strictly ensures that per sample loss or per token loss is correct. Therefore, enabling dynamic batch size will not affect loss calculation, and it is strongly recommended to enable it. ```bash PERF_ARGS=( --tensor-model-parallel-size 2 --sequence-parallel --pipeline-model-parallel-size 1 --context-parallel-size 2 --expert-model-parallel-size 1 --expert-tensor-parallel-size 1 --recompute-granularity full --recompute-method uniform --recompute-num-layers 1 # --micro-batch-size 1 # This item is ignored when dynamic batching is enabled --use-dynamic-batch-size --max-tokens-per-gpu 4608 ) ``` ### GRPO_ARGS: GRPO Algorithm Parameters - `--use-kl-loss`: Enabling this option will load a reference model and calculate the KL divergence between the current model and the reference model as a monitoring metric. Whether KL divergence is included in the final training loss depends on the `--kl-loss-coef` parameter. If this parameter is set to 0, KL divergence will only be displayed as an observation metric and will not participate in loss calculation. ```bash GRPO_ARGS=( --advantage-estimator grpo --use-kl-loss --kl-loss-coef 0.00 --kl-loss-type low_var_kl --entropy-coef 0.00 --eps-clip 0.2 --eps-clip-high 0.28 ) ``` - `--advantage-estimator`: In addition to [GRPO](https://arxiv.org/abs/2402.03300), slime also supports several other training algorithms, such as [GSPO](https://arxiv.org/abs/2507.18071), [Reinforce++](https://arxiv.org/abs/2501.03262) and [Reinforce++ Baseline](https://arxiv.org/abs/2501.03262), and [PPO](https://arxiv.org/abs/1707.06347). - `--calculate-per-token-loss`: By default, slime calculates the loss on a per-sample basis, i.e., `mean(sum(sample_i) / len(sample_i))`. To calculate the loss on a per-token basis, i.e., `sum(sum(sample_i)) / sum(len(sample_i))`, you can enable this flag. - `--use-tis`: Enable this setting to use TIS (Truncated Importance Sampling), which is introduced by this [blog](https://fengyao.notion.site/off-policy-rl). ### OPTIMIZER_ARGS: Optimizer Parameters ```bash OPTIMIZER_ARGS=( --optimizer adam --lr 1e-6 --lr-decay-style constant --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 ) ``` ### SGLANG_ARGS: SGLang Service Parameters This part of parameters is used to configure SGLang inference service. - `--rollout-num-gpus-per-engine`: Basically equivalent to SGLang's `tp_size`. - Other SGLang parameters can be passed to slime by adding the `--sglang-` prefix, and slime will automatically forward them to SGLang. For example, to set SGLang's `--log-level INFO` parameter, just use `--sglang-log-level INFO`. > ⚠️ **Note**: > slime uses `sgl-router` to schedule multiple SGLang Servers. Without enabling DP Attention, `dp_size` will be calculated through `rollout-num-gpus/rollout-num-gpus-per-engine`. ```bash SGLANG_ARGS=( --rollout-num-gpus-per-engine 2 ) ``` ## Feature Introduction ### Colocated Actor and Rollout Under the default configuration, training (Actor) and inference (Rollout) resources are specified separately. Ray allocates `actor_num_nodes * actor_num_gpus_per_node` GPUs to the training part and `rollout_num_gpus` GPUs to inference, that is, training and inference are separated. When `--rollout-num-gpus` is explicitly set to `0`, slime still parses SGLang arguments and launches the router, but does not launch local SGLang servers. **Standard (Disaggregated) Configuration**: ```bash ray job submit ... \ -- python3 train.py \ --actor-num-nodes 1 \ --actor-num-gpus-per-node 4 \ --rollout-num-gpus 4 \ ... ``` In the above configuration, Actor uses 4 cards, and Rollout also uses 4 cards, running in parallel. **Training-Inference Integration (Colocated) Configuration**: To deploy training and inference on the same group of GPUs, please add the `--colocate` parameter. By default, this makes the number of cards for training and inference equal. You can explicitly set a different positive `--rollout-num-gpus`, for example to use more rollout GPUs than actor GPUs; the extra GPUs are used as rollout-only resources. If `--rollout-num-gpus 0` is set explicitly, slime launches only the router and no local SGLang servers. ```bash ray job submit ... \ -- python3 train.py \ --actor-num-nodes 1 \ --actor-num-gpus-per-node 8 \ --colocate \ ... ``` At this time, training and inference will share all 8 GPUs. > ⚠️ **Note**: > In training-inference integration mode, Megatron will occupy a certain amount of GPU memory before it can be offloaded after initialization. You need to adjust the `--sglang-mem-fraction-static` parameter to reduce SGLang's GPU memory usage ratio to avoid insufficient GPU memory. We usually recommend 0.8. ### Dynamic Sampling slime supports more complex sampling strategies, such as dynamic sampling used in [DAPO](https://dapo-sia.github.io/). To enable this feature, you need to configure the following parameters: ```bash --over-sampling-batch-size 64 \ --dynamic-sampling-filter-path \ slime.rollout.filter_hub.dynamic_sampling_filters.check_reward_nonzero_std ``` Here `over_sampling_batch_size` needs to be greater than `rollout_batch_size`, for example, configured as: ```bash --rollout-batch-size 32 \ --n-samples-per-prompt 8 \ --over-sampling-batch-size 64 \ ``` Then each sampling will directly sample 64 prompts, and each prompt will be sampled 8 times. Because slime performs asynchronous sampling internally, we will successively obtain 8 responses for each prompt. When receiving responses, the function corresponding to `dynamic_sampling_filter_path` will be used for filtering. If it passes, these 8 pieces of data will be kept; otherwise, they will be discarded. The filtering function `check_reward_nonzero_std` in the example will check whether the standard deviation of rewards for a group of samples is greater than zero, ensuring that the reward scores of each group of samples left have differences, thereby avoiding overly homogeneous data and improving data diversity. ```python def check_reward_nonzero_std(args, samples: list[Sample], **kwargs): rewards = [sample.get_reward_value(args) for sample in samples] keep = torch.tensor(rewards, dtype=torch.float).std() > 0.0 return DynamicFilterOutput( keep=keep, reason=None if keep else f"zero_std_{round(rewards[0], 1)}", ) ``` If the filtering function is very strict, causing a large number of prompt groups to be discarded, the system will monitor the number of pending tasks in `remaining_batch_size`. Once the number of pending tasks drops below the target number (32) due to too many being discarded, the system will automatically trigger a new round of oversampling, requesting `over_sampling_batch_size` (64) new prompts again to repeat the above process. ### Partial Rollout During dynamic sampling, a large number of requests may be aborted early, causing waste of computational resources. By enabling the `--partial-rollout` parameter, these half-generated samples can be cached and continued to be generated in the next Rollout phase, thereby improving performance. You can also customize the strategy for extracting data from the cache through `--buffer-filter-path`. The default strategy is `pop_first`, which extracts the required number of samples in first-in-first-out order. ```python def pop_first(args, rollout_id, buffer: list[list[Sample]], num_samples: int) -> list[list[Sample]]: num_to_pop = min(len(buffer), num_samples) samples = buffer[:num_to_pop] del buffer[:num_to_pop] return samples ``` That is, take out the first `num_samples` prompts corresponding to `num_samples * n_samples_per_prompt` pieces of data each time. > 💡 **Tip**: > The `sample.metadata` of each partial rollout sample stores the rollout id of the first generation, which can be used for data filtering. ### bf16 Training fp8 Inference slime directly supports bf16 training and fp8 inference. For Qwen3-4B model, you only need to download the following model: ```bash hf download Qwen/Qwen3-4B-FP8 --local-dir /root/Qwen3-4B-FP8 ``` And replace `--hf-checkpoint` with: ```bash # Used to load tokenizer and other information, actually won't use model weight parameters from hf path --hf-checkpoint /root/Qwen3-4B-FP8 # The megatron checkpoint still needs to be the dist weights converted from bf16 huggingface at the beginning, not modified because of FP8 rollout. --ref-load /root/Qwen3-4B_torch_dist ``` This will trigger fp8 inference. Currently, we will directly cast bf16 weights to fp8, and we will gradually add quantization schemes with less impact on accuracy in the future. For long-context rollout, you can also enable FP8 KV cache in SGLang to increase effective KV cache capacity: ```bash --sglang-kv-cache-dtype fp8_e4m3 ``` ⚠️ The training megatron checkpoint still needs to be the one converted from bf16 huggingface at the beginning. ## Multiturn Adaptation The slime framework is highly extensible and supports complex Agent scenarios (such as multi-turn interaction and tool calling). Its core mechanism is to rewrite the default data generation (Rollout) and reward calculation (Reward) logic through custom functions. This section uses an implementation based on [Search-R1](https://github.com/PeterGriffinJin/Search-R1) as an example to illustrate how to adapt slime to support multi-turn interaction. ### Adaptation Strategy Summary Adapting slime to support multi-turn interaction mainly includes three steps: 1. **Data Preparation**: Adapt the multi-turn interaction dataset to slime's `Sample` objects. Map conversation history, real labels, etc. to `prompt` and `label` fields, and store additional information such as tool definitions and intermediate states in the `metadata` field for subsequent function calls. 2. **Implement Custom Generation Function**: Write functions to simulate the interaction loop of "model generates action → executes tool → concatenates observation results", and correctly handle Loss Masking. 3. **Implement Custom Reward Function**: Write functions to evaluate complete interaction trajectories and return final reward scores. ### Data Preparation and Mapping To pass complex contextual information to custom functions, you need to aggregate all relevant additional fields during the **data preprocessing stage**. **Core Idea**: Merge all additional information in the dataset except `prompt` and `label` (such as `session_id`, `user_profile`, `tool_code`, etc.) to construct a **single, structured field** (for example, a column named `metadata` with JSON string content). ### Step One: Construct `metadata` Field in Dataset Before training starts, you need to process the original dataset. For example, your original data might be as follows: | question | final_answer | session_id | tool_code | | :--- | :--- | :--- | :--- | | "..." | "..." | "sess_123" | "code_A" | You need to convert it to: | question | final_answer | metadata | | :--- | :--- | :--- | | "..." | "..." | `{"session_id": "sess_123", "tool_code": "code_A"}` | ### Step Two: Specify Mapping in Training Script After completing data preparation, in the training script, map this preprocessed `metadata` column to slime's `Sample.metadata` field through `ROLLOUT_ARGS`. ```bash ROLLOUT_ARGS=( # 1. Specify the preprocessed dataset file --prompt-data /root/nq_search/train_processed.json # 2. Map "question" column to input prompt --input-key question # 3. Map "final_answer" column to evaluation label --label-key final_answer # 4. Load the pre-constructed "metadata" column into Sample.metadata # slime will automatically parse it as a Python dictionary --metadata-key metadata ) ``` Through this approach, you can easily access all pre-prepared structured information through methods like `sample.metadata['session_id']` in custom `generate` or `reward` functions. ### Writing Custom Generation Function First, specify a custom asynchronous Python function through the `--custom-generate-function-path` parameter. **Function Signature**: `async def generate(args, sample: Sample, sampling_params) -> Sample:` **Core Implementation Points**: 1. **Build Interaction Loop**: Create a loop to control maximum interaction rounds (such as `for _ in range(max_turns):`). 2. **Call Model to Generate Action**: In each round of the loop, call SGLang service to let the model generate the next action (such as `query`) based on the current conversation history. 3. **Parse and Execute Action**: Parse model output, identify actions and parameters, and call external tools or APIs (such as Google search). 4. **Build Observation Results**: Format the results returned by tools and append them to the conversation history as input for the next round. 5. **Handle Loss Masking**: This is the key to Agent training. - Note: `loss_mask` should be the same length as `response`, where tokens that need to calculate loss are 1, and masked ones are 0 - **Model-generated** tokens (such as thinking, action instructions) → set `loss_mask` to `1`, participate in loss calculation. - **Tool or environment returned** tokens (such as API results) → set `loss_mask` to `0`, do not participate in loss calculation. 6. **Termination Conditions**: End the loop when the model generates termination tags (such as `...`) or reaches maximum rounds. 7. **Encapsulate Return**: Fill the complete interaction history, token IDs, and `loss_masks` into the `Sample` object and return. **Code Example (Pseudocode)**: ```python async def generate(args, sample: Sample, sampling_params) -> Sample: # ... initialization ... prompt, full_response, loss_masks = sample.prompt, "", [] for _ in range(max_turns): # 1. Model generates action model_output = await call_sglang(prompt + full_response, ...) # ... tokenization and appending ... loss_masks += [1] * len(model_tokens) # loss_mask = 1 full_response += model_output # 2. Parse and execute action action, content = parse_action(model_output) if action == "search": # 3 & 4. Get and append observation results tool_output = await google_search(content) # ... tokenization and appending ... loss_masks += [0] * len(tool_tokens) # loss_mask = 0 full_response += tool_output elif action == "answer": break # end loop # 7. Fill and return Sample object sample.response = full_response sample.tokens = ... sample.loss_mask = loss_masks return sample ``` ### Writing Custom Reward Function Similarly, specify a custom reward function through `--custom-rm-path`. **Function Signature**: `async def reward_func(args, sample: Sample, **kwargs) -> float:` This function receives a complete `Sample` object and calculates scores based on the final interaction results. You can implement custom scoring logic here or call external Reward Model services. ### Configure in Training Script Finally, in the training script, enable the above custom functions through the following parameters: ```bash CUSTOM_ARGS=( # Specify the path of custom generation function (format: path.to.your.file:function_name) --custom-generate-function-path your_module.multiturn_logic.generate # Specify the path of custom reward function --custom-rm-path your_module.multiturn_logic.reward_func ) ``` ## Multi-Node Training for Large-Scale MOE Models To start a multi-node task, you need to first start a Ray cluster. On node 0, run: ```bash # Node0 (HEAD) ray start --head --node-ip-address ${MASTER_ADDR} \ --num-gpus 8 --disable-usage-stats # Other Nodes ray start --address=${MASTER_ADDR}:6379 --num-gpus 8 ``` After the Ray cluster has started, you can submit a job from node 0, for example: ```bash ray job submit --address="http://127.0.0.1:8265" \ --runtime-env-json='{ "env_vars": { "PYTHONPATH": "/root/Megatron-LM/", ... # e.g., no_proxy, API variables, etc. } }' \ -- python3 train.py \ --... # Other Megatron/SGLang/slime arguments ``` Optionally, the following environment variables may be needed based on your environment. For example, when there are multiple IPs and the wrong one is chosen in a Docker or SLURM envionment. We provide an example used in a SLURM + enroot multi-node system as follows: ``` export SLIME_HOST_IP=$(hostname -I | awk '{print $1}') export GLOO_SOCKET_IFNAME=$(ip -o -4 addr show | awk '$4 ~ /^10\\./ {print $2}') export NCCL_SOCKET_IFNAME=$(ip -o -4 addr show | awk '$4 ~ /^10\\./ {print $2}') export NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME=$(ip -o -4 addr show | awk '$4 ~ /^10\./ {print $2}') ``` slime has been deeply optimized for distributed training of large-scale Mixture of Experts (MoE) models. We provide some end-to-end training cases for reference: - [Example: 8xH100 Training GLM-4.7-Flash](../examples/glm4.7-30B-A3B.md) - [Example: 32xH100 Training GLM-5.2](../examples/glm5.2-744B-A40B.md) - [Example: 64xH100 Training GLM-4.7](../examples/glm4.7-355B-A32B.md) - [Example: 128xH100 Training DeepSeek-R1](../examples/deepseek-r1.md) - Scripts such as `scripts/run_qwen3_30b_a3b.py` and `scripts/run_glm45_355b_a32b.py` also support multi-node training. Their documentation is still being expanded. --- ### En/Get Started/Usage # Usage Guide ## Introduction to slime Parameters When using slime, parameters are primarily passed for the following purposes: 1. To allocate a portion of the GPUs in the cluster for training and another portion for inference. 2. To load Megatron for the training portion. 3. To load SGLang for the inference portion. 4. To configure the hyperparameters required for RL training. Following this order, we need to configure these parameters: ### Cluster Resource Allocation There are four main parameters for cluster resource allocation: - `--actor-num-nodes`: The number of nodes required for RL actor training. - `--actor-num-gpus-per-node`: The number of GPUs per node for RL actor training. - `--rollout-num-gpus`: The total number of GPUs required for rollout (inference). Set it to `0` to still parse SGLang arguments and launch the router without launching local SGLang servers. - `--rollout-num-gpus-per-engine`: The number of GPUs per inference engine. This parameter is similar to SGLang's `tp_size`. When performing multi-node serving, this value should be the total number of GPUs. For example, if serving one model with 2 nodes and 16 GPUs, this value should be 16. The reason for not using a parameter like `--sglang-tp-size` is that we might consider supporting SGLang's `dp_size` parameter in the future, which means an engine could contain multiple SGLang servers (currently, only `--sglang-dp-size` under the `--sglang-enable-dp-attention` condition is supported). With the default configuration, we use these parameters to allocate `actor_num_nodes * actor_num_gpus_per_node` GPUs for training and `rollout_num_gpus` GPUs for inference via Ray, thus achieving a separation of training and inference resources. For co-located training and inference, you also need to configure: - `--colocate`: Enables co-located training and inference. By default, this makes the number of GPUs for training and inference equal. You can explicitly set a different positive `--rollout-num-gpus`, for example to use more rollout GPUs than actor GPUs; the extra GPUs are used as rollout-only resources. If `--rollout-num-gpus 0` is set explicitly, slime launches only the router and no local SGLang servers. Additionally, slime supports Prefill and Decode disaggregation (PD Disaggregation). You can set the number of servers used for Prefill by setting the `--prefill-num-servers` argument. ### Choosing Training Backend slime supports multiple training backends, which can be selected via the `--train-backend` parameter: - `megatron` (default): Uses Megatron-LM as the training backend, supporting efficient training of large-scale models. ### Loading Megatron Unlike tools such as SGLang, vLLM, or Hugging Face Trainer, Megatron cannot directly read Hugging Face checkpoints. Instead, the user must configure the parameters for the model to be trained and load Megatron's own checkpoint format. Generally, we need to perform three preparatory steps: - Configure model parameters. - Configure parallelism and other optimizations. - Configure the checkpoint to be loaded. For details on some of Megatron's customizations and the principles behind how slime incorporates Megatron, please see the "How to Use Megatron" section. #### Configuring Model Parameters Taking qwen3 4B as an example, we need these parameters: ```bash MODEL_ARGS=( --num-layers 36 --hidden-size 2560 --ffn-hidden-size 9728 --swiglu --vocab-size 151936 --disable-bias-linear # attn head --num-attention-heads 32 --group-query-attention --num-query-groups 8 --kv-channels 128 --qk-layernorm # norm --normalization "RMSNorm" --norm-epsilon 1e-6 # rope --use-rotary-position-embeddings --rotary-base 1000000 ) ``` We provide configurations for common models in [scripts/models](../../../scripts/models), which you can reuse directly. If you are also using Megatron for pre-training/SFT, you can directly reuse the model configurations from your pre-training/SFT setup. Note: - slime will load all parameters of Megatron found in the `PYTHONPATH`, so you can find parameters and their descriptions within the Megatron in your environment. - slime uses data packing (also known as varlen or thd) for training. There is no need to configure `--seq-length` or `--max-positional-embedding`, as these parameters do not affect the maximum context length of the trained model. #### Setting Up Parallelism and Recomputation Megatron is currently the most comprehensively optimized training framework. A major reason for using Megatron is to pursue its excellent performance. Here is a brief introduction to configuring Megatron's parallelism and recomputation. - Here we list Megatron's parallelism strategies. For a more detailed discussion on the trade-offs between these strategies, please refer to more specialized discussions: - `--tensor-model-parallel-size`: TP - `--sequence-parallel`: Megatron's SP is an optimization for TP. It is recommended to always enable SP when using TP. - `--pipeline-model-parallel-size`: PP - `--context-parallel-size`: Megatron's CP, also known as sequence parallelism, generally corresponds to ring attention. - `--expert-model-parallel-size`: EP for MoE, where each GPU has `num_experts / ep_size` experts. - `--expert-tensor-parallel-size`: Megatron supports using a different `tp_size` for the MoE experts than for other parts of the model, which we generally call ETP. - For recomputation, the following flags are commonly configured in Megatron: - `--recompute-granularity`: This can be set to `full` or `selective`. `full` means complete recomputation, while `selective` recomputes less. If not configured, no recomputation is done. - `--recompute-method`: `uniform` is generally sufficient. - `--recompute-num-layers`: The number of layers per group for recomputation. A value of 1 is usually fine. #### Loading Megatron Checkpoints Megatron supports several of its custom checkpoint formats. Here are two of the more common ones: - The once mainstream `torch` format (corresponding to `--ckpt-format torch`). - The currently recommended `torch_dist` format (corresponding to `--ckpt-format torch_dist`). The `torch` format is Megatron's older storage format. Its structure consists of directories like `mp_rank_xxx`, where each directory corresponds to the checkpoint stored by each rank under a specific parallel partitioning. Because of this, when loading a `torch` format checkpoint, you must ensure that the checkpoint's parallelism strategy matches that of the training task. We recommend using the `torch_dist` format because it supports automatic parallel sharding, meaning that training tasks with different parallelism settings can share the same checkpoint, which is much more convenient. `torch_dist` is also the default format in the open-source Megatron. A `torch_dist` format checkpoint typically contains a set of `.distcp` files. When using `torch_dist`, you can convert from Hugging Face to `torch_dist` and vice versa using the checkpoint conversion method described in the [README](../../../README.md). In terms of storage structure, a Megatron checkpoint typically looks like this, assuming the storage path is `/ckpt/`: ```bash --/ckpt/ |-- latest_checkpointed_iteration.txt |-- iter_0000100/ |-- _0_0.distcp |-- _0_1.distcp |-- ... |-- iter_0000200/ |-- iter_0000300/ |-- ... ``` The `latest_checkpointed_iteration.txt` file records the latest training step. When loading a model, you should not directly pass `/ckpt/iter_xxxxxxx`, but rather pass `/ckpt/` and use `--ckpt-step` to select the corresponding training step (if `--ckpt-step` is not used, the step will be read from `latest_checkpointed_iteration.txt`). When using slime, there are three parameters for loading and saving checkpoints: - `--ref-load`: The Megatron checkpoint for the reference model. - `--load`: The Megatron checkpoint for the actor. If `--load` is not set, or if the specified directory does not exist or does not contain `latest_checkpointed_iteration.txt`, the actor will be initialized from the `--ref-load` checkpoint. - `--save`: The path where the actor's checkpoints are saved. Note: - Regardless of the checkpoint storage method (i.e., however `--ckpt-format` is set), Megatron can load both `torch` and `torch_dist` formats. ### Loading SGLang Loading SGLang is very simple. You only need: - `--hf-checkpoint`: The Hugging Face checkpoint used to initialize SGLang. Note: - Before the first training step, slime will synchronize the parameters from Megatron to SGLang. Therefore, the `--hf-checkpoint` does not need to contain the latest training parameters, and you do not need to change the HF checkpoint when resuming training. - By default, SGLang reads the maximum context length from the `config.json` in the Hugging Face checkpoint. You can use the `--sglang-context-length` parameter to override this value to support longer inference. - During co-located training and inference, although Megatron and SGLang will offload sequentially, they still need to leave some memory for each other. You need to adjust SGLang's total VRAM usage by reducing `--sglang-mem-fraction-static`. - slime supports passing through sgl-router parameters by adding a `router` prefix to the original parameter name. For example, sgl-router's `--balance-abs-threshold` parameter should be set as `--router-balance-abs-threshold`. Since sgl-router uses cache-aware routing by default, it may cause uneven request distribution. You can set `--router-balance-abs-threshold 0` to force balanced distribution, but this may affect prefix cache hit rate in multi-turn conversation scenarios. - If SGLang engines are pre-launched by an external system, connect to them with `--rollout-external-engine-addrs host1:port host2:port`. When the trainer and engines cannot form an NCCL weight-update group, use `--update-weight-mode full --update-weight-transport disk --update-weight-disk-dir /shared/fs/updates`; slime writes a complete HF checkpoint and asks SGLang to hot-load it through `update_weights_from_disk`. For large models or cross-cluster deployments, use `--update-weight-mode delta --update-weight-transport disk` instead. See [External Rollout Engines Roadmap](../advanced/external-rollout-engines.md) and [Delta Weight Sync](../advanced/delta-weight-sync.md). For details on some of SGLang's customizations and the principles behind how slime incorporates SGLang, please see the "How to Use SGLang" section. ### Data Format Currently, slime only supports loading files in `.jsonl` format, where each line of the file is a JSON object. An example of a single data entry (expanded) is as follows: ```json { "prompt": [ { "content": "Solve the following math problem step by step. The last line of your response should be of the form Answer: \\boxed{$Answer} where $Answer is the answer to the problem.\n\nIn triangle $ABC$, $\\sin \\angle A = \\frac{4}{5}$ and $\\angle A < 90^\\circ$. Let $D$ be a point outside triangle $ABC$ such that $\\angle BAD = \\angle DAC$ and $\\angle BDC = 90^\\circ$. Suppose that $AD = 1$ and that $\\frac{BD}{CD} = \\frac{3}{2}$. If $AB + AC$ can be expressed in the form $\\frac{a\\sqrt{b}}{c}$ where $a, b, c$ are pairwise relatively prime integers, find $a + b + c$.\n\nRemember to put your answer on its own line after \"Answer:\".", "role": "user", "step_loss_mask": 1, } ], "label": "34" } ``` This corresponds to the following configuration: ```bash --input-key prompt --label-key label --apply-chat-template ``` Please note that the `step_loss_mask` (default=1) here is for SFT phase. If it is set to 0, the turn will not contibute to the final loss; if it is set to 1, slime will use the normal `loss_mask`. Additionally, we provide a `metadata_key`, which defaults to `"metadata"`. When read, slime will load the metadata from the data, which can be helpful for custom data generation or creating custom reward models. If one run mixes multiple data sources, put `source_name` in the sample metadata: ```json { "prompt": "...", "label": "...", "metadata": { "source_name": "math" } } ``` The recommended contract is to put the source identifier in `metadata["source_name"]`; slime also recognizes a dynamically set `sample.source` from custom data sources. When rollout samples are converted to training data, slime carries one `source_names` entry per sample to the training side. The source lookup order is dynamic `sample.source`, then `metadata["source_name"]`; if neither is set, the source is `"unknown"`. This is useful for custom rewards, filters, logging, and future per-source routing such as OPD teacher selection. ### Hyperparameters for RL Training - `--advantage-estimator`: Specifies the RL algorithm for the training process. Currently supported algorithms include: - `grpo` ([https://arxiv.org/abs/2402.03300](https://arxiv.org/abs/2402.03300)) - `gspo` ([https://arxiv.org/abs/2507.18071](https://arxiv.org/abs/2507.18071)) - `cispo` ([https://arxiv.org/abs/2506.13585](https://arxiv.org/abs/2506.13585)) - `reinforce_plus_plus` and `reinforce_plus_plus_baseline` ([https://arxiv.org/abs/2501.03262](https://arxiv.org/abs/2501.03262)) - `ppo` ([https://arxiv.org/abs/1707.06347](https://arxiv.org/abs/1707.06347)) Note: On-policy distillation (OPD) is now orthogonal to the advantage estimator. Use `--use-opd` and `--opd-kl-coef` to enable OPD on top of any estimator. - `--calculate-per-token-loss`: By default, slime calculates loss on a per-sample basis, i.e., `mean(sum(sample_i) / len(sample_i))`. Enable this flag to calculate loss on a per-token basis, i.e., `sum(sum(sample_i)) / sum(len(sample_i))`. - `--use-tis`: Enable this setting to use TIS (Truncated Importance Sampling) (https://fengyao.notion.site/off-policy-rl). #### GRPO Algorithm GRPO (Group Relative Policy Optimization) is an RL algorithm proposed in DeepSeek-Math. Its core idea is to compute advantage through intra-group relative comparisons, eliminating the need for a separate critic model. To use GRPO, set: ```bash --advantage-estimator grpo ``` Key features of GRPO: - **No Critic Model Required**: GRPO samples multiple responses for the same prompt and estimates advantage by computing relative rewards within the group, avoiding the overhead of training and maintaining a critic model. - **Resource Efficient**: Since no critic model is needed, GPU resources can be fully utilized for actor training and inference. - **Simple to Use**: Easy configuration - just set `--advantage-estimator grpo`. Related parameters: - `--n-samples-per-prompt`: Number of responses sampled per prompt for intra-group comparison. - `--normalize-advantages`: Whether to normalize advantages. - `--eps-clip`: PPO-style clip range. #### PPO Algorithm PPO (Proximal Policy Optimization) is a classic RL algorithm that uses a critic model to estimate the value function for computing advantages. To use PPO, set: ```bash --advantage-estimator ppo ``` **Note: In PPO, the critic and actor share the same training GPU group.** You do not need to reserve a separate set of GPUs for the critic. Specifically: - PPO creates separate actor and critic training process groups, but places them on the same train placement group. - The critic training scale follows the actor configuration, and the actor / critic Megatron parallel topology must currently stay identical. - PPO forces train-side offload so that actor and critic can wake up and release memory on the same GPUs in turn. - There are currently no separate CLI arguments for configuring critic training resources; the critic node count and GPUs per node are derived from the actor configuration. PPO-related parameters: - `--megatron-config-path`: YAML config for role-specific Megatron overrides, such as setting critic-specific `load`, `save`, `lr`, or warmup parameters. - `--num-critic-only-steps`: Number of steps to train only the critic at the beginning of training. - `--eps-clip`: PPO clip range. - `--value-clip`: Clip range for value loss. - `--kl-coef`: KL penalty coefficient for reward shaping. ### Advanced Megatron Configuration (--megatron-config-path) For PPO workflows, you can use `--megatron-config-path` with a YAML file to override Megatron arguments separately for actor and critic. Common use cases include setting a different critic `lr`, or giving actor and critic different `load` / `save` paths. ```yaml megatron: - name: default role: actor overrides: lr: 1e-6 - name: default role: critic overrides: lr: 1e-5 ``` > **Note:** This configuration currently only supports PPO, and in current PPO the actor and critic must use the same Megatron parallel topology. The recommended pattern is to keep parallelism-related settings in the shared CLI arguments and put only role-specific differences in YAML. See [Megatron Config: Role-Based Training Overrides](../advanced/megatron-config.md) for details. ## Custom Rollout Function slime supports customizing data generation (rollout) to various degrees. - By default, it uses the `generate_rollout` function from [slime/rollout/sglang_rollout.py](https://github.com/THUDM/slime/blob/main/slime/rollout/sglang_rollout.py) for data generation. This file implements an asynchronous (asyncio) data generation flow based on SGLang and supports features like dynamic sampling and partial rollout. - You can completely replace the `generate_rollout` in sglang\_example.py by using the `--rollout-function-path` parameter. You just need to ensure that the function signature passed via `--rollout-function-path` is as follows: ```python def generate_rollout(args, rollout_id, data_source, evaluation=False) -> RolloutFnTrainOutput | RolloutFnEvalOutput: """ Args: args: the whole args rollout_id: int, the id of the rollout, used for deterministic data generation data_source: the data source to get and store samples evaluation: bool, whether the rollout is for evaluation or not Returns: RolloutFnTrainOutput | RolloutFnEvalOutput: the output of the rollout """ ... return output ``` Where: - `args`: The complete arguments used for the slime run. - `rollout_id`: The ID of the current data generation round, used to ensure data order when resuming training. - `data_source`: A globally unique data source in slime, which can be used to get initial prompts, data IDs, and store partially generated samples for later use. - `evaluation`: A boolean indicating if the rollout is for evaluation. You can configure a separate evaluation function using `--eval-function-path`. - The returned `Sample` type is defined in [slime/utils/types.py](https://github.com/THUDM/slime/blob/main/slime/utils/types.py). When implementing, you need to ensure the following fields are correctly set: - `tokens`: The tokens for the prompt + response. - `response_length`: The total length of the response. For multi-turn tasks, this is the length of the tokens remaining after the first-turn prompt. - `reward`: The reward for this data sample. - `status`: The status of this data sample (e.g., `Sample.Status.COMPLETED`, `Sample.Status.TRUNCATED`, `Sample.Status.ABORTED`, `Sample.Status.FAILED`). - `loss_mask` should be the same length as `response_length`, with `1` for tokens that should be included in the loss calculation and `0` for those that should be masked out. - In some cases, you may only need to replace the data generation logic. You can do this using `--custom-generate-function-path`. A simplified implementation of this function is as follows: ```python async def generate(args, sample: Sample, sampling_params) -> Sample: global TOKENIZER if TOKENIZER is None: TOKENIZER = AutoTokenizer.from_pretrained(args.hf_checkpoint, trust_remote_code=True) # send request to router output = await post( f"http://{args.sglang_router_ip}:{args.sglang_router_port}/generate", { "text": sample.prompt, "sampling_params": sampling_params, } ) prompt_tokens_ids = TOKENIZER(sample.prompt, add_special_tokens=False)["input_ids"] response_token_ids = TOKENIZER(output["text"], add_special_tokens=False)["input_ids"] # set sample sample.tokens = prompt_tokens_ids + response_token_ids sample.response_length = len(response_token_ids) finish_reason = output["meta_info"]["finish_reason"]["type"] if finish_reason == "length": sample.status = Sample.Status.TRUNCATED elif finish_reason == "abort": sample.status = Sample.Status.ABORTED else: sample.status = Sample.Status.COMPLETED sample.response = output["text"] return sample ``` For a more complete version, please refer to [slime/rollout/sglang_rollout.py](https://github.com/THUDM/slime/blob/main/slime/rollout/sglang_rollout.py). - Sometimes, you may also need to support a custom reward model. This can be configured by setting `--custom-rm-path`. ## How to Use SGLang slime implements a server-based engine using SGLang via the `HttpServerEngineAdapter` as an intermediary. ### Parameter Configuration slime incorporates almost all SGLang parameters by using SGLang's `ServerArgs.add_cli_args`. When setting an SGLang parameter, you need to add the `--sglang-` prefix. For example: - In co-located training and inference, you often need to limit `--mem-fraction-static`. This parameter should be changed to `--sglang-mem-fraction-static`. - During training, if you want SGLang to infer beyond the maximum context length specified in the Hugging Face checkpoint's `config.json`, you need to use `--context-length`, which becomes `--sglang-context-length` in slime. - For multi-node large EP inference, you might need `--ep-size`, `--enable-dp-attention`, `--dp-size`, `--moe-a2a-backend deepep`, etc. These can be passed as `--sglang-ep-size`, `--sglang-enable-dp-attention`, `--sglang-dp-size`, and `--sglang-moe-a2a-backend deepep` respectively. Some parameters related to slime's resource scheduling are configured by slime itself, for example: - `--tp-size` in slime is set using `--rollout-num-gpus-per-engine`. - `--model-path` in slime is set using `--hf-checkpoint`. The way SGLang parameters are integrated into slime can be found in [slime/backends/sglang_utils/arguments.py](https://github.com/THUDM/slime/blob/main/slime/backends/sglang_utils/arguments.py). ### How to Use the Router slime uses [sglang-router](https://github.com/sgl-project/sglang/tree/main/sgl-model-gateway) to manage the SGLang servers during the training process. You can configure the address of the [sglang-router](https://github.com/sgl-project/sglang/tree/main/sgl-model-gateway) using `--sglang-router-ip` and `--sglang-router-port`. If not configured, a router will be started by default within the cluster. After starting, all SGLang servers will register with the router via the `/add_worker` endpoint. When actually generating data, you only need to send HTTP requests to the router, which will perform load balancing and forward the requests to the servers. When you configure an external router using `--sglang-router-ip` and `--sglang-router-port`, slime will not start an internal router. Instead, it will register all its servers with this external router. You can then use this external router's address to implement more complex data generation workflows. Note that the router supports OpenAI-compatible APIs. ### Advanced Engine Configuration (--sglang-config) For advanced deployments, you can use `--sglang-config` with a YAML file to configure server groups, multi-model serving, and selective weight updates. **Multi-model deployment** allows serving multiple models simultaneously (e.g., an actor model that receives weight updates and a frozen reference/reward model): ```yaml sglang: - name: actor update_weights: true # receives weight updates from training (default) server_groups: - worker_type: regular num_gpus: 8 num_gpus_per_engine: 4 - name: ref model_path: /path/to/ref_model update_weights: false # frozen, no weight updates server_groups: - worker_type: regular num_gpus: 4 num_gpus_per_engine: 2 ``` Each model gets its own router. The per-model router info is accessible via `args.sglang_model_routers` (a dict mapping model name to `(ip, port)` tuples). Custom rollout functions can use `get_model_url(args, "ref")` from `slime.rollout.sglang_rollout` to route requests to a specific model. **Server group features:** - `worker_type`: `regular`, `prefill`, `decode`, or `placeholder` (reserves GPU slots without creating engines) - `overrides`: Dict of SGLang `ServerArgs` field overrides applied on top of `--sglang-*` CLI args - `num_gpus_per_engine`: Per-group TP size override ## How to Use Megatron slime supports different and lightly modified versions of Megatron by reusing common functions from the `megatron.training` directory, such as `parse_args`, `save_checkpoint`, and `load_checkpoint`. Therefore, when using it, you must ensure that Megatron is accessible in the `PYTHONPATH`, for example, by adding `export PYTHONPATH=/root/Megatron-LM` at runtime. ### Parameter Configuration slime directly imports all parameters of the Megatron in the current environment by using `from megatron.training.arguments import parse_args`. If the version of Megatron you are using has parameters defined outside of `parse_args`, you can configure them by passing them in, similar to how it's done in [train.py](https://github.com/THUDM/slime/blob/main/train.py), for example: ```python if __name__ == "__main__": try: from pretrain_gpt import extra_args_provider except: extra_args_provider = None args = parse_args(extra_args_provider) train(args) ``` ### Custom Parameters In some customized Megatron implementations, special operations need to be performed during initialization or before/after a training step. We have added the following plugins for this purpose: - `--custom-megatron-init-path`: Adds some initialization calls. - `--custom-megatron-before-log-prob-hook-path`: Is called before calculating the log probability. - `--custom-megatron-before-train-step-hook-path`: Is called before each training step. You could use this to mix in special training losses, for example. --- ### En/Examples/Deepseek R1 # DeepSeek R1 with 128xH100 This is an example of doing DeepSeek R1 RL training using 128xH100 GPUs. We will use bf16 for training, and an fp8 format with 128x128 blockwise quantization for inference. The maximum response length is 32k, and dynamic sampling will be used to filter data during training. Regarding parallelism, for sglang we will enable EP64, activate dp attention, and deepep. For the Megatron part, we will use TP8, PP4, EP32, and CP4. ⚠️ To save GPU memory, we will use CPU Adam. Each node (8xH100) will occupy 1.4\~1.5TB of host memory. If a single machine's host memory is insufficient, this can be resolved by adding more GPUs to expand the parallelism. ## Environment Setup For instructions on setting up the environment and downloading data, please refer to [Example: Qwen3-4B](qwen3-4B.md). To prepare the DeepSeek R1 checkpoint, first you will need to download DeepSeek-R1 to a directory accessible by all machines (hereinafter referred to as `$BASE_DIR`): ```bash hf download deepseek-ai/DeepSeek-R1 --local-dir $BASE_DIR/DeepSeek-R1 ``` The Hugging Face checkpoint for DeepSeek-R1 is in a block-quantized fp8 format. To convert it into a torch_dist format that Megatron can load, you first need to convert it to a bf16 Hugging Face checkpoint: ```bash cd slime/ python tools/fp8_cast_bf16.py --input-fp8-hf-path $BASE_DIR/DeepSeek-R1 --output-bf16-hf-path $BASE_DIR/DeepSeek-R1-bf16/ ``` Next, we need to convert the bf16 version of DeepSeek-R1 into the torch_dist format. Specifically, execute the following on 4 separate nodes: ```bash cd slime/ source scripts/models/deepseek-v3.sh PYTHONPATH=/root/Megatron-LM/ torchrun \ --nproc-per-node 8 \ --master-addr ${MASTER_ADDR} --master-port 12345 \ --nnodes=4 --node-rank ${NODE_RANK} \ tools/convert_hf_to_torch_dist.py \ ${MODEL_ARGS[@]} \ --tensor-model-parallel-size 1 \ --pipeline-model-parallel-size 8 \ --expert-tensor-parallel-size 1 \ --expert-model-parallel-size 4 \ --decoder-first-pipeline-num-layers 7 \ --decoder-last-pipeline-num-layers 6 \ --hf-checkpoint $BASE_DIR/DeepSeek-R1-bf16/ \ --save $BASE_DIR/DeepSeek-R1_torch_dist/ ``` Here, `MASTER_ADDR` is the IP of node0, and `NODE_RANK` indicates the node's index, both configured similarly to a multi-node `torchrun` setup. ## Executing the Training On node0, run: ```bash cd slime/ bash scripts/run-deepseek-r1.sh ``` On other nodes, you need to join the Ray cluster with the following command: ```bash ray start --address=${MASTER_ADDR}:6379 --num-gpus 8 --node-ip-address ${WORKER_IP} --disable-usage-stats" ``` Alternatively, if you have a list of all node IPs, for example, an MPI hostfile (where each line is `ip slot=8`), you can add the following commands after the `ray start --head` command in `scripts/run-deepseek-r1.sh`. This allows you to execute the training entirely from node0: ```bash for WORKER_IP in $(awk '{print $1}' $BASE_DIR/mpi_hostfile); do if [[ "$WORKER_IP" == "$MASTER_ADDR" ]]; then continue fi echo "Starting Ray worker on ${WORKER_IP}" ssh root@"${WORKER_IP}" \ "pkill -9 sglang ; ray stop --force ; pkill -9 python ; ray start --address=${MASTER_ADDR}:6379 --num-gpus 8 --node-ip-address ${WORKER_IP} --disable-usage-stats" & done wait ``` ### Parameter Introduction ```bash SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)" source "${SCRIPT_DIR}/models/deepseek-v3.sh" ``` This reads the model's config from [scripts/models/deepseek-v3.sh](https://github.com/THUDM/slime/blob/main/scripts/models/deepseek-v3.sh). These configs are all Megatron parameters. When training with Megatron, it cannot read the model config from the checkpoint, so we need to configure it ourselves. We provide some examples in [scripts/models](https://github.com/THUDM/slime/tree/main/scripts/models/). #### CKPT\_ARGS ```bash CKPT_ARGS=( # HF ckpt required by sglang, we also read the tokenizer from here --hf-checkpoint $BASE_DIR/DeepSeek-R1/ #--hf-checkpoint $BASE_DIR/DeepSeek-R1-bf16/ --ref-load $BASE_DIR/DeepSeek-R1_torch_dist/ # Actor's load directory, if empty, it will read from `ref_load` --load $BASE_DIR/DeepSeek-R1_slime/ --save $BASE_DIR/DeepSeek-R1_slime/ --save-interval 20 ) ``` slime will perform online quantization during training based on the quantization configuration in `hf_checkpoint`. For instance, in the current example, we are using the fp8 checkpoint of DeepSeek R1. This means that when updating parameters, we will first perform blockwise quantization on the parameters before passing them to sglang. #### PERF\_ARGS A set of Megatron parallelism parameters. Only `--use-dynamic-batch-size` and `--max-tokens-per-gpu` are added by slime. For the Megatron part, we have configured TP8, PP4, CP4, and EP32. Since DeepSeek-R1 has 61 layers, which is not divisible by 4, we have specifically configured the last pipeline stage to have 13 layers. `max_tokens_per_gpu` refers to the maximum number of tokens each GPU can process. When `use_dynamic_batch_size` is enabled, it will pack data of varying lengths within a batch as close to `max_tokens_per_gpu`. If a single data item exceeds `max_tokens_per_gpu`, it will form its own batch without truncation. When context parallelism (CP) is enabled, it allows CP GPUs to share a total length of `CP * max_tokens_per_gpu` tokens. When `dynamic_batch_size` is enabled, the traditional `micro_batch_size` is ignored. ⚠️ slime always trains the model using data packing and strictly guarantees per-sample or per-token loss. This means enabling dynamic batch size will not affect the loss calculation. It is recommended to enable it. ```bash PERF_ARGS=( --tensor-model-parallel-size 8 --sequence-parallel --pipeline-model-parallel-size 4 --context-parallel-size 4 --expert-model-parallel-size 32 --expert-tensor-parallel-size 1 --decoder-last-pipeline-num-layers 13 --recompute-granularity full --recompute-method uniform --recompute-num-layers 1 --use-dynamic-batch-size --max-tokens-per-gpu 16384 ) ``` #### GRPO\_ARGS Currently, these are some GRPO-related parameters in slime: ```bash GRPO_ARGS=( --advantage-estimator grpo --use-kl-loss --kl-loss-coef 0.00 --kl-loss-type low_var_kl --entropy-coef 0.00 --eps-clip 0.2 --eps-clip-high 0.28 ) ``` If you wish to train without loading the reference model, you need to remove `--use-kl-loss` and set `--kl-coef 0.00` (the default value is 0). #### OPTIMIZER\_ARGS We have configured CPU Adam with the following parameters to save GPU memory. ```bash OPTIMIZER_ARGS=( ... --optimizer-cpu-offload --overlap-cpu-optimizer-d2h-h2d --use-precision-aware-optimizer ) ``` #### SGLANG\_ARGS These are the parameters required by sglang. Here, `--rollout-num-gpus-per-engine` basically corresponds to sglang's `tp_size`. Other sglang parameters are passed to slime by adding a `--sglang-` prefix. To fully leverage sglang's large EP inference capabilities, we have added configurations like ep64, dp\_attention dp8, and deepep mode auto. The final `--sglang-server-concurrency` is a parameter specific to slime. It is used to prevent the sglang server's concurrent requests from becoming too large and crashing the HTTP server. The default is 512. However, since we now have one server for 8 nodes, we have adjusted it to 1024 to ensure that each dp rank can have a concurrency of 128. ```bash SGLANG_ARGS=( --rollout-num-gpus-per-engine 64 --sglang-mem-fraction-static 0.7 ----sglang-ep-size 64 # dp attention --sglang-enable-dp-attention --sglang-dp-size 8 --sglang-moe-dense-tp-size 1 --sglang-enable-dp-lm-head # enable deepep for sglang --sglang-moe-a2a-backend deepep --sglang-deepep-mode auto # make every dp rank have 128 concurrency --sglang-server-concurrency 1024 ) ``` #### MISC\_ARGS Some additional Megatron configurations. Note that Megatron's deepep is configured here. ```bash MISC_ARGS=( ... # use deepep for megatron --moe-enable-deepep --moe-token-dispatcher-type flex ) ``` --- ### En/Examples/Glm4.7 30B A3B # GLM-4.7-Flash with 8×H100 ## Environment Preparation The environment setup, data, and checkpoint conversion are the same as for the Qwen3-4B model. You can refer to [Example: Qwen3-4B Model](qwen3-4B.md), replacing mentions of Qwen3-4B with GLM-4.7-Flash. ### Download Model ```bash hf download THUDM/GLM-4.7-Flash --local-dir /root/GLM-4.7-Flash ``` ### Convert Checkpoint To convert the Hugging Face checkpoint to torch_dist format: ```bash cd /root/slime pip install -e . --no-deps source scripts/models/glm4.7-30B-A3B.sh PYTHONPATH=/root/Megatron-LM/ torchrun --nproc-per-node 8 \ tools/convert_hf_to_torch_dist.py \ ${MODEL_ARGS[@]} \ --hf-checkpoint /root/GLM-4.7-Flash/ \ --save /root/GLM-4.7-Flash_torch_dist/ ``` ## Run Training Execute the training script: ```bash cd /root/slime bash scripts/run-glm4.7-30B-A3B-8gpus.sh ``` ### Parameter Introduction Here, we will briefly introduce the key parts in the [run-glm4.7-30B-A3B-8gpus.sh](https://github.com/THUDM/slime/blob/main/scripts/run-glm4.7-30B-A3B-8gpus.sh) script. #### MoE Configuration GLM-4.7-Flash is a Mixture-of-Experts (MoE) model with 64 routed experts (top-4 activation) and 1 shared expert. It has 47 layers: 1 dense layer + 46 MoE layers. 1. To support running GLM-4.7-Flash on 8×H100, we need to enable Megatron's CPU Adam to save GPU memory: ```bash OPTIMIZER_ARGS=( ... --optimizer-cpu-offload --overlap-cpu-optimizer-d2h-h2d --use-precision-aware-optimizer ) ``` 2. Enable MoE optimization in Megatron. For single-node 8×H100, we use TP=1, EP=8: ```bash PERF_ARGS=( --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --context-parallel-size 1 --expert-model-parallel-size 8 --expert-tensor-parallel-size 1 ... ) ``` 3. Enable MoE optimization in SGLang with DP attention: ```bash SGLANG_ARGS=( --rollout-num-gpus-per-engine 8 --sglang-mem-fraction-static 0.7 --sglang-enable-dp-attention --sglang-dp-size 8 --sglang-enable-dp-lm-head --sglang-moe-dense-tp-size 1 ... ) ``` #### MTP Speculative Decoding (Inference Acceleration) GLM-4.7-Flash includes 1 MTP (Multi-Token Prediction) layer, which can be used for speculative decoding during inference to speed up rollout generation. To enable this, add the following to `SGLANG_ARGS`: ```bash SGLANG_ARGS=( ... # MTP speculative decoding (EAGLE) --sglang-speculative-algorithm EAGLE --sglang-speculative-num-steps 3 --sglang-speculative-eagle-topk 1 --sglang-speculative-num-draft-tokens 4 ) ``` This enables SGLang to use the model's MTP layer as a draft model for EAGLE-style speculative decoding. The MTP layer predicts multiple future tokens, and SGLang verifies them in parallel, leading to faster generation. > ⚠️ **Note**: Speculative decoding requires additional GPU memory. If you encounter OOM issues, try reducing `--sglang-mem-fraction-static` or disabling speculative decoding. #### MTP Training slime also supports training MTP layers jointly with the main model for models that have MTP weight conversion implemented (e.g., MiMo, GLM-4.7). When enabled, the relevant arguments are: ```bash # Add MTP layer count to model config MODEL_ARGS+=(--mtp-num-layers 1) # Enable MTP training SPEC_ARGS=( --enable-mtp-training --mtp-loss-scaling-factor 0.2 ) ``` - `--mtp-num-layers 1`: Tells Megatron to load the MTP layer from the checkpoint. - `--enable-mtp-training`: Enables gradient computation for MTP layers. Without this flag, the MTP layer is loaded but frozen. - `--mtp-loss-scaling-factor 0.2`: Weight of the MTP loss relative to the main policy loss. Default is 0.2. > **Note**: The native DeepSeek-layout loader uses the model's configured layer count when mapping MTP weights, including GLM-4.7-Flash's 47-layer architecture. > > For other models with MTP training support (e.g., MiMo), see `scripts/run-mimo-7B-rl-eagle.sh` as a reference. ### Multi-Node Support For multi-node training (e.g., 2×8 H100), use the multi-node script: ```bash cd /root/slime export BASE_DIR=/shared/path # accessible by all nodes bash scripts/run-glm4.7-30B-A3B.sh ``` Key modifications for multi-node: - Place the model and data on a path accessible by all nodes. - Set `MASTER_ADDR` to an address accessible by all nodes. - Remove CPU Adam configurations (distributed optimizer reduces per-GPU memory usage). - Adjust parallelism: e.g., TP=4, PP=2, EP=8, CP=2. When the total number of GPUs is not a multiple or divisor of the total number of experts (64), you can use `--sglang-ep-num-redundant-experts` to add redundant experts. For example, in a 24-GPU scenario: ```bash SGLANG_ARGS=( --rollout-num-gpus-per-engine 24 --sglang-mem-fraction-static 0.7 --sglang-ep-size 24 --sglang-enable-dp-attention --sglang-dp-size 3 --sglang-moe-dense-tp-size 1 --sglang-enable-dp-lm-head --sglang-ep-num-redundant-experts 16 ) ``` --- ### En/Examples/Glm4.7 355B A32B # GLM-4.7 with 64xH100 ## Environment Preparation The environment setup and dataset download are the same as for the Qwen3-4B model. You can refer to [Example: Qwen3-4B Model](qwen3-4B.md), replacing mentions of Qwen3-4B with GLM-4.7. ### Prerequisites GLM-4.7 follows the standard slime Docker environment. For multi-node launches, make sure all nodes can access the same `$BASE_DIR` path and unset proxy variables before starting Ray workers: ```bash unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY ``` ### Download Model ```bash hf download zai-org/GLM-4.7 --local-dir $BASE_DIR/GLM-4.7-355B-A32B ``` ### Convert Checkpoint To convert the Hugging Face checkpoint to torch_dist format, use 2 nodes x 8 GPUs: ```bash cd /root/slime pip install -e . --no-deps source scripts/models/glm4.5-355B-A32B.sh PYTHONPATH=/root/Megatron-LM/ torchrun \ --nproc-per-node 8 \ --master-addr ${MASTER_ADDR} --master-port 12345 \ --nnodes=2 --node-rank ${NODE_RANK} \ tools/convert_hf_to_torch_dist.py \ ${MODEL_ARGS[@]} \ --hf-checkpoint $BASE_DIR/GLM-4.7-355B-A32B/ \ --save $BASE_DIR/GLM-4.7-355B-A32B_torch_dist/ ``` Here, `MASTER_ADDR` is the IP of node0, and `NODE_RANK` is the node index, configured just like a multi-node `torchrun` job. ## Run Training Execute the training script from node0: ```bash cd /root/slime export BASE_DIR=/shared/path # accessible by all nodes bash scripts/run-glm4.7-355B-A32B.sh ``` ### Parameter Introduction Here, we briefly introduce the key parts in the [run-glm4.7-355B-A32B.sh](https://github.com/THUDM/slime/blob/main/scripts/run-glm4.7-355B-A32B.sh) script. #### MoE Configuration GLM-4.7 is a Mixture-of-Experts (MoE) model with 160 routed experts (top-8 activation) and shared experts. It has 92 layers: 3 dense layers + 89 MoE layers. 1. To support GLM-4.7 on 64xH100, we enable Megatron's CPU Adam to save GPU memory: ```bash OPTIMIZER_ARGS=( ... --optimizer-cpu-offload --overlap-cpu-optimizer-d2h-h2d --use-precision-aware-optimizer ) ``` 2. Enable MoE optimization in Megatron. For the provided 64xH100 example, we use TP=8, PP=4, CP=2, and EP=16: ```bash PERF_ARGS=( --tensor-model-parallel-size 8 --sequence-parallel --pipeline-model-parallel-size 4 --context-parallel-size 2 --expert-model-parallel-size 16 --expert-tensor-parallel-size 1 ... --use-dynamic-batch-size --max-tokens-per-gpu 16384 ) ``` 3. Enable MoE optimization in SGLang with DP attention: ```bash SGLANG_ARGS=( --rollout-num-gpus-per-engine 32 --sglang-mem-fraction-static 0.7 --sglang-enable-dp-attention --sglang-dp-size 4 --sglang-ep-size 32 --sglang-enable-dp-lm-head --sglang-moe-dense-tp-size 1 ... ) ``` #### MTP Speculative Decoding (Inference Acceleration) GLM-4.7 includes MTP (Multi-Token Prediction) layers that can be used for speculative decoding during inference to speed up rollout generation. To enable this, add the following to `SGLANG_ARGS`: ```bash SGLANG_ARGS=( ... # MTP speculative decoding (EAGLE) --sglang-speculative-algorithm EAGLE --sglang-speculative-num-steps 3 --sglang-speculative-eagle-topk 1 --sglang-speculative-num-draft-tokens 4 ) ``` This lets SGLang use the model's MTP layer as the draft model for EAGLE-style speculative decoding. > ⚠️ **Note**: Speculative decoding requires additional GPU memory. If you encounter OOM issues, try reducing `--sglang-mem-fraction-static` or disabling speculative decoding. #### MTP Training slime also supports training the MTP layers jointly with the main model for GLM-4.7. When enabled, the relevant arguments are: ```bash # Add MTP layer count to model config MODEL_ARGS+=(--mtp-num-layers 1) # Enable MTP training MTP_ARGS=( --enable-mtp-training --mtp-loss-scaling-factor 0.2 ) ``` - `--mtp-num-layers 1`: Tells Megatron to load the MTP layer from the checkpoint. - `--enable-mtp-training`: Enables gradient computation for MTP layers. Without this flag, the MTP layer is loaded but frozen. - `--mtp-loss-scaling-factor 0.2`: Weight of the MTP loss relative to the main policy loss. Default is 0.2. > **Note**: The native loader in `slime/backends/megatron_utils/hf_to_megatron/glm.py` maps both regular and MTP weights. #### Multi-Node Support This example already targets multi-node training. Before launching: - Place the model checkpoints and datasets on a path accessible by all nodes. - Set `MASTER_ADDR` to an address reachable by all nodes. - Unset proxy variables before starting Ray workers. - Provide a `HOSTFILE` listing worker IPs (one per line) and export `HOSTFILE=/path/to/hostfile` before launching. - Adjust parallelism coherently. The default example uses TP=8, PP=4, EP=16, CP=2, while rollout uses 32 GPUs per engine with SGLang DP attention. If your rollout GPU count does not divide the expert count cleanly, you can use `--sglang-ep-num-redundant-experts` to add redundant experts. ## FP8 Rollout The open-source FP8 checkpoint of GLM-4.7 uses per-channel quantization, which cannot currently enable DeepEP in SGLang. You can convert it to a 128x128 per-block FP8 checkpoint with the tool provided in slime: ```bash cd /root/slime python tools/convert_hf_to_fp8.py \ --model-dir $BASE_DIR/GLM-4.7-355B-A32B/ \ --save-dir $BASE_DIR/GLM-4.7-355B-A32B-FP8/ \ --strategy block --block-size 128 128 \ --max-workers 4 ``` Then switch `--hf-checkpoint` to `$BASE_DIR/GLM-4.7-355B-A32B-FP8/` to enable FP8 rollout. An example FP8 `SGLANG_ARGS` setup is: ```bash SGLANG_ARGS=( --rollout-num-gpus-per-engine 32 --sglang-mem-fraction-static 0.7 --sglang-enable-dp-attention --sglang-dp-size 32 --sglang-ep-size 32 --sglang-moe-dense-tp-size 1 --sglang-enable-dp-lm-head --sglang-cuda-graph-bs 1 2 4 8 $(seq 16 8 128) --sglang-speculative-algorithm EAGLE --sglang-speculative-num-steps 3 --sglang-speculative-eagle-topk 1 --sglang-speculative-num-draft-tokens 4 --sglang-moe-a2a-backend deepep --sglang-deepep-mode auto ) ``` --- ### En/Examples/Glm4 9B # GLM4-9B with 8xH100 ## Environment Setup After pulling the `slimerl/slime:latest` image, initialize the image environment as follows: ```bash cd /root/ git clone https://github.com/THUDM/slime.git cd slime/ pip install -e . --no-deps ``` Download the model and data: ```bash # hf checkpoint hf download zai-org/GLM-Z1-9B-0414 --local-dir /root/GLM-Z1-9B-0414 # train data hf download --repo-type dataset zhuzilin/dapo-math-17k \ --local-dir /root/dapo-math-17k # eval data hf download --repo-type dataset zhuzilin/aime-2024 \ --local-dir /root/aime-2024 ``` Convert the Hugging Face checkpoint to a Megatron-loadable Hugging Face checkpoint: ```bash # mcore checkpoint cd /root/slime source scripts/models/glm4-9B.sh PYTHONPATH=/root/Megatron-LM python tools/convert_hf_to_torch_dist.py \ ${MODEL_ARGS[@]} \ --hf-checkpoint /root/GLM-Z1-9B-0414 \ --save /root/GLM-Z1-9B-0414_torch_dist ``` ## Run Training Execute the training: ```bash cd /root/slime bash scripts/run-glm4-9B.sh ``` ### Parameter Introduction Here, we will briefly introduce the various components of the [run-glm4-9B.sh](https://github.com/THUDM/slime/blob/main/scripts/run-glm4-9B.sh) script: #### MODEL\_ARGS ```bash SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)" source "${SCRIPT_DIR}/models/glm4-9B.sh" ``` Reads the model's config from [scripts/models/glm4-9B.sh](https://github.com/THUDM/slime/blob/main/scripts/models/glm4-9B.sh). These configs are all Megatron parameters. When training with Megatron, it cannot read the model config from the checkpoint, so we need to configure it ourselves. We provide some examples in [scripts/models](https://github.com/THUDM/slime/tree/main/scripts/models/). ⚠️ Ensure that settings such as `--rotary-base` in the model configuration file match the settings of the model you are currently training. This is because different models, even with the same architecture, might use different values. If needed, you can override these parameters in your script after loading the model weights. For instance: ```bash source "${SCRIPT_DIR}/models/glm4-9B.sh" MODEL_ARGS += ( --rotary-base 10000 ) ``` #### CKPT\_ARGS ```bash CKPT_ARGS=( # HF checkpoint required by sglang; we also read the tokenizer from here --hf-checkpoint /root/GLM-Z1-9B-0414 # Checkpoint for the reference model --ref-load /root/GLM-Z1-9B-0414_torch_dist # Load directory for the actor; if empty, it will be loaded from `ref_load` --load /root/GLM-Z1-9B-0414_slime/ --save /root/GLM-Z1-9B-0414_slime/ --save-interval 20 ) ``` #### ROLLOUT\_ARGS ```bash ROLLOUT_ARGS=( # Prompt dataset, each line is a JSON object --prompt-data /root/dapo-math-17k/dapo-math-17k.jsonl --input-key prompt --label-key label # If the `input_key` in the prompt contains an OpenAI message, # tokenizer.apply_chat_template(...) will be executed --apply-chat-template # Whether to shuffle the data --rollout-shuffle # Reward model type. # slime provides many types and --custom-rm-path for custom models --rm-type deepscaler # Total number of rollouts to train --num-rollout 3000 # Number of prompts in one rollout --rollout-batch-size 32 # Number of responses to sample per prompt # A rollout will have rollout_batch_size * n_samples_per_prompt items --n-samples-per-prompt 8 # Rollout sampling parameters --rollout-max-response-len 8192 --rollout-temperature 1 # Number of training steps corresponding to one rollout --num-steps-per-rollout 1 # Whether to balance data during training, which might improve speed --balance-data ) ``` #### EVAL\_ARGS During evaluation, most rollout parameters are inherited, but we provide some parameters that can override the rollout configuration, allowing for different sampling strategies for training and evaluation. ```bash EVAL_ARGS=( --eval-interval 5 --eval-prompt-data /root/aime-2024/aime-2024.jsonl --n-samples-per-eval-prompt 16 --eval-max-response-len 16384 --eval-top-p 1 ) ``` #### PERF\_ARGS A set of Megatron's parallelism parameters. Only `--use-dynamic-batch-size` and `--max-tokens-per-gpu` are added by slime. `max_tokens_per_gpu` specifies the maximum number of tokens each GPU can process. When `use_dynamic_batch_size` is enabled, it will try to pack data of varying lengths within a batch up to `max_tokens_per_gpu`, thus forming a dynamic micro-batch size. If a single data item's length exceeds `max_tokens_per_gpu`, it will form its own batch without being truncated. When context parallelism (CP) is enabled, it allows the CP GPUs to share data with a total length of `CP * max_tokens_per_gpu` tokens. When `dynamic_batch_size` is enabled, the traditional `micro_batch_size` is ignored. ⚠️ slime always trains the model using data packing and strictly guarantees per-sample or per-token loss. This means enabling dynamic batch size will not affect the loss calculation. It is recommended to enable it. ```bash PERF_ARGS=( --tensor-model-parallel-size 2 --sequence-parallel --pipeline-model-parallel-size 1 --context-parallel-size 2 --expert-model-parallel-size 1 --expert-tensor-parallel-size 1 --recompute-granularity full --recompute-method uniform --recompute-num-layers 1 # --micro-batch-size 1 --use-dynamic-batch-size --max-tokens-per-gpu 4608 ) ``` #### GRPO\_ARGS Here are some GRPO-related parameters: ```bash GRPO_ARGS=( --advantage-estimator grpo --use-kl-loss --kl-loss-coef 0.00 --kl-loss-type low_var_kl --entropy-coef 0.00 --eps-clip 0.2 --eps-clip-high 0.28 ) ``` #### OPTIMIZER\_ARGS ```bash OPTIMIZER_ARGS=( --optimizer adam --lr 1e-6 --lr-decay-style constant --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 ) ``` #### SGLANG\_ARGS Parameters required by sglang. Here, `--rollout-num-gpus-per-engine` basically corresponds to sglang's `tp_size`. Other sglang parameters are passed to slime by adding the `--sglang-` prefix. ```bash SGLANG_ARGS=( --rollout-num-gpus-per-engine 2 ) ``` ⚠️ slime uses `sgl-router` to schedule multiple sglang servers. `dp_size` is not supported when DP attention is disabled. ### Co-located Training and Inference In the original script, the resource configuration is as follows: ```bash ray job submit ... \ -- python3 train.py \ --actor-num-nodes 1 \ --actor-num-gpus-per-node 4 \ --rollout-num-gpus 4 \ ... ``` This enables decoupled training and inference, where the training part will use 1 machine with 4 GPUs, and the inference will use another 4 GPUs. If you want to use the co-located feature, you need to add `--colocate` and remove `--rollout-num-gpus`: ```bash ray job submit ... \ -- python3 train.py \ --actor-num-nodes 1 \ --actor-num-gpus-per-node 8 \ --colocate \ ... ``` In this case, both training and inference will share these 8 GPUs. ⚠️ When using co-located training and inference, Megatron will always occupy some GPU memory. Therefore, you need to adjust `--sglang-mem-fraction-static` to reduce the proportion of memory occupied by sglang. ### Dynamic Sampling slime supports more complex sampling schemes, such as the dynamic sampling in [DAPO](https://dapo-sia.github.io/). To enable dynamic sampling, you need to configure: ```bash --over-sampling-batch-size ${OVER_SAMPLING_BS} \ --dynamic-sampling-filter-path \ slime.rollout.filter_hub.dynamic_sampling_filters.check_reward_nonzero_std \ ``` Here, `over_sampling_batch_size` needs to be greater than `rollout_batch_size`. For example: ```bash --rollout-batch-size 32 \ --n-samples-per-prompt 8 \ --over-sampling-batch-size 64 \ ``` The sampling will then directly sample 64 prompts, with 8 samples per prompt. Since slime performs asynchronous sampling internally, we will receive the 8 responses for each prompt sequentially. Upon receiving responses, they will be filtered using the function specified by `dynamic_sampling_filter_path`. If they pass, these 8 data points are kept; otherwise, they are discarded. The function in the example checks if the answers are all correct or all incorrect: ```python def check_reward_nonzero_std(args, samples: list[Sample], **kwargs): rewards = [sample.reward for sample in samples] return torch.tensor(rewards, dtype=torch.float).std() > 0.0 ``` When we have received 32 \* 8 data points, we will immediately stop sampling and will not wait for the remaining data to be sampled. If more than 32 prompts' worth of data is discarded (leaving fewer than 32 prompts' worth), we will then sample another 64 prompts. ### Partial Rollout During the process of dynamic sampling, a large number of requests are aborted prematurely. We can configure the `--partial-rollout` parameter to save these partially generated requests to a data buffer. In the next rollout, these requests can be retrieved to continue data generation, thereby further optimizing performance. You can customize how data is retrieved from the buffer by configuring the `--buffer-filter-path`. The default function is: ```python def pop_first(args, rollout_id, buffer: list[list[Sample]], num_samples: int) -> list[list[Sample]]: num_to_pop = min(len(buffer), num_samples) samples = buffer[:num_to_pop] del buffer[:num_to_pop] return samples ``` This means that each time, the data corresponding to the first `num_samples` prompts is retrieved, totaling `num_samples * n_samples_per_prompt` items. ⚠️ The `sample.metadata` of each partial rollout sample stores the rollout ID from its initial generation, which can be used for data filtering. --- ### En/Examples/Glm5.2 744B A40B # GLM-5.2 744B-A40B with 256xH100 This is the recommended 32-node, 256-H100 training example for [GLM-5.2](https://z.ai/blog/glm-5.2). The recipe uses the GLM-5.2 BF16 checkpoint for Megatron training and the FP8 checkpoint for SGLang rollout. It assumes two Hugging Face repositories will be available: - BF16: `zai-org/GLM-5.2` - FP8: `zai-org/GLM-5.2-FP8` ## Environment Setup For environment setup and dataset download, see [Example: Qwen3-4B](qwen3-4B.md). For multi-node training, make sure every node can access the same `$BASE_DIR` path. ### Download Model ```bash hf download zai-org/GLM-5.2 --local-dir $BASE_DIR/GLM-5.2 hf download zai-org/GLM-5.2-FP8 --local-dir $BASE_DIR/GLM-5.2-FP8 ``` The open-source GLM-5.2 config uses `model_type: glm_moe_dsa`, which slime maps onto the native DeepSeek-V3.2 loader since the two share the same DSA weight layout. ### Convert Checkpoint The training side needs the BF16 Hugging Face checkpoint converted to the Megatron torch_dist format. The torch_dist format is reshardable, so the conversion parallel layout does **not** need to match training; we use a layout that satisfies Megatron's expert-group constraint on the conversion node count. Run the following on 4 nodes / 32 GPUs: ```bash cd /root/slime pip install -e . --no-deps source scripts/models/glm5.2-744B-A40B.sh PYTHONPATH=/root/Megatron-LM/ torchrun \ --nproc-per-node 8 \ --master-addr ${MASTER_ADDR} --master-port 12345 \ --nnodes=4 --node-rank ${NODE_RANK} \ tools/convert_hf_to_torch_dist.py \ ${MODEL_ARGS[@]} \ --tensor-model-parallel-size 8 \ --pipeline-model-parallel-size 2 \ --decoder-last-pipeline-num-layers 40 \ --expert-model-parallel-size 16 \ --expert-tensor-parallel-size 1 \ --hf-checkpoint $BASE_DIR/GLM-5.2/ \ --save $BASE_DIR/GLM-5.2_torch_dist/ ``` Here, `MASTER_ADDR` is the IP of node0, and `NODE_RANK` is the current node index. `MODEL_ARGS` includes `--allgather-cp`, a slime-only flag, so `tools/convert_hf_to_torch_dist.py` registers it too (it is a no-op for conversion). On 32 GPUs, Megatron requires `expert_tp(1) * expert_model_parallel * pp` to divide the world size, so we convert with `EP=16` (`1*16*2=32`). The resulting checkpoint still loads at training-time `EP=32` because torch_dist is reshardable. ## Run Training From node0: ```bash cd /root/slime export BASE_DIR=/shared/path export MASTER_ADDR= export HOSTFILE=$BASE_DIR/hostfile # one worker IP per line, all 32 nodes bash scripts/run-glm5.2-744B-A40B.sh ``` If `HOSTFILE` is not set, join the other nodes to the Ray cluster manually. ### Parameter Introduction #### Model Configuration `scripts/models/glm5.2-744B-A40B.sh` contains the GLM-5.2 DSA + cross-layer index sharing configuration: 256 routed experts, top-8 activation, 1 shared expert, and 78 layers total (3 dense + 75 MoE). The DSA index sharing schedule, such as `index_topk_freq=4` and `index_skip_topk_offset=3`, is read from the Hugging Face config. The Megatron side uses the shared `slime_plugins.models.glm5.glm5:get_glm5_spec` provider and enables: ```bash --allgather-cp ``` This makes DSA + context parallel use the allgather-CP layout, and the index-share provider gathers index K/V across the CP group. #### Training Parallelism The default script targets 32 nodes and 256 GPUs: ```bash PERF_ARGS=( --tensor-model-parallel-size 4 --pipeline-model-parallel-size 8 --decoder-first-pipeline-num-layers 14 --decoder-last-pipeline-num-layers 16 --context-parallel-size 8 --expert-model-parallel-size 32 --expert-tensor-parallel-size 1 ... ) ``` `TP=4 * PP=8 * CP=8 = 256` GPUs form one training group (`DP=1`). The expert group constraint `expert_tp(1) * EP(32) * PP(8) = 256` divides the world size exactly (`expert_dp=1`). DSA cross-layer index sharing requires every pipeline stage to **start** on a "computing" layer. With `index_topk_freq=4` / `index_skip_topk_offset=3`, the computing layers are 1, 2, 3, 7, 11, ..., 75. A uniform `78/8` split would start stages on skip layers and fail the index-share assertion in `get_glm5_spec`. We therefore use `--decoder-first-pipeline-num-layers 14` and `--decoder-last-pipeline-num-layers 16`, leaving 6 middle stages of `(78-14-16)/6 = 8` layers each. The stage starts land on global layers 1, 15, 23, 31, 39, 47, 55, 63 — all computing layers. #### BF16 Training + FP8 Rollout The launcher writes the default paths directly in `CKPT_ARGS` and `ROLLOUT_ARGS`, matching the style of the other example scripts: ```bash CKPT_ARGS=( --hf-checkpoint $BASE_DIR/GLM-5.2-FP8 --ref-load $BASE_DIR/GLM-5.2_torch_dist --load $BASE_DIR/GLM-5.2_slime --save $BASE_DIR/GLM-5.2_slime --save-interval 20 ) ROLLOUT_ARGS=( --prompt-data $BASE_DIR/dapo-math-17k/dapo-math-17k.jsonl ... ) ``` `--hf-checkpoint` provides FP8 weights and the tokenizer for SGLang rollout; `--ref-load` is the Megatron torch_dist checkpoint converted from BF16. To debug BF16 rollout, change `--hf-checkpoint` in the script to `$BASE_DIR/GLM-5.2`. #### SGLang Configuration The rollout side runs with **prefill/decode (PD) disaggregation**: 1 prefill engine (64 GPU) + 3 decode engines (192 GPU) = 256 GPUs total (which must equal the colocated `rollout_num_gpus`). Each engine spans 64 GPUs with DP attention and `EP=64` (DeepEP's dispatch config map supports up to 160 EP ranks, so a single 256-GPU engine would be invalid). Prefill uses the `auto` DeepEP path; decode uses `low_latency` + `deep_gemm`. The split is configured via the `--sglang-config` YAML: ```yaml sglang: - name: default server_groups: - worker_type: prefill num_gpus: 64 num_gpus_per_engine: 64 overrides: { deepep_mode: auto, ... } - worker_type: decode num_gpus: 192 num_gpus_per_engine: 64 overrides: { deepep_mode: low_latency, moe_runner_backend: deep_gemm, ... } ``` PD transfer runs over RDMA/IB with the mooncake backend: ```bash --sglang-disaggregation-transfer-backend mooncake --sglang-disaggregation-ib-device mlx5_100,...,mlx5_107 ``` The rest of the rollout uses FP8 KV cache and the NSA + DeepEP backends: ```bash SGLANG_ARGS=( --sglang-enable-dp-attention --sglang-ep-size 64 --sglang-dp-size 64 --sglang-kv-cache-dtype fp8_e4m3 --sglang-nsa-decode-backend flashmla_kv --sglang-nsa-prefill-backend flashmla_sparse --sglang-attention-backend nsa ... ) ``` MTP / EAGLE speculative decoding is enabled using the model's own next-token-prediction layer (the GLM-5.2 checkpoint ships an MTP layer), so no separate draft model is needed: ```bash --sglang-speculative-algorithm EAGLE --sglang-speculative-num-steps 4 --sglang-speculative-eagle-topk 1 --sglang-speculative-num-draft-tokens 5 ``` `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK` must cover the largest decode batch: `max cuda_graph_max_bs (decode group = 12) * speculative_num_draft_tokens (5) = 60`, rounded up to `64`. A value below this trips the DeepEP low-latency dispatch buffer assertion during the decode group's CUDA-graph capture. #### Networking DeepEP/NVSHMEM communication across nodes needs the IB-aware NCCL settings in the Ray runtime env (`NCCL_SOCKET_IFNAME`, `NCCL_IB_*`, `NCCL_NET_GDR_LEVEL`, `NCCL_P2P_LEVEL=NVL`, `NCCL_NVLS_ENABLE=0`, `MC_IB_PCI_RELAXED_ORDERING`, ...). The script defaults to `SOCKET_IFNAME=eth0`; set `SOCKET_IFNAME` before launch if your environment differs, and it will be written to `GLOO_SOCKET_IFNAME`, `TP_SOCKET_IFNAME`, and `NCCL_SOCKET_IFNAME`. DeepEP also requires `NVSHMEM_DISABLE_NCCL=1`. --- ### En/Examples/Qwen3 4B # Qwen3-4B with 8xH100 ## Environment Setup After pulling the `slimerl/slime:latest` image, initialize the image environment as follows: ```bash cd /root/ git clone https://github.com/THUDM/slime.git cd slime/ pip install -e . --no-deps ``` Download the model and data: ```bash # hf checkpoint hf download Qwen/Qwen3-4B --local-dir /root/Qwen3-4B # train data hf download --repo-type dataset zhuzilin/dapo-math-17k \ --local-dir /root/dapo-math-17k # eval data hf download --repo-type dataset zhuzilin/aime-2024 \ --local-dir /root/aime-2024 ``` Convert the Hugging Face checkpoint into a format that Megatron can load: ```bash # mcore checkpoint cd /root/slime source scripts/models/qwen3-4B.sh PYTHONPATH=/root/Megatron-LM python tools/convert_hf_to_torch_dist.py \ ${MODEL_ARGS[@]} \ --hf-checkpoint /root/Qwen3-4B \ --save /root/Qwen3-4B_torch_dist ``` ## Run Training Execute the training script: ```bash cd /root/slime bash scripts/run-qwen3-4B.sh ``` ### Parameter Introduction Here, we will briefly introduce the various components of the [run-qwen3-4B.sh](https://github.com/THUDM/slime/blob/main/scripts/run-qwen3-4B.sh) script: #### MODEL\_ARGS ```bash SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)" source "${SCRIPT_DIR}/models/qwen3-4B.sh" ``` This reads the model's configuration from [scripts/models/qwen3-4B.sh](https://github.com/THUDM/slime/blob/main/scripts/models/qwen3-4B.sh). These are all Megatron parameters. When training with Megatron, it cannot read the model config from the checkpoint, so we need to configure it ourselves. We provide some examples in [scripts/models](https://github.com/THUDM/slime/tree/main/scripts/models/). ⚠️ Ensure that settings such as `--rotary-base` in the model configuration file match the settings of the model you are currently training. This is because different models, even with the same architecture, might use different values. If needed, you can override these parameters in your script after loading the model weights. For instance: ```bash source "${SCRIPT_DIR}/models/qwen3-4B.sh" MODEL_ARGS += ( --rotary-base 10000 ) ``` #### CKPT\_ARGS ```bash CKPT_ARGS=( # HF checkpoint required by sglang; we also read the tokenizer from here --hf-checkpoint /root/Qwen3-4B # Checkpoint for the reference model --ref-load /root/Qwen3-4B_torch_dist # Load directory for the actor; if empty, it will be loaded from `ref_load` --load /root/Qwen3-4B_slime/ --save /root/Qwen3-4B_slime/ --save-interval 20 ) ``` #### ROLLOUT\_ARGS ```bash ROLLOUT_ARGS=( # Prompt dataset, each line is a JSON object --prompt-data /root/dapo-math-17k/dapo-math-17k.jsonl --input-key prompt --label-key label # If the `input_key` in the prompt contains an OpenAI message, # tokenizer.apply_chat_template(...) will be executed --apply-chat-template # Whether to shuffle the data --rollout-shuffle # Reward model type. # slime provides many types and --custom-rm-path for custom models --rm-type deepscaler # Total number of rollouts to train --num-rollout 3000 # Number of prompts in one rollout --rollout-batch-size 32 # Number of responses to sample per prompt # A rollout will have rollout_batch_size * n_samples_per_prompt samples --n-samples-per-prompt 8 # Rollout sampling parameters --rollout-max-response-len 8192 --rollout-temperature 1 # Number of training steps corresponding to one rollout --num-steps-per-rollout 1 # Whether to balance data during training, which might improve speed --balance-data ) ``` #### EVAL\_ARGS During evaluation, most rollout parameters are inherited, but we provide some parameters that can override the rollout configuration to allow for different sampling strategies for training and evaluation. ```bash EVAL_ARGS=( --eval-interval 5 --eval-prompt-data /root/aime-2024/aime-2024.jsonl --n-samples-per-eval-prompt 16 --eval-max-response-len 16384 --eval-top-p 1 ) ``` #### PERF\_ARGS This is a set of Megatron's parallelism parameters. Only `--use-dynamic-batch-size` and `--max-tokens-per-gpu` are added by slime. `max_tokens_per_gpu` specifies the maximum number of tokens each GPU can process. When `use_dynamic_batch_size` is enabled, it attempts to pack data of varying lengths within a batch as close to `max_tokens_per_gpu` as possible, thus forming a dynamic micro-batch size. If a single data item exceeds `max_tokens_per_gpu`, it forms its own batch without being truncated. When context parallelism (CP) is enabled, it allows the CP GPUs to share a total of `CP * max_tokens_per_gpu` tokens. When `dynamic_batch_size` is enabled, the traditional `micro_batch_size` is ignored. ⚠️ slime always trains the model using data packing and strictly guarantees per-sample or per-token loss. This means enabling dynamic batch size will not affect the loss calculation. It is recommended to enable it. ```bash PERF_ARGS=( --tensor-model-parallel-size 2 --sequence-parallel --pipeline-model-parallel-size 1 --context-parallel-size 1 --expert-model-parallel-size 1 --expert-tensor-parallel-size 1 --recompute-granularity full --recompute-method uniform --recompute-num-layers 1 # --micro-batch-size 1 --use-dynamic-batch-size --max-tokens-per-gpu 9216 ) ``` #### GRPO\_ARGS Here are some GRPO-related parameters: ```bash GRPO_ARGS=( --advantage-estimator grpo --use-kl-loss --kl-loss-coef 0.00 --kl-loss-type low_var_kl --entropy-coef 0.00 --eps-clip 0.2 --eps-clip-high 0.28 ) ``` #### OPTIMIZER\_ARGS ```bash OPTIMIZER_ARGS=( --optimizer adam --lr 1e-6 --lr-decay-style constant --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 ) ``` #### SGLANG\_ARGS These are the parameters required by sglang. Here, `--rollout-num-gpus-per-engine` basically corresponds to sglang's `tp_size`. Other sglang parameters are passed to slime by adding the `--sglang-` prefix. ```bash SGLANG_ARGS=( --rollout-num-gpus-per-engine 2 --sglang-mem-fraction-static 0.7 ) ``` ⚠️ slime uses `sgl-router` to schedule multiple sglang servers. `dp_size` is not supported when DP attention is disabled. ### Dynamic Sampling slime supports more complex sampling schemes, such as the dynamic sampling in [DAPO](https://dapo-sia.github.io/). To enable dynamic sampling, you need to configure: ```bash --over-sampling-batch-size ${OVER_SAMPLING_BS} \ --dynamic-sampling-filter-path \ slime.rollout.filter_hub.dynamic_sampling_filters.check_reward_nonzero_std \ ``` Here, `over_sampling_batch_size` needs to be greater than `rollout_batch_size`. For example, you can configure it as: ```bash --rollout-batch-size 32 \ --n-samples-per-prompt 8 \ --over-sampling-batch-size 64 \ ``` In this case, the sampling process will directly sample 64 prompts, with 8 samples per prompt. Since slime performs asynchronous sampling internally, we will receive the 8 responses for each prompt sequentially. Upon receiving the responses, the function specified by `dynamic_sampling_filter_path` is used for filtering. If the samples pass the filter, these 8 data points are kept; otherwise, they are discarded. The function in the example checks if the rewards for the samples are not all identical (i.e., not all correct or all incorrect): ```python def check_reward_nonzero_std(args, samples: list[Sample], **kwargs): rewards = [sample.reward for sample in samples] return torch.tensor(rewards, dtype=torch.float).std() > 0.0 ``` When we have received 32 \* 8 data points, we will immediately stop the current sampling round and will not wait for the remaining data to be sampled. If more than 32 prompts' worth of data is discarded (leaving fewer than 32 prompts' worth), we will then sample another 64 prompts. ### Partial Rollout During the process of dynamic sampling, a large number of requests are aborted prematurely. We can configure the `--partial-rollout` parameter to save these partially generated requests to a data buffer. In the next rollout, these requests can be retrieved to continue data generation, thereby further optimizing performance. You can customize how data is retrieved from the buffer by configuring the `--buffer-filter-path`. The default function is: ```python def pop_first(args, rollout_id, buffer: list[list[Sample]], num_samples: int) -> list[list[Sample]]: num_to_pop = min(len(buffer), num_samples) samples = buffer[:num_to_pop] del buffer[:num_to_pop] return samples ``` This means that each time, the data corresponding to the first `num_samples` prompts is retrieved, totaling `num_samples * n_samples_per_prompt` items. ⚠️ The `sample.metadata` of each partial rollout sample stores the rollout ID from its initial generation, which can be used for data filtering. ### Decoupled Training and Inference In the original script, the resource configuration is as follows: ```bash ray job submit ... \ -- python3 train.py \ --actor-num-nodes 1 \ --actor-num-gpus-per-node 8 \ --colocate \ ... ``` This enables co-located training and inference, where the training part uses 1 machine with 8 GPUs, and inference shares these 8 GPUs with training. If you want to use the decoupled training and inference feature, you need to remove `--colocate` and configure `--rollout-num-gpus`. For example: ```bash ray job submit ... \ -- python3 train.py \ --actor-num-nodes 1 \ --actor-num-gpus-per-node 2 \ --rollout-num-gpus 6 \ ... ``` In this case, 2 GPUs will be allocated for training, and 6 GPUs will be allocated for inference. ⚠️ If the concurrency on each sglang server is too high, it may exceed sglang's default CUDA graph concurrency limit (the default maximum is 160), which will affect inference speed. You can adjust this in the following two ways: 1. Use `--sglang-server-concurrency` to limit the maximum number of concurrent requests sent to a single sglang server. For example: ```bash --sglang-server-concurrency 160 ``` 2. Use `--sglang-cuda-graph-bs` (which corresponds to sglang's native `--cuda-graph-bs` argument) to increase the number of CUDA graphs initialized by sglang. For example: ```bash --sglang-cuda-graph-bs 1 2 4 8 $(seq 16 8 256) ``` ### Asynchronous Training When you separate training and inference, you may notice that the training and inference GPUs are always waiting for each other. To prevent these resources from being idle, we can enable asynchronous training. This can be done by changing `train.py` to `train_async.py` in the startup script. By doing this, slime will generate data for the next rollout while training on the current one. The only difference between `train.py` and `train_async.py` lies in the synchronization logic of the training loop. We achieve this by using Ray's asynchronous features (`.remote`, `ray.get`). --- ### En/Examples/Qwen3 4b Base Openhermes # SFT Qwen3-4B-Base ## Environment Preparation First, we need to create a mirror environment and convert the `Qwen3-4B-Base` model by following the [Example: Qwen3-4B Model](qwen3-4B.md). After that, we will process the SFT data. Here, we use the classic [OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5) as an example. First, we process the data into a format suitable for `slime` to load. You can use the following script to add a column that conforms to the OpenAI message format and save it to `/root/openhermes2_5.parquet`. ```python from datasets import load_dataset ds = load_dataset("teknium/OpenHermes-2.5")["train"] def convert(sample): conversations = sample["conversations"] def convert_role(role): if role == "human": return "user" elif role == "gpt": return "assistant" elif role == "system": return "system" else: raise ValueError(f"Unknown role: {role}") messages = [ { "role": convert_role(turn["from"]), "content": turn["value"], } for turn in conversations ] return {"messages": messages} ds = ds.map(convert) ds.to_parquet("/root/openhermes2_5.parquet") ``` ## Execute Training Execute the training: ```bash cd /root/slime bash script/run-qwen3-4B-base-sft.sh ``` ### Parameter Introduction You can compare [run-qwen3-4B-base-sft.sh](https://github.com/THUDM/slime/blob/main/scripts/run-qwen3-4B-base-sft.sh) with [run-qwen3-4B.sh](https://github.com/THUDM/slime/blob/main/scripts/run-qwen3-4B.sh). You will find that besides changing the model from the instruct version to the base model, the main adjustments are as follows: 1. Removed `SGLANG_ARGS` and `GRPO_ARGS`. This is because it is not necessary to start SGLang or configure GRPO-related settings during the SFT process. 2. Renamed `ROLLOUT_ARGS` to `SFT_ARGS` and configured it as follows: ```bash SFT_ARGS=( --rollout-function-path slime.rollout.sft_rollout.generate_rollout --prompt-data /root/openhermes2_5.parquet --input-key messages --rollout-shuffle --num-epoch 3 --rollout-batch-size 128 --global-batch-size 128 --loss-type sft_loss --calculate-per-token-loss --disable-compute-advantages-and-returns --debug-train-only ) ``` SFT actually reuses the custom rollout functionality of slime. By using `--rollout-function-path`, the data generation part is switched from the RL rollout that uses `sglang` to the SFT version that reads data from a file, which is `slime.rollout.sft_rollout.generate_rollout`. For SFT, it is recommended to set `rollout_batch_size` and `global_batch_size` to the same value and not to configure `n_samples_per_prompt`. This is equivalent to training one batch right after reading one batch. `slime` also supports different loss types, and we configure the SFT loss using `--loss-type sft_loss`. As for `--calculate-per-token-loss`, this is because `slime` defaults to calculating the per-sample mean for GRPO. In general SFT training, the average is taken over all unmasked tokens in a batch, so it is recommended to configure this. Finally, `--disable-compute-advantages-and-returns` indicates that there is no need to pre-calculate log probabilities during the SFT process, and `--debug-train-only` means that `sglang` does not need to be initialized. 3. Used `train_async.py` instead of `train.py`. This is to leverage the asynchronous training process to implement data prefetching. --- ### En/Examples/Qwen3 30B A3B # Qwen3-30B-A3B with 8xH100 ## Environment Preparation The environment setup, model download, data, and checkpoint conversion are the same as for the Qwen3-4B model. You can refer to [Example: Qwen3-4B Model](qwen3-4B.md), replacing mentions of Qwen3-4B with Qwen3-30B-A3B. To convert huggingface checkpoint to torch_dist, please try: ```bash cd slime/ pip install -e . --no-deps source scripts/models/qwen3-30B-A3B.sh PYTHONPATH=/root/Megatron-LM/ torchrun --nproc-per-node 8 \ tools/convert_hf_to_torch_dist.py \ ${MODEL_ARGS[@]} \ --hf-checkpoint /root/Qwen3-30B-A3B/ \ --save /root/Qwen3-30B-A3B_torch_dist/ ``` ## Run Training Execute the training script: ```bash cd /root/slime bash scripts/run-qwen3-30B-A3B.sh ``` ### Parameter Introduction Here, we will briefly introduce the MoE-related parts in the [run-qwen3-30B-A3B.sh](https://github.com/THUDM/slime/blob/main/scripts/run-qwen3-30B-A3B.sh) script. 1. To support running Qwen3-30B-A3B in an 8xH800 environment, we need to enable Megatron's CPU Adam to save GPU memory. The corresponding configuration is: ```bash OPTIMIZER_ARGS=( ... --optimizer-cpu-offload --overlap-cpu-optimizer-d2h-h2d --use-precision-aware-optimizer ) ``` 2. Enable MoE optimization supported by Megatron. The current configuration is tp4, ep8: ```bash PERF_ARGS=( --tensor-model-parallel-size 4 --sequence-parallel --pipeline-model-parallel-size 1 --context-parallel-size 1 --expert-model-parallel-size 8 --expert-tensor-parallel-size 1 ... ) ``` 3. Enable MoE optimization supported by SGLang. The current configuration is ep8: ```bash SGLANG_ARGS=( --rollout-num-gpus-per-engine 8 --sglang-mem-fraction-static 0.7 --sglang-ep-size 8 --sglang-cuda-graph-bs 1 2 4 8 $(seq 16 8 256) ) ``` Similarly, you can also add DP attention, for example, by configuring: ```bash --sglang-enable-dp-attention --sglang-dp-size 8 ``` ### BF16 Training with FP8 Inference slime also supports BF16 training with FP8 inference. For the Qwen3-30B-A3B model, you just need to download the following model: ```bash hf download Qwen/Qwen3-30B-A3B-FP8 --local-dir /root/Qwen3-30B-A3B-FP8 ``` And replace `--hf-checkpoint` with: ```bash #--hf-checkpoint /root/Qwen3-30B-A3B --hf-checkpoint /root/Qwen3-30B-A3B-FP8 ``` This will trigger FP8 inference. Currently, we directly cast the BF16 weights to FP8. In the future, we will gradually add more sophisticated quantization schemes that have less impact on precision. ⚠️ The Megatron checkpoint for training still needs to be the one that was originally converted from the BF16 Hugging Face model. ### Multi-Node Support For a multi-node environment, the following modifications are necessary: - Place the training model and data on a path accessible by all nodes. - Set the `MASTER_ADDR` to an address that is accessible by all nodes. - Remove configurations related to CPU Adam. This is because a distributed optimizer is used, which significantly reduces the optimizer's video memory (VRAM) usage in a multi-node setup. In addition, you can make the following changes: - When the total number of GPUs is not a multiple or divisor of the total number of experts, you can use `--sglang-ep-num-redundant-experts` to add redundant experts. For example, in a 24-GPU scenario, you can configure it as follows: ```bash SGLANG_ARGS=( --rollout-num-gpus-per-engine 24 --sglang-mem-fraction-static 0.7 --sglang-ep-size 24 --sglang-enable-dp-attention --sglang-dp-size 3 --sglang-moe-dense-tp-size 1 --sglang-enable-dp-lm-head --sglang-ep-num-redundant-experts 16 ) ``` --- ### En/Developer Guide/Ci # CI (Continuous Integration) slime CI has two layers: 1. **Always-on CPU correctness tests** that run on every PR, every push to `main`, and manual `workflow_dispatch`. 2. **Label-gated GPU end-to-end tests** that validate real Megatron + SGLang training and rollout paths on self-hosted GPU runners. This split is intentional. Most invariants should be checked quickly without waiting for the GPU fleet, while full training/rollout behavior is still covered by GPU e2e jobs. ## How It Works The workflow is defined in `.github/workflows/pr-test.yml`, which is auto-generated from `.github/workflows/pr-test.yml.j2`. ### CPU Jobs CPU jobs run on GitHub-hosted `ubuntu-latest` runners: - `cpu-unittest` installs CPU PyTorch and lightweight dependencies, then runs registered unit and contract tests with `python tests/.py`. - `agent-adapter-test` does the same for agent adapter tests, with extra dependencies such as `openai`, `openai-agents`, and `anthropic`. CPU jobs do not use Docker, do not acquire GPUs, and do not call `tests/ci/gpu_lock_exec.py`. ### GPU E2E Jobs GPU jobs run on self-hosted GPU runners. Each job: 1. Starts a Docker container, usually `slimerl/slime:latest`; image validation uses `slimerl/slime-test:latest`. 2. Installs slime with `pip install -e . --no-deps`. 3. Acquires the requested GPUs with `tests/ci/gpu_lock_exec.py --count `. 4. Executes the registered test file with `python tests/.py`. GPU tests usually follow the e2e pattern: `prepare()` downloads models/datasets, and `execute()` builds CLI arguments and calls `U.execute_train(...)`. ### Changed-Test Job `run-ci-changed` dynamically detects added or modified files under `tests/test_*.py` and `tests/plugin_contracts/test_*.py` relative to `origin/main`. For each changed test file, it extracts a top-level `NUM_GPUS = ` constant and builds a matrix. If `NUM_GPUS` is missing, CI defaults to `8`, so CPU-only tests should declare: ```python NUM_GPUS = 0 ``` The changed-test job itself runs through the self-hosted Docker path. When `NUM_GPUS = 0`, it runs the test without acquiring GPUs. ## CI Jobs and Triggers | Trigger | Job | Type | Description | |---|---|---|---| | Automatic | `cpu-unittest` | CPU | Always-on unit and contract tests for argument validation, schedules, rewards, samples, rollout validation, checkpoint utilities, and plugin contracts. | | Automatic | `agent-adapter-test` | CPU | Always-on agent adapter tests with optional provider SDK dependencies. | | `run-ci-sglang-config` | `e2e-test-sglang-config` | GPU | SGLang config tests for advanced rollout engine deployment and mixed/offload scenarios. | | `run-ci-megatron` | `e2e-test-megatron` | GPU | Core Megatron training tests covering dense, MoE, PPO, MTP, OPD, async rollout, PD/Mooncake, and debug replay paths. | | `run-ci-precision` | `e2e-test-precision` | GPU | Numerical precision validation and parallel consistency checks. | | `run-ci-ckpt` | `e2e-test-ckpt` | GPU | Checkpoint save/load correctness, including CPU/GPU optimizer states and async save. | | `run-ci-image` | `e2e-test-image` | GPU | Runs the `run-ci-megatron` matrix on `slimerl/slime-test:latest`. | | `run-ci-changed` | `e2e-test-changed` | Mixed | Runs only changed tests, using each file's `NUM_GPUS` value. | `workflow_dispatch` can be used from the Actions page for manual validation. It runs the registered jobs according to the workflow conditions. ## CPU Unit Tests The CPU suite is the first line of defense for correctness. It is designed to catch silent RL infrastructure bugs before a change reaches expensive GPU runs. The registered CPU suite currently covers: - Megatron argument and HF config validation; - DP/CP scheduling utilities and CP loss invariance; - metric reporting and distributed metric aggregation; - reward-model grading utilities for math, GPQA, F1, DeepScaler, and DAPO-style math; - `Sample` behavior, rollout validation, and agent trajectory merging; - HF checkpoint saver behavior; - customization hook contracts for rollout functions, generate functions, runtime hooks, and path loading. Agent adapter tests are kept in a separate CPU job because they need extra SDK dependencies. Useful local commands: ```bash python tests/test_agent_trajectory.py python -m pytest tests/test_megatron_argument_validation.py tests/plugin_contracts/test_plugin_generate_contracts.py ``` ## GPU E2E Tests GPU e2e tests validate the integrated training/rollout behavior that CPU tests cannot cover: - `run-ci-sglang-config`: advanced SGLang deployment paths, including config-based engine layouts. - `run-ci-megatron`: main Megatron backend coverage for dense/MoE recipes, async rollout, OPD, PPO-style paths, PD/Mooncake, and debug rollout-then-train replay. - `run-ci-precision`: numerical consistency across parallel settings. - `run-ci-ckpt`: checkpoint save/load combinations and async save. - `run-ci-image`: the same matrix as `run-ci-megatron`, but on the release/test image. Use targeted labels for routine PRs. Use `run-ci-image` sparingly because it consumes significantly more GPU time. ## Writing a New Test ### CPU Tests For CPU-only tests: 1. Add the test under `tests/test_*.py`, `tests/utils/test_*.py`, or `tests/plugin_contracts/test_*.py`, following nearby patterns. 2. Add a top-level `NUM_GPUS = 0` if the file may be run by `run-ci-changed`. 3. Make the file executable directly: ```python if __name__ == "__main__": raise SystemExit(pytest.main([__file__])) ``` 4. If the test should run permanently, register it in the `cpu-unittest` or `agent-adapter-test` job in `.github/workflows/pr-test.yml.j2`, then regenerate the workflow. ### GPU E2E Tests For GPU e2e tests: 1. Create `tests/test_.py` following the existing `prepare()` / `execute()` pattern. 2. Declare the required GPU count with `NUM_GPUS = `. 3. Download required models/datasets in `prepare()`. 4. Build arguments and call `U.execute_train(...)` in `execute()`. 5. Register the test in the appropriate GPU job in `.github/workflows/pr-test.yml.j2`, then regenerate the workflow. Example skeleton: ```python import os import slime.utils.external_utils.command_utils as U MODEL_NAME = "Qwen2.5-0.5B-Instruct" MODEL_TYPE = "qwen2.5-0.5B" NUM_GPUS = 4 def prepare(): U.exec_command("mkdir -p /root/models /root/datasets") U.exec_command(f"hf download Qwen/{MODEL_NAME} --local-dir /root/models/{MODEL_NAME}") def execute(): # Build argument strings and call U.execute_train(...) ... if __name__ == "__main__": prepare() for proxy_var in ("http_proxy", "https_proxy", "HTTP_PROXY", "HTTPS_PROXY"): os.environ.pop(proxy_var, None) execute() ``` ## Workflow Generation The workflow file `pr-test.yml` is auto-generated from the Jinja2 template `pr-test.yml.j2`. Do not edit `pr-test.yml` directly. To change the permanent CI matrix: 1. Edit `.github/workflows/pr-test.yml.j2`. 2. Run: ```bash python .github/workflows/generate_github_workflows.py ``` 3. Commit both `.github/workflows/pr-test.yml.j2` and the generated `.github/workflows/pr-test.yml`. ## Choosing Checks for a PR - Pure argument parsing, reward, schedule, sample, trajectory, or hook-contract changes: rely on CPU tests first. - SGLang topology or rollout engine deployment changes: use `run-ci-sglang-config`. - Megatron training, loss, checkpoint conversion, or model recipe changes: use `run-ci-megatron`; add `run-ci-precision` or `run-ci-ckpt` when relevant. - Docker image or dependency changes: use `run-ci-image`. - New or modified tests: use `run-ci-changed` for quick targeted validation. --- ### En/Developer Guide/Debug # Debugging ## Aligning Precision During the development of slime, it is often necessary to check if the model's precision is correct. This can be verified in the following ways: 1. **First Training Step** 1. Check if the generated `rollout` is coherent. If not, there are two possible reasons: * Parameters were not loaded correctly. You need to check the logs for a confirmation that Megatron successfully loaded the checkpoint (ckpt). * There was an error in updating the parameters. You can check if all parameters were converted and mapped correctly, or if the parameter names were converted according to the parallelization strategy (e.g., when `pp_size > 1`, check if the layer IDs for the parameters provided by the second stage are correct). A thorough method is to save all parameters in the `load_weights` implementation of the corresponding model in SGLang and verify that they are consistent with the loaded checkpoint. * If all parameters are updated correctly and the problem persists, it's possible that some special buffers in SGLang were released during the release process. * If you are testing with a pretrained model, you can switch to an instruct version of a model with the same architecture to see if this garbled output is specific to the pretrained model. 2. Check the printed rollout stats to see if `log_probs` and `ref_log_probs` are exactly equal (meaning KL divergence is 0 in the first step) and their values are small. * If they are not exactly equal, it is usually caused by certain non-deterministic kernels in the Transformer Engine, for example: * In some versions of Transformer Engine (TE), Megatron requires `--attention-backend flash` to enforce the use of Flash Attention, thereby avoiding numerical instability from the fused attention under Context Parallelism (CP). * If the values are large (e.g., > 1), there are generally two possibilities: * If the value is extremely large, there is likely a problem with the training configuration. * If the value is only slightly larger than the SFT loss, for example, if the log probability of an instruct model reaches 0.8, it might be because the data does not conform to the trained chat template or does not match the cold-start distribution. 3. When running one inference step per training step (`num_steps_per_rollout == 1`), check if the KL divergence is 0 and if the `grad_norm` is small. * This is basically due to some Megatron / TE related bugs, for example: * Mixture of Experts (MoE) requires enabling `--moe-permute-fusion`. 2. **Second Training Step** 1. For integrated training and inference, check if the second step can be loaded correctly and whether it results in an Out of Memory (OOM) error. ## Separate Debugging for Training and Inference slime supports debugging the training and inference parts separately, which allows for the following: * When tuning/debugging the inference part, you can start the task with only a few GPUs. * When tuning/debugging the training part, you can ensure the model input is fixed, removing the randomness of rollouts. Specifically, slime currently provides the following parameters for separate debugging: 1. `--debug-rollout-only` When enabled, slime will not load Megatron and will only initialize SGLang. You can use this method to debug the inference part. 2. `--debug-train-only` When enabled, slime will not load SGLang and will only initialize Megatron. You can use this method to debug the training part. 3. `--save-debug-rollout-data /your/saved/debug/data_{rollout_id}.pt` When enabled, the results of each rollout will be saved. This can be used in conjunction with `--debug-rollout-only`. Note that the data is saved using the format: `args.save_debug_rollout_data.format(rollout_id=rollout_id)`. 4. `--load-debug-rollout-data /your/saved/debug/data_{rollout_id}.pt` When enabled, data will be loaded from `args.load_debug_rollout_data.format(rollout_id=rollout_id)`, and SGLang will not be initialized (automatically setting `debug_train_only=True`). This method allows you to fix the input for the training part to tune it, for example, by switching between different parallelization strategies. 5. `--save-debug-train-data /your/saved/debug/train_{rollout_id}.pt` Saves one train-side file per rollout. Only the last Pipeline Parallel stage and Tensor Parallel rank 0 participate. They restore response-token fields such as `log_probs`, `ref_log_probs`, `values`, `advantages`, `returns`, `kl`, and `entropy` across Context Parallel ranks. Context Parallel rank 0 moves each restored tensor to CPU immediately, so complete tensors do not accumulate on the GPU, and then gathers the distinct Data Parallel shards to one writer. The version-2 payload mirrors the rollout debug dump: a top-level `samples` list holds one dict per training sample (`sample_index`, `data_parallel_rank`, and its per-sample fields such as `tokens`, `log_probs`, `advantages`), sorted by `sample_index` so it lines up one-to-one with the rollout dump's `samples` (join on `sample_index` ↔ the rollout side's `index`). A parallel `dp_shards` key preserves the DP/micro-batch layout — each entry records `rank`, `data_parallel_rank`, that shard's `sample_indices`, and the DP-local schedule (`micro_batch_indices`, `num_microbatches`, `global_batch_sizes`) — without duplicating any per-sample tensor. Whole-batch fields such as `raw_reward` are stored once at the top level. If any sample lacks a `sample_index` (custom rollouts that build fresh `Sample` objects leave it `None`), the samples stay in DP-gather order and a warning is logged. With or without CP, response-token fields use the same full-response format. In configs that skip the separate actor log-prob recompute (`can_reuse_log_probs_in_loss` or `--use-rollout-logprobs`), the actor `log_probs` are snapshotted from the training forward itself (keyed by rollout position, at no extra forward), so the dump still carries them. ## INT4 / Compressed-Tensors Quantization Checkpoint Issues When using INT4-quantized models (e.g., `compressed-tensors` with `W4A16`), the checkpoint's `config.json` contains a `quantization_config.ignore` list that specifies which parameters should **not** be quantized. During online weight updates (Megatron → SGLang), slime also reads this ignore list to decide which parameters to INT4-quantize. An incorrect ignore list can cause silent errors: 1. **MoE router weights (`mlp.gate.weight`) become all zeros** The MoE router weight (`mlp.gate.weight`, shape `[num_experts, hidden_size]`) is a plain 2D weight tensor, but it is **not** a Linear layer weight. If it is not in the ignore list, the online quantizer will INT4-quantize it into `weight_packed`, `weight_scale`, `weight_zero_point`, etc. However, SGLang does not expect quantized names for the router, so these parameters are silently skipped during `load_weights`, resulting in all-zero gate weights. **Fix**: Ensure `config.json` contains `"re:.*mlp\\.gate\\..*"` in the ignore list. 2. **Other non-Linear 2D weights** Similar issues can occur with any 2D `.weight` tensor that is not a true Linear layer, such as `model.embed_tokens.weight`. Always verify the ignore list covers all non-Linear weights. **Recommended ignore patterns** (for GLM-style MoE models): ```json "ignore": [ "lm_head", "model.embed_tokens.weight", "re:.*self_attn.*", "re:.*mlp\\.shared_experts.*", "re:.*mlp\\.gate_up_proj.*", "re:.*mlp\\.gate_proj.*", "re:.*mlp\\.up_proj.*", "re:.*mlp\\.down_proj.*", "re:.*eh_proj.*", "re:.*mlp\\.gate\\..*" ] ``` 3. **Missing safetensors shards** Conversion tools may occasionally produce an incomplete checkpoint (e.g., a missing `model-00010-of-00093.safetensors`). After conversion, always verify: - The number of `.safetensors` files matches the expected count. - The `model.safetensors.index.json` contains entries for every layer. - Spot-check that critical layers (e.g., the first MoE layer) have the expected number of keys. 4. **How to diagnose** - Use `--check-weight-update-equal` to verify that weights after a Megatron → SGLang sync match the expected values. If a parameter shows all zeros on the SGLang side, it was likely incorrectly quantized or missing from the checkpoint. - Use `--debug-rollout-only` with a small number of GPUs to quickly test whether SGLang can generate coherent text from the quantized checkpoint alone. ## Debug sglang illegal memory access (IMA) When running large scale RL, we will occationally meet the IMA in SGLang, there are some debug suggestions based on our experience: 1. Enable `CUDA_LAUNCH_BLOCKING=1` 2. Enable or disable speculative decoding and cuda graph to see if anything changed IMA always appears in the padding in cuda graph replay, or the difference between draft model and main model. We can minimize the scope by tuning them. 3. Turn off deepep If you are using deepep during training or inference, you can try turn it off. 4. Try CUDA Core Dump to find the error kernel We recommend reading the blog from the vLLM team: [CUDA Core Dump: An Effective Tool to Debug Memory Access Issues and Beyond](https://blog.vllm.ai/2025/08/11/cuda-debugging.html) ## Step-by-Step Debugging with Ray Distributed Debugger Ray provides a [distributed debugger](https://docs.ray.io/en/latest/ray-observability/ray-distributed-debugger.html) based on debugpy that lets you set breakpoints in the driver process and step through code interactively. 1. Install debugpy: ```bash pip install debugpy==1.8.0 ``` 2. Enable `RAY_DEBUG_POSTMORTEM` in your launch script: ```bash export RAY_DEBUG_POSTMORTEM=1 RUNTIME_ENV_JSON="{ \"env_vars\": { ... \"RAY_DEBUG_POSTMORTEM\": \"${RAY_DEBUG_POSTMORTEM:-0}\" } }" ray job submit --address="http://127.0.0.1:8265" \ --runtime-env-json="${RUNTIME_ENV_JSON}" \ -- python3 train.py [args...] ``` 3. Add `ray.init()` before `breakpoint()` in `train.py`: ```python if __name__ == "__main__": ray.init() breakpoint() args = parse_args() train(args) ``` `ray.init()` is required because the distributed debugger depends on `core_worker`, which is only available after Ray initialization. Without it, `breakpoint()` raises `AttributeError: 'Worker' object has no attribute 'core_worker'`. 4. Connect via VS Code: Install the [Ray Distributed Debugger](https://marketplace.visualstudio.com/items?itemName=ray-project.ray-distributed-debugger) extension in VS Code. Run your launch script to submit the job. Once the job hits `breakpoint()`, open the Ray Dashboard panel in VS Code and click the active breakpoint to attach the debugger. You can then step through code, inspect variables, and set additional breakpoints directly in the editor. > **Note**: Remove `ray.init()` and `breakpoint()` after debugging. An explicit `ray.init()` without arguments may cause issues in multi-node training where Ray injects specific namespace and runtime environment configurations via `ray job submit`. --- ### En/Developer Guide/Profiling # Profiling In slime, we can perform detailed performance analysis of the rollout process using the profiling interface provided by SGLang. ## 1. Sleeping the Rollout Process For more flexible stress testing and profiling, it is often useful to make the slime rollout process enter a waiting state after initialization, instead of starting generation immediately. You can achieve this by replacing the `rollout_function_path` in your startup arguments without modifying the source code: ```bash python train.py \ --rollout-function-path slime.rollout.sleep_rollout.sleep \ ... (other arguments) ``` This function will make the rollout process enter an infinite wait loop, allowing you to manually send requests or run stress testing tools. ## 2. Obtaining SGLang Engine List SGLang engines (workers) are registered with the router. You can retrieve the list of all active engines by accessing the `/workers` endpoint of the router. The router address is typically printed in the startup logs: ``` Router launched at 127.0.0.1:3000 ``` You can use `curl` to view the workers: ```bash curl http://127.0.0.1:3000/workers ``` ## 3. Using Automated Profiling Tool To simplify profiling across multiple engines simultaneously, we provide an automated script: `tools/profile_rollout.py`. ### Starting Profiling By default, this tool starts profiling on all workers and will automatically stop after 3 steps: ```bash python tools/profile_rollout.py --router-url http://127.0.0.1:3000 --action start --num-steps 3 ``` **Key Parameters:** * `--router-url`: The URL of the Router. * `--num-steps`: Number of steps to record, defaults to 3. * `--output-dir`: Directory where trace files will be saved. * `--activities`: Activities to monitor, e.g., `GPU` `CPU`. * `--profile-by-stage`: Whether to profile by stage (prefill/decode). ### Stopping Profiling Manually If you did not set `num_steps` or wish to stop early: ```bash python tools/profile_rollout.py --router-url http://127.0.0.1:3000 --action stop ``` ## 4. Running Stress Tests While the Rollout process is in a waiting state via `sleep_rollout`, you can: 1. Start profiling using `tools/profile_rollout.py`. 2. Use stress testing tools (such as SGLang's built-in benchmark tools) to send requests to the router or directly to the engines. 3. Wait for profiling to complete (if `num_steps` was set) or stop it manually. 4. Collect the `.json` trace files from the `output_dir` and view them using `chrome://tracing` in Chrome or [Perfetto](https://ui.perfetto.dev/). --- ### En/Developer Guide/Trace # Trace Viewer slime can attach lightweight execution traces to each rollout sample. These traces capture span-style events such as generation and reward-model calls, and they can be inspected later from a saved rollout debug dump. ## Save rollout trace data To inspect traces later, save rollout debug data during a run: ```bash python train.py \ ... \ --save-debug-rollout-data /path/to/debug/rollout_{rollout_id}.pt ``` Each saved `.pt` file contains the rollout samples together with their `trace` payloads. You can also replay the same dump later with `--load-debug-rollout-data`. ## Open the timeline viewer Use the trace viewer script on a saved rollout dump: ```bash python tools/trace_timeline_viewer.py /path/to/debug/rollout_0.pt ``` The script generates: - `rollout_0.trace_timeline_cache.json` - `rollout_0.trace_timeline_viewer.html` By default it also starts a local static server so you can open the generated HTML immediately. If you only want the files, use `--no-serve`. ## How to read the viewer - Each row corresponds to one sample. - Bars represent spans, while point markers represent instant events. - Span attributes recorded at the start or end of `trace_span(...)` are shown in the details panel. - When SGLang returns PD disaggregation timings, the viewer adds synthetic `[P]` and `[D]` lanes to break out prefill/decode work. - When PD is not enabled, those virtual lanes are omitted automatically and the base trace still renders normally. ## Instrument custom code For custom rollout or reward code — including custom agent steps, tool calls, sandbox execution, and verifier calls in agentic workflows — reuse helpers from `slime.utils.trace_utils`: - `trace_span(target, name, attrs=...)`: record a duration span. - `trace_event(target, name, attrs=...)`: record an instant event. - `trace_function(name, ...)`: wrap a whole sync/async function in a span. - `bind_trace(sample)`: ensure a sample already has a trace carrier before passing it across helpers or tasks. ### `trace_span` vs `trace_function` Use `trace_span(...)` when you only want to trace part of a function body, or when you need to update end-of-span attrs from inside the block. Use `trace_function(...)` when the whole function should be represented as one span. Internally it resolves the trace target and then opens a `trace_span(...)` around the function call, so it works for both sync and async functions. The decorator is what slime uses for the main rollout pipeline. For example, `generate_and_rm(...)` is traced per sample and `generate_and_rm_group(...)` is traced per sample group: ```python from slime.utils.trace_utils import trace_function @trace_function("generate_and_rm", target="sample") async def generate_and_rm(args, sample, sampling_params, evaluation=False): ... @trace_function( "generate_and_rm_group", target="group", attrs_getter=lambda args, group, sampling_params, evaluation=False: {"group_size": len(group)}, ) async def generate_and_rm_group(args, group, sampling_params, evaluation=False): ... ``` ### Choosing a target `trace_function(...)` needs a trace target, usually a `Sample`, `TraceHandle`, or a list of them. - Prefer `target="sample"` or `target="group"` when the target is already one of the function arguments. - Use `target_getter=...` when the trace target has to be derived from arguments. - Avoid relying on automatic inference unless the function signature is simple. The implementation can infer a target from arguments or the current trace context, but explicit targets are more stable and avoid ambiguous traces. ### Recording attrs on decorated functions If you want to attach attributes at span start, use `attrs_getter=...`: ```python @trace_function( "custom_rollout_batch", target="samples", attrs_getter=lambda samples, **_: {"batch_size": len(samples)}, ) async def custom_rollout_batch(samples, **kwargs): ... ``` If you need to add attrs after part of the function has executed, use an inner `trace_span(...)` instead of only relying on the decorator. A common pattern is: - `trace_function(...)` for the outer function-level lifecycle span - nested `trace_span(...)` for important sub-steps such as generation, RM, filtering, or post-processing If you want to record SGLang generation metadata in a consistent way, reuse `build_sglang_meta_trace_attrs`: ```python from slime.utils.trace_utils import build_sglang_meta_trace_attrs, trace_span with trace_span(sample, "sglang_generate") as span: output = await post(url, payload) span.update(build_sglang_meta_trace_attrs(output["meta_info"])) ``` ## Tips - Save a small number of rollouts first; the viewer is easiest to read when each dump contains a manageable number of samples. - The viewer is built from the saved `.pt` dump, so traces can be inspected offline on another machine. - For GPU/kernel-level SGLang profiling traces, see [Profiling](./profiling.md). --- ### En/Blogs/Introducing Slime # slime: An SGLang-Native Post-Training Framework for RL Scaling > This article was first released in [lmsys.org](https://lmsys.org/blog/2025-07-09-slime/). ## Vision That Drives slime We believe in RL. We believe RL is the final piece toward AGI. If you feel the same way, you'll share our vision: - Every field should be end-to-end RLed and every task should become an agent environment. - Every RL run should last longer, and every model should scale larger. - RL systems should integrate seamlessly with existing infrastructure, letting us focus on new ideas instead of boilerplate engineering. That's why we present [slime](https://github.com/THUDM/slime), a post-training framework designed to be: - **Versatile** – with a fully customizable rollout interface and flexible training setups (colocated or decoupled, synchronous or asynchronous, RL or SFT cold start). - **Performant** - integrating SGLang for inference and Megatron-LM for training, natively. - **Maintainable** - with a lightweight codebase and smooth transition from Megatron pretraining to SGLang deployment. In short, a post-training framework for RL scaling. Here’s how we made it happen. ## Customizability Brings Freedom > We should stop trying to find simple ways to think about the contents of minds, such as simple ways to think about space, objects, multiple agents, or symmetries. > > > — *The Bitter Lesson* > A prevailing misconception within the RL community is the need for separate frameworks for different tasks: one for plain math, one for multi-turn tool calling, one for asynchronous training, one for agentic tasks, and so on. Forking and maintaining multiple frameworks is dreadful, leading to time-wasting bugfix cherry-picking, or worse, training crashes by missing patches. It wasn’t always like this: no one forks PyTorch just for a new dataloader. We believe the current chaos stems from the trap of dictating how people should build their applications. If we insist on defining a universal template for every rollout scenario, we’ll inevitably create an RL framework that meets only a fraction of real-world needs. slime views the data sampling in RL differently. We manage all SGLang servers within slime with [sgl-router](https://github.com/sgl-project/sglang/tree/main/sgl-router) and provide an interface for the data generation component, **allowing users to inject custom logic and freely interact with SGLang servers**. Unleash their creativity. With the sgl-router, users only need to send HTTP requests to a single endpoint. By exposing this endpoint, complex agent environments can directly interact with slime through an OpenAI-compatible API — no need to modify the environment, and training-deployment consistency is preserved. Regarding training schemes, slime uses Ray for resource management, enabling **colocated** (same GPUs) or **decoupled** (separate GPUs) setups with a single flag (`--colocate`). And with Ray's asynchronous execution via `.remote()`, slime naturally supports asynchronous training. Changing synchronization behavior is as simple as moving the `ray.get` operation. And to make experimenting with different strategies easy, we didn't wrap the code with trainer classes, but simply exposed the training loop in entrypoint `train.py`. ## Built for Performance **A decent RL framework must be fast and consistently fast.** **Fast** means leveraging the fastest inference and training frameworks. Unlike pre-training, RL workloads involve tons of online sampling during training, which makes the inference performance crucial. Therefore, slime exclusively integrates SGLang, and deliberately delivers an SGLang-native experience. So what does ‘SGLang-native’ mean? It means you can take full advantage of all SGLang optimizations — using SGLang inside slime feels just like using it standalone. To make that possible: - slime internally launches SGLang servers in a **server-based mode**. - slime implements **seamless pass-through** for all SGLang parameters (with a `--sglang` prefix), ensuring that all optimization options can be enabled. For instance, you can pass `--sglang-enable-ep-moe`, `--sglang-enable-dp-attention` and `--sglang-enable-deepep-moe` for the powerful multi-node MoE inference capabilities. - slime provides an **SGLang-only debug mode** (`--debug-rollout-only`) for easy performance tuning. Together, we can reproduce the standalone performance of SGLang within slime. Even the base image of slime is built on `lmsysorg/sglang:dev`. For training, slime integrates the battle-tested Megatron-LM, aiming for a similarly native pre-training experience: - slime also implements **seamless pass-through** for all Megatron parameters. - slime supports **all Megatron parallelisms** (TP, PP, EP, CP) and monitors training MFU. - slime offers a **Megatron-only debug mode** (`--debug-train-only`) and supports storing sampling data for reproducibility. Megatron can be notoriously complex, so we also provide checkpoint conversion tools to simplify its use. **Consistently fast** means keeping pace with the evolving inference and training frameworks. If you ever followed the [SGLang PR list](https://github.com/sgl-project/sglang/pulls), you will be astonished by its rapid evolution. Megatron, on the other hand, is often heavily customized, with every organization maintaining its own fork. slime is designed to keep pace with upstream changes in SGLang and adapt to optimizations in in-house Megatron variants. This is another reason why we pursue native support for SGLang and Megatron. The parameter pass-through makes upgrading effortless. Beyond optimizing inference and training frameworks, we also tackled RL-specific workloads. When SGLang needs changes to support these workflows, we work closely with the SGLang team to upstream patches—so slime can stay native, even as RL logic evolves. Examples include: **Optimizing weight updates**: Unlike inference tasks, RL training involves frequent updates to model weights. To address this, we’ve introduced several optimizations in SGLang: - Parameter updates for MoE models under various parallelism strategies ([#6265](https://github.com/sgl-project/sglang/pull/6265), [#6308](https://github.com/sgl-project/sglang/pull/6308), [#6311](https://github.com/sgl-project/sglang/pull/6311)). - Bucketed parameter update support to reduce overhead ([#7292](https://github.com/sgl-project/sglang/pull/7292)). **`/abort_request` for dynamic sampling**: In RL algorithms that require oversampling, such as [DAPO](https://arxiv.org/abs/2503.14476), some requests may continue running even after sufficient data has been collected. In collaboration with the [AReal](https://github.com/inclusionAI/AReaL) team, we designed an new endpoint: `/abort_request`. This endpoint enables: - Immediate termination of on-going requests. - Reclaiming partially generated content, which enables partial rollouts. Implemented in [#6698](https://github.com/sgl-project/sglang/pull/6698), [#6855](https://github.com/sgl-project/sglang/pull/6855), [#6184](https://github.com/sgl-project/sglang/pull/6184), [#5966](https://github.com/sgl-project/sglang/pull/5966). ## Lightweight and Extensible Focusing on customization and performance, slime: 1. Provides a customizable rollout interface. 2. Uses Ray for GPU management and asynchronous execution. 3. Integrates SGLang for inference and Megatron for training. 4. Provides weight updates between training and inference. Pretty straightforward, right? slime transfers complexity from the framework to user-defined pipelines and core libraries (SGLang and Megatron), resulting in a lightweight, easily maintainable codebase. But it doesn’t stop at RL. Thanks to its modular design and powerful backends, slime can naturally extend to other post-training workflows with minimal extra code: - **SFT**: Load Megatron and use token prediction loss. - **Rejection Sampling**: Use SGLang for filter, followed by Megatron SFT. *(Note that SFT feature is now in experimental state.)* Beyond that, slime's native integration **seamlessly bridges pre-training to online services**. We can use Megatron for pre-training, switch to slime (which integrates both Megatron and SGLang) for post-training, and finally use SGLang directly for evaluation and deployment. This eliminates the cumbersome and error-prone steps of converting checkpoint formats and aligning precision between frameworks. The unified pipeline saves us from tedious glue code, freeing us to focus on what really matters: better RL. Hurray! ## Roadmap The journey of RL scaling has just begun, and slime is continuously evolving. In the next phase, we will focus on: 1. Collaborating with the SGLang team to explore optimal RL training strategies for large-scale MoE models. 2. Supporting broader post-training workflows, strengthening the pre-training-to-production bridge. 3. Adding native PyTorch training backend support to lower the entry barrier. We hope slime accelerates your RL scaling journey and turns your innovative ideas into reality. Contributions and conversations are always welcome! Special thanks to the AMD GenAI - Foundation Model Team for Day-1 AMD hardware support. --- ### En/Blogs/Release V0.1.0 # v0.1.0: Redefining High-Performance RL Training Frameworks > The origin version of this article is in Chinese and was first released in [zhihu](https://zhuanlan.zhihu.com/p/1945237948166547268). With the help of the community, we've finally released the first version of **slime**, **v0.1.0**, just two months after it was open-sourced. In a nutshell, this version can be summarized as follows: > **slime v0.1.0 provides all the essential performance optimizations needed for large-scale MoE RL training.** Specifically, this version brings the following improvements: - **Performance**: - Provides **efficient inference for MoE models**, especially with **fp8 rollout + deepep + mtp**. - Designed a generic **training framework memory offload solution** to save more KV Cache space, thus increasing inference concurrency. - **Faster parameter updates**. - Achieves more training with fewer GPUs through **CPU Adam**. - Supports **all of Megatron's parallel strategies** as well as **deepep**. - **Features**: - Added support for **GSPO** for MoE model training. - Added support for **TIS** for fp8 rollout. - **Correctness**: - Implemented **Dense and MoE model CI** to strictly check metrics like kl. We hope to use slime v0.1.0 to demonstrate our understanding of high-performance RL training frameworks and have it become a baseline for future performance comparisons. Next, I'll elaborate on the design philosophy behind these features. ----- ## Performance Optimization: Pushing the Limits of RL Training Speed In traditional deep learning training, there's a universal solution for speedup: **add more GPUs**. By reducing the amount of data processed per GPU, you can significantly lower end-to-end training latency. However, this method doesn't work for RL training because **inference latency cannot be reduced by adding more GPUs**. Even with more GPUs, we still have to wait for the longest sample to finish decoding. While increasing throughput can improve the amount of training data per rollout, the off-policy issues caused by an excessively large inference batch size still have some limitations. I believe this is the biggest challenge for infrastructure under the current RL paradigm, which is: > **We want to scale inference compute, but we cannot scale inference latency.** The decoding speed of a single data point determines the upper limit of RL training speed. For larger MoE models, there are currently three common optimization methods to push this limit, and we've tried all of them: 1. **Reduce memory access through quantization**: Considering that long calibration is not feasible in RL training, slime opts for fp8 quantization. 2. **Use deepep low-latency mode to reduce all2all latency across machines**: To work with deepep, slime recommends using blockwise quantization with fp8 to enable related SGLang configurations. 3. **Enable Speculative Sampling**: slime allows loading any draft model for the inference part (currently, it doesn't support updating the draft model during training). By using the three optimizations mentioned above, we can increase a model like **GLM4.5 355B-A32B** from less than 10 tokens/s for a single data point to **60-70 tokens/s**, which significantly raises the upper limit of RL training speed. In addition to monitoring inference throughput, slime also monitors `perf/longest_sample_tokens_per_sec` to better understand the potential for performance optimization in the inference part. ----- ## Doing More Experiments with Fewer GPUs: Fully Offloading Megatron After optimizing the upper limit, we noticed another characteristic of RL training: as long as the **KV Cache doesn't overflow**, increasing the inference batch size doesn't significantly affect training latency. **KV Cache overflow** occurs during inference when the response lengths of the data are all very long, leading to insufficient KV Cache space. This requires kicking out some half-generated data from the queue and then re-running prefill and subsequent inference steps after other data has been processed and freed up space. If a data point with a response length of 64k has to wait for 32k tokens to be decoded by other data during its inference, its total time is equivalent to decoding 96k tokens. This greatly impacts the RL training speed. Therefore, a more suitable training configuration is to calculate the minimum number of GPUs needed to prevent KV Cache overflow based on the inference batch size, the average response length, and the available KV Cache space on a single server. A group of these GPUs is then used for training. For example, if we have 512 GPUs and the calculation shows that 256 GPUs provide enough KV Cache, we should run two experiments in parallel instead of launching one experiment with all 512 GPUs. Based on this consideration, we noticed two points for optimization: 1. **The optimal number of GPUs may not be sufficient to load the training part**. Inference only needs to load the fp8 parameters, while training generally requires more than 18 times the parameter size of GPU memory (bf16 param, fp32 grad, fp32 master param, fp32 m and v). To solve this, slime uses **Megatron's built-in CPU Adam** to save GPU memory for the training part. This strategy allowed us to provide solutions for training GLM 4.5 355B-A32B with 8 nodes and DeepSeek R1 with 16 nodes. 2. **Increase the KV Cache space available per SGLang Server**, which means increasing `mem_fraction`. For the more common integrated training and inference tasks, the main limitation for a larger `mem_fraction` is the residual GPU memory after offloading the training part to the CPU. Therefore, we need to find a generic way to offload the GPU memory used by the Megatron part. ### How to Offload GPU Tensors Generically One crude approach is to find all the GPU Tensors allocated by Megatron and call `.to("cpu")` on all of them. This method has three difficulties: - It's hard to capture all GPU Tensors allocated by Megatron. - Because Megatron's distributed optimizer reorganizes all parameters into some contiguous GPU buffers and then divides them with various slices, it's difficult to properly handle all references to correctly free the GPU Tensors. - It requires checking the source code again with every new Megatron version, which is hard to maintain. Is there a more generic solution? We noticed that SGLang's `torch_memory_saver` and VLLM's `cumem_allocator` provide a more general offload solution. Their principle is that CUDA 10.2 provides a **series of Virtual Memory Management APIs**, similar to an operating system's virtual and physical addresses (VA and PA). When allocating GPU memory, they return a handle to a memory mapping instead of the actual physical address. Therefore, when offloading, we only need to "secretly" release the memory corresponding to this mapping and reallocate it when this memory is needed. The upper-level application doesn't need to be aware of this. A natural idea is to use this method to take over the entire training process in RL. However, this prevents the reuse of PyTorch's `CUDACachingAllocator`, and without the cache, memory fragmentation becomes more pronounced, easily leading to **OOM** during training. To continue reusing the native, cached allocator, we cannot use `CUDAPluggableAllocator`. Noticing again that slime's architecture has training and inference in different processes, we only need to **directly replace `cudaMalloc` and `cudaFree` used by `CUDACachingAllocator` in the training process with VMM APIs via `LD_PRELOAD`**. This allows us to completely and generically offload all GPU Tensors allocated by PyTorch. At the same time, we must also note one detail: VMM APIs and cudaIPC APIs (such as `cudaIpcGetMemHandle`) are incompatible. Therefore, for integrated training and inference tasks and DeepEP, we need to disable the `LD_PRELOAD` replacement and switch back to `cudaMalloc`. With help from the SGLang community, we updated `torch_memory_saver` for slime's needs, implementing this offload solution. ### How to Offload NCCL After thoroughly offloading the GPU Tensors in Megatron, we found that a large amount of GPU memory still remained, which was caused by **NCCL**. In PyTorch, each NCCL group involved in communication allocates a substantial buffer. This issue is particularly noticeable for larger MoE models due to the various parallel strategies, potentially taking up **more than 10GB**. The `LD_PRELOAD` solution mentioned above doesn't handle the NCCL issue well, and we don't want to modify the NCCL source code to avoid having to maintain a separate NCCL fork in addition to slime. So, slime's approach is to use `destroy_process_group` to destroy the NCCL group when offloading Megatron and then recreate it before loading Megatron. To do this, we mimicked the VMM API and monkey patched `dist.new_group` to add a layer of `ReloadableProcessGroup`. In this way, we achieved a generic **NCCL offload**. However, because we need to rebuild the NCCL group, this operation has a slight impact on the speed of the first communication in each training iteration. But we believe this approach offers a significant advantage in terms of maintainability and the GPU memory it saves. Combining these two optimizations, we reduced Megatron's residual GPU memory from around **15-18GB** to **3-5GB**, which allows us to increase the `mem_fraction` for MoE models to **0.7-0.8**. This significantly boosts the available KV Cache, increases the concurrency each server can support, and allows us to launch more training tasks with fewer GPUs. ----- ## Parameter Update Optimization Parameter update is another special step in RL training. For this, slime v0.1.0 provides the best optimization solution for scenarios where training and inference are in different processes. This work was heavily optimized by Biao He. I recommend reading his blog post: - [Efficient Reinforcement Learning Training - Optimizing Weight Synchronization in slime](https://hebiao064.github.io/rl-weight-sync) Currently, slime can complete weight synchronization for a GLM4.5 355B-A32B model with bf16 weights in **48s** and complete fp8 blockwise quantization + parameter update in **100s** (the fp8 branch is still being optimized). ----- ## Training Optimization For the pure training part of slime, we believe Megatron already provides ample optimizations, so our main focus was to **ensure compatibility with all of Megatron's parallel strategies**. During this adaptation, we found an interesting bugfix: we discovered that when SGLang enabled mtp, the Megatron part couldn't start DeepEP. It turned out that when mtp is enabled, SGLang disables the overlap schedule, which causes a certain metadata communication to use nccl instead of gloo after being offloaded to the CPU, and this conflicts with DeepEP. ----- ## Performance Optimization Check List Since the release of slime, I've often been asked about its performance comparison with other frameworks. My understanding of benchmarks is that they should not be used as a weapon for frameworks to attack each other, but rather as a **tool for identifying gaps**. To that end, we will gradually release performance benchmarks that slime focuses on to improve ourselves. I also believe that before running benchmarks, you can analyze a framework's focus on performance from a qualitative perspective. Here's a basic feature check list for optimizations: - Does it support MoE training? (Currently, large-scale experiments are focused on MoE) - Can the internal sglang `mem_fraction` or vllm `gpu_utilization` be adjusted to over 0.7? (Ensures KV Cache space) - Does it support fp8 or lower precision inference? (Reduces inference memory access, boosts speed) - Does it support enabling deepep for both training and inference? (Optimizes MoE all2all communication) - Does it support speculative sampling? (Improves inference latency and throughput) - Does it have an efficient training backend, such as Megatron or torchtitan, and support all necessary parallelization strategies? (Reuses mature training optimizations) slime v0.1.0 has made preliminary attempts at all the above optimizations, and there's still a lot of room for improvement. We hope this version can serve as a baseline for future slime versions or for performance comparisons between different frameworks. We also welcome all friends who share our pursuit of performance to try out slime and join the slime community\! ----- ## New Algorithm Support To better train MoE models and perform fp8 rollouts, we implemented **GSPO** and **TIS**. Additionally, community experts have helped implement algorithms like reinforce++ and reinforce++ baseline. ----- ## Correctness Verification slime v0.1.0 adds **end-to-end CI**: we run single-machine GLM4 9B and Qwen3 30B-A3B training for each PR, ensuring correctness through strict checks. For example, we explicitly require: - The recomputed log prob of the first rollout must be exactly equal to the log prob of the reference model. - The ppo_kl of the first training step within each rollout must be exactly 0. Such precise verification is rarely achieved in training frameworks, and it's something we are very proud of. ----- This is a brief introduction to slime v0.1.0. I hope it sparks your curiosity about slime and that it can be helpful in your work. Everyone is welcome to join the slime community. Let's work together to build an open RL Infra and contribute to RL scaling\! --- ### En/Advanced/Arch Support Beyond Megatron # Supporting Model Architectures Beyond Megatron-LM While the Megatron-LM framework is highly efficient for parallel training, it can lack the flexibility to support rapidly evolving model architectures like Qwen3Next. Natively supporting the unique structures of these models, such as Gated-Delta-Net, often requires invasive and time-consuming modifications to Megatron's core codebase. To accelerate the adoption of these cutting-edge models, slime introduces a more agile approach: **instead of deeply re-engineering Megatron, we directly import and wrap the model's official HuggingFace implementation**, embedding it as a "black-box" module into Megatron's parallel training pipeline. This document uses Qwen3Next 80B-A3B as an example to illustrate this concept. ## Principle and Core Components Megatron's model instantiation is a two-step process: first, it generates a "layer specification" (`ModuleSpec`) based on the configuration, and then it instantiates the actual PyTorch modules according to that spec. slime leverages this mechanism by **hijacking the spec generation stage to replace Megatron's native modules** with an external implementation (in this case, from HuggingFace). This process involves the coordination of three core components: 1. **Replacing the Megatron Module Spec** This is the entry point for our solution. We use a custom function (e.g., `get_qwen3_next_spec`) to modify the standard `ModuleSpec`, swapping out Megatron's native Attention layer with our custom wrapper. * **Implementation**: It retrieves the standard Decoder Block Spec, points its `self_attention` field to our custom module, and enables model-specific configurations like `qk_layernorm` as needed. * **Corresponding File**: `slime_plugins/models/qwen3_next.py` 2. **Wrapping the HuggingFace Implementation** The spec modified in the previous step now points to a wrapper layer, such as `HuggingfaceAttention`. This layer inherits from Megatron's `MegatronModule`. Its core responsibility is to act as a bridge, handling the data alignment required by parallelism strategies (like sequence parallelism), and then internally calling the native `Qwen3NextAttention` module loaded from HuggingFace. * **Corresponding File**: `slime_plugins/models/hf_attention.py` 3. **Aligning Model Weights** Once the model architecture is integrated, we must ensure that the weights can be loaded correctly. slime keeps the HuggingFace-to-Megatron name mapping and tensor transforms next to its checkpoint loader. * **Corresponding File**: `slime/backends/megatron_utils/hf_to_megatron/qwen3_next.py` Through the coordination of these three components, we can successfully run a complex model architecture not natively supported by Megatron—using its HuggingFace implementation as the vehicle—on top of Megatron's parallel framework. This is achieved while fully retaining all key capabilities like model parallelism, MoE acceleration, and pipeline scheduling. ## Current Limitations * This approach does not currently support Tensor Parallelism (TP) within the replaced module itself (e.g., the Attention layer in this case). * **Impact**: In most large-scale MoE models, the parameter count of the Attention layer is relatively small, so this limitation typically has a minimal effect on memory footprint and training throughput. * **Alternative**: If TP for the module is critical, the only alternative is to revert to the more invasive approach of modifying Megatron's native implementation. --- ### En/Advanced/Delta Weight Sync # Delta Weight Sync Delta weight sync keeps non-colocated rollout engines up to date by shipping only the bytes that changed between two syncs, instead of a full checkpoint each time. It targets large-model training/inference disaggregation across clusters or datacenters, where writing the whole actor every sync is the dominant cost. It is **disk-transport only**. The trainer publishes each sync as a canonical HF checkpoint directory; the engine's `/pull_weights` endpoint (shipped in slime's sglang patch) fans the apply out to **every host the engine spans** and verifies it, then the engine reloads the patched local checkpoint through the **ordinary** `update_weights_from_disk` endpoint. slime only ever talks to one endpoint per engine, so multi-node serving and external rollout engines need nothing extra on the slime side. ## Configuration ```bash --update-weight-mode delta --update-weight-transport disk --update-weight-disk-dir /shared/fs/delta-updates --update-weight-local-checkpoint-dir /local/nvme/rollout-ckpt --update-weight-delta-encoding xor # or: overwrite --update-weight-delta-checksum xxh3-128 # or: blake3, adler32 ``` | Flag | Role | |---|---| | `--update-weight-disk-dir` | Shared filesystem directory the trainer publishes deltas to and the rollout hosts read from. | | `--update-weight-local-checkpoint-dir` | Host-local (e.g. NVMe) full HF checkpoint that `/pull_weights` keeps in sync — deltas are applied into it in place; a published full checkpoint replaces it. Each host seeds it from the engine's model path on the first `/pull_weights`. | | `--update-weight-delta-encoding` | On-disk delta encoding: `xor` (default) or `overwrite`. | | `--update-weight-delta-checksum` | Per-tensor integrity checksum: `xxh3-128` (default), `blake3`, or `adler32`. | Deltas are always zstd-compressed (level 1); profiling showed it dominates lz4 / gzip / snappy / brotli on both wire size and decompress speed for this data, so it is not a knob. ## How it works 1. **Seed.** On the first sync the trainer captures a CPU snapshot of every parameter — seeded from `--hf-checkpoint`, which is exactly what each rollout host materializes its local checkpoint from. Nothing is published; this snapshot is the base the next sync diffs against. The trainer also issues `/pull_weights` with `target_version=0` so every host materializes its local base now, overlapped with the snapshot capture. 2. **Publish.** On every later sync the trainer diffs each gathered HF tensor against the snapshot, encodes and compresses the change, and writes a new version directory `weight_v{N:06d}/` under `--update-weight-disk-dir`. The directory is a canonical HF checkpoint — `model-NNNNN.safetensors` files holding the compressed diff tensors plus a `model.safetensors.index.json` (tensor name → file) carrying the apply metadata — so the artifact is portable, not tied to the trainer's parallelism layout. The snapshot is then advanced to the new values for the next diff. 3. **Pull.** The trainer calls `/pull_weights` on each engine. Inside the engine the request is broadcast to every rank on every node; each host applies the new version's delta into its local checkpoint in place (a per-host file lock collapses co-located ranks to one apply). The apply is parallelized across tensors and verified per-tensor (see Integrity); the call only reports success once **every host** holds a checksum-verified checkpoint. `/pull_weights` is not delta-specific: each published version is self-describing, and a version that is an ordinary full HF checkpoint (no delta metadata in its index) is pulled by copying it as-is — resetting the chain, so a fresh host joining late seeds from the newest full version instead of replaying every delta, and older deltas can be pruned. slime's full-mode disk sync uses exactly this when `--update-weight-local-checkpoint-dir` is set. 4. **Reload.** The engines reload the patched local checkpoint through the vanilla `update_weights_from_disk` path — the weight-loading code never sees the delta format. Because the snapshot is seeded from `--hf-checkpoint` (the engine's actual base) rather than from the current GPU weights, the scheme is correct for any model even where the Megatron→HF round-trip is not byte-exact (e.g. trimmed vocab-padding rows in the embedding / LM head). ## Encodings Both encodings are byte-level and dtype-blind, so the same path works for quantized checkpoints. The engine reads the choice from each version's index metadata. - **`xor`** (default): writes `new ^ old`. Smallest wire and fastest to apply (sequential, cache-friendly; the unchanged bytes are zeros the compressor crushes). It is an involution, so it must be applied **exactly once** against the correct base — applying it twice reverts. - **`overwrite`**: writes the changed positions and their new absolute values. Larger on the wire and a less cache-friendly scattered apply, but **idempotent**: re-applying it (or finishing a partially-applied delta) converges to the same state regardless of how many times it runs. Use it when re-applicability matters more than wire size. ## Integrity The trainer stores a per-tensor checksum of each tensor's new state in the version. After applying, every host recomputes the checksum and **raises on any mismatch** — the failure propagates through the `/pull_weights` response, so a corrupt delta or a wrong base fails loud instead of serving bad weights. The apply also refuses to run out of order: a version only applies on top of its declared base version. `--update-weight-delta-checksum` selects the algorithm. The checksum is not the apply bottleneck (the apply is decompress + XOR bound), so this is a digest-property choice, not a speed one: `xxh3-128` (default) is the widest fast non-cryptographic digest; `blake3` is cryptographic, for untrusted storage; `adler32` is for interop with systems that expect it. ## Shared-filesystem visibility hooks On a POSIX shared filesystem (NFS, Lustre, …) no extra step is needed. Object-store-backed mounts that need an explicit publish/refresh to make writes visible across hosts can supply two optional hooks, loaded by import path — no vendor-specific code lives in slime or sglang: - `--custom-update-weight-post-write-path` (slime, trainer side): called after a version's files are written, before the engines are told to read it (e.g. upload pending writes to the backing object store). Signature: `hook(args, version_dir, rollout_engines)`. - `--sglang-custom-pull-weights-pre-read-hook` (sglang server arg, engine side): called on each host inside the engine before `/pull_weights` reads the delta directory (e.g. refresh the mount's view). Signature: `hook(delta_dir, target_version)`. ---