## File: README.md
## π° News
- **[2026/4/15]** π We sincerely thank [Fireworks AI](https://fireworks.ai) for its generous support of this project, which has enabled more experiments and faster iteration.
- **[2026/4/11]** β¨ Qwen3.5-4B/9B/27B is supported now, both text and multi-modal!
- **[2026/4/4]** π¨βπ¦βπ¦ We support optimizing a single model based on feedback from a group of people.
- **[2026/3/25]** π We sincerely thank [Tinker](https://thinkingmachines.ai/tinker/) for its generous support of this project, which has enabled more experiments and faster iteration.
- **[2026/3/20]** π» You can use your own openclaw now, simply install [this extension](https://github.com/Gen-Verse/OpenClaw-RL/tree/main/extensions/rl-training-headers).
- **[2026/3/13]** βοΈ OpenClaw-RL now supports both local GPU and cloud ([Tinker](https://thinkingmachines.ai/tinker/)) deployment. Launch with [**one line of code**](#combinemethod) β Hybrid RL, OPD, and Binary RL all supported!
- **[2026/3/12]** β‘ We support LoRA training now!
- **[2026/3/10]** π We have released our [**Technical Report**](https://arxiv.org/abs/2603.10165)! π Ranked **#1** on [HuggingFace Daily Papers](https://huggingface.co/papers/2603.10165)!
- **[2026/3/10]** π₯ Huge updates today! We released a [new combination method](./openclaw-combine), along with an [interesting evaluation](./openclaw-test) of these OpenClaw-RL methods. Track 2 is released too, featuring scalable RL implementations for general agent settings across [terminal](./terminal-rl), [GUI](./gui-rl), [SWE](./swe-rl), and [tool-call](./toolcall-rl) scenarios. We only focus on real-world settings!
- **[2026/3/3]** π Working with the authors of [SDFT](https://arxiv.org/abs/2601.19897) and [SDPO](https://arxiv.org/abs/2601.20802), we have integrated their methods into [openclaw-opd](./openclaw-opd). We welcome the integration of novel and effective methods!
- **[2026/3/3]** πΊ Check out these community tutorial videos on OpenClaw-RL: [Video 1](https://www.youtube.com/watch?v=5xnm1vB7G64) | [Video 2](https://www.youtube.com/watch?v=ZtN6Gg_bdJE)
- **[2026/2/26]** π₯ We release **OpenClaw-RL v1** β a fully asynchronous RL framework for training personalized AI agents from natural conversation feedback.
---
## π‘ TL;DR
> **OpenClaw-RL** is a fully asynchronous reinforcement learning framework that turns everyday conversations into training signals for personalized AI agents, and supports training general agents with large-scale environment parallelization.
Most RL-for-LLM systems assume centralized, batch-mode training with pre-collected datasets. **OpenClaw-RL** takes a fundamentally different approach: it wraps your self-hosted model in [OpenClaw](https://openclaw.ai) as an OpenAI-compatible API, intercepts live multi-turn conversations, and continuously optimizes the policy in the background β all without interrupting your usage.
> **Highlights:** Fully async 4-component loop Β· Self-hosted & private Β· Zero manual labeling Β· Three learning paradigms (Binary RL / OPD / Combine) Β· Personal + General agent support
**π Features**
### Fully Asynchronous 4-Component Architecture
OpenClaw-RL decouples **agent serving**, **rollout collection**, **PRM/judge evaluation**, and **policy training** into independent async loops. None of them block one another: the model continues serving requests while training runs in the background, and judging happens concurrently with new interactions.
### Self-Hosted & Private by Design
The entire stack, including the **policy model**, **judge/PRM**, and **trainer**, runs on **your own infrastructure**. Conversation data stays within your system, and no third-party model API is required.
### From Feedback to Gradient β Automatically
You do not need to manually label data. The system automatically:
- Organizes multi-turn interactions into session-aware training trajectories
- Classifies API messages into **main-line** (trainable) vs. **side** (non-trainable) turns
- Uses the next user, environment, or tool feedback as a natural "next-state" signal
- Runs PRM/judge evaluation asynchronously, with majority voting when needed for more robust scoring
- Submits ready samples to the trainer as they become available
### Three Optimization Methods in One Framework
**Binary RL (GRPO):** A Process Reward Model scores each turn based on next-state feedback. The scalar reward is then used with GRPO advantage estimation and a PPO-style clipped surrogate loss.
**On-Policy Distillation (OPD):** When the next state reveals useful hindsight, a judge model extracts a textual hint. This hint augments the original prompt to create an enhanced teacher, whose token-level log-probability gap with the student becomes a directional advantage signal richer than any scalar reward.
**Hybrid Method:** OpenClaw-RL further combines Binary RL and OPD in a unified training recipe, leveraging the dense scalar supervision of Binary RL together with the richer token-level directional signal from OPD. This combination achieves stronger and more robust optimization than either method alone.
### From Personal Agents to Real-World Agentic RL
The same framework supports both personalized OpenClaw optimization and scalable RL for **terminal**, **GUI**, **SWE**, and **tool-call** agents in real-world settings.
---
## π― Roadmap
Our long-term goal is to **advance personalized, practically useful agents with reinforcement learning**. The roadmap has two tracks:
#### Track 1 β [Personal Agent Optimization](#personalagent) (Small-Scale but Personal)
β
**Release Track 1:** Fully async OpenClaw-RL framework with Binary RL + OPD
β
Best recipe discovery via demonstration experiments
β
Support LoRA Training
β
Deploy training on [Tinker](https://thinkingmachines.ai/tinker/)
β
Deploy training on [Fireworks AI](https://fireworks.ai)
#### Track 2 β [General Agents Optimization](#generalagent) (Scalable Infra)
β
**Release Track 2:** Scalable agentic RL infra for general agents
β
Support Qwen3.5
β¬ Support more cloud services
## π Contents
- [Personal OpenClaw Optimization](#personalagent)
- [Bybrid RL](#combinemethod)
- [Method Evaluation](#evalmethod)
- [Agentic RL in Real World Settings](#agentrl)
- [Terminal Agent](#terminal)
- [GUI Agent](#gui)
- [SWE Agent](#swe)
- [Tool-call Agent](#toolcall)
---
## π§ Personal Agent Optimization Quick Start
### 1. Deployment Requirements
- **Hardware:** 8Γ GPUs (default; configurable via `NUM_GPUS`, `ACTOR_GPUS`, `ROLLOUT_GPUS`, `PRM_GPUS`)
- **Software:** CUDA 12.9, Python 3.12
- **Framework:** [Slime](https://github.com/THUDM/slime) (our base RL framework)
For detailed environment setup, see [Slime](https://github.com/THUDM/slime) or [`./instructions/README.md`](./instructions/README.md).
### 2. Start the RL Server
```bash
cd slime
bash ../openclaw-combine/run_qwen3_4b_openclaw_topk_select.sh
```
This method combines binary RL and OPD to achieve the best optimization.
See [`./openclaw-combine/README.md`](./openclaw-combine/README.md) for algorithm details.
Once running, the model is served as an OpenAI-compatible API at:
```
http://:30000/v1
```
where `` is the **IP address** of the machine running the RL server (e.g. `115.190.98.251`). The port `30000` is the default and can be changed via the `PORT` environment variable.
**Take note of this endpoint** β you will need it when configuring OpenClaw in the next step.
We also provide an interesting case for evaluation. A student who uses OpenClaw to do homework, does not want to be found using AI. A teacher who also uses OpenClaw to grade student's homework, wants the comments to be specific and friendly.
**Evaluation Setting** β Both student, TA and teacher use AI!
See [`./openclaw-test/README.md`](./openclaw-test/README.md) for setup and algorithm details. Example of evaluation [results](https://github.com/Gen-Verse/OpenClaw-RL/blob/main/openclaw-test/results.txt).
### 3. OpenClaw Setup
You can use your own openclaw, just install [this extension](https://github.com/Gen-Verse/OpenClaw-RL/tree/main/extensions/rl-training-headers).
**Then configure OpenClaw to route requests to your RL server. **
Open your `openclaw.json` (or the equivalent settings file) and add a provider entry under `"models"` β `"providers"`:
Example of Slime-based RL server:
```json
{
"models": {
"providers": {
"qwen": {
"baseUrl": "http://:30000/v1",
"apiKey": "apiKey",
"api": "openai-completions",
"models": [
{
"id": "qwen3-4b",
"name": "Qwen3 4B",
"reasoning": true,
"input": ["text"],
"cost": {
"input": 0,
"output": 0,
"cacheRead": 0,
"cacheWrite": 0
},
"contextWindow": 32768,
"maxTokens": 8192
}
]
}
}
}
}
```
Replace `` with the IP address of your RL server machine. The `apiKey` should match the `SGLANG_API_KEY` you set when starting the server.
Example of Tinker-based RL server:
```json
{
"models": {
"providers": {
"openclaw-rl": {
"baseUrl": "http://localhost:30000/v1",
"apiKey": "no-auth-needed",
"api": "openai-completions",
"models": [
{
"id": "qwen3-4b-lora",
"name": "Qwen3 4B (OpenClaw-RL LoRA)",
"reasoning": true,
"input": ["text"],
"cost": {
"input": 0,
"output": 0,
"cacheRead": 0,
"cacheWrite": 0
},
"contextWindow": 32768,
"maxTokens": 8192
}
]
}
}
}
}
```
That's it β start chatting with your OpenClaw agent. The RL server will automatically collect conversation trajectories, compute rewards, and train the model. Your agent gets better the more you use it.
---
## π§ Agentic RL in Real-world Settings
The same asynchronous RL backbone that powers our personal-agent setting can also support large-scale optimization for these broader real-world environments.
| Setting | Environment | Next-state signal | Horizon |
|---|---|---|---|
| Terminal | Shell execution sandbox | stdout/stderr, exit code | Long |
| GUI | Screen state + accessibility tree | Visual state diff, task progress | Long |
| SWE | Code repository + test suite | Test verdicts, diff, lint output | Long |
| Tool-call | API/function execution | Return values, error traces | Medium |
### π₯οΈ Terminal Agent β the most widely used computer-use agent
```bash
cd slime
bash ../terminal-rl/terminal_qwen3_8b_rl.sh
```
See [`./terminal-rl/README.md`](./terminal-rl/README.md) for setup details.
### π GUI Agent β the most general computer-use agent
```bash
cd slime
bash ../gui-rl/gui_qwen3vl_8b_rl.sh
```
See [`./gui-rl/README.md`](./gui-rl/README.md) for setup details.
### π¨βπ» SWE Agent β software engineering agent
```bash
cd slime
bash ../swe-rl/run_swe_rl_32b_remote_8nodes.sh
```
See [`./swe-rl/README.md`](./swe-rl/README.md) for setup details.
### π οΈ Tool-call Agent β the most practical agent
```bash
cd slime
bash ../toolcall-rl/retool_qwen3_4b_rl.sh
```
See [`./toolcall-rl/README.md`](./toolcall-rl/README.md) for setup details.
## π Citation
```
@article{wang2026openclawrl,
title={OpenClaw-RL: Train Any Agent Simply by Talking},
author={Wang, Yinjie and Chen, Xuyang and Jin, Xiaolong and Wang, Mengdi and Yang, Ling},
journal={arXiv preprint arXiv:2603.10165},
year={2026}
}
@article{wang2026rlanything,
title={RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System},
author={Wang, Yinjie and Xie, Tianbao and Shen, Ke and Wang, Mengdi and Yang, Ling},
journal={arXiv preprint arXiv:2602.02488},
year={2026}
}
```
## π Acknowledgements
This work aims to explore more effective paradigms for Agentic RL. Our implementation builds upon the excellent codebases of [slime](https://github.com/THUDM/slime), [OpenClaw](https://github.com/openclaw/openclaw), [Tinker](https://thinkingmachines.ai/tinker/) and [Open-AgentRL](https://github.com/Gen-Verse/Open-AgentRL).
We also build terminal RL using [SETA](https://github.com/camel-ai/seta)'s dataset and agent framework, GUI RL using [OSWorld](https://github.com/xlang-ai/OSWorld)'s evaluation scripts, SWE RL using [mini-swe-agent](https://github.com/SWE-agent/mini-swe-agent)'s evaluation scripts, and tool-call RL based on the work of [Retool](https://github.com/ReTool-RL/ReTool).
We sincerely thank these projects for their valuable insights and high-quality implementations, which have greatly facilitated our research.
## β οΈ Reminder
When using OpenClaw-RL, please do not provide sensitive personal information during conversations with the model. Also, make sure to keep your API keys secure and never expose them in prompts, logs, or shared files.
---
---
## File: extensions/rl-training-headers/README.md
# RL Training Headers
OpenClaw plugin that injects **`X-Session-Id`** and **`X-Turn-Type`** HTTP headers into every outgoing LLM API request, enabling downstream RL training pipelines to classify and segment training data.
## Headers
| Header | Value | Description |
|---|---|---|
| `X-Session-Id` | `` | The active agent session identifier |
| `X-Turn-Type` | `main` | User-initiated conversation turn |
| `X-Turn-Type` | `side` | Housekeeping turn (heartbeat, memory flush, cron) |
## Install
1. Copy (or symlink) this folder into `extensions/rl-training-headers` inside your OpenClaw repo.
2. Enable the plugin:
```bash
corepack pnpm start -- plugins enable rl-training-headers
```
3. Restart the gateway:
```bash
corepack pnpm start -- gateway restart
```
That's it. Every LLM API request now carries the two extra headers.
## Configuration (optional)
You can customize the header names in `~/.openclaw/openclaw.json` under `plugins.entries`:
```json
{
"plugins": {
"entries": {
"rl-training-headers": {
"enabled": true,
"config": {
"sessionIdHeader": "X-Session-Id",
"turnTypeHeader": "X-Turn-Type"
}
}
}
}
}
```
## How it works
The plugin hooks into the `before_prompt_build` lifecycle event to capture the current session ID and turn type (derived from the `trigger` field: `"user"` β `main`, `"heartbeat"` / `"memory"` / `"cron"` β `side`).
It then patches `globalThis.fetch` to inject these headers into outgoing POST requests during an active agent run. Header state is stored with async-local per-run context, so parallel sessions keep separate session IDs and turn types.
## Extracting training data
On the API proxy / logging side, use these headers to:
- **Group** requests by `X-Session-Id` to reconstruct full conversation sessions.
- **Filter** by `X-Turn-Type: main` to keep only user-facing dialogue for reward modelling, or include `side` turns for full trajectory analysis.
---
## File: gui-rl/desktop_env/evaluators/README.md
# Evaluator Setup Details
Setup scaffolding for the evaluators in the desktop environment for those who want to know the details of the evaluator setup for customized evaluation and extension
## Overall
Inside the virtual machine, disable the system crash report by:
```
sudo vim /etc/default/apport
```
and then change the `enabled` to `0`.
## VSCode
todo
## LibreOffice
For LibreOffice, please enter into the app first, and then enable the no pop-up when 'ctrl + s'.
## LibreOffice Press
### Setting Up the python-pptx Library
```shell
pip install python-pptx
```
## LibreOffice Writer
### Setting Up the python-docx and odfpy Library
```shell
pip install python-docx
pip install odfpy
```
## LibreOffice Calc
### Required Libraries
```
openpyxl
pandas
lxml
xmltodict
```
### How to Generate CSV from XLSX
```sh
libreoffice --convert-to "csv:Text - txt - csv (StarCalc):44,34,UTF8,,,,false,true,true,false,false,1" --out-dir /home/user /home/user/abc.xlsx
```
This command will generate `abc-Sheet1.csv` under `/home/user`. The last `1` in
the conversion options indicates the sheet number (starting from 1) to export.
Detailed usage should be referred to at [CSV Filter
Options](https://help.libreoffice.org/latest/ro/text/shared/guide/csv_params.html).
Refer to `libreoffice_calc/21df9241-f8d7-4509-b7f1-37e501a823f7.json` for an
example.
### About `compare_table`
Evaluation to xlsx files mainly relies on `compare_table`. It accepts two file
names and a list of rules defined as `options`. Refer to
`libreoffice_calc/21df9241-f8d7-4509-b7f1-37e501a823f7.json` for an example.
In each rule, there is a required field `type`. The supported types are defined
in `compare_table` function. The most common two are `sheet_data` and
`sheet_print`. `sheet_data` compares the internal cell values through pandoc,
while `sheet_print` compares the shown cell values through csv. A csv should be
generated and downloaded for `sheet_print`. See the previous section and
example in `libreoffice_calc/21df9241-f8d7-4509-b7f1-37e501a823f7.json`.
Other fields in a rule are described for each evaluation type in
`compare_table` function. `sheet_idx0` (or `sheet_idx1`, `sheet_idx`) is a
common field to indicate which sheet is to extracted from the workbook. If an
integer i is given, then it extracts the i-th sheet from result xlsx (i starts
from 0). If a string is given, it should be preceded with "RI", "RN", "EI", or
"EN". "R" indicates to extract from result xlsx while "E" indicates to extract
from expected (golden) xlsx. "I" indicates a sheet number (starting from 0) and
"N" indicates a sheet name (usually, they're like "Sheet1", "Sheet2", ...).
Some rules use a atructure like `{"method": "eq", "ref": "abc"}`. These rules
are checked through `utils._match_value_to_rule` function. Check it for the
implemented matching methods.
## Chrome
### Starting Chrome with Remote Debugging for Python
To enable remote debugging in Chrome, which allows tools like Playwright for Python to connect to and control an existing Chrome instance, follow these steps:
#### Manually Enabling Remote Debugging in Chrome
1. **Locate the Chrome Shortcut**:
- Find the Chrome shortcut that you usually use to open the browser. This could be on your desktop, start menu, or taskbar.
2. **Edit Shortcut Properties**:
- Right-click on the Chrome shortcut and select `Properties`.
3. **Modify the Target Field**:
- In the `Target` field, add `--remote-debugging-port=9222` at the end of the path. Ensure there is a space between the path and the flag you add.
- It should look something like this: `"C:\Path\To\Chrome.exe" --remote-debugging-port=9222`.
4. **Apply and Close**:
- Click `Apply` and then `OK` to close the dialog.
5. **Start Chrome**:
- Use this modified shortcut to start Chrome. Chrome will now start with remote debugging enabled on port 9222.
6. **Confirm Remote Debugging**:
- Open a browser and navigate to `http://localhost:9222`. If you see a webpage with information about active tabs, remote debugging is working.
---
### Setting Up Playwright for Python
Playwright for Python is a browser automation library to control Chromium, Firefox, and WebKit with a single API.
#### Installing Playwright
- Ensure you have Python installed on your system. If not, download and install it from the [Python official website](https://www.python.org/).
- Install Playwright using pip (Python Package Installer). Open a command line or terminal and run:
```bash
pip install playwright
```
- After installing Playwright, you need to run the install command to download the necessary browser binaries:
```bash
playwright install
```
#### Writing a Playwright Script in Python
- Create a Python file for your automation script.
- Import the Playwright module at the beginning of your script:
```python
from playwright.sync_api import sync_playwright
```
- You can now use Playwright's API to control browsers.
#### Example Playwright Script
Here is a simple example to open a page using Playwright:
```python
from playwright.sync_api import sync_playwright
def run(playwright):
browser = playwright.chromium.launch()
page = browser.new_page()
page.goto("http://example.com")
## other actions...
browser.close()
with sync_playwright() as playwright:
run(playwright)
```
- This script launches Chromium, opens a new page, navigates to `example.com`, and then closes the browser.
#### Troubleshooting
- If you encounter issues with Playwright, ensure that your Python environment is correctly set up and that you have installed Playwright and its dependencies correctly.
- For detailed documentation, visit the [Playwright for Python Documentation](https://playwright.dev/python/docs/intro).
## VLC Media Player
### Bugs fix
One thing on Ubuntu need to do, enter into the `meida`>`convert/save`>select files>`convert/save`
Then enter the profile of `Audio - MP3`, change the profile for mp3, section audiocodec from "MP3" to "MPEG Audio"
Otherwise the mp3 file will be created but with 0 bytes. It's a bug of VLC.
### Setting Up VLC's HTTP Interface
To enable and use the HTTP interface in VLC Media Player for remote control and status checks, follow these steps:
#### 1. Open VLC Preferences
- Open VLC Media Player.
- Go to `Tools` > `Preferences` from the menu.
#### 2. Show All Settings
- In the Preferences window, at the bottom left corner, select `All` under `Show settings` to display advanced settings.
#### 3. Enable Main Interfaces
- In the advanced preferences, expand the `Interface` section.
- Click on `Main interfaces`.
- Check the box for `Web` to enable the HTTP interface.
#### 4. Configure Lua HTTP
- Expand the `Main interfaces` node and select `Lua`.
- Under `Lua HTTP`, set a password `password` in the `Lua HTTP` section. This password will be required to access the HTTP interface.
#### 5. Save and Restart VLC
- Click `Save` to apply the changes.
- Restart VLC Media Player for the changes to take effect.
#### 6. Accessing the HTTP Interface
- Open a web browser and go to `http://localhost:8080`.
- You will be prompted for a password. Enter the password you set in the Lua HTTP settings.
- Once logged in, you will have access to VLC's HTTP interface for remote control.
#### Packages
```bash
pip install opencv-python-headless Pillow imagehash
```
#### Troubleshooting
- If you cannot access the HTTP interface, check if your firewall or security software is blocking the connection.
- Ensure VLC is running and the correct port (default is 8080) is being used.
- If the port is in use by another application, you may change the port number in VLC's settings.
## GIMP
Click on the "Keep" of the image loading pop-up.
---
## File: gui-rl/desktop_env/server/README.md
# Server setup
This README is useful if you want to set up your own machine for the environment. This README is not yet finished. Please contact the author if you need any assistance.
## Configuration Overview
The following sections contain guidelines for configuring the system image to ensure benchmark examples can run properly.
The main configuration requirements include:
1. **Account Credentials**:
Our benchmark configurations are based on specific username and password settings (with username `user` and password `password`).
Please ensure these settings remain consistent or update the corresponding configuration files.
2. **Service Setup**:
Our environment operates through a service that automatically starts at boot time, as shown in the figure below. The service needs to be properly configured and placed.
3. **Accessibility Tree Support**:
Benchmark examples rely on accessibility tree functionality. The necessary support packages need to be installed.
4. **System Service Management**:
Certain system services that may cause interference need to be disabled, such as automatic updates and notification pop-ups.
5. **Required Software Installation**:
Ensure all necessary software packages required by the benchmark examples are properly installed.
6. **Software Configuration**:
Various software packages require specific configurations, such as disabling certain auto-save features or installing additional plugins.
7. **Port Configuration**:
To monitor and control software states from the host machine, specific port configurations are needed for various applications.
8. **Miscellaneous Settings**:
Additional system-specific settings need to be configured, such as desktop environment settings and display resolution.
Detailed instructions for each of these requirements will be provided in the following sections.
## [Ubuntu](https://huggingface.co/datasets/xlangai/ubuntu_osworld)
Make a new VM with the Ubuntu 20.04 LTS image.
### How to install Ubuntu Desktop (package: ubuntu-desktop) with GNOME desktop environment on Ubuntu 22.04 system.
```bash
sudo apt update
sudo apt install ubuntu-desktop
sudo systemctl set-default graphical.target
```
### Account Credentials
Download the iso file from the [Ubuntu website](https://ubuntu.com/download/alternative-downloads) and install it in the VM.
Using GUI:
The default username should be `user` and the password should be `password` when you are asked to set up the account. Give the user sudo permission.
Using Command Line:
```bash
sudo adduser user
usermod -aG sudo user
```
### Installation and Auto-login Setup
1. Download the iso file from the [Ubuntu website](https://ubuntu.com/download/alternative-downloads) and install it in the VM.
The default username should be `user` and the password should be `password` when you are asked to set up the account.
2. To enable automatic login:
Using GUI:
```bash
# Open Settings -> Users
# Click Unlock button and enter password
# Toggle "Automatic Login" to ON for user 'user'
```
Or using Command Line:
```bash
# Edit the custom configuration file
sudo nano /etc/gdm3/custom.conf
# Under [daemon] section, add or modify these lines:
AutomaticLoginEnable=true
AutomaticLogin=user
# Save the file and restart the system
sudo systemctl restart gdm3
```
After setting up automatic login, the system will boot directly into the desktop environment without requiring password input, which enables seamless startup experience for automated testing environments.
### VNC Configuration
1. Install x11vnc
```
sudo apt update
sudo apt install x11vnc
```
2. Install noVNC
```
sudo snap install novnc
```
3. Create system services for x11vnc and novnc:
- Go to directory cd `/etc/systemd/user/`
- Write a file `novnc.service` with the following content:
```
[Unit]
Description=noVNC Service
After=x11vnc.service network.target snap.novnc.daemon.service
Wants=x11vnc.service
[Service]
Type=simple
ExecStart=/snap/bin/novnc --vnc localhost:5900 --listen 5910
Restart=on-failure
RestartSec=3
Environment=DISPLAY=:0
Environment=XAUTHORITY=/home/user/.Xauthority
Environment=SNAP_COOKIE=/run/snap.cookie
Environment=SNAP_NAME=novnc
Environment=SNAP_REVISION=current
[Install]
WantedBy=default.target
```
Write a file `x11vnc.service` with the following content:
```
[Unit]
Description=X11 VNC Server
After=display-manager.service network.target
Wants=display-manager.service
[Service]
Type=simple
ExecStart=x11vnc -display :0 -rfbport 5900 -forever
Restart=on-failure
RestartSec=3
Environment=DISPLAY=:0
Environment=XAUTHORITY=/home/user/.Xauthority
[Install]
WantedBy=default.target
```
4. Enable both services:
```
systemctl --user daemon-reload
systemctl --user enable novnc.service
systemctl --user enable x11vnc.service
systemctl --user start x11vnc.service
systemctl --user start novnc.service
```
5. Allow VNC port:
Expose 5910 port in the firewall and any other security tools you are using.
6. Access the VNC server:
Connect to the VNC via `http://[Instance IP]:5910/vnc.html`
### Display Configuration
> **β οΈ IMPORTANT NOTE**: The display configuration is critical for proper system operation. Incorrect settings can prevent the graphical environment from starting and potentially crash the X server. Make sure to follow these steps carefully and verify each configuration file. If you encounter any issues, check the X server logs at `/var/log/Xorg.0.log` for troubleshooting. Backup your existing X11 configuration before making any changes.
1. Install dummy video driver:
```
sudo apt-get install xserver-xorg-video-dummy
```
Go to `/etc/X11/` and create a file named `xorg.conf` with the following content:
```
/* Detailed source-code truncated for AI context efficiency. */
```
2. In the same directory as the previous step, go to its sub-directory `xorg.conf.d` , create a file named `10-dummy.conf` with the following content:
```
Section "Device"
Identifier "DummyDevice"
Driver "dummy"
VideoRam 32768
EndSection
Section "Monitor"
Identifier "DummyMonitor"
HorizSync 28.0-80.0
VertRefresh 48.0-75.0
Modeline "1920x1080" 172.80 1920 2048 2248 2576 1080 1083 1088 1120
EndSection
Section "Screen"
Identifier "DummyScreen"
Device "DummyDevice"
Monitor "DummyMonitor"
DefaultDepth 24
SubSection "Display"
Depth 24
Modes "1920x1080"
EndSubSection
EndSection
```
3. Reload the display manager:
```
sudo systemctl restart display-manager
```
### Set up the OSWorld server service in VM
Upload the OSWorld server to the home directory (/home/user) of user (via scp or git clone).
1. Copy the `main.py` and `pyxcursor.py` and to the `/home/user-name` where the `user-name` is your username of the ubuntu, here we make it `user` as default. If you customize the path of placing these files in this step, you should change the parameters in the service file we will mention later accordingly.
2. First please set up the environment:
```shell
sudo apt install python3
pip3 install -r requirements.txt
sudo apt-get install python3-tk python3-dev
sudo apt install gnome-screenshot
sudo apt install wmctrl
sudo apt install ffmpeg
sudo apt install socat
sudo apt install xclip
```
If you encounter an error about python not being found, run:
```
sudo ln -s /usr/bin/python3 /usr/bin/python
```
if you customize the environment in this step, you should change the parameters in the service file we will mention later accordingly.
3. Due to some configuration issues, you need to modify the `osworld_server.service` file:
1) In our released version, the X server is set to :1, but the default X server is actually :0. You need to modify the `osworld_server.service` file to change the `DISPLAY` variable from `:1` to `:0`.
Change the following line:
```
Environment="DISPLAY=:1"
```
to
```
Environment="DISPLAY=:0"
```
2) Need to add environment variables to enable DBUS to change wallpaper.
Change the following line:
```
Environment="DISPLAY=:0"
```
to
```
Environment="DISPLAY=:0;DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus"
```
4. Copy the `osworld_server.service` to the systemd configuration directory at `/etc/systemd/system/`:
```shell
sudo cp osworld_server.service /etc/systemd/system/
```
Reload the systemd daemon to recognize the new service:
```shell
sudo systemctl daemon-reload
```
Enable the service to start on boot:
```shell
sudo systemctl enable osworld_server.service
```
Start the service:
```shell
sudo systemctl start osworld_server.service
```
Verify the service is running correctly:
```shell
sudo systemctl status osworld_server.service
```
You should see output indicating the service is active and running. If there are errors, review the logs with `journalctl -xe` for further troubleshooting.
If you need to make adjustments to the service configuration, you can edit the `/etc/systemd/system/osworld_server.service` file:
```shell
sudo nano /etc/systemd/system/osworld_server.service
```
After making changes, reload the daemon and restart the service:
```shell
sudo systemctl daemon-reload
sudo systemctl enable osworld_server.service
sudo systemctl start osworld_server.service
```
### Accessibility Tree Support
To support the accessibility tree functionality, you'll need to install pyastpi2 in your Ubuntu environment. This package enables access to accessibility information and tree structures.
Installation steps:
```bash
# Update package list and ensure pip is installed
sudo apt-get update
sudo apt-get install python3-pip
# Install pyastpi2 using pip
pip3 install pyastpi2
```
### Xorg Configuration
Regarding the graphics display system, we need to ensure that Ubuntu displays images using the **Xorg** protocol instead of **Wayland**. Since **Wayland** is typically the default setting for Ubuntu, we will need to manually change the settings.
1. Click the user menu in the upper right corner and select "Log Out" or "Sign Off."
2. On the login screen, click on the username.
3. Before entering the password, click the gear icon in the lower right or upper right corner of the screen (it may need to be displayed after clicking the username first).
4. Select "Ubuntu on Xorg" from the pop-up menu.
You can run the following command to check if **Xorg** is being used:
```bash
echo $XDG_SESSION_TYPE
```
### System Service Management (Optional)
The automatic software update service can interfere with benchmark examples. To disable this service, you can refer to the https://www.makeuseof.com/disable-automatic-updates-in-ubuntu/ for the solution.
You can check and manage system services using systemctl commands. For example, to verify if a service like unattended-upgrades is installed and running on your system:
```bash
# Check service status
sudo systemctl status unattended-upgrades.service
```
If the output is `x11`, it means you have switched to **Xorg**.
To disable a system service:
```bash
# Disable and stop the service
sudo systemctl disable unattended-upgrades
sudo systemctl stop unattended-upgrades
```
To verify service configurations, you can use apt-config:
```bash
# Check current configurations
apt-config dump APT::Periodic::Update-Package-Lists
apt-config dump APT::Periodic::Unattended-Upgrade
```
### Software Installation
#### Software Installation Source
Since for some examples like change the settings of certain software, we hardcode some paths in our evaluation file, which means you need to install the software to the specific path. Here we provide a list of software that you need to install and the certain source which default the path you should install them to.
1. Chrome: If you are using ARM System, download the chromium using `sudo snap install chromium` and make sure your Chromium config files are under `~/snap/chromium`; otherwise, download the chrome from the [Chromium](https://www.chromium.org/Home) and make sure your Chromium config files are under `~/.config/google-chrome`.
2. LibreOffice: Go to [LibreOffice Website](https://www.libreoffice.org/), select "Download Libreoffice", select "older versions" in the bottom of the page, and download `7.3.7.2` version.
3. GIMP: Search "GIMP" in "Ubuntu Software" and install it. Our GIMP version is `2.10.30`.
4. VLC: Search "VLC" in "Ubuntu Software" and install it. Our VLC version is `3.0.16`.
5. VSCode: Go to [VSCode Website](https://code.visualstudio.com/download), download the `.deb` file, and install it. Our VSCode version is `1.91.1`.
#### Additional Inner Software Installation
> **β οΈ IMPORTANT NOTE**: The software installation and configuration steps described in this section are crucial for maintaining consistent task execution and performance. Skipping or incorrectly configuring these components may lead to task failures or degraded performance. Please follow the installation instructions carefully and verify each component is properly set up before proceeding.
##### LibreOffice font installation
Some examples in LibreOffice Impress use non-default system fonts, and you need to download the corresponding **TTF files** and put them in the system fonts directory.
[Here](https://huggingface.co/datasets/xlangai/ubuntu_osworld_file_cache/resolve/main/fonts_20250608_fixed.zip) we provides all the fonts downloaded, just download it, and unzip to the system fonts directory (which usually `usr/share/fonts/`).
```bash
unzip fonts.zip -d /usr/share/fonts/
```
And then run the following command to refresh the font cache:
```bash
sudo fc-cache -fv
```
##### Customized Plugin Installation
**VS Code plugin installation:**
To extract relevant internal information and configurations from the VS Code environment, we principally leverage the capabilities offered by the VS Code Extension API. Here's how to install the extension developed by ourselves:
```bash
1. Download the extension from: https://github.com/xlang-ai/OSWorld/blob/04a9df627c7033fab991806200877a655e895bfd/vscodeEvalExtension/eval-0.0.1.vsix
2. Open VS Code
3. Go to Extensions -> ... -> Install from VSIX... -> choose the downloaded eval-0.0.1.vsix file
```
### Software Configuration
1. LibreOffice Default Format Settings:
```bash
# Open LibreOffice Writer/Calc/Impress
# Go to Tools -> Options -> Load/Save -> General
# Under "Default file format and ODF settings":
# Change "Document type" to "Text document"
# Set "Always save as" to "Word 2007-365 (.docx)"
# Change "Document type" to "Spreadsheet"
# Set "Always save as" to "Excel 2007-365 (.xlsx)"
# Change "Document type" to "Presentation"
# Set "Always save as" to "PowerPoint 2007-365 (.pptx)"
```
2. Chrome password requirement removal:
Chrome requests a password input when first opened after system startup, which can interfere with our experiments. Here's how to disable this feature:
```bash
# Prevent Chrome from using keyring
mkdir -p ~/.local/share/keyrings
touch ~/.local/share/keyrings/login.keyring
```
Or you can use any ways to disable the keyring service, which will prevent Chrome from requesting a password input.
3. VSCode Trust Settings:
To prevent VSCode from showing "Do you trust this workspace?" dialog when opening projects, configure the following setting:
```bash
# Open VSCode
# Go to File -> Preferences -> Settings (or press Ctrl+,)
# In the search bar, type "workspace trust"
# Find "Security: Workspace Trust Enabled" and uncheck it
# Find "Security: Workspace Trust Banner" and set to "never"
# Find "Security: Workspace Trust Empty Window" and uncheck it
# Find "Security: Workspace Trust Startup Prompt" and set to "never"
```
This disables the workspace trust feature and prevents trust dialog prompts when opening projects.
### Network Configuration
#### Firewall Configuration
In OSWorld, we need the following ports to be open:
```
server_port = 5000
chromium_port = 9222
vnc_port = 8006
vlc_port = 8080
novnc_port = 5910
```
Please open the corresponding ports in the firewall and any other security tools you are using.
#### socat Installation
Ensure `socat` is installed to enable port forwarding.
```sh
sudo apt install socat
```
#### Network Configuration for Remote Control
##### VLC Configuration
To enable remote control of VLC media player, follow these configuration steps:
1. Enable HTTP interface:
```bash
# Open VLC
# Go to Tools -> Preferences
# Show Settings: All (bottom left)
# Navigate to Interface -> Main interfaces
# Check 'Web' option
```
2. Configure HTTP interface settings:
```bash
# Still in Preferences
# Navigate to Interface -> Main interfaces -> Lua
# Under Lua HTTP:
# - Set Password to 'password'
```
The following is the screenshot of the VLC configuration:
When VLC is open, the service will be running on port 8080.
##### Chrome Configuration
When you open Chrome through the GUI interface, it doesn't automatically enable remote debugging port 1337 (the port we set for controlling the client machine's Chrome through the host, which gets forwarded to 9222).
This is because GUI startup and command-line startup use different parameters.
The consequence is that once Chrome is closed and reopened, it will fail to connect.
To ensure Chrome uses consistent debugging ports even after being closed and reopened, follow these steps:
1. Create or edit Chrome desktop entry:
```bash
sudo vim ~/.local/share/applications/google-chrome.desktop
```
If that doesn't work, try:
```
sudo vim /usr/share/applications/google-chrome.desktop
```
2. Modify **all** the `Exec` lines to include debugging port, for example, change:
```
Exec=/usr/bin/google-chrome-stable %U
```
into:
```
Exec=/usr/bin/google-chrome-stable --remote-debugging-port=1337 --remote-debugging-address=0.0.0.0 %U
```
In cases where Chrome is needed, the 1337 will be forwarded to 9222 in the virtual machine via socat.
### Miscellaneous Settings
#### Screen Resolution
The required screen resolution for the virtual machine is 1920x1080 in OSWorld and we did make some hardcode related to this resolution in our configuration file in some examples, but only a few.
So please set the screen resolution to 1920x1080 in the virtual machine settings.
#### Automatic Suspend
To close automatic suspend, open Setting app and enter "Power" section. Switch "Screen Blank" to "Never" and "Automatic Suspend" to "Off".
#### Additional Installation
Activating the window manager control requires the installation of `wmctrl`:
```bash
sudo apt install wmctrl
```
Otherwise, you cannot control the window manager in the virtual machine when running the experiments. Some cases will be effected.
To enable recording in the virtual machine, you need to install `ffmpeg`:
```bash
sudo apt install ffmpeg
```
Otherwise you cannot get the video recording of the virtual machine when running the experiments.
### Others Information
#### About the Converted Accessibility Tree
For several applications like Firefox or Thunderbird, you should first enable
```sh
gsettings set org.gnome.desktop.interface toolkit-accessibility true
```
to see their accessibility tree.
##### Example of AT
An example of a node:
```xml
ζ‘θΏδ½Ώη¨ζ°η Outlook.com εΈ³ζΆ
```
An example of a tree:
```xml
...
...
```
##### Useful attributes
1. `name` - shows the name of application, title of window, or name of some
component
2. `attr:class` - somewhat the same role as `class` in HTML
3. `attr:id` - somewhat the same role as `id` in HTML
4. `cp:screencoord` - absolute coordinator on the screen
5. `cp:windowcoord` - relative coordinator in the window
6. `cp:size` - the size
Also several states like `st:enabled` and `st:visible` can be indicated. A full
state list is available at
.
##### How to use it in evaluation
See example `thunderbird/12086550-11c0-466b-b367-1d9e75b3910e.json` and
function `check_accessibility_tree` in `metrics/general.py`. You can use CSS
selector or XPath to reference a target nodes. You can also check its text
contents.
An example of a CSS selector:
```css
application[name=Thunderbird] page-tab-list[attr|id="tabmail-tabs"]>page-tab[name="About Profiles"]
```
This selector will select the page tab of profile manager in Thunderbird (if open).
For usage of CSS selector: . For usage of XPath: .
##### Manual check
You can use accerciser to check the accessibility tree on GNOME VM.
```sh
sudo apt install accerciser
```
## [MacOS](https://huggingface.co/datasets/xlangai/macos_osworld)
Coming soon...
---
## File: gui-rl/README.md
# GUI-RL: Reinforcement Learning for Desktop GUI Agents
This directory trains and evaluates **vision-language models** (Qwen3-VL) to operate real desktop environments through screenshots and `pyautogui` actions. The agent observes the screen, reasons about the task, and emits keyboard/mouse actions β all inside cloud VMs managed by a pool server.
## Architecture
```
ββββββββββββββββββββββ ββββββββββββββββββββββββββββββββββββββ
β Env Pool Server β β Ray Cluster β
β (Flask, port β HTTP β ββββββββββββ βββββββββββββββββ β
β 18080) βββββββββΊβ β Rollout β β Actor / β β
β β β β (SGLang) β β Trainer β β
β allocate / reset β β ββββββββββββ βββββββββββββββββ β
β step / evaluate β β β² β² β
β close β β β screenshots β weights β
ββββββββββ¬ββββββββββββ βββββββββΌββββββββββββββββΌβββββββββββββ
β β β
β VM lifecycle β β
βΌ β β
ββββββββββββββββββββββ βββββββββ΄ββββββββββββββββ΄ββββ
β Cloud VMs β β generate_with_gui.py β
β (Volcengine / β β (trajectory loop: β
β AWS / Aliyun) β β screenshot β agent β β
β Ubuntu desktops β β action β step β reward)β
ββββββββββββββββββββββ βββββββββββββββββββββββββββββ
```
**Env Pool Server** β A Flask service that manages a pool of cloud VM instances. It handles allocation, task reset (snapshot restore + setup scripts), action execution, evaluation, and teardown. VMs are pre-warmed at launch to eliminate cold-start latency during training.
**Agent** (`Qwen3VLAgentLocal`) β Converts screenshot observations into multi-turn VLM conversations (system prompt + image history + action output), and parses the model's response into executable `pyautogui` calls.
**Reward Agent** (`Qwen3VLRewardAgent`, PRM mode only) β A separate VLM that scores each agent step by comparing the action's intent against the next observation, producing +1/β1 per-step rewards for step-wise advantage estimation.
**Trajectory Loop** (`generate_with_gui.py`) β Orchestrates multi-step episodes: allocate a VM, reset to a task, loop (screenshot β model inference β parse action β execute β observe), compute final reward via OSWorld evaluators, and return the trajectory for training.
## Three Operating Modes
| Mode | Script | Description |
|---|---|---|
| **Eval-only** | `gui_qwen3vl_4b_eval.sh` | Load a checkpoint, run all eval tasks, report accuracy. No training. |
| **RL Training** | `gui_qwen3vl_8b_rl.sh` | GRPO-based RL with sequence-level binary rewards from OSWorld evaluators. |
| **PRM + RL Training** | `gui_qwen3vl_8b_prm_rl.sh` | RL with an additional Process Reward Model that provides per-step rewards via a separate VLM judge. |
## Prerequisites
- 8 GPUs (default; configurable via `NUM_GPUS`)
- A cloud provider account (Volcengine by default) with VM image and network configured
- Volcengine secrets (`VOLC_ACCESSKEY`, `VOLC_SECRETKEY`) exported in your shell
- Python environment with `requirements.txt` installed
- SLIME framework and Megatron-LM available in sibling directories
Install the GUI-specific dependencies:
```bash
pip install -r requirements.txt
playwright install chromium
```
## Step-by-Step Guide
### Mode 1: Evaluation Only
Run a checkpoint on the full eval task suite without any training:
```bash
# Optional: point to your checkpoint (default: Qwen3-VL-4B-Thinking)
export HF_CKPT="/path/to/checkpoint"
# Optional: W&B logging
export WANDB_KEY="your-wandb-key"
cd slime
bash ../gui-rl/gui_qwen3vl_4b_eval.sh
```
The script:
1. Starts the env pool server and pre-warms VMs.
2. Launches a Ray cluster with all GPUs dedicated to rollout (no actor).
3. Runs every task in `evaluation_examples/test_nochrome.json`.
4. Reports accuracy per domain.
### Mode 2: RL Training (GRPO)
Train with sequence-level binary rewards β the agent gets +1 if the OSWorld evaluator marks the task as successful, β1 otherwise:
```bash
export HF_CKPT="/path/to/Qwen3-VL-8B-Thinking"
export WANDB_KEY="your-wandb-key"
# Optional GPU split (default: 4 actor + 4 rollout)
export ACTOR_GPUS=4
export ROLLOUT_GPUS=4
cd slime
bash ../gui-rl/gui_qwen3vl_8b_rl.sh
```
The script partitions GPUs between the training actor and the SGLang rollout engine, then alternates between:
1. **Rollout**: generate trajectories across `n_samples_per_prompt` parallel attempts per task.
2. **Train**: update the policy using GRPO with KL regularization.
3. **Eval**: periodically run the eval task set and log accuracy to W&B.
### Mode 3: PRM-Augmented RL Training
Adds a **Process Reward Model** that scores each step, enabling step-wise advantage estimation instead of sequence-level:
```bash
export HF_CKPT="/path/to/Qwen3-VL-8B-Thinking"
export PRM_MODEL_PATH="/path/to/Qwen3-VL-4B-Thinking"
export WANDB_KEY="your-wandb-key"
# GPU partition: actor + rollout + PRM must equal NUM_GPUS
export ACTOR_GPUS=4
export ROLLOUT_GPUS=3
export PRM_GPUS=1
cd slime
bash ../gui-rl/gui_qwen3vl_8b_prm_rl.sh
```
The PRM agent takes the interaction history plus the next observation after each step, and assigns +1 (progress) or β1 (no effect / regression). These per-step rewards feed into a `step_wise` advantage estimator, providing denser gradient signal than the final-outcome-only GRPO.
## Key Environment Variables
### GPU & Training
| Variable | Default | Description |
|---|---|---|
| `NUM_GPUS` | `8` | Total GPUs on the node |
| `ACTOR_GPUS` | `4` | GPUs for the Megatron training actor |
| `ROLLOUT_GPUS` | `4` (`3` for PRM) | GPUs for the SGLang rollout engine |
| `PRM_GPUS` | `1` | GPUs for the PRM judge engine (PRM mode) |
| `HF_CKPT` | *(model-specific)* | HuggingFace checkpoint path |
| `PRM_MODEL_PATH` | β | Checkpoint for the PRM reward model |
| `ROLLOUT_BATCH_SIZE` | `8` | Tasks per rollout batch |
| `N_SAMPLES_PER_PROMPT` | `8` | Parallel attempts per task |
### GUI Environment
| Variable | Default | Description |
|---|---|---|
| `GUI_POOL_MAX_ENVS` | `64` | Max concurrent VM instances |
| `GUI_PREWARM_ENVS` | `64` | VMs to pre-warm before training starts |
| `GUI_PROVIDER_NAME` | `volcengine` | Cloud provider (`volcengine` / `aws` / `aliyun`) |
| `GUI_REGION` | `cn-beijing` | Cloud region for VM provisioning |
| `GUI_ACTION_SPACE` | `pyautogui` | Action space for the agent |
| `GUI_OBSERVATION_TYPE` | `screenshot` | Observation modality |
| `GUI_SCREEN_WIDTH` | `1920` | VM screen resolution width |
| `GUI_SCREEN_HEIGHT` | `1080` | VM screen resolution height |
### Logging
| Variable | Default | Description |
|---|---|---|
| `WANDB_KEY` | β | Weights & Biases API key |
| `WANDB_PROJECT` | `slime_gui` | W&B project name |
| `GUI_RESULT_DIR` | `./results` | Directory for trajectory logs and eval results |
### Cloud Provider (Volcengine)
Secrets must be exported before running:
```bash
export VOLC_ACCESSKEY="your-access-key"
export VOLC_SECRETKEY="your-secret-key"
```
Non-secret configs (`VOLCENGINE_IMAGE_ID`, `VOLCENGINE_SUBNET_ID`, etc.) have defaults in the shell scripts and can be overridden.
## File Structure
```
/* Detailed source-code truncated for AI context efficiency. */
```
---
## File: instructions/README.md
### Slime Env Setup
```bash
# cuda 12.9 (nvcc -V, nvidia-smi)
cd OpenClaw-RL
conda create --name openclaw-rl python=3.12 -y
conda activate openclaw-rl
pip install \
torch==2.9.1+cu129 \
torchvision==0.24.1+cu129 \
torchaudio==2.9.1+cu129 \
--index-url https://download.pytorch.org/whl/cu129
pip install -r requirements.txt
# DeepEP
git clone https://github.com/deepseek-ai/DeepEP.git
cd DeepEP
pip install -e . --no-build-isolation
cd ..
pip install -e slime/slime/backends/megatron_utils/kernels/int4_qat --no-build-isolation
# apex
git clone https://github.com/NVIDIA/apex.git
cd apex
APEX_CPP_EXT=1 APEX_CUDA_EXT=1 pip install -v --no-build-isolation .
cd ..
# flash_attn
export MAX_JOBS=8
pip install --no-build-isolation -v flash-attn==2.7.4.post1
# flashinfer
pip install "flashinfer-jit-cache==0.6.3" --index-url https://flashinfer.ai/whl/cu129
# megatron-bridge
pip install "megatron-bridge @ git+https://github.com/fzyzcjy/Megatron-Bridge.git@35b4ebfc486fb15dcc0273ceea804c3606be948a" --no-build-isolation
# TransformerEngine
export NVTE_FRAMEWORK=pytorch
pip install --no-build-isolation "transformer_engine[pytorch,core_cu12]==2.10.0"
# apt
apt-get update
apt-get install -y python3-apt
```
If you want to use Qwen3.5 (omit this if only use Qwen3)
```
# upgrade transformers
pip install transformers==5.3.0
```
If you want to use Qwen3.5 for multimodal (omit if only use Qwen3, or use Qwen3.5 text only)
```
# Megatron-Bridge for qwen35 vl
cd OpenClaw-RL
git clone --recursive https://github.com/NVIDIA-NeMo/Megatron-Bridge.git Megatron-Bridge-qwen35
cd Megatron-Bridge-qwen35
git checkout ebca893607d48388a6c083bfc143bc05621cc753
git submodule update --init --recursive
# Megatron-LM comes from the Megatron-Bridge submodule
cd 3rdparty/Megatron-LM
git checkout 17a67b9a97fb11a75933fd7f76ad76e1ac98a53d
cd /path/to/Megatron-Bridge-qwen35
python3 -m pip uninstall -y megatron-bridge megatron-core mbridge
python3 -m pip install --no-deps -e ./3rdparty/Megatron-LM
python3 -m pip install --no-deps -e ./
```
---
## File: Megatron-LM/docs/discussions/deepseek-v3-gb200-optimization/deepseek-v3-gb200-reproduce-guide.md
# A Guide to Reproduce DeepSeek-V3 Pre-training Performance on GB200
## 1. Dockerfile
Requirements:
- Transformer Engine: We recommend using commit [d2945c6](https://github.com/NVIDIA/TransformerEngine/commit/d2945c6a571e3978677614d1fe08779966a5a4ef) with PR [2146](https://github.com/NVIDIA/TransformerEngine/pull/2146) and [2150](https://github.com/NVIDIA/TransformerEngine/pull/2150). You could prepare the branch by yourself, or use this [branch](https://github.com/hxbai/TransformerEngine/commits/dev_20251024/) based on TE v2.9 plus the above three commits/PRs.
- cuDNN: v9.14 is required.
- HybridEP: Install it from [here](https://github.com/deepseek-ai/DeepEP/commits/3f601f7ac1c062c46502646ff04c535013bfca00).
Dockerfile for reference.
```
/* Detailed source-code truncated for AI context efficiency. */
```
> [!Tip]
>
> If you prefer to use CUDA 12.9, please change the base container to `nvcr.io/nvidia/pytorch:25.06-py3` and the cuDNN to be installed to `libcudnn9-cuda-12`.
## 2. Megatron-Core
We recommend using the [dev branch](https://github.com/NVIDIA/Megatron-LM/tree/dev) after PR [1917](https://github.com/NVIDIA/Megatron-LM/pull/1917).
```bash
git clone https://github.com/NVIDIA/Megatron-LM.git && \
cd Megatron-LM &&
git checkout effebd81f410bc6566fffee6c320b6f8f762e06d
```
## 3. Cluster Configuration
Since we're using EP 32 on NVL72, it's important to make sure
> [!Important]
> **Every 32 GB200 GPUs (8 nodes) are in the same NVL domain (or rack)**.
Usually you can make it via your cluster workload manager. Taking Slurm as an example, you could pass `--segment 8` to the sbatch command to ensure that every segment of 8 nodes will be scheduled to a rack.
## 4. Training scripts
### Environment variables
```bash
CUDA_DEVICE_MAX_CONNECTIONS=1
NVTE_FWD_LAYERNORM_SM_MARGIN=0
NVTE_BWD_LAYERNORM_SM_MARGIN=0
NVLINK_DOMAIN_SIZE=72
NVTE_ALLOW_NONDETERMINISTIC_ALGO=1
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
NCCL_NVLS_ENABLE=0
NVTE_FUSED_ATTN=1
NVTE_NORM_FWD_USE_CUDNN=1
NVTE_NORM_BWD_USE_CUDNN=1
PYTHONWARNINGS=ignore
NCCL_DEBUG=VERSION
NCCL_GRAPH_REGISTER=0
```
### bindpcie
Download [bindpcie](https://github.com/NVIDIA/mlperf-common/blob/main/client/bindpcie) to your workdir, make it executable,
```bash
wget https://raw.githubusercontent.com/NVIDIA/mlperf-common/refs/heads/main/client/bindpcie &&
chmod 755 bindpcie
```
and then
> [!Important]
> **Place it at the beginning of your launch command in every process.**
Taking Slurm as an example, your script should look like
```bash
#!/bin/bash
#SBATCH [... sbatch args]
srun [... srun args] /path/to/bindpcie /path/to/pretrain_gpt.py [... mcore arguments]
```
This is a very important step on GB200.
### Launch script
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Explanation of arguments
The following arguments indicate key optimizations.
- Pipeline parallel layout
```bash
--pipeline-model-parallel-layout "Et|(tt|)*30L"
```
`E` stands for embedding, `t` for transformer layer, `L` for Loss. So it's interpreted as a total of 32 stages, where the first stage is Embedding + 1 transformer layer, the last stage is Loss, and the middle 30 stages are 2 transformer layers.
- Fine-grained recompute
```bash
--recompute-granularity selective \
--recompute-modules moe_act mlp \
```
- Partial CUDA Graphs
```bash
--cuda-graph-impl transformer_engine \
--cuda-graph-scope attn moe_router moe_preprocess \
--te-rng-tracker \
```
- Force load balancing for performance benchmark
```bash
--moe-router-force-load-balancing \
```
- HybridEP
```bash
--moe-token-dispatcher-type flex \
--moe-flex-dispatcher-backend hybridep \
--moe-hybridep-num-sms 32 \
```
- MXFP8 recipe
```bash
--fp8-recipe mxfp8 \
--fp8-format e4m3 \
--fp8-param-gather \
--reuse-grad-buf-for-mxfp8-param-ag \
```
- BF16 optimizer states
```bash
--use-precision-aware-optimizer \
--main-grads-dtype fp32 \
--main-params-dtype fp32 \
--exp-avg-dtype bf16 \
--exp-avg-sq-dtype bf16 \
```
- Kernel fusions
```bash
--cross-entropy-loss-fusion \
--cross-entropy-fusion-impl te \
--moe-permute-fusion \
--moe-router-fusion \
```
- Manual GC to make ranks better synchronized
```bash
--manual-gc \
--manual-gc-interval 10 \
```
---
## File: Megatron-LM/docs/discussions/README.md
# Megatron Discussions
This directory contains in-depth guides, tutorials, and discussions about optimizing and using Megatron for various use cases.
## Available Guides
### Performance Optimization
- **[Optimizing DeepSeek-V3 Training Performance on NVIDIA GB200 NVL72](deepseek-v3-gb200-optimization/deepseek-v3-gb200-optimization.md)**
A comprehensive guide on optimizing DeepSeek-V3 model training on NVIDIA GB200 NVL72 systems, covering profiling techniques, performance bottlenecks, and optimization strategies.
- **[A Guide to Reproduce DeepSeek-V3 Pre-training Performance on GB200](deepseek-v3-gb200-optimization/deepseek-v3-gb200-reproduce-guide.md)**
A detailed guide on how to reproduce the DeepSeek-V3 pre-training performance on GB200, incluing the dockerfile, package requirements and training scripts.
## Contributing
If you'd like to contribute a guide or tutorial, please follow this structure:
1. Create a new directory: `docs/discussions/your-guide-name/`
2. Add your main guide: `docs/discussions/your-guide-name/your-guide-name.md`
3. Create an images directory: `docs/discussions/your-guide-name/images/`
4. Update this README.md with a link to your guide
Each guide should be self-contained with its own images and supporting files.
---
## File: Megatron-LM/docs/source/api-guide/context_parallel.rst
context\_parallel package
=========================
Context parallelism overview
----------------------------
.. figure:: ../images/context_parallel/CP_overview.png
:alt: cp_overview
:align: center
Figure 1: A transformer layer running with TP2CP2. Communications next to Attention are for CP, others are for TP. (AG/RS: all-gather in forward and reduce-scatter in backward, RS/AG: reduce-scatter in forward and all-gather in backward, /AG: no-op in forward and all-gather in backward).
Context Parallelism ("CP") is a parallelization scheme on the dimension of sequence length. Unlike prior SP (sequence parallelism) which only splits the sequence of Dropout and LayerNorm activations, CP partitions the network inputs and all activations along sequence dimension. With CP, all modules except attention (e.g., Linear, LayerNorm, etc.) can work as usual without any changes, because they do not have inter-token operations. As for attention, the Q (query) of each token needs to compute with the KV (key and value) of all tokens in the same sequence. Hence, CP requires additional all-gather across GPUs to collect the full sequence of KV. Correspondingly, reduce-scatter should be applied to the activation gradients of KV in backward propagation. To reduce activation memory footprint, each GPU only stores the KV of a sequence chunk in forward and gathers KV again in backward. KV communication happens between a GPU and its counterparts in other TP groups. The all-gather and reduce-scatter are transformed to point-to-point communications in ring topology under the hood. Exchanging KV also can leverage MQA/GQA to reduce communication volumes, as they only have one or few attention heads for KV.
For example, in Figure 1, assuming sequence length is 8K, each GPU processes 4K tokens. GPU0 and GPU2 compose a CP group, they exchange KV with each other. Same thing also happens between GPU1 and GPU3. CP is similar to `Ring Attention `_ but provides better performance by (1) leveraging the latest OSS and cuDNN flash attention kernels; (2) removing unnecessary computation resulted from low-triangle causal masking and achieving optimal load balance among GPUs.
Context parallelism benefits
----------------------------
.. figure:: ../images/context_parallel/CP_results.png
:alt: cp_results
:align: center
Figure 2: Speedup of 175B GPT with various TP+CP combinations vs. full recompute (i.e., TP8CP1).
LLM encounters OOM (out of memory) issue with long context (i.e., long sequence length) because of linearly increasing memory footprint of activations. Recomputing activations in backward can avoid OOM but also introduce significant overheads (~30% with full recompute). Enlarging TP (tensor model parallelism) can fix the OOM issue as well, but it potentially makes compute (e.g., Linear) too short to overlap communication latencies. To be clear, scaling out to more GPUs with bigger TP can hit the overlapping problem no matter if OOM happens.
CP can better address the issues. With CP, each GPU only computes on a part of the sequence, which reduces both computation and communication by CP times. Therefore, there are no concerns about the overlapping between them. The activation memory footprint per GPU is also CP times smaller, hence no OOM issue anymore. As Figure 2 shows, the combinations of TP and CP can achieve optimal performance by eliminating recompute overheads and making the best tradeoff between computation and communications.
Enabling context parallelism
----------------------------
CP support has been added to GPT. All models that share GPT code path also should be able to benefit from CP, such as Llama. CP can work with TP (tensor model parallelism), PP (pipeline model parallelism), and DP (data parallelism), where the total number of GPUs equals TPxCPxPPxDP. CP also can work with different attention variants, including MHA/MQA/GQA, uni-directional and bi-directional masking.
CP is enabled by simply setting context_parallel_size= in command line. Default context_parallel_size is 1, which means CP is disabled. Running with CP requires Megatron-Core (>=0.5.0) and Transformer Engine (>=1.1).