OpenClaw-RL Documentation

File: README.md Copy Chapter

📰 News Copy Chapter

• [2026/4/15] 🙌 We sincerely thank Fireworks AI for its generous support of this project, which has enabled more experiments and faster iteration.
• [2026/4/11] ✨ Qwen3.5-4B/9B/27B is supported now, both text and multi-modal!
• [2026/4/4] 👨‍👦‍👦 We support optimizing a single model based on feedback from a group of people.
• [2026/3/25] 🙌 We sincerely thank Tinker for its generous support of this project, which has enabled more experiments and faster iteration.
• [2026/3/20] 💻 You can use your own openclaw now, simply install this extension.
• [2026/3/13] ☁️ OpenClaw-RL now supports both local GPU and cloud (Tinker) deployment. Launch with one line of code — Hybrid RL, OPD, and Binary RL all supported!
• [2026/3/12] ⚡ We support LoRA training now!
• [2026/3/10] 📃 We have released our Technical Report! 🏆 Ranked #1 on HuggingFace Daily Papers!
• [2026/3/10] 🔥 Huge updates today! We released a new combination method, along with an interesting evaluation of these OpenClaw-RL methods. Track 2 is released too, featuring scalable RL implementations for general agent settings across terminal, GUI, SWE, and tool-call scenarios. We only focus on real-world settings!
• [2026/3/3] 🙌 Working with the authors of SDFT and SDPO, we have integrated their methods into openclaw-opd. We welcome the integration of novel and effective methods!
• [2026/3/3] 📺 Check out these community tutorial videos on OpenClaw-RL: Video 1 | Video 2
• [2026/2/26] 🔥 We release OpenClaw-RL v1 — a fully asynchronous RL framework for training personalized AI agents from natural conversation feedback.

💡 TL;DR Copy Chapter

OpenClaw-RL is a fully asynchronous reinforcement learning framework that turns everyday conversations into training signals for personalized AI agents, and supports training general agents with large-scale environment parallelization.

Most RL-for-LLM systems assume centralized, batch-mode training with pre-collected datasets. OpenClaw-RL takes a fundamentally different approach: it wraps your self-hosted model in OpenClaw as an OpenAI-compatible API, intercepts live multi-turn conversations, and continuously optimizes the policy in the background — all without interrupting your usage.

Highlights: Fully async 4-component loop · Self-hosted & private · Zero manual labeling · Three learning paradigms (Binary RL / OPD / Combine) · Personal + General agent support

🌈 Features

Fully Asynchronous 4-Component Architecture Copy Section

OpenClaw-RL decouples agent serving, rollout collection, PRM/judge evaluation, and policy training into independent async loops. None of them block one another: the model continues serving requests while training runs in the background, and judging happens concurrently with new interactions.

Self-Hosted & Private by Design Copy Section

The entire stack, including the policy model, judge/PRM, and trainer, runs on your own infrastructure. Conversation data stays within your system, and no third-party model API is required.

From Feedback to Gradient — Automatically Copy Section

You do not need to manually label data. The system automatically:

• Organizes multi-turn interactions into session-aware training trajectories
• Classifies API messages into main-line (trainable) vs. side (non-trainable) turns
• Uses the next user, environment, or tool feedback as a natural "next-state" signal
• Runs PRM/judge evaluation asynchronously, with majority voting when needed for more robust scoring
• Submits ready samples to the trainer as they become available

Three Optimization Methods in One Framework Copy Section

Binary RL (GRPO): A Process Reward Model scores each turn based on next-state feedback. The scalar reward is then used with GRPO advantage estimation and a PPO-style clipped surrogate loss.

On-Policy Distillation (OPD): When the next state reveals useful hindsight, a judge model extracts a textual hint. This hint augments the original prompt to create an enhanced teacher, whose token-level log-probability gap with the student becomes a directional advantage signal richer than any scalar reward.

Hybrid Method: OpenClaw-RL further combines Binary RL and OPD in a unified training recipe, leveraging the dense scalar supervision of Binary RL together with the richer token-level directional signal from OPD. This combination achieves stronger and more robust optimization than either method alone.

From Personal Agents to Real-World Agentic RL Copy Section

The same framework supports both personalized OpenClaw optimization and scalable RL for terminal, GUI, SWE, and tool-call agents in real-world settings.

🎯 Roadmap Copy Chapter

Our long-term goal is to advance personalized, practically useful agents with reinforcement learning. The roadmap has two tracks:

Track 1 — [Personal Agent Optimization](#personalagent) (Small-Scale but Personal)

✅ Release Track 1: Fully async OpenClaw-RL framework with Binary RL + OPD
✅ Best recipe discovery via demonstration experiments
✅ Support LoRA Training
✅ Deploy training on Tinker
✅ Deploy training on Fireworks AI

Track 2 — [General Agents Optimization](#generalagent) (Scalable Infra)

✅ Release Track 2: Scalable agentic RL infra for general agents
✅ Support Qwen3.5
⬜ Support more cloud services

📝 Contents Copy Chapter

• Personal OpenClaw Optimization
  • Bybrid RL
  • Method Evaluation
• Agentic RL in Real World Settings
  • Terminal Agent
  • GUI Agent
  • SWE Agent
  • Tool-call Agent

🔧 Personal Agent Optimization Quick Start Copy Chapter

1. Deployment Requirements Copy Section

• Hardware: 8× GPUs (default; configurable via NUM_GPUS, ACTOR_GPUS, ROLLOUT_GPUS, PRM_GPUS)
• Software: CUDA 12.9, Python 3.12
• Framework: Slime (our base RL framework)

For detailed environment setup, see Slime or ./instructions/README.md.

2. Start the RL Server Copy Section

cd slime
bash ../openclaw-combine/run_qwen3_4b_openclaw_topk_select.sh

This method combines binary RL and OPD to achieve the best optimization.

See ./openclaw-combine/README.md for algorithm details.

Once running, the model is served as an OpenAI-compatible API at:

http://<HOST_IP>:30000/v1

where <HOST_IP> is the IP address of the machine running the RL server (e.g. 115.190.98.251). The port 30000 is the default and can be changed via the PORT environment variable.

Take note of this endpoint — you will need it when configuring OpenClaw in the next step.

We also provide an interesting case for evaluation. A student who uses OpenClaw to do homework, does not want to be found using AI. A teacher who also uses OpenClaw to grade student's homework, wants the comments to be specific and friendly.

Evaluation Setting — Both student, TA and teacher use AI!

See ./openclaw-test/README.md for setup and algorithm details. Example of evaluation results.

3. OpenClaw Setup Copy Section

You can use your own openclaw, just install this extension.

**Then configure OpenClaw to route requests to your RL server. **

Open your openclaw.json (or the equivalent settings file) and add a provider entry under "models" → "providers":

Example of Slime-based RL server:

{
  "models": {
    "providers": {
      "qwen": {
        "baseUrl": "http://<HOST_IP>:30000/v1",
        "apiKey": "apiKey",
        "api": "openai-completions",
        "models": [
          {
            "id": "qwen3-4b",
            "name": "Qwen3 4B",
            "reasoning": true,
            "input": ["text"],
            "cost": {
              "input": 0,
              "output": 0,
              "cacheRead": 0,
              "cacheWrite": 0
            },
            "contextWindow": 32768,
            "maxTokens": 8192
          }
        ]
      }
    }
  }
}

Replace <HOST_IP> with the IP address of your RL server machine. The apiKey should match the SGLANG_API_KEY you set when starting the server.

Example of Tinker-based RL server:

{
  "models": {
    "providers": {
      "openclaw-rl": {
        "baseUrl": "http://localhost:30000/v1",
        "apiKey": "no-auth-needed",
        "api": "openai-completions",
        "models": [
          {
            "id": "qwen3-4b-lora",
            "name": "Qwen3 4B (OpenClaw-RL LoRA)",
            "reasoning": true,
            "input": ["text"],
            "cost": {
              "input": 0,
              "output": 0,
              "cacheRead": 0,
              "cacheWrite": 0
            },
            "contextWindow": 32768,
            "maxTokens": 8192
          }
        ]
      }
    }
  }
}

That's it — start chatting with your OpenClaw agent. The RL server will automatically collect conversation trajectories, compute rewards, and train the model. Your agent gets better the more you use it.

🔧 Agentic RL in Real-world Settings Copy Chapter

The same asynchronous RL backbone that powers our personal-agent setting can also support large-scale optimization for these broader real-world environments.

🖥️ Terminal Agent — the most widely used computer-use agent Copy Section

cd slime
bash ../terminal-rl/terminal_qwen3_8b_rl.sh

See ./terminal-rl/README.md for setup details.

📟 GUI Agent — the most general computer-use agent Copy Section

cd slime
bash ../gui-rl/gui_qwen3vl_8b_rl.sh

See ./gui-rl/README.md for setup details.

👨‍💻 SWE Agent — software engineering agent Copy Section

cd slime
bash ../swe-rl/run_swe_rl_32b_remote_8nodes.sh

See ./swe-rl/README.md for setup details.

🛠️ Tool-call Agent — the most practical agent Copy Section

cd slime
bash ../toolcall-rl/retool_qwen3_4b_rl.sh

See ./toolcall-rl/README.md for setup details.

📖 Citation Copy Chapter

@article{wang2026openclawrl,
  title={OpenClaw-RL: Train Any Agent Simply by Talking},
  author={Wang, Yinjie and Chen, Xuyang and Jin, Xiaolong and Wang, Mengdi and Yang, Ling},
  journal={arXiv preprint arXiv:2603.10165},
  year={2026}
}

@article{wang2026rlanything,
  title={RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL System},
  author={Wang, Yinjie and Xie, Tianbao and Shen, Ke and Wang, Mengdi and Yang, Ling},
  journal={arXiv preprint arXiv:2602.02488},
  year={2026}
}

🙏 Acknowledgements Copy Chapter

This work aims to explore more effective paradigms for Agentic RL. Our implementation builds upon the excellent codebases of slime, OpenClaw, Tinker and Open-AgentRL.

We also build terminal RL using SETA's dataset and agent framework, GUI RL using OSWorld's evaluation scripts, SWE RL using mini-swe-agent's evaluation scripts, and tool-call RL based on the work of Retool.

We sincerely thank these projects for their valuable insights and high-quality implementations, which have greatly facilitated our research.

⚠️ Reminder Copy Chapter

When using OpenClaw-RL, please do not provide sensitive personal information during conversations with the model. Also, make sure to keep your API keys secure and never expose them in prompts, logs, or shared files.

File: extensions/rl-training-headers/README.md Copy Chapter

RL Training Headers

OpenClaw plugin that injects X-Session-Id and X-Turn-Type HTTP headers into every outgoing LLM API request, enabling downstream RL training pipelines to classify and segment training data.

Headers Copy Chapter

Install Copy Chapter

1. Copy (or symlink) this folder into extensions/rl-training-headers inside your OpenClaw repo.

2. Enable the plugin:

corepack pnpm start -- plugins enable rl-training-headers

3. Restart the gateway:

corepack pnpm start -- gateway restart

That's it. Every LLM API request now carries the two extra headers.

Configuration (optional) Copy Chapter

You can customize the header names in ~/.openclaw/openclaw.json under plugins.entries:

{
  "plugins": {
    "entries": {
      "rl-training-headers": {
        "enabled": true,
        "config": {
          "sessionIdHeader": "X-Session-Id",
          "turnTypeHeader": "X-Turn-Type"
        }
      }
    }
  }
}

How it works Copy Chapter

The plugin hooks into the before_prompt_build lifecycle event to capture the current session ID and turn type (derived from the trigger field: "user" → main, "heartbeat" / "memory" / "cron" → side).

It then patches globalThis.fetch to inject these headers into outgoing POST requests during an active agent run. Header state is stored with async-local per-run context, so parallel sessions keep separate session IDs and turn types.

Extracting training data Copy Chapter

On the API proxy / logging side, use these headers to:

• Group requests by X-Session-Id to reconstruct full conversation sessions.
• Filter by X-Turn-Type: main to keep only user-facing dialogue for reward modelling, or include side turns for full trajectory analysis.

File: gui-rl/desktop_env/evaluators/README.md Copy Chapter

Evaluator Setup Details

Setup scaffolding for the evaluators in the desktop environment for those who want to know the details of the evaluator setup for customized evaluation and extension

Overall Copy Chapter

Inside the virtual machine, disable the system crash report by:

sudo vim /etc/default/apport

and then change the enabled to 0.

VSCode Copy Chapter

todo

LibreOffice Copy Chapter

For LibreOffice, please enter into the app first, and then enable the no pop-up when 'ctrl + s'.

LibreOffice Press Copy Chapter

Setting Up the python-pptx Library Copy Section

pip install python-pptx

LibreOffice Writer Copy Chapter

Setting Up the python-docx and odfpy Library Copy Section

pip install python-docx
pip install odfpy

LibreOffice Calc Copy Chapter

Required Libraries Copy Section

openpyxl
pandas
lxml
xmltodict

How to Generate CSV from XLSX Copy Section

libreoffice --convert-to "csv:Text - txt - csv (StarCalc):44,34,UTF8,,,,false,true,true,false,false,1" --out-dir /home/user /home/user/abc.xlsx

This command will generate abc-Sheet1.csv under /home/user. The last 1 in
the conversion options indicates the sheet number (starting from 1) to export.
Detailed usage should be referred to at CSV Filter
Options.

Refer to libreoffice_calc/21df9241-f8d7-4509-b7f1-37e501a823f7.json for an
example.

About `compare_table` Copy Section

Evaluation to xlsx files mainly relies on compare_table. It accepts two file
names and a list of rules defined as options. Refer to
libreoffice_calc/21df9241-f8d7-4509-b7f1-37e501a823f7.json for an example.

In each rule, there is a required field type. The supported types are defined
in compare_table function. The most common two are sheet_data and
sheet_print. sheet_data compares the internal cell values through pandoc,
while sheet_print compares the shown cell values through csv. A csv should be
generated and downloaded for sheet_print. See the previous section and
example in libreoffice_calc/21df9241-f8d7-4509-b7f1-37e501a823f7.json.

Other fields in a rule are described for each evaluation type in
compare_table function. sheet_idx0 (or sheet_idx1, sheet_idx) is a
common field to indicate which sheet is to extracted from the workbook. If an
integer i is given, then it extracts the i-th sheet from result xlsx (i starts
from 0). If a string is given, it should be preceded with "RI", "RN", "EI", or
"EN". "R" indicates to extract from result xlsx while "E" indicates to extract
from expected (golden) xlsx. "I" indicates a sheet number (starting from 0) and
"N" indicates a sheet name (usually, they're like "Sheet1", "Sheet2", ...).

Some rules use a atructure like {"method": "eq", "ref": "abc"}. These rules
are checked through utils._match_value_to_rule function. Check it for the
implemented matching methods.

Chrome Copy Chapter

Starting Chrome with Remote Debugging for Python Copy Section

To enable remote debugging in Chrome, which allows tools like Playwright for Python to connect to and control an existing Chrome instance, follow these steps:

Manually Enabling Remote Debugging in Chrome

1. Locate the Chrome Shortcut:

   • Find the Chrome shortcut that you usually use to open the browser. This could be on your desktop, start menu, or taskbar.

2. Edit Shortcut Properties:

   • Right-click on the Chrome shortcut and select Properties.

3. Modify the Target Field:

   • In the Target field, add --remote-debugging-port=9222 at the end of the path. Ensure there is a space between the path and the flag you add.
   • It should look something like this: "C:\Path\To\Chrome.exe" --remote-debugging-port=9222.

4. Apply and Close:

   • Click Apply and then OK to close the dialog.

5. Start Chrome:

   • Use this modified shortcut to start Chrome. Chrome will now start with remote debugging enabled on port 9222.

6. Confirm Remote Debugging:

   • Open a browser and navigate to http://localhost:9222. If you see a webpage with information about active tabs, remote debugging is working.

Setting Up Playwright for Python Copy Section

Playwright for Python is a browser automation library to control Chromium, Firefox, and WebKit with a single API.

Installing Playwright

• Ensure you have Python installed on your system. If not, download and install it from the Python official website.

• Install Playwright using pip (Python Package Installer). Open a command line or terminal and run:

  bash Copy

  pip install playwright

• After installing Playwright, you need to run the install command to download the necessary browser binaries:

  bash Copy

  playwright install

Writing a Playwright Script in Python

• Create a Python file for your automation script.

• Import the Playwright module at the beginning of your script:

  python Copy

  from playwright.sync_api import sync_playwright

• You can now use Playwright's API to control browsers.

Example Playwright Script

Here is a simple example to open a page using Playwright:

from playwright.sync_api import sync_playwright

def run(playwright):
    browser = playwright.chromium.launch()
    page = browser.new_page()
    page.goto("http://example.com")
    ## other actions...
    browser.close()

with sync_playwright() as playwright:
    run(playwright)

• This script launches Chromium, opens a new page, navigates to example.com, and then closes the browser.

Troubleshooting

• If you encounter issues with Playwright, ensure that your Python environment is correctly set up and that you have installed Playwright and its dependencies correctly.
• For detailed documentation, visit the Playwright for Python Documentation.

VLC Media Player Copy Chapter

Bugs fix Copy Section

One thing on Ubuntu need to do, enter into the meida>convert/save>select files>convert/save
Then enter the profile of Audio - MP3, change the profile for mp3, section audiocodec from "MP3" to "MPEG Audio"
Otherwise the mp3 file will be created but with 0 bytes. It's a bug of VLC.

Setting Up VLC's HTTP Interface Copy Section

To enable and use the HTTP interface in VLC Media Player for remote control and status checks, follow these steps:

1. Open VLC Preferences

• Open VLC Media Player.
• Go to Tools > Preferences from the menu.

2. Show All Settings

• In the Preferences window, at the bottom left corner, select All under Show settings to display advanced settings.

3. Enable Main Interfaces

• In the advanced preferences, expand the Interface section.
• Click on Main interfaces.
• Check the box for Web to enable the HTTP interface.

4. Configure Lua HTTP

• Expand the Main interfaces node and select Lua.
• Under Lua HTTP, set a password password in the Lua HTTP section. This password will be required to access the HTTP interface.

5. Save and Restart VLC

• Click Save to apply the changes.
• Restart VLC Media Player for the changes to take effect.

6. Accessing the HTTP Interface

• Open a web browser and go to http://localhost:8080.
• You will be prompted for a password. Enter the password you set in the Lua HTTP settings.
• Once logged in, you will have access to VLC's HTTP interface for remote control.

Packages

pip install opencv-python-headless Pillow imagehash

Troubleshooting

• If you cannot access the HTTP interface, check if your firewall or security software is blocking the connection.
• Ensure VLC is running and the correct port (default is 8080) is being used.
• If the port is in use by another application, you may change the port number in VLC's settings.

GIMP Copy Chapter

Click on the "Keep" of the image loading pop-up.

File: gui-rl/desktop_env/server/README.md Copy Chapter

Server setup

This README is useful if you want to set up your own machine for the environment. This README is not yet finished. Please contact the author if you need any assistance.

Configuration Overview Copy Chapter

The following sections contain guidelines for configuring the system image to ensure benchmark examples can run properly.

The main configuration requirements include:

1. Account Credentials:
   Our benchmark configurations are based on specific username and password settings (with username user and password password).
   Please ensure these settings remain consistent or update the corresponding configuration files.

2. Service Setup:
   Our environment operates through a service that automatically starts at boot time, as shown in the figure below. The service needs to be properly configured and placed.

3. Accessibility Tree Support:
   Benchmark examples rely on accessibility tree functionality. The necessary support packages need to be installed.

4. System Service Management:
   Certain system services that may cause interference need to be disabled, such as automatic updates and notification pop-ups.

5. Required Software Installation:
   Ensure all necessary software packages required by the benchmark examples are properly installed.

6. Software Configuration:
   Various software packages require specific configurations, such as disabling certain auto-save features or installing additional plugins.

7. Port Configuration:
   To monitor and control software states from the host machine, specific port configurations are needed for various applications.

8. Miscellaneous Settings:
   Additional system-specific settings need to be configured, such as desktop environment settings and display resolution.

Detailed instructions for each of these requirements will be provided in the following sections.

[Ubuntu](https://huggingface.co/datasets/xlangai/ubuntu_osworld) Copy Chapter

Make a new VM with the Ubuntu 20.04 LTS image.

How to install Ubuntu Desktop (package: ubuntu-desktop) with GNOME desktop environment on Ubuntu 22.04 system. Copy Section

sudo apt update
sudo apt install ubuntu-desktop
sudo systemctl set-default graphical.target

Account Credentials Copy Section

Download the iso file from the Ubuntu website and install it in the VM.

Using GUI:
The default username should be user and the password should be password when you are asked to set up the account. Give the user sudo permission.

Using Command Line:

sudo adduser user
usermod -aG sudo user

Installation and Auto-login Setup Copy Section

1. Download the iso file from the Ubuntu website and install it in the VM.
   The default username should be user and the password should be password when you are asked to set up the account.

2. To enable automatic login:

Using GUI:

# Open Settings -> Users
# Click Unlock button and enter password
# Toggle "Automatic Login" to ON for user 'user'

Or using Command Line:

# Edit the custom configuration file
sudo nano /etc/gdm3/custom.conf

# Under [daemon] section, add or modify these lines:
AutomaticLoginEnable=true
AutomaticLogin=user

# Save the file and restart the system
sudo systemctl restart gdm3

After setting up automatic login, the system will boot directly into the desktop environment without requiring password input, which enables seamless startup experience for automated testing environments.

VNC Configuration Copy Section

1. Install x11vnc

sudo apt update
sudo apt install x11vnc

2. Install noVNC

sudo snap install novnc

3. Create system services for x11vnc and novnc:

• Go to directory cd /etc/systemd/user/
• Write a file novnc.service with the following content:

[Unit]
Description=noVNC Service
After=x11vnc.service network.target snap.novnc.daemon.service
Wants=x11vnc.service

[Service]
Type=simple
ExecStart=/snap/bin/novnc --vnc localhost:5900 --listen 5910
Restart=on-failure
RestartSec=3
Environment=DISPLAY=:0
Environment=XAUTHORITY=/home/user/.Xauthority
Environment=SNAP_COOKIE=/run/snap.cookie
Environment=SNAP_NAME=novnc
Environment=SNAP_REVISION=current

[Install]
WantedBy=default.target

Write a file x11vnc.service with the following content:

[Unit]
Description=X11 VNC Server
After=display-manager.service network.target
Wants=display-manager.service

[Service]
Type=simple
ExecStart=x11vnc -display :0 -rfbport 5900 -forever
Restart=on-failure
RestartSec=3
Environment=DISPLAY=:0
Environment=XAUTHORITY=/home/user/.Xauthority

[Install]
WantedBy=default.target

4. Enable both services:

systemctl --user daemon-reload
systemctl --user enable novnc.service
systemctl --user enable x11vnc.service
systemctl --user start x11vnc.service
systemctl --user start novnc.service

5. Allow VNC port:
   Expose 5910 port in the firewall and any other security tools you are using.

6. Access the VNC server:
   Connect to the VNC via http://[Instance IP]:5910/vnc.html

Display Configuration Copy Section

⚠️ IMPORTANT NOTE: The display configuration is critical for proper system operation. Incorrect settings can prevent the graphical environment from starting and potentially crash the X server. Make sure to follow these steps carefully and verify each configuration file. If you encounter any issues, check the X server logs at /var/log/Xorg.0.log for troubleshooting. Backup your existing X11 configuration before making any changes.

1. Install dummy video driver:

sudo apt-get install xserver-xorg-video-dummy

Go to /etc/X11/ and create a file named xorg.conf with the following content:

/* Detailed source-code truncated for AI context efficiency. */

2. In the same directory as the previous step, go to its sub-directory xorg.conf.d , create a file named 10-dummy.conf with the following content:

Section "Device"
    Identifier "DummyDevice"
    Driver "dummy"
    VideoRam 32768
EndSection

Section "Monitor"
    Identifier "DummyMonitor"
    HorizSync 28.0-80.0
    VertRefresh 48.0-75.0
    Modeline "1920x1080" 172.80 1920 2048 2248 2576 1080 1083 1088 1120
EndSection

Section "Screen"
    Identifier "DummyScreen"
    Device "DummyDevice"
    Monitor "DummyMonitor"
    DefaultDepth 24
    SubSection "Display"
        Depth 24
        Modes "1920x1080"
    EndSubSection
EndSection

3. Reload the display manager:

sudo systemctl restart display-manager

Set up the OSWorld server service in VM Copy Section

Upload the OSWorld server to the home directory (/home/user) of user (via scp or git clone).

1. Copy the main.py and pyxcursor.py and to the /home/user-name where the user-name is your username of the ubuntu, here we make it user as default. If you customize the path of placing these files in this step, you should change the parameters in the service file we will mention later accordingly.

2. First please set up the environment:

sudo apt install python3
pip3 install -r requirements.txt
sudo apt-get install python3-tk python3-dev
sudo apt install gnome-screenshot
sudo apt install wmctrl
sudo apt install ffmpeg
sudo apt install socat
sudo apt install xclip

If you encounter an error about python not being found, run:

sudo ln -s /usr/bin/python3 /usr/bin/python

if you customize the environment in this step, you should change the parameters in the service file we will mention later accordingly.

3. Due to some configuration issues, you need to modify the osworld_server.service file:

1. In our released version, the X server is set to :1, but the default X server is actually :0. You need to modify the osworld_server.service file to change the DISPLAY variable from :1 to :0.
   Change the following line:

Environment="DISPLAY=:1"

to

Environment="DISPLAY=:0"

2. Need to add environment variables to enable DBUS to change wallpaper.
   Change the following line:

Environment="DISPLAY=:0"

to

Environment="DISPLAY=:0;DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus"

4. Copy the osworld_server.service to the systemd configuration directory at /etc/systemd/system/:

sudo cp osworld_server.service /etc/systemd/system/

Reload the systemd daemon to recognize the new service:

sudo systemctl daemon-reload

Enable the service to start on boot:

sudo systemctl enable osworld_server.service

Start the service:

sudo systemctl start osworld_server.service

Verify the service is running correctly:

sudo systemctl status osworld_server.service

You should see output indicating the service is active and running. If there are errors, review the logs with journalctl -xe for further troubleshooting.

If you need to make adjustments to the service configuration, you can edit the /etc/systemd/system/osworld_server.service file:

sudo nano /etc/systemd/system/osworld_server.service

After making changes, reload the daemon and restart the service:

sudo systemctl daemon-reload
sudo systemctl enable osworld_server.service
sudo systemctl start osworld_server.service

Accessibility Tree Support Copy Section

To support the accessibility tree functionality, you'll need to install pyastpi2 in your Ubuntu environment. This package enables access to accessibility information and tree structures.

Installation steps:

# Update package list and ensure pip is installed
sudo apt-get update
sudo apt-get install python3-pip

# Install pyastpi2 using pip
pip3 install pyastpi2

Xorg Configuration Copy Section

Regarding the graphics display system, we need to ensure that Ubuntu displays images using the Xorg protocol instead of Wayland. Since Wayland is typically the default setting for Ubuntu, we will need to manually change the settings.

1. Click the user menu in the upper right corner and select "Log Out" or "Sign Off."
2. On the login screen, click on the username.
3. Before entering the password, click the gear icon in the lower right or upper right corner of the screen (it may need to be displayed after clicking the username first).
4. Select "Ubuntu on Xorg" from the pop-up menu.

You can run the following command to check if Xorg is being used:

echo $XDG_SESSION_TYPE

System Service Management (Optional) Copy Section

The automatic software update service can interfere with benchmark examples. To disable this service, you can refer to the https://www.makeuseof.com/disable-automatic-updates-in-ubuntu/ for the solution.

You can check and manage system services using systemctl commands. For example, to verify if a service like unattended-upgrades is installed and running on your system:

# Check service status
sudo systemctl status unattended-upgrades.service

If the output is x11, it means you have switched to Xorg.

To disable a system service:

# Disable and stop the service
sudo systemctl disable unattended-upgrades
sudo systemctl stop unattended-upgrades

To verify service configurations, you can use apt-config:

# Check current configurations
apt-config dump APT::Periodic::Update-Package-Lists
apt-config dump APT::Periodic::Unattended-Upgrade

Software Installation Copy Section

Software Installation Source

Since for some examples like change the settings of certain software, we hardcode some paths in our evaluation file, which means you need to install the software to the specific path. Here we provide a list of software that you need to install and the certain source which default the path you should install them to.

1. Chrome: If you are using ARM System, download the chromium using sudo snap install chromium and make sure your Chromium config files are under ~/snap/chromium; otherwise, download the chrome from the Chromium and make sure your Chromium config files are under ~/.config/google-chrome.
2. LibreOffice: Go to LibreOffice Website, select "Download Libreoffice", select "older versions" in the bottom of the page, and download 7.3.7.2 version.
3. GIMP: Search "GIMP" in "Ubuntu Software" and install it. Our GIMP version is 2.10.30.
4. VLC: Search "VLC" in "Ubuntu Software" and install it. Our VLC version is 3.0.16.
5. VSCode: Go to VSCode Website, download the .deb file, and install it. Our VSCode version is 1.91.1.

Additional Inner Software Installation

⚠️ IMPORTANT NOTE: The software installation and configuration steps described in this section are crucial for maintaining consistent task execution and performance. Skipping or incorrectly configuring these components may lead to task failures or degraded performance. Please follow the installation instructions carefully and verify each component is properly set up before proceeding.

LibreOffice font installation

Some examples in LibreOffice Impress use non-default system fonts, and you need to download the corresponding TTF files and put them in the system fonts directory.
Here we provides all the fonts downloaded, just download it, and unzip to the system fonts directory (which usually usr/share/fonts/).

unzip fonts.zip -d /usr/share/fonts/

And then run the following command to refresh the font cache:

sudo fc-cache -fv

Customized Plugin Installation

VS Code plugin installation:
To extract relevant internal information and configurations from the VS Code environment, we principally leverage the capabilities offered by the VS Code Extension API. Here's how to install the extension developed by ourselves:

1. Download the extension from: https://github.com/xlang-ai/OSWorld/blob/04a9df627c7033fab991806200877a655e895bfd/vscodeEvalExtension/eval-0.0.1.vsix
2. Open VS Code
3. Go to Extensions -> ... -> Install from VSIX... -> choose the downloaded eval-0.0.1.vsix file

Software Configuration Copy Section

1. LibreOffice Default Format Settings:

# Open LibreOffice Writer/Calc/Impress
# Go to Tools -> Options -> Load/Save -> General
# Under "Default file format and ODF settings":
# Change "Document type" to "Text document"
# Set "Always save as" to "Word 2007-365 (.docx)"
# Change "Document type" to "Spreadsheet"
# Set "Always save as" to "Excel 2007-365 (.xlsx)"
# Change "Document type" to "Presentation"
# Set "Always save as" to "PowerPoint 2007-365 (.pptx)"

2. Chrome password requirement removal:
   Chrome requests a password input when first opened after system startup, which can interfere with our experiments. Here's how to disable this feature:

# Prevent Chrome from using keyring
mkdir -p ~/.local/share/keyrings
touch ~/.local/share/keyrings/login.keyring

Or you can use any ways to disable the keyring service, which will prevent Chrome from requesting a password input.

3. VSCode Trust Settings:
   To prevent VSCode from showing "Do you trust this workspace?" dialog when opening projects, configure the following setting:

# Open VSCode
# Go to File -> Preferences -> Settings (or press Ctrl+,)
# In the search bar, type "workspace trust"
# Find "Security: Workspace Trust Enabled" and uncheck it
# Find "Security: Workspace Trust Banner" and set to "never"
# Find "Security: Workspace Trust Empty Window" and uncheck it
# Find "Security: Workspace Trust Startup Prompt" and set to "never"

This disables the workspace trust feature and prevents trust dialog prompts when opening projects.

Network Configuration Copy Section

Firewall Configuration

In OSWorld, we need the following ports to be open:

server_port = 5000
chromium_port = 9222
vnc_port = 8006
vlc_port = 8080
novnc_port = 5910

Please open the corresponding ports in the firewall and any other security tools you are using.

socat Installation

Ensure socat is installed to enable port forwarding.

sudo apt install socat

Network Configuration for Remote Control

VLC Configuration

To enable remote control of VLC media player, follow these configuration steps:

1. Enable HTTP interface:

# Open VLC
# Go to Tools -> Preferences
# Show Settings: All (bottom left)
# Navigate to Interface -> Main interfaces
# Check 'Web' option

2. Configure HTTP interface settings:

# Still in Preferences
# Navigate to Interface -> Main interfaces -> Lua
# Under Lua HTTP:
# - Set Password to 'password'

The following is the screenshot of the VLC configuration:

When VLC is open, the service will be running on port 8080.

Chrome Configuration

When you open Chrome through the GUI interface, it doesn't automatically enable remote debugging port 1337 (the port we set for controlling the client machine's Chrome through the host, which gets forwarded to 9222).
This is because GUI startup and command-line startup use different parameters.
The consequence is that once Chrome is closed and reopened, it will fail to connect.
To ensure Chrome uses consistent debugging ports even after being closed and reopened, follow these steps:

1. Create or edit Chrome desktop entry:

sudo vim ~/.local/share/applications/google-chrome.desktop

If that doesn't work, try:

sudo vim /usr/share/applications/google-chrome.desktop

2. Modify all the Exec lines to include debugging port, for example, change:

Exec=/usr/bin/google-chrome-stable %U

into:

Exec=/usr/bin/google-chrome-stable --remote-debugging-port=1337 --remote-debugging-address=0.0.0.0 %U

In cases where Chrome is needed, the 1337 will be forwarded to 9222 in the virtual machine via socat.

Miscellaneous Settings Copy Section

Screen Resolution

The required screen resolution for the virtual machine is 1920x1080 in OSWorld and we did make some hardcode related to this resolution in our configuration file in some examples, but only a few.
So please set the screen resolution to 1920x1080 in the virtual machine settings.

Automatic Suspend

To close automatic suspend, open Setting app and enter "Power" section. Switch "Screen Blank" to "Never" and "Automatic Suspend" to "Off".

Additional Installation

Activating the window manager control requires the installation of wmctrl:

sudo apt install wmctrl

Otherwise, you cannot control the window manager in the virtual machine when running the experiments. Some cases will be effected.

To enable recording in the virtual machine, you need to install ffmpeg:

sudo apt install ffmpeg

Otherwise you cannot get the video recording of the virtual machine when running the experiments.

Others Information Copy Section

About the Converted Accessibility Tree

For several applications like Firefox or Thunderbird, you should first enable

gsettings set org.gnome.desktop.interface toolkit-accessibility true

to see their accessibility tree.

Example of AT

An example of a node:

<section xmlns:attr="uri:deskat:attributes.at-spi.gnome.org" attr:class="subject" st:enabled="true" cp:screencoord="(1525, 169)", cp:windowcoord="(342, 162)", cp:size="(327, 21)">
    歡迎使用新的 Outlook.com 帳戶
</section>

An example of a tree:

<desktop-frame ...>
    <application name="Thunderbird" ...>
        ... 
    </application>
    ...
</desktop-frame>

Useful attributes

1. name - shows the name of application, title of window, or name of some
   component
2. attr:class - somewhat the same role as class in HTML
3. attr:id - somewhat the same role as id in HTML
4. cp:screencoord - absolute coordinator on the screen
5. cp:windowcoord - relative coordinator in the window
6. cp:size - the size

Also several states like st:enabled and st:visible can be indicated. A full
state list is available at
https://gitlab.gnome.org/GNOME/pyatspi2/-/blob/master/pyatspi/state.py?ref_type=heads.

How to use it in evaluation

See example thunderbird/12086550-11c0-466b-b367-1d9e75b3910e.json and
function check_accessibility_tree in metrics/general.py. You can use CSS
selector or XPath to reference a target nodes. You can also check its text
contents.

An example of a CSS selector:

application[name=Thunderbird] page-tab-list[attr|id="tabmail-tabs"]>page-tab[name="About Profiles"]

This selector will select the page tab of profile manager in Thunderbird (if open).

For usage of CSS selector: https://www.w3.org/TR/selectors-3/. For usage of XPath: https://www.w3.org/TR/xpath-31/.

Manual check

You can use accerciser to check the accessibility tree on GNOME VM.

sudo apt install accerciser

[MacOS](https://huggingface.co/datasets/xlangai/macos_osworld) Copy Chapter

Coming soon...

File: gui-rl/README.md Copy Chapter

GUI-RL: Reinforcement Learning for Desktop GUI Agents

This directory trains and evaluates vision-language models (Qwen3-VL) to operate real desktop environments through screenshots and pyautogui actions. The agent observes the screen, reasons about the task, and emits keyboard/mouse actions — all inside cloud VMs managed by a pool server.

Architecture Copy Chapter

┌────────────────────┐        ┌────────────────────────────────────┐
│  Env Pool Server   │        │         Ray Cluster                │
│  (Flask, port      │  HTTP  │  ┌──────────┐  ┌───────────────┐  │
│   18080)           │◄──────►│  │  Rollout  │  │   Actor /     │  │
│                    │        │  │  (SGLang) │  │   Trainer     │  │
│  allocate / reset  │        │  └──────────┘  └───────────────┘  │
│  step / evaluate   │        │       ▲               ▲            │
│  close             │        │       │  screenshots  │  weights   │
└────────┬───────────┘        └───────┼───────────────┼────────────┘
         │                            │               │
         │  VM lifecycle              │               │
         ▼                            │               │
┌────────────────────┐        ┌───────┴───────────────┴───┐
│  Cloud VMs         │        │   generate_with_gui.py    │
│  (Volcengine /     │        │   (trajectory loop:       │
│   AWS / Aliyun)    │        │    screenshot → agent →   │
│  Ubuntu desktops   │        │    action → step → reward)│
└────────────────────┘        └───────────────────────────┘

Env Pool Server — A Flask service that manages a pool of cloud VM instances. It handles allocation, task reset (snapshot restore + setup scripts), action execution, evaluation, and teardown. VMs are pre-warmed at launch to eliminate cold-start latency during training.

Agent (Qwen3VLAgentLocal) — Converts screenshot observations into multi-turn VLM conversations (system prompt + image history + action output), and parses the model's response into executable pyautogui calls.

Reward Agent (Qwen3VLRewardAgent, PRM mode only) — A separate VLM that scores each agent step by comparing the action's intent against the next observation, producing +1/−1 per-step rewards for step-wise advantage estimation.

Trajectory Loop (generate_with_gui.py) — Orchestrates multi-step episodes: allocate a VM, reset to a task, loop (screenshot → model inference → parse action → execute → observe), compute final reward via OSWorld evaluators, and return the trajectory for training.

Three Operating Modes Copy Chapter

Prerequisites Copy Chapter

• 8 GPUs (default; configurable via NUM_GPUS)
• A cloud provider account (Volcengine by default) with VM image and network configured
• Volcengine secrets (VOLC_ACCESSKEY, VOLC_SECRETKEY) exported in your shell
• Python environment with requirements.txt installed
• SLIME framework and Megatron-LM available in sibling directories

Install the GUI-specific dependencies:

pip install -r requirements.txt
playwright install chromium

Step-by-Step Guide Copy Chapter

Mode 1: Evaluation Only Copy Section

Run a checkpoint on the full eval task suite without any training:

# Optional: point to your checkpoint (default: Qwen3-VL-4B-Thinking)
export HF_CKPT="/path/to/checkpoint"

# Optional: W&B logging
export WANDB_KEY="your-wandb-key"

cd slime
bash ../gui-rl/gui_qwen3vl_4b_eval.sh

The script:

1. Starts the env pool server and pre-warms VMs.
2. Launches a Ray cluster with all GPUs dedicated to rollout (no actor).
3. Runs every task in evaluation_examples/test_nochrome.json.
4. Reports accuracy per domain.

Mode 2: RL Training (GRPO) Copy Section

Train with sequence-level binary rewards — the agent gets +1 if the OSWorld evaluator marks the task as successful, −1 otherwise:

export HF_CKPT="/path/to/Qwen3-VL-8B-Thinking"
export WANDB_KEY="your-wandb-key"

# Optional GPU split (default: 4 actor + 4 rollout)
export ACTOR_GPUS=4
export ROLLOUT_GPUS=4

cd slime
bash ../gui-rl/gui_qwen3vl_8b_rl.sh

The script partitions GPUs between the training actor and the SGLang rollout engine, then alternates between:

1. Rollout: generate trajectories across n_samples_per_prompt parallel attempts per task.
2. Train: update the policy using GRPO with KL regularization.
3. Eval: periodically run the eval task set and log accuracy to W&B.

Mode 3: PRM-Augmented RL Training Copy Section

Adds a Process Reward Model that scores each step, enabling step-wise advantage estimation instead of sequence-level:

export HF_CKPT="/path/to/Qwen3-VL-8B-Thinking"
export PRM_MODEL_PATH="/path/to/Qwen3-VL-4B-Thinking"
export WANDB_KEY="your-wandb-key"

# GPU partition: actor + rollout + PRM must equal NUM_GPUS
export ACTOR_GPUS=4
export ROLLOUT_GPUS=3
export PRM_GPUS=1

cd slime
bash ../gui-rl/gui_qwen3vl_8b_prm_rl.sh

The PRM agent takes the interaction history plus the next observation after each step, and assigns +1 (progress) or −1 (no effect / regression). These per-step rewards feed into a step_wise advantage estimator, providing denser gradient signal than the final-outcome-only GRPO.

Key Environment Variables Copy Chapter

GPU & Training Copy Section

GUI Environment Copy Section

Logging Copy Section

Cloud Provider (Volcengine) Copy Section

Secrets must be exported before running:

export VOLC_ACCESSKEY="your-access-key"
export VOLC_SECRETKEY="your-secret-key"

Non-secret configs (VOLCENGINE_IMAGE_ID, VOLCENGINE_SUBNET_ID, etc.) have defaults in the shell scripts and can be overridden.

File Structure Copy Chapter

/* Detailed source-code truncated for AI context efficiency. */

File: instructions/README.md Copy Chapter

Slime Env Setup Copy Section

# cuda 12.9 (nvcc -V, nvidia-smi)
cd OpenClaw-RL
conda create --name openclaw-rl python=3.12 -y
conda activate openclaw-rl
 
pip install \
  torch==2.9.1+cu129 \
  torchvision==0.24.1+cu129 \
  torchaudio==2.9.1+cu129 \
  --index-url https://download.pytorch.org/whl/cu129
 
pip install -r requirements.txt

# DeepEP
git clone https://github.com/deepseek-ai/DeepEP.git
cd DeepEP
pip install -e . --no-build-isolation
cd ..

pip install -e slime/slime/backends/megatron_utils/kernels/int4_qat --no-build-isolation
 
# apex
git clone https://github.com/NVIDIA/apex.git
cd apex
APEX_CPP_EXT=1 APEX_CUDA_EXT=1 pip install -v --no-build-isolation .
cd ..

# flash_attn
export MAX_JOBS=8
pip install --no-build-isolation -v flash-attn==2.7.4.post1
 
# flashinfer
pip install "flashinfer-jit-cache==0.6.3" --index-url https://flashinfer.ai/whl/cu129

# megatron-bridge
pip install "megatron-bridge @ git+https://github.com/fzyzcjy/Megatron-Bridge.git@35b4ebfc486fb15dcc0273ceea804c3606be948a" --no-build-isolation

# TransformerEngine
export NVTE_FRAMEWORK=pytorch
pip install --no-build-isolation "transformer_engine[pytorch,core_cu12]==2.10.0"

# apt
apt-get update
apt-get install -y python3-apt

If you want to use Qwen3.5 (omit this if only use Qwen3)

# upgrade transformers
pip install transformers==5.3.0

If you want to use Qwen3.5 for multimodal (omit if only use Qwen3, or use Qwen3.5 text only)

# Megatron-Bridge for qwen35 vl
cd OpenClaw-RL
git clone --recursive https://github.com/NVIDIA-NeMo/Megatron-Bridge.git Megatron-Bridge-qwen35
cd Megatron-Bridge-qwen35
git checkout ebca893607d48388a6c083bfc143bc05621cc753
git submodule update --init --recursive

# Megatron-LM comes from the Megatron-Bridge submodule
cd 3rdparty/Megatron-LM
git checkout 17a67b9a97fb11a75933fd7f76ad76e1ac98a53d
cd /path/to/Megatron-Bridge-qwen35

python3 -m pip uninstall -y megatron-bridge megatron-core mbridge
python3 -m pip install --no-deps -e ./3rdparty/Megatron-LM
python3 -m pip install --no-deps -e ./

File: Megatron-LM/docs/discussions/deepseek-v3-gb200-optimization/deepseek-v3-gb200-reproduce-guide.md Copy Chapter

A Guide to Reproduce DeepSeek-V3 Pre-training Performance on GB200

1. Dockerfile Copy Chapter

Requirements:

• Transformer Engine: We recommend using commit d2945c6 with PR 2146 and 2150. You could prepare the branch by yourself, or use this branch based on TE v2.9 plus the above three commits/PRs.
• cuDNN: v9.14 is required.
• HybridEP: Install it from here.

Dockerfile for reference.

/* Detailed source-code truncated for AI context efficiency. */

[!Tip]

If you prefer to use CUDA 12.9, please change the base container to nvcr.io/nvidia/pytorch:25.06-py3 and the cuDNN to be installed to libcudnn9-cuda-12.

2. Megatron-Core Copy Chapter

We recommend using the dev branch after PR 1917.

git clone https://github.com/NVIDIA/Megatron-LM.git && \
cd Megatron-LM &&
git checkout effebd81f410bc6566fffee6c320b6f8f762e06d

3. Cluster Configuration Copy Chapter

Since we're using EP 32 on NVL72, it's important to make sure

[!Important]
Every 32 GB200 GPUs (8 nodes) are in the same NVL domain (or rack).

Usually you can make it via your cluster workload manager. Taking Slurm as an example, you could pass --segment 8 to the sbatch command to ensure that every segment of 8 nodes will be scheduled to a rack.

4. Training scripts Copy Chapter

Environment variables Copy Section

CUDA_DEVICE_MAX_CONNECTIONS=1
NVTE_FWD_LAYERNORM_SM_MARGIN=0
NVTE_BWD_LAYERNORM_SM_MARGIN=0
NVLINK_DOMAIN_SIZE=72
NVTE_ALLOW_NONDETERMINISTIC_ALGO=1
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
NCCL_NVLS_ENABLE=0
NVTE_FUSED_ATTN=1
NVTE_NORM_FWD_USE_CUDNN=1
NVTE_NORM_BWD_USE_CUDNN=1
PYTHONWARNINGS=ignore
NCCL_DEBUG=VERSION
NCCL_GRAPH_REGISTER=0

bindpcie Copy Section

Download bindpcie to your workdir, make it executable,

wget https://raw.githubusercontent.com/NVIDIA/mlperf-common/refs/heads/main/client/bindpcie &&
chmod 755 bindpcie

and then

[!Important]
Place it at the beginning of your launch command in every process.

Taking Slurm as an example, your script should look like

#!/bin/bash

#SBATCH [... sbatch args]

srun [... srun args] /path/to/bindpcie /path/to/pretrain_gpt.py [... mcore arguments]

This is a very important step on GB200.

Launch script Copy Section

/* Detailed source-code truncated for AI context efficiency. */

Explanation of arguments Copy Section

The following arguments indicate key optimizations.

• Pipeline parallel layout

--pipeline-model-parallel-layout "Et|(tt|)*30L"

E stands for embedding, t for transformer layer, L for Loss. So it's interpreted as a total of 32 stages, where the first stage is Embedding + 1 transformer layer, the last stage is Loss, and the middle 30 stages are 2 transformer layers.

• Fine-grained recompute

--recompute-granularity selective \
--recompute-modules moe_act mlp \

• Partial CUDA Graphs

--cuda-graph-impl transformer_engine \
--cuda-graph-scope attn moe_router moe_preprocess \
--te-rng-tracker \

• Force load balancing for performance benchmark

--moe-router-force-load-balancing \

• HybridEP

--moe-token-dispatcher-type flex \
--moe-flex-dispatcher-backend hybridep \
--moe-hybridep-num-sms 32 \

• MXFP8 recipe

--fp8-recipe mxfp8 \
--fp8-format e4m3 \
--fp8-param-gather \
--reuse-grad-buf-for-mxfp8-param-ag \

• BF16 optimizer states

--use-precision-aware-optimizer \
--main-grads-dtype fp32 \
--main-params-dtype fp32 \
--exp-avg-dtype bf16 \
--exp-avg-sq-dtype bf16 \

• Kernel fusions

--cross-entropy-loss-fusion \
--cross-entropy-fusion-impl te \
--moe-permute-fusion \
--moe-router-fusion \

• Manual GC to make ranks better synchronized

--manual-gc \
--manual-gc-interval 10 \

File: Megatron-LM/docs/discussions/README.md Copy Chapter

Megatron Discussions

This directory contains in-depth guides, tutorials, and discussions about optimizing and using Megatron for various use cases.

Available Guides Copy Chapter

Performance Optimization Copy Section

• Optimizing DeepSeek-V3 Training Performance on NVIDIA GB200 NVL72

  A comprehensive guide on optimizing DeepSeek-V3 model training on NVIDIA GB200 NVL72 systems, covering profiling techniques, performance bottlenecks, and optimization strategies.

• A Guide to Reproduce DeepSeek-V3 Pre-training Performance on GB200

  A detailed guide on how to reproduce the DeepSeek-V3 pre-training performance on GB200, incluing the dockerfile, package requirements and training scripts.

Contributing Copy Chapter

If you'd like to contribute a guide or tutorial, please follow this structure:

1. Create a new directory: docs/discussions/your-guide-name/
2. Add your main guide: docs/discussions/your-guide-name/your-guide-name.md
3. Create an images directory: docs/discussions/your-guide-name/images/
4. Update this README.md with a link to your guide

Each guide should be self-contained with its own images and supporting files.

File: Megatron-LM/docs/source/api-guide/context_parallel.rst Copy Chapter

context\_parallel package

Context parallelism overview Copy Chapter

.. figure:: ../images/context_parallel/CP_overview.png
:alt: cp_overview
:align: center

Figure 1: A transformer layer running with TP2CP2. Communications next to Attention are for CP, others are for TP. (AG/RS: all-gather in forward and reduce-scatter in backward, RS/AG: reduce-scatter in forward and all-gather in backward, /AG: no-op in forward and all-gather in backward).

Context Parallelism ("CP") is a parallelization scheme on the dimension of sequence length. Unlike prior SP (sequence parallelism) which only splits the sequence of Dropout and LayerNorm activations, CP partitions the network inputs and all activations along sequence dimension. With CP, all modules except attention (e.g., Linear, LayerNorm, etc.) can work as usual without any changes, because they do not have inter-token operations. As for attention, the Q (query) of each token needs to compute with the KV (key and value) of all tokens in the same sequence. Hence, CP requires additional all-gather across GPUs to collect the full sequence of KV. Correspondingly, reduce-scatter should be applied to the activation gradients of KV in backward propagation. To reduce activation memory footprint, each GPU only stores the KV of a sequence chunk in forward and gathers KV again in backward. KV communication happens between a GPU and its counterparts in other TP groups. The all-gather and reduce-scatter are transformed to point-to-point communications in ring topology under the hood. Exchanging KV also can leverage MQA/GQA to reduce communication volumes, as they only have one or few attention heads for KV.

For example, in Figure 1, assuming sequence length is 8K, each GPU processes 4K tokens. GPU0 and GPU2 compose a CP group, they exchange KV with each other. Same thing also happens between GPU1 and GPU3. CP is similar to Ring Attention <https://arxiv.org/abs/2310.01889>_ but provides better performance by (1) leveraging the latest OSS and cuDNN flash attention kernels; (2) removing unnecessary computation resulted from low-triangle causal masking and achieving optimal load balance among GPUs.

Context parallelism benefits Copy Chapter

.. figure:: ../images/context_parallel/CP_results.png
:alt: cp_results
:align: center

Figure 2: Speedup of 175B GPT with various TP+CP combinations vs. full recompute (i.e., TP8CP1).

LLM encounters OOM (out of memory) issue with long context (i.e., long sequence length) because of linearly increasing memory footprint of activations. Recomputing activations in backward can avoid OOM but also introduce significant overheads (~30% with full recompute). Enlarging TP (tensor model parallelism) can fix the OOM issue as well, but it potentially makes compute (e.g., Linear) too short to overlap communication latencies. To be clear, scaling out to more GPUs with bigger TP can hit the overlapping problem no matter if OOM happens.

CP can better address the issues. With CP, each GPU only computes on a part of the sequence, which reduces both computation and communication by CP times. Therefore, there are no concerns about the overlapping between them. The activation memory footprint per GPU is also CP times smaller, hence no OOM issue anymore. As Figure 2 shows, the combinations of TP and CP can achieve optimal performance by eliminating recompute overheads and making the best tradeoff between computation and communications.

Enabling context parallelism Copy Chapter

CP support has been added to GPT. All models that share GPT code path also should be able to benefit from CP, such as Llama. CP can work with TP (tensor model parallelism), PP (pipeline model parallelism), and DP (data parallelism), where the total number of GPUs equals TPxCPxPPxDP. CP also can work with different attention variants, including MHA/MQA/GQA, uni-directional and bi-directional masking.

CP is enabled by simply setting context_parallel_size= in command line. Default context_parallel_size is 1, which means CP is disabled. Running with CP requires Megatron-Core (>=0.5.0) and Transformer Engine (>=1.1).