Qwen-Agent Documentation

1. Project Overview & Quickstart (QwenLM/Qwen-Agent) Copy Chapter

File: README.md Copy Chapter

中文 ｜ English

Qwen-Agent is a framework for developing LLM applications based on the instruction following, tool usage, planning, and
memory capabilities of Qwen.
It also comes with example applications such as Browser Assistant, Code Interpreter, and Custom Assistant.
Now Qwen-Agent plays as the backend of Qwen Chat.

News

• 🔥🔥🔥Feb 16, 2026: Open-sourced Qwen3.5. For usage examples, refer to Qwen3.5 Agent Demo.
• Jan 27, 2026: Open-sourced agent evaluation benchmark DeepPlanning and added Qwen-Agent documentation.
• Sep 23, 2025: Added Qwen3-VL Tool-call Demo, supporting tools such as zoom in, image search, and web search.
• Jul 23, 2025: Add Qwen3-Coder Tool-call Demo; Added native API tool call interface support, such as using vLLM's built-in tool call parsing.
• May 1, 2025: Add Qwen3 Tool-call Demo, and add MCP Cookbooks.
• Mar 18, 2025: Support for the reasoning_content field; adjust the default Function Call template, which is applicable to the Qwen2.5 series general models and QwQ-32B. If you need to use the old version of the template, please refer to the example for passing parameters.
• Mar 7, 2025: Added QwQ-32B Tool-call Demo. It supports parallel, multi-step, and multi-turn tool calls.
• Dec 3, 2024: Upgrade GUI to Gradio 5 based. Note: GUI requires Python 3.10 or higher.
• Sep 18, 2024: Added Qwen2.5-Math Demo to showcase the Tool-Integrated Reasoning capabilities of Qwen2.5-Math. Note: The python executor is not sandboxed and is intended for local testing only, not for production use.

Getting Started

Installation Copy Chapter

• Install the stable version from PyPI:

pip install -U "qwen-agent[gui,rag,code_interpreter,mcp]"
# Or use `pip install -U qwen-agent` for the minimal requirements.
# The optional requirements, specified in double brackets, are:
#   [gui] for Gradio-based GUI support;
#   [rag] for RAG support;
#   [code_interpreter] for Code Interpreter support;
#   [mcp] for MCP support.

• Alternatively, you can install the latest development version from the source:

git clone https://github.com/QwenLM/Qwen-Agent.git
cd Qwen-Agent
pip install -e ./"[gui,rag,code_interpreter,mcp]"
# Or `pip install -e ./` for minimal requirements.

Preparation: Model Service Copy Chapter

You can either use the model service provided by Alibaba
Cloud's DashScope, or deploy and use your own
model service using the open-source Qwen models.

• If you choose to use the model service offered by DashScope, please ensure that you set the environment
  variable DASHSCOPE_API_KEY to your unique DashScope API key.

• Alternatively, if you prefer to deploy and use your own model service, please follow the instructions provided in the README of Qwen2 for deploying an OpenAI-compatible API service.
  Specifically, consult the vLLM section for high-throughput GPU deployment or the Ollama section for local CPU (+GPU) deployment.
  For the QwQ and Qwen3 model, it is recommended to do not add the --enable-auto-tool-choice and --tool-call-parser hermes parameters, as Qwen-Agent will parse the tool outputs from vLLM on its own.
  For Qwen3-Coder, it is recommended to enable both of the above parameters, use vLLM's built-in tool parsing, and combine with the use_raw_api parameter usage.

Developing Your Own Agent Copy Chapter

Qwen-Agent offers atomic components, such as LLMs (which inherit from class BaseChatModel and come with function calling) and Tools (which inherit
from class BaseTool), along with high-level components like Agents (derived from class Agent).

The following example illustrates the process of creating an agent capable of reading PDF files and utilizing tools, as
well as incorporating a custom tool:

/* Detailed source-code truncated for AI context efficiency. */

In addition to using built-in agent implementations such as class Assistant, you can also develop your own agent implemetation by inheriting from class Agent.

The framework also provides a convenient GUI interface, supporting the rapid deployment of Gradio Demos for Agents.
For example, in the case above, you can quickly launch a Gradio Demo using the following code:

from qwen_agent.gui import WebUI
WebUI(bot).run()  # bot is the agent defined in the above code, we do not repeat the definition here for saving space.

Now you can chat with the Agent in the web UI. Please refer to the examples directory for more usage examples.

FAQ

How to Use the Code Interpreter Tool? Copy Chapter

We implement a code interpreter tool based on local Docker containers. You can enable the built-in code interpreter tool for your agent, allowing it to autonomously write code according to specific scenarios, execute it securely within an isolated sandbox environment, and return the execution results.

⚠️ Note: Before using this tool, please ensure that Docker is installed and running on your local operating system. The time required to build the container image for the first time depends on your network conditions. For Docker installation and setup instructions, please refer to the official documentation.

How to Use MCP? Copy Chapter

You can select the required tools on the open-source MCP server website and configure the relevant environment.

Example of MCP invocation format:

{
    "mcpServers": {
        "memory": {
            "command": "npx",
            "args": ["-y", "@modelcontextprotocol/server-memory"]
        },
        "filesystem": {
            "command": "npx",
            "args": ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/allowed/files"]
        },
        "sqlite" : {
            "command": "uvx",
            "args": [
                "mcp-server-sqlite",
                "--db-path",
                "test.db"
            ]
        }
    }
}

For more details, you can refer to the MCP usage example

The dependencies required to run this example are as follows:

# Node.js (Download and install the latest version from the Node.js official website)
# uv 0.4.18 or higher (Check with uv --version)
# Git (Check with git --version)
# SQLite (Check with sqlite3 --version)

# For macOS users, you can install these components using Homebrew:
brew install uv git sqlite3

# For Windows users, you can install these components using winget:
winget install --id=astral-sh.uv -e
winget install git.git sqlite.sqlite

Do you have function calling (aka tool calling)? Copy Chapter

Yes. The LLM classes provide function calling. Additionally, some Agent classes also are built upon the function calling capability, e.g., FnCallAgent and ReActChat.

The current default tool calling template natively supports Parallel Function Calls.

How to pass LLM parameters to the Agent? Copy Chapter

llm_cfg = {
    # The model name being used:
    'model': 'qwen3-32b',
    # The model service being used:
    'model_type': 'qwen_dashscope',
    # If 'api_key' is not set here, it will default to reading the `DASHSCOPE_API_KEY` environment variable:
    'api_key': 'YOUR_DASHSCOPE_API_KEY',

    # Using an OpenAI API compatible model service, such as vLLM or Ollama:
    # 'model': 'qwen3-32b',
    # 'model_server': 'http://localhost:8000/v1',  # base_url, also known as api_base
    # 'api_key': 'EMPTY',

    # (Optional) LLM hyperparameters:
    'generate_cfg': {
        # This parameter will affect the tool-call parsing logic. Default is False:
          # Set to True: when content is `this is the thoughtthis is the answer`
          # Set to False: when response consists of reasoning_content and content
        # 'thought_in_content': True,

        # tool-call template: default is nous (recommended for qwen3):
        # 'fncall_prompt_type': 'nous'

        # Maximum input length, messages will be truncated if they exceed this length, please adjust according to model API:
        # 'max_input_tokens': 58000

        # Parameters that will be passed directly to the model API, such as top_p, enable_thinking, etc., according to the API specifications:
        # 'top_p': 0.8

        # Using the API's native tool call interface
        # 'use_raw_api': True,
    }
}

How to do question-answering over super-long documents involving 1M tokens? Copy Chapter

We have released a fast RAG solution, as well as an expensive but competitive agent, for doing question-answering over super-long documents. They have managed to outperform native long-context models on two challenging benchmarks while being more efficient, and perform perfectly in the single-needle "needle-in-the-haystack" pressure test involving 1M-token contexts. See the blog for technical details.

Application: BrowserQwen

BrowserQwen is a browser assistant built upon Qwen-Agent. Please refer to its documentation for details.

Disclaimer

The Docker container-based code interpreter mounts only the specified working directory and implements basic sandbox isolation, but it should still be used with caution in production environments.

File: benchmark/code_interpreter/README.md Copy Chapter

Code Interpreter Benchmark

Introduction Copy Chapter

To assess LLM's ability to use the Python Code Interpreter for tasks such as mathematical problem solving, data visualization, and other general-purpose tasks such as file handling and web scraping, we have created and open-sourced a benchmark specifically designed for evaluating these capabilities.

Metrics Copy Section

The metrics are divided into two parts: code executability and code correctness.

• Code executability: evaluating the ability of the LLM-generated code to be executed.
• Code correctness: evaluating whether the LLM-generated code runs correctly.

Domain Copy Section

When evaluating the accuracy of the code execution results for code correctness, we further divide it into two specific domains: Math, Visualization.
In terms of code executability, we calculate executable rate of the generated code for General problem-solving.

Results Copy Chapter

• Qwen-7B-Chat refers to the version updated after September 25, 2023.
• The code correctness judger model for Visualization has changed from Qwen-vl-chat to gpt-4-vision-preview in the version 20231206.

Furthermore, we also provide the results of Qwen-vl-plus as the code correctness judger model for Visualization task to serve as a reference.

Usage Copy Chapter

Installation Copy Section

git clone https://github.com/QwenLM/Qwen-Agent.git
cd benchmark
pip install -r requirements.txt

Dataset Download Copy Section

cd benchmark
wget https://qianwen-res.oss-cn-beijing.aliyuncs.com/assets/qwen_agent/benchmark_code_interpreter_data.zip
unzip benchmark_code_interpreter_data.zip
mkdir eval_data
mv eval_code_interpreter_v1.jsonl eval_data/

Evaluation Copy Section

To reproduce the comprehensive results of benchmark, you can run the following script:

python inference_and_execute.py --model {model_name}

{model_name}:

• qwen-1.8b-chat
• qwen-7b-chat
• qwen-14b-chat
• qwen-72b-chat
• llama-2-7b-chat
• llama-2-13b-chat
• codellama-7b-instruct
• codellama-13b-instruct
• internlm-7b-chat-1.1
• internlm-20b-chat

The benchmark will run the test cases and generate the performance results. The results will be saved in the output_data directory.

Notes:
Please install simhei.ttf font for proper display in matplotlib when evaluating visualization task. You can do this by preparing simhei.ttf (which can be found on any Windows PC) and then running the following code snippet:

import os
import matplotlib
target_font_path = os.path.join(
    os.path.abspath(
        os.path.join(matplotlib.matplotlib_fname(), os.path.pardir)),
        'fonts', 'ttf', 'simhei.ttf')
os.system(f'cp simhei.ttf {target_font_path}')
font_list_cache = os.path.join(matplotlib.get_cachedir(), 'fontlist-*.json')
os.system(f'rm -f {font_list_cache}')

Code Executable Rate

python inference_and_execute.py --task {task_name} --model {model_name}

{task_name}:

• general: General problem-solving task

Code Correctness Rate

python inference_and_execute.py --task {task_name} --model {model_name}

{task_name}:

• visualization: Visualization task
• gsm8k: Math task

Configuration Copy Chapter

The inference_and_exec.py file contains the following configurable options:

• --model: The model to test which can be one of qwen-72b-chat, qwen-14b-chat, qwen-7b-chat, qwen-1.8b-chat, qwen-7b-chat, llama-2-7b-chat, llama-2-13b-chat, codellama-7b-instruct, codellama-13b-instruct, internlm-7b-chat-1.1, internlm-20b-chat.
• --task: The test task which can be one of all, visualization, general, gsm8k.
• --output-path: The path for saving evaluation result.
• --input-path: The path for placing evaluation data.
• --output-fname: The file name for evaluation result.
• --input-fname: The file name for evaluation data.
• --force: Force generation and will overwrite the cached results.
• --eval-only: Only calculate evaluation metrics without re-inference.
• --eval-code-exec-only: Only evaluate code executable rate
• --gen-exec-only: Only generate and execuate code without calculating evaluation metrics.
• --gen-only: Only generate without execuating code and calculating evaluation metrics.
• --vis-judger: The model to judge the result correctness for Visualization task which can be one of gpt-4-vision-preview, qwen-vl-chat, qwen-vl-plus. It is set to gpt-4-vision-preview by default in the version 20231206, and Qwen-vl-chat has been deprecated.

2. Official Technical Reference & Guides (QwenLM/QwenLM.github.io) Copy Chapter

File: README.md Copy Chapter

Qwen's Blog

[!IMPORTANT]
This blog is no longer being updated. For the latest news, please visit qwen.ai.

File: content/about/index.md Copy Chapter

title: "About Us"
search_hidden: true
show_bread_crumbs: false
hide_meta: true
disable_share: true
header:
background: "linear-gradient(to top, #30cfd0 0%, #330867 100%);"

Qwen Team

The Qwen team aims at chasing artificial general intelligence and now focuses on building generalist models, including large language models and large multimodal models. We embrace opensource and previously we have released the Qwen model series, including the language models (Qwen), expert models (Qwen-Coder and Qwen-Math), and multimodal models (Qwen-VL and Qwen-Audio). Additionally, we have built web service and APP for users to benefit from the assistance of Qwen for your daily work and life. We are a group of people with diverse talents and interests. Feel free to chat with us and welcome to join us!

Links Copy Chapter

In the following, we provide important links for you to refer to our opensource resources.

Resources Copy Section

{{< button href="https://github.com/QwenLM/Qwen" label="GITHUB" external=true >}}
{{< button href="https://modelscope.com/organization/qwen" label="MODELSCOPE" external=true >}}
{{< button href="https://huggingface.co/Qwen" label="HUGGING FACE" external=true >}}
{{< button href="https://www.kaggle.com/models/qwen-lm" label="KAGGLE" external=true >}}

Social Media Copy Section

{{< button href="https://x.com/Alibaba_Qwen" label="X" external=true >}}
{{< button href="https://www.linkedin.com/company/qwen/" label="LINKEDIN" external=true >}}
{{< button href="https://discord.com/invite/CV4E9rpNSD" label="DISCORD" external=true >}}

API Copy Section

{{< button href="https://www.alibabacloud.com/en/product/modelstudio" label="ALIBABA CLOUD" external=true >}}

{{< button href="https://www.aliyun.com/product/bailian" label="ALIYUN" external=true >}}

File: content/blog/chinese-clip/index.md Copy Chapter

title: "Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese"
date: 2022-12-24T14:54:19+08:00

weight: 1

aliases: ["/first"]

tags: ["Research"]

author: ["Junyang Lin", "Binyuan Hui"]

comments: false

disable_share: false

hide_meta: false

hide_summary: false # to hide summary in list

hide_footer: false

math: false

search_hidden: false # to hide from search page

show_reading_time: true
show_bread_crumbs: true
show_post_nav_links: false # the prev/next after the content
show_code_copy_buttons: true
show_word_count: true

use_hugo_toc: true

show_toc: true

toc_open: true # default expand all

cover:

image: "cnclip_banner.jpg"

# can also paste direct link from external site

# ex. https://i.ibb.co/K0HVPBd/paper-mod-profilemode.png

alt: ""

caption: ""

relative: true # To use relative path for cover image, used in hugo Page-bundles

responsive_images: true

hidden: false

header:

background: "" # background css value

background_image: ""

gradient: false

blur: false

CLIP[^1] is a phenomenal playmaker in vision and multimodal representation learning. It plays not only as a foundation model but also a bridge between vision and language. It has triggered a series of research in different fields, especially text-to-image generation. However, we find that there is a necessity for a language-specific CLIP for applications, especially cross-modal retrieval, and there is no opensourced Chinese CLIP with good performance. We therefore launched this project to promote the Chinese multimodal representation learning.

{{< button href="https://arxiv.org/abs/2211.01335" label="Paper" external=true >}}
{{< button href="https://github.com/OFA-Sys/Chinese-CLIP" label="GitHub" external=true >}}
{{< button href="https://www.modelscope.cn/models/damo/multi-modal_clip-vit-base-patch16_zh/summary" label="ModelScope" external=true >}}
{{< button href="https://huggingface.co/spaces/OFA-Sys/chinese-clip-zero-shot-image-classification" label="Demo" external=true >}}

Background Copy Chapter

In real-world vision-language applications, e.g., cross-modal retrieval, the language plays an important role. Suppose we directly use CLIP and translation for texts, the quality of translation will significantly impact the downstream performance. Furthermore, another significant issue is the domains of pretraining data. If we hope the model achieve good performance on the Chinese data, there is also a necessity for CLIP to adapt to the domain of images in the Chinese websites, which reflect cultural values, social landscape, etc.

Here is an example of the search with mCLIP[^2]. We find that it is really hard for the model to understand some concepts in Chinese, and it can only retrieve relevant items that belong to western culture.

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/cnclip/search.jpg#center" width="80%" >}}

Also, we have conducted experiments on cross-modal retrieval with the original CLIP plus machine translation. The performance significantly degrades and falls far back behind our Chinese CLIP. This is also an evidence to support why we need a language-specific CLIP.

Method Copy Chapter

In general, we follow the setups of the original CLIP, and we propose a two-stage pretraining method that shows better performance than training from scratch. We believe this is a more cost-effective way to transfer CLIP to another language.

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/cnclip/chinese-clip-model-v3.jpg#center" width="80%" >}}

In the first stage, we initialize the two towers with pretrained models, which are the vision encoder of CLIP, e.g., ViT-B, ResNet, etc., and Chinese RoBERTa RoBERTA-wwm-Chinese. We freeze the image encoder and contrastively tune the language encoder that maps its representation to the output space of CLIP vision encoder. In the second stage, we unlock the vision encoder and contrastively tune the two towers so that the vision encoder can learn to model the distribution of the images of Chinese data.

To make this research reproducible, we mostly use the public datasets for pretraining, including the part marked with "zh" in LAION-5B[^3], Wukong dataset[^4], the translated data from Visual Genome and MSCOCO, etc. The total amount of the image-text pairs reaches 200 million.

We released 5 versions of Chinese CLIP, including ResNet-50, ViT-B/16, ViT-L/14, ViT-L/14
@336px, and ViT-H/14. The statistics are listed below.

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/cnclip/model_variants.jpg#center" width="80%" >}}

Experiments Copy Chapter

The experiments are conducted on 3 cross-modal retrieval datasets, including the Chinese native dataset MUGE, and the English-native datasets (which means the images and texts are not from the Chinese websites) Flickr30K-CN and COCO-CN. On all datasets, Chinese CLIP performs the best, and its gap with the previous best models in MUGE is much larger than those in the other datasets. This demonstrates that our method is contributive to building a language specific CLIP model that can perform much better on native datasets.

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/cnclip/muge.jpg#center" width="80%" >}}

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/cnclip/flickr.jpg#center" width="80%" >}}

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/cnclip/coco.jpg#center" width="80%" >}}

We also try Chinese CLIP on zero-shot image classification, and we participate in the ELEVATER benchmark[^5] by translating all labels and prompts to Chinese manually. Results show that Chinese CLIP can also achieve a competitive performance in the English-native benchmark.

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/cnclip/elevater.jpg#center" width="80%" >}}

For the ablation, it can be found that in comparison with training from scratch, the two-stage pretraining method demonstrates much better performance, and the second-stage pretraining can further level up the model performance in cross-modal retrieval.

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/cnclip/ablation.jpg#center" title="Ablation studies." >}}

Limitations and Future Work Copy Chapter

Though the above contents show the effectiveness of Chinese CLIP, we still need to work on validating its role as a vision foundation model. Empirically, it should be a strong foundation model for tasks of Chinese-native data. Thus, in the next step, we will work on building a benchmark for Chinese multimodal representation learning and vision representation learning.

Feel free to visit our GitHub repo and use the codes and checkpoints. Hope they will be helpful for your research or applications!

[^1]: Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021).
Learning Transferable Visual Models From Natural Language Supervision.
International Conference on Machine Learning.

[^2]: Carlsson, F., Eisen, P., Rekathati, F., & Sahlgren, M. (2022).
Cross-lingual and Multilingual CLIP.
International Conference on Language Resources and Evaluation.

[^3]: Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S., Crowson, K., Schmidt, L., Kaczmarczyk, R., & Jitsev, J. (2022).
LAION-5B: An open large-scale dataset for training next generation image-text models.
arXiv, abs/2210.08402.

[^4]: Gu, J., Meng, X., Lu, G., Hou, L., Niu, M., Xu, H., Liang, X., Zhang, W., Jiang, X., & Xu, C. (2022).
Wukong: 100 Million Large-scale Chinese Cross-modal Pre-training Dataset and A Foundation Framework.
arXiv, abs/2202.06767.

[^5]: Li, C., Liu, H., Li, L., Zhang, P., Aneja, J., Yang, J., Jin, P., Lee, Y.J., Hu, H., Liu, Z., & Gao, J. (2022).
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models.
arXiv, abs/2204.08790.

File: content/blog/chinese-clip/index.zh.md Copy Chapter

title: "Chinese CLIP: 中文图文对比学习预训练"
date: 2022-12-24T14:54:19+08:00

weight: 1

aliases: ["/first"]

tags: ["Research"]

author: ["Junyang Lin", "Binyuan Hui"]

comments: false

disable_share: false

hide_meta: false

hide_summary: false # to hide summary in list

hide_footer: false

math: false

search_hidden: false # to hide from search page

show_reading_time: true
show_bread_crumbs: true
show_post_nav_links: false # the prev/next after the content
show_code_copy_buttons: true
show_word_count: true

use_hugo_toc: true

show_toc: true

toc_open: true # default expand all

cover:

image: "cnclip_banner.jpg"

# can also paste direct link from external site

# ex. https://i.ibb.co/K0HVPBd/paper-mod-profilemode.png

alt: ""

caption: ""

relative: true # To use relative path for cover image, used in hugo Page-bundles

responsive_images: true

hidden: false

header:

background: "" # background css value

background_image: ""

gradient: false

blur: false

lang: zh-CNs Copy Chapter

CLIP[^1]是多模态表示学习领域一个现象级的模型。它不仅扮演基础模型，并且建立了视觉和语言的桥梁。它还推动了很多其他领域技术的发展，尤其是文本生成图像。然而，我们还需要特定语言的CLIP，尤其在现实应用中，比如跨模态检索。在此之前还没有效果较好的开源中文CLIP。因此我们希望通过这个项目推动中文多模态的发展。

{{< button href="https://arxiv.org/abs/2211.01335" label="论文" external=true >}}
{{< button href="https://github.com/OFA-Sys/Chinese-CLIP" label="Github" external=true >}}
{{< button href="https://www.modelscope.cn/models/damo/multi-modal_clip-vit-base-patch16_zh/summary" label="ModelScope" external=true >}}
{{< button href="https://huggingface.co/spaces/OFA-Sys/chinese-clip-zero-shot-image-classification" label="体验" external=true >}}

背景 Copy Chapter

在诸如跨模态检索的图文应用中，语言往往扮演重要的角色。假设直接使用CLIP和翻译文本，翻译质量将严重影响下游任务表现。此外，预训练数据的分布也同样重要。我们希望能够有一个模型较好地建模中文世界的数据，那么CLIP的图像学习同样需要适应到中文世界图像的领域中，这些图像在很多维度都有自己的特色，比如它所蕴含的文化价值、展现的社会风貌等。

下面是一个多语言CLIP[^2]检索的例子。不难看出，模型难以理解一些中文概念，甚至只能召回一些西方文化类似的概念的图片。

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/cnclip/search.jpg#center" width="80%" >}}

并且，我们还做了相关实验，对比原始CLIP配合翻译文本和中文CLIP在跨模态检索任务上的表现。原始CLIP的效果远低于中文CLIP，这也一定程度反映语言特定的CLIP模型的必要性。

方法 Copy Chapter

我们尽可能采用和原始CLIP一致的设定，不去增加模块或者设计复杂化的训练方法。而为了更加高效的训练，包括训练效率和最终迁移效果的提升，我们没有选择从头开始预训练，而是提出一个两阶段预训练的方法。

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/cnclip/chinese-clip-model-v3.jpg#center" width="80%" >}}

在第一阶段中，我们将CLIP的双塔用已有的预训练模型进行初始化，分别是CLIP的视觉侧（如ViT-B、ResNet等）以及中文RoBERTa RoBERTA-wwm-Chinese。我们冻结图像塔，通过对比学习让文本塔的输出表示和图像塔的一致。在第二阶段中，我们解冻图像塔，进行对比学习继续训练，从而图像塔也能学习建模中文领域的图像数据分布。

在可复现性方面，我们尽可能采用公开数据集，其中包括LAION-5B[^3]的中文部分、悟空数据集[^4]、来自VG和MSCOCO等的翻译数据。总数据量大约2亿。

我们推出了5个规模的中文CLIP模型，其中包括ResNet-50、ViT-B/16、ViT-L/14、ViT-L/14@336px和ViT-H/14。具体数据详见下表。

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/cnclip/model_variants.jpg#center" width="80%" >}}

实验 Copy Chapter

我们做了3个跨模态检索的实验，其中包括中文原生的MUGE和英文原生的Flickr30K-CN和COCO-CN。在所有数据集上，中文CLIP都取得了最好的效果，而尤其在MUGE上中文CLIP相比此前模型的优势最为巨大。这也反映中文CLIP在中文原生数据集上能够取得更好的表现。

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/cnclip/muge.jpg#center" width="80%" >}}

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/cnclip/flickr.jpg#center" width="80%" >}}

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/cnclip/coco.jpg#center" width="80%" >}}

我们也尝试了中文CLIP的零样本分类能力，并在ELEVATER[^5]上做了测试，具体实现包括翻译标签和提示词。实验结果也反映中文CLIP在英文原生的基准上同样能取得有竞争力的表现。

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/cnclip/elevater.jpg#center" width="80%" >}}

我们补充了实验说明两阶段训练方法的有效性。对比从头训练，不管是收敛效率还是最终迁移效果上，两阶段的方法都取得更好的效果，并且对比单纯的一阶段联合训练，第二阶段预训练的加入还能进一步提升效果。这也意味着当我们打造一个语言特定的中文CLIP其实不需要从头来，可以在已有模型的基础上站得更高。

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/cnclip/ablation.jpg#center" title="Ablation studies." >}}

局限性及未来工作 Copy Chapter

尽管上文介绍了中文CLIP的强大之处，但我们仍需要认识到当前中文CLIP还没有充分验证其作为视觉基础模型的作用。经验上，它应当在中文原生数据上表现更好。因此，在下一阶段的工作中，我们将研究构造一个针对中文多模态表示学习和视觉表示学习的基准。

欢迎大家访问我们的GitHub存储库和ModelScope模型库，并使用我们的代码和模型。希望能够帮助到你们的研究和应用！

[^1]: Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021).
Learning Transferable Visual Models From Natural Language Supervision.
International Conference on Machine Learning.

[^2]: Carlsson, F., Eisen, P., Rekathati, F., & Sahlgren, M. (2022).
Cross-lingual and Multilingual CLIP.
International Conference on Language Resources and Evaluation.

[^3]: Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S., Crowson, K., Schmidt, L., Kaczmarczyk, R., & Jitsev, J. (2022).
LAION-5B: An open large-scale dataset for training next generation image-text models.
arXiv, abs/2210.08402.

[^4]: Gu, J., Meng, X., Lu, G., Hou, L., Niu, M., Xu, H., Liang, X., Zhang, W., Jiang, X., & Xu, C. (2022).
Wukong: 100 Million Large-scale Chinese Cross-modal Pre-training Dataset and A Foundation Framework.
arXiv, abs/2202.06767.

[^5]: Li, C., Liu, H., Li, L., Zhang, P., Aneja, J., Yang, J., Jin, P., Lee, Y.J., Hu, H., Liu, Z., & Gao, J. (2022).
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models.
arXiv, abs/2204.08790.

File: content/blog/codeqwen1.5/index.md Copy Chapter

title: "Code with CodeQwen1.5"
date: 2024-04-16T13:33:00+08:00
weight: 1

aliases: ["/first"]

tags: ["Research"]

draft: true

comments: false

disable_share: false

hide_meta: false

hide_summary: false # to hide summary in list

hide_footer: false

math: true

search_hidden: false # to hide from search page

show_reading_time: true
show_bread_crumbs: true
show_post_nav_links: false # the prev/next after the content
show_code_copy_buttons: true
show_word_count: true

use_hugo_toc: true

show_toc: true

toc_open: true # default expand all

cover:

image: "path"

# can also paste direct link from external site

# ex. https://i.ibb.co/K0HVPBd/paper-mod-profilemode.png

alt: ""

caption: ""

relative: true # To use relative path for cover image, used in hugo Page-bundles

responsive_images: true

hidden: false

header:

background: "" # background css value

background_image: ""

gradient: false

blur: false

{{< button href="https://github.com/QwenLM/CodeQwen1.5" label="GITHUB" external=true >}}
{{< button href="https://huggingface.co/Qwen" label="HUGGING FACE" external=true >}}
{{< button href="https://modelscope.cn/organization/qwen" label="MODELSCOPE" external=true >}}
{{< button href="https://huggingface.co/spaces/Qwen/CodeQwen1.5-7b-Chat-demo" label="DEMO" external=true >}}
{{< button href="https://discord.gg/yPEP2vHTu4" label="DISCORD" external=true >}}

Introduction

The advent of advanced programming tools, which harnesses the power of large language models (LLMs), has significantly enhanced programmer productivity and accuracy. Notwithstanding these advancements, dominant coding assistants like Github Copilot, built upon proprietary LLMs, pose notable challenges in terms of cost, privacy, security, and potential copyright infringement. Recognizing the imperative for a more transparent and accessible alternative, the open-source community has embarked on a concerted endeavor to develop open codeLLMs. This initiative has already given rise to several promising open-source models, including StarCoder2, CodeLlama, and DeepSeek-Coder, offering a path forward, albeit one that necessitates continued refinement.

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/codeqwen1.5/intro.png#center" width="100%">}}

Today, we are delighted to introduce a new member of the Qwen1.5 open-source family, the CodeQwen1.5-7B, a specialized codeLLM built upon the Qwen1.5 language model. CodeQwen1.5-7B has been pretrained with around 3 trillion tokens of code-related data. It supports an extensive repertoire of 92 programming languages, and it exhibits exceptional capacity in long-context understanding and generation with the ability to process information of 64K tokens. In terms of performance, CodeQwen1.5 demonstrates impressive capabilities in basic code generation, long-context modeliing, code editation and SQL. We believe this model can significantly enhance developer productivity and streamline software development workflows within diverse technological environments.

CodeQwen is a Basic Coder

Code generation is a key competence for large language models, as they are tasked with translating natural language instructions into executable code with unwavering precision. CodeQwen1.5, with only 7 billion parameters, has surpassed larger models in basic code generation capabilities, further narrowing the gap in coding proficiency between GPT-4 and opensource code LLMs. We conducted a thorough evaluation on HumanEval and MBPP to provide a clear and fair comparison as follows.

| Model | Size | HumanEval
0-shot | HumanEval+
0-shot | MBPP
0-shot | MBPP+
0-shot | MBPP
3-shot |

| --- | --- | --- | --- | --- | --- | --- |
| Base Model | | | | | | |
| CodeLlama-Base | 7B | 33.5 | 25.6 | 52.1 | 41.6 | 38.6 |
| StarCoder2 | 7B | 35.4 | 29.9 | 54.4 | 45.6 | 51.0 |
| DeepSeek-Coder-Base | 6.7B | 47.6 | 39.6 | 70.2 | 56.6 | 60.6 |
| CodeQwen1.5 | 7B | 51.8 | 45.7 | 72.2 | 60.2 | 61.8 |
| Chat Model | | | | | | |
| GPT-3.5-Turbo | - | 76.8 | 70.7 | 82.5 | 69.7 | 70.8 |
| GPT-4-Turbo (Nov 2023) | - | 85.4 | 81.7 | 83.5 | 70.7 | 80.0 |
| DeepSeek-Coder-Instruct | 6.7B | 78.6 | 70.1 | 73.2 | 63.4 | 65.4 |
| CodeQwen1.5-Chat | 7B | 83.5 | 78.7 | 77.7 | 67.2 | 70.6 |

In addition to the widely recognized HumanEval and MBPP benchmarks, we explored LiveCodeBench. This benchmark assesses code performance by introducing fresh challenges sourced from coding competitions such as LeetCode, AtCoder, and CodeForces over time. Our evaluation of CodeQwen1.5 on LiveCodeBench spanned from September 1, 2023, to April 1, 2024. The findings indicate that CodeQwen1.5 ranks among the top open-access models currently available. Note: it is possible that the inclusion of LeetCode data in our pretraining corpus may contribute to the performance in LiveCodeBench.

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/codeqwen1.5/livecode.png#center" width="75%">}}

The evaluations mentioned primarily revolve around Python capabilities; however, CodeQwen1.5 is not merely a Python specialist but also an expert across multiple programming languages. We conducted a comprehensive evaluation of CodeQwen1.5 in the eight mainstream languages featured in MultiPL-E, including Python, C++, Java, PHP, TypeScript, C#, Bash, and JavaScript. The results highlight the exceptional programming capabilities of CodeQwen1.5.

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/codeqwen1.5/radar-vertical.png#center" width="40%">}}

CodeQwen is a Long Context Coder

Long context capability is crucial for code LLMs, serving as the core skill for understanding repository-level code and becoming a code agent. However, current code models still have very limited support for length, which hinders their potential for practical application. CodeQwen1.5 aims to further advance the progress of open-source code models in long context modeling. To achieve this, we have collected and constructed long sequence code data at the repository level for pre-training. Through careful data proportioning and organization, we have enabled it to support input lengths of up to 64K tokens.

Evaluation 1: We collected high-quality repo from GitHub Trending repositories on 2024-3-28 that were not included in CodeQwen1.5's training data to observe the effectiveness of long context modeling. The following figure demonstrates that as the sequence length increases, CodeQwen1.5's Perplexity (PPL) still manages to maintain a downward trend.

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/codeqwen1.5/longcontext-ppl.png#center" width="90%">}}

Evaluation 2: We created a synthetic task called Needle in the Code, inspired by popular long-context evaluations in the text domain. In this task, we inserted a very simple custom function at various positions within a longer codebase (we chose Megatron to honor its contributions to open-source LLMs!) and tested whether the model could replicate this function at the end of the codebase. The figure below shows that CodeQwen is capable of successfully completing this task within a 64k length range.

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/codeqwen1.5/longcontext-needle.png#center" width="90%">}}

Both Evaluation 1 and Evaluation 2 serve as initial and foundational assessments. For the Chat model, we aim to evaluate its long context capabilities with more practical tasks. However, our objective is to examine the Chat model's capability to handle long contexts through more pragmatic, real-world evaluation tasks.

Evaluation 3: SWE Bench is a benchmark designed to assess the ability of Large Language Models (LLMs) or agents to tackle practical software development challenges. It presents contestants with a code repository and an associated issue, tasking them with generating a commit patch that resolves the issue effectively. The benchmark uniquely emphasizes the long-context processing capabilities of code LLMs, necessitating both deep comprehension of the given codebase and the generation of extensive, unit-test-passing code.

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/codeqwen1.5/swe-bench.png#center" width="95%">}}

Currently, participants in the SWE Bench competition predominantly are proprietary models. We introduce CodeQwen1.5 as an open-source model entry. Despite achieving a score of 0.89, CodeQwen1.5 surpasses ChatGPT-3.5, demonstrating the nascent yet promising competitiveness of open-source code models against their proprietary counterparts.

CodeQwen is a Debugger

An effective code assistant must demonstrate proficiency in both generating code in response to given specifications and adeptly modifying or debugging existing code to accommodate evolving requirements or rectify errors. In assessing CodeQwen1.5's proficiency in code modification tasks, we concentrated our evaluation on the CodeEditorBench suite, encompassing four distinct dimensions: Debugging, Translation, Language Switching, and Code Polishing. The results indicate that CodeQwen1.5 achieves the SOTA performance at the 7 billion parameter scale.

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/codeqwen1.5/codeedit.jpeg#center" width="90%">}}

CodeQwen is a SQLer

CodeQwen1.5 serves as a solution to bridge the gap between non-programming professionals and efficient data interaction. It alleviates the steep learning curve associated with SQL by enabling users without coding expertise to query databases through natural language. We evaluated CodeQwen1.5-Chat's performance on two popular Text-to-SQL benchmarks, Spider and Bird. Experimental results pose CodeQwen1.5 a second position close to GPT-4 (results come from DIN-SQL, a SOTA prompting method). This outstanding performance is attributed to the utilization of synthetic data throughout both pre-training and fine-tuning stages. Synthetic data, characterized by its scalability, verifiability, and variety, emerges as a compelling area for future research due to its proven effectiveness in enhancing CodeQwen1.5's SQL capabilities.

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/codeqwen1.5/sql-score.png#center" width="90%">}}

Develop with CodeQwen1.5

CodeQwen1.5 is part of the Qwen1.5 open-source family. We advise you to read our blog for Qwen1.5 to figure out the usages with Transformers, vLLM, llama.cpp, Ollama, etc.

Conclusion

We have released CodeQwen1.5-7B and CodeQwen1.5-7B-Chat, an open and versatile code LLM. The models are intended to aid progress in code assistance and code agents, benefiting the research community. We'll keep investing heavily in smart code development, with the ultimate goal of creating AI programmers.

Citation

@misc{codeqwen1.5,
    title = {Code with CodeQwen1.5},
    url = {https://qwenlm.github.io/blog/codeqwen1.5/},
    author = {Qwen Team},
    month = {April},
    year = {2024}
}

File: content/blog/codeqwen1.5/index.zh.md Copy Chapter

title: "与 CodeQwen1.5 结对编程"
date: 2024-04-16T13:33:00+08:00
weight: 1

aliases: ["/first"]

tags: ["Research"]

draft: false

comments: false

disable_share: false

hide_meta: false

hide_summary: false # to hide summary in list

hide_footer: false

math: true

search_hidden: false # to hide from search page

show_reading_time: true
show_bread_crumbs: true
show_post_nav_links: false # the prev/next after the content
show_code_copy_buttons: true
show_word_count: true

use_hugo_toc: true

show_toc: true

toc_open: true # default expand all

cover:

image: "path"

# can also paste direct link from external site

# ex. https://i.ibb.co/K0HVPBd/paper-mod-profilemode.png

alt: ""

caption: ""

relative: true # To use relative path for cover image, used in hugo Page-bundles

responsive_images: true

hidden: false

header:

background: "" # background css value

background_image: ""

gradient: false

blur: false

{{< button href="https://github.com/QwenLM/CodeQwen1.5" label="GITHUB" external=true >}}
{{< button href="https://huggingface.co/Qwen" label="HUGGING FACE" external=true >}}
{{< button href="https://modelscope.cn/organization/qwen" label="MODELSCOPE" external=true >}}
{{< button href="https://huggingface.co/spaces/Qwen/CodeQwen1.5-7b-Chat-demo" label="DEMO" external=true >}}
{{< button href="https://discord.gg/yPEP2vHTu4" label="DISCORD" external=true >}}

简介

代码助手，是一种基于 LLMs 的智能化的编程工具，它可以帮助程序员更高效、更准确的编写代码，使得整个软件开发过程更加流畅和高效。然而流行的代码助手，比如 Github Copilot，依赖于闭源的商业模型，不仅昂贵还会引起如隐私、安全、版权等方面的担忧。幸运的是，开源社区正在致力于打造开放代码模型来实现开放的代码助手。近期涌现出了一批优秀的 Open CodeLLMs，比如 StarCoder2、CodeLlama、DeepSeek-Coder 等，提供了一条新的路径，但仍然值得探索。

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/codeqwen1.5/intro.png#center" width="100%">}}

今天，我们非常激动地和大家介绍来自 Qwen1.5 开源家族的新成员，一个代码专家模型 CodeQwen1.5! CodeQwen1.5 基于 Qwen 语言模型初始化，拥有 7B 参数的模型，其拥有 GQA 架构，经过了 ~3T tokens 代码相关的数据进行预训练，共计支持 92 种编程语言、且最长支持 64K 的上下文输入。效果方面，CodeQwen1.5 展现出了非凡的代码生成、长序列建模、代码修改、SQL 能力等,该模型可以大大提高开发人员的工作效率，并在不同的技术环境中简化软件开发工作流程。

CodeQwen 是基础的 Coder

代码生成是大语言模型的关键能力之一，期待模型将自然语言指令转换为具有精确的、可执行的代码。仅拥有 70 亿参数的 CodeQwen1.5 在基础代码生成能力上已经超过了更尺寸的模型，进一步缩小了开源代码 LLM 和 GPT-4 之间的编码能力差距。我们对 HumanEval 和 MBPP 进行了评估，下面是具体的比较。

| Model | Size | HumanEval
0-shot | HumanEval+
0-shot | MBPP
0-shot | MBPP+
0-shot | MBPP
3-shot |

| --- | --- | --- | --- | --- | --- | --- |
| Base Model | | | | | | |
| CodeLlama-Base | 7B | 33.5 | 25.6 | 52.1 | 41.6 | 38.6 |
| StarCoder2 | 7B | 35.4 | 29.9 | 54.4 | 45.6 | 51.0 |
| DeepSeek-Coder-Base | 6.7B | 47.6 | 39.6 | 70.2 | 56.6 | 60.6 |
| CodeQwen1.5 | 7B | 51.8 | 45.7 | 72.2 | 60.2 | 61.8 |
| Chat Model | | | | | | |
| GPT-3.5-Turbo | - | 76.8 | 70.7 | 82.5 | 69.7 | 70.8 |
| GPT-4-Turbo (Nov 2023) | - | 85.4 | 81.7 | 83.5 | 70.7 | 80.0 |
| DeepSeek-Coder-Instruct | 6.7B | 78.6 | 70.1 | 73.2 | 63.4 | 65.4 |
| CodeQwen1.5-Chat | 7B | 83.5 | 78.7 | 77.7 | 67.2 | 70.6 |

除了流行的 Humaneval 与 MBPP 外，我们还注意到了 LiveCodeBench，一个对 LLM 代码能力进行更全面、随着时间动态更新的评估。LiveCodeBench 不断地从 LeetCode、AtCoder 和 CodeForces 三个竞赛平台中收集问题来测试模型的泛化能力。我们选择在 LiveCodeBench (2023-09-01->2024-04-01)上对 CodeQwen1.5 进行评估，结果展示出了 CodeQwen1.5 极具竞争力的效果。但值得注意的是，在预训练语料中包含的 LeetCode 数据可能对该评测有帮助。

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/codeqwen1.5/livecode.png#center" width="75%">}}

上述的评估主要围绕 Python 能力，但 CodeQwen1.5 不仅仅是 Python 专家，还是一个多编程语言专家。我们在 MultiPL-E 的 8 种主流语言（Python、C++、Java、PHP、TypeScript、C#、Bash，JavaScript）上对 CodeQwen1.5 进行全面评估。这些结果证明了 CodeQwen1.5 强大的编程能力。

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/codeqwen1.5/radar-vertical.png#center" width="40%">}}

CodeQwen 是长序列 Coder

长序列能力对于代码模型来说至关重要，是理解仓库级别代码、成为 Code Agent 的核心能力。而当前的代码模型对于长度的支持仍然非常有限，阻碍了其实际应用的潜力。CodeQwen1.5 希望进一步推进开源代码模型在长序列建模上的进展，我们收集并构造了仓库级别的长序列代码数据进行预训练，通过精细的数据配比和组织方式，使其最终可以最长支持 64K 的输入长度。

评估一：我们选择了不在 CodeQwen1.5 训练数据、最新产生的高质量 github 仓库 （来自 2024-3-28 的 Github Trending 仓库），来观测其长序列建模的有效性。下图可以发现在序列不断增长的情况下， CodeQwen1.5 的 PPL 仍然可以保持下降的趋势。

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/codeqwen1.5/longcontext-ppl.png#center" width="90%">}}

评估二：一个名为 Needle in the Code 的合成任务， 其效仿文本领域流行的长序列评测。我们在一个较长的代码库（我们选择了 Megatron，向其对开源 LLMs 的贡献致敬）的不同位置中插入非常简单的一个自定义函数，测试模型能否在代码库最后重复这个函数。下图可以发现，CodeQwen 能够在 64k 长度范围内仍然可以很好的完成这个任务。

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/codeqwen1.5/longcontext-needle.png#center" width="90%">}}

无论是评估一还是评估二，都是初步的、基础的评估方式，仅仅是一个起点而非全部。但是，对于 Chat 模型，我们希望用更实际的任务来评估其长序列能力。

评估三：SWE Bench 的目的是解决真实软件开发中的问题，给定一个代码仓库和 issue，期待 LLMs/Agents 能够给出相应的 commit patch 来解决这个 issue。SWE Bench 对 Code LLMs 的长序列能力提出了更高的要求，不仅需要理解代码仓库，还要生成可通过单测的代码。

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/codeqwen1.5/swe-bench.png#center" width="95%">}}

目前 SWEBench 竞技场上的玩家都依赖闭源模型，我们让 CodeQwen1.5 首次入局，尽管仅有 0.89 的分数但仍强于 ChatGPT3.5，这展示了开源代码模型与专有模型的竞争力尽管尚处于初期，但具有潜力。

CodeQwen 是优秀的代码修改者

一个好的代码助手不仅可以根据指令生成代码，还能够针对已有代码或者新的需求进行修改或错误修复。为此，我们评估了 CodeQwen1.5 在代码修改方面的能力。我们首先在关注 CodeEditorBench，涉及到 Debug、Translate、Switch、Polish 等四个方面的代码修改能力，结果表明 CodeQwen1.5 在 7B 规模上达到了最好的效果。

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/codeqwen1.5/codeedit.jpeg#center" width="90%">}}

CodeQwen 是出色的 SQL 专家

CodeQwen1.5 可以作为一个智能的 SQL 专家，弥合了非编程专业人士与高效数据交互之间的差距。它通过自然语言使无编程专业知识的用户能够查询数据库，从而缓解了与SQL相关的陡峭学习曲线。我们在两个流行的文本到SQL基准测试Spider和Bird上评估了CodeQwen1.5-Chat的性能。实验结果显示，CodeQwen1.5在接近GPT-4的位置排名第二（结果来自DIN-SQL，一种 SOTA 的提示方法）。这一出色的表现得益于在预训练和微调阶段均广泛利用了合成数据。合成数据具有可扩展性、可验证性和多样性的特点，在增强CodeQwen1.5的SQL能力方面已被证明是一项具有吸引力的未来研究领域。

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/assets/blog/codeqwen1.5/sql-score.png#center" width="90%">}}

部署 CodeQwen1.5

CodeQwen1.5 隶属 Qwen1.5 开源家族，我们推荐您阅读我们的 Qwen1.5 文档来了解具体的使用方式，目前已支持 Transformers, vLLM, llama.cpp, Ollama, 等等。

结论

我们发布了 CodeQwen1.5-7B 及 CodeQwen1.5-7B-Chat，一个开放的、多面体的 Code LLM，我们希望这个模型能在 Code 助手、Code Agent 等方面为社区贡献。未来我们仍然会积极的投入代码智能建设，实现真正的 AI 程序员。

引用

@misc{codeqwen1.5,
    title = {Code with CodeQwen1.5},
    url = {https://qwenlm.github.io/blog/codeqwen1.5/},
    author = {Qwen Team},
    month = {April},
    year = {2024}
}

File: content/blog/global-load-balance/index.md Copy Chapter

title: "Global-batch load balance almost free lunch to improve your MoE LLM training"
date: 2025-01-21T00:00:03+08:00
weight: 1

aliases: ["/first"]

tags: ["Research"]

draft: true

comments: false

disable_share: false

hide_meta: false

hide_summary: false # to hide summary in list

hide_footer: false

math: true

search_hidden: false # to hide from search page

show_reading_time: true
show_bread_crumbs: true
show_post_nav_links: false # the prev/next after the content
show_code_copy_buttons: true
show_word_count: true

use_hugo_toc: true

show_toc: true

toc_open: true # default expand all

cover:

image: "path"

# can also paste direct link from external site

# ex. https://i.ibb.co/K0HVPBd/paper-mod-profilemode.png

alt: ""

caption: ""

relative: true # To use relative path for cover image, used in hugo Page-bundles

responsive_images: true

hidden: false

header:

background: "" # background css value

background_image: ""

gradient: false

blur: false

{{< figure src="https://qianwen-res.oss-cn-beijing.aliyuncs.com/assets/balance/main_results.png#center" width="100%">}}

{{< button href="https://github.com/QwenLM/Qwen2-Math" label="GITHUB" external=true >}}
{{< button href="https://huggingface.co/collections/Qwen/qwen25-math-66eaa240a1b7d5ee65f1da3e" label="HUGGING FACE" external=true >}}
{{< button href="https://modelscope.cn/organization/qwen" label="MODELSCOPE" external=true >}}
{{< button href="https://discord.gg/yPEP2vHTu4" label="DISCORD" external=true >}}

Background Copy Chapter

The Mixture-of-Experts (MoEs) architecture has become a popular model-parameter-scale-up technique. Typically, one MoE layer consists of a router (often parameterized as one single Linear layer) and a group of experts (for transformer-based models, each expert is one feedforward layer). Given an input, only a subset of experts will be activated, and then their outputs will be aggregated based on the scores the router assigned. Specifically,

$$
\mathbb{y}=\sum_{i\in N_E,g_i\in\operatorname{topK}}g_i(\mathbb{x})E_i(\mathbb{x})
$$

Load Balancing loss Copy Section

Load balancing loss is an essential regularization technique in training MoE-based networks, and high-level intuition encourages the balanced activation of all experts. It can be calculated as:

$$
L_{\text{balance}}=N_E \sum_{i=1}^{N_E} f_ip_i
$$

where $f_i$ is the activation frequency of the expert $E_i$, and the $p_i$ is the the average gating score that the expert $E_i$ is assigned.

However, most existing MoE training frameworks (e.g., Megatron-core), implement micro-batch level balance, which means the $L_{\text{balance}}$ is calculated within every micro-batch and is then averaged on the global batch level.

Our key point is that this implementation could be problematic if one micro-batch does not contain diverse data. For instance, imagine one micro-batch only contains some code data; the aforementioned load-balancing loss still pushes the router to distribute these code tokens to all experts uniformly, potentially hurting the model performance and preventing expert specialization.

This situation is even more common in training MoE-based LLMs: the data in one micro-batch is often from the same domain. This partially explains why most existing open-source MoE-based LLMs do not achieve notable expert specialization.

This drawback motivates us to extend the current method to the global-batch level balance.

From micro-batch balance to global-batch balance Copy Chapter

One easy way to calculate global-batch balance loss is to 1) Synchronize expert selection frequency $f_i$ across all parallel groups; 2) Calculate the load-balancing loss in each parallel group (e.g., one GPU); 3) Aggregate the loss across all micro-batches.

Specifically,

$$
L_{\text{global}}=N_E\sum_{i=1}^{N_E}f_i^{\text{global}}p_i^{\text{global}}=N_E\sum_{i=1}^{n_E}f_i^{\text{global}}\cdot(\frac{1}{N_p}\sum_{j=1}^{N_p}p_j)= \frac{1}{N_P} \sum_{j=1}^{N_p}(N_E \sum_{i=1}^{N_E} \bar{f_i} \cdot P^j_i)
$$

Note that the expert selection frequency is just one expert-num-dimentional vector! It is almost free to synchronize them across micro-batches.

Results: More Performant and Interpretable MoE Copy Chapter

We experiment with three MoE configs (3.4B with 0.6B activated, 15B with 2.54B activated, and 43B with 6.6B activated) and two data configs (120B tokens and 400B tokens). The results are shown in the following figure and chart.

In short, compared to the micro-batch-level loss, the global batch one achieves better performance in all settings (model, data, and tasks). More importantly, the MoE model achieves significant domain specialization with the global batch balance. In Figure (b) left, almost all experts are uniformly activated regardless of the domain. But in Figure (b) right, some experts are frequently activated by a specific domain, demonstrating their specialization.

{{< figure src="https://qianwen-res.oss-cn-beijing.aliyuncs.com/assets/balance/table_results.png#center" width="90%">}}

We further compare the model performance with balance batch size on a 3.4B with 0.6B activated model. The ptr-training PPL decreases rapidly from balance BSZ 2 to 128, and gradually saturates after 128. In the current mainstream MoE framework, even with cross expert-parallel groups communication, the balance BSZ is generally between 8 and 16 for larger models, which further reflects the significance of our method.

{{< figure src="https://qianwen-res.oss-cn-beijing.aliyuncs.com/assets/balance/different_balance_BSZ.png#center" width="70%">}}

Using global batch balance may result in a degradation in micro-batch balance, potentially affecting the computational efficiency of MoE. We further experimented with the effect of adding micro-batch balance loss on top of the global-batch balance loss (with constant weights of 0.01 of the global batch loss). It can be seen that adding local equalization improves the speed of the model (from 1.64 to 1.59 seconds per update step) while the effectiveness of the model is almost unaffected.

{{< figure src="https://qianwen-res.oss-cn-beijing.aliyuncs.com/assets/balance/efficiency.png#center" width="70%">}}

Conclusion Copy Chapter

In summary, we investigate the challenges associated with the LBL in training MoEs models. By introducing global-batch balance loss, we achieve improved performance and foster expert specialization within the MoE model. We believe this advancement addresses an essential limitation in existing MoE training, offering a novel perspective for MoEs model optimization.
Though mainly experimenting with language-based tasks, we hope our work could pave the way for training more substantial and specialised MoE models in various domains.

Citation

If you find our work helpful, feel free to give us a citation.

@article{qiu2025demonsdetailimplementingload,
  title={Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models}, 
  author={
    Zihan Qiu and Zeyu Huang and Bo Zheng and Kaiyue Wen and Zekun Wang and Rui Men and Ivan Titov and Dayiheng Liu and Jingren Zhou and Junyang Lin
  },
  journal={arXiv preprint arXiv:2501.11873},
  year={2025}
}

File: content/blog/global-load-balance/index.zh.md Copy Chapter

title: "通过全局负载均衡提升混合专家模型的性能和特异化程度"
date: 2025-01-21T00:00:03+08:00
weight: 1

aliases: ["/first"]

tags: ["Research"]

draft: true

comments: false

disable_share: false

hide_meta: false

hide_summary: false # to hide summary in list

hide_footer: false

math: true

search_hidden: false # to hide from search page

show_reading_time: true
show_bread_crumbs: true
show_post_nav_links: false # the prev/next after the content
show_code_copy_buttons: true
show_word_count: true

use_hugo_toc: true

show_toc: true

toc_open: true # default expand all

cover:

image: "path"

# can also paste direct link from external site

# ex. https://i.ibb.co/K0HVPBd/paper-mod-profilemode.png

alt: ""

caption: ""

relative: true # To use relative path for cover image, used in hugo Page-bundles

responsive_images: true

hidden: false

header:

background: "" # background css value

background_image: ""

gradient: false

blur: false

{{< figure src="https://qianwen-res.oss-cn-beijing.aliyuncs.com/assets/balance/main_results.png#center" width="100%">}}

{{< button href="https://github.com/QwenLM/Qwen2-Math" label="GITHUB" external=true >}}
{{< button href="https://huggingface.co/collections/Qwen/qwen25-math-66eaa240a1b7d5ee65f1da3e" label="HUGGING FACE" external=true >}}
{{< button href="https://modelscope.cn/organization/qwen" label="MODELSCOPE" external=true >}}
{{< button href="https://discord.gg/yPEP2vHTu4" label="DISCORD" external=true >}}

**引言** Copy Chapter

混合专家模型（MoEs）通过路由机制动态并稀疏地激活模型参数，使得能高效地增大模型参数规模。基于 TopK 机制的稀疏激活会在训练中会遇到专家激活不均衡的问题：少数被频繁选择的专家会被优化得更多，进一步使得这些专家被更频繁地选择，最终导致只选择少数专家，造成剩余专家的冗余。因此，MoE 在训练中需要引入额外的辅助损失（load balance loss，LBL）来鼓励专家的选择趋于均衡。

目前主流 MoE 训练框架中 LBL 实现其实是局部的负载均衡，这使得模型需要将局部的输入都均匀分配给不同的专家。然而，局部的输入往往只来自个别领域，局部负载均衡会让模型将所有领域的输入都均匀分配。这种均匀分配会阻碍某些专家更多处理特定领域的数据，也即阻碍专家出现领域层次的分化特征。我们发现，将局部的负载均衡放松到全局的负载均衡，能显著增强专家的特异化并提高模型性能。

从局部均衡到全局均衡 Copy Chapter

LBL 的计算公式为 $N_E \sum_{i=1}^{N_E} f_ip_i$ ，其中 $N_E$ 为专家数， $f_i$ 为专家 i 被选择的频率， $p_i$ 为路由赋予专家 i 的平均分数。目前 Megatron-mcore 等主流框架中的 LBL 都是在每一张卡上统计计算后再全局平均，这使得 $f_i$ 也是在局部统计，而优化 LBL 也鼓励模型将每个局部的输入都均匀分配给所有的专家。这也解释了为什么目前大部分 MoE 工作没有观察到领域层面的专家分化。

得益于 LBL 计算的格式，我们可以通过通信不同节点的 $f_i$ 来将局部的 LBL 转化为全局的 LBL。因为 $f_i$ 只是一个专家数大小的向量，即使是在全局通信的情况下也不会带来明显的开销。此外由于 LBL 的计算与模型其它部分的计算相对独立，还可以用计算掩盖等策略进一步消除同步 $f_i$ 的通信开销。

扩大均衡的范围带来稳定的提升 Copy Chapter

我们在三种参数规模（3.4B 激活 0.6B, 15B 激活 2.54B，43B 激活 6.6B）下分别训练了 120B 和 400B tokens，对比了不同的均衡范围（Balance BSZ）对模型性能的影响。所有模型都使用了细粒度专家、共享专家及 dropless 策略（专家不会抛弃超过容量的tokens）。可以看到，将均衡范围从一般框架实现的 4，8 或者 16 增大到 128 以上后模型在 Benchmark 指标和 PPL 都有明显提升。

{{< figure src="https://qianwen-res.oss-cn-beijing.aliyuncs.com/assets/balance/table_results.png#center" width="90%">}}

我们在 3.4B 激活 0.6B 的模型训练 400B tokens 到设置上进一步对比了模型效果随着均衡范围的变化，可以看到 balance BSZ 从 2 到 128 模型的 PPL 在快速降低，在 128 后逐渐饱和。目前主流 MoE 框架中即使是进行了机内通信，对于较大的模型 balance BSZ 也一般在 8 到 16 的，这进一步体现了我们通信方法的意义。

{{< figure src="https://qianwen-res.oss-cn-beijing.aliyuncs.com/assets/balance/different_balance_BSZ.png#center" width="70%">}}

添加少量局部均衡损失能提高模型效率 Copy Chapter

只使用全局均衡会导致局部均衡状况有所降低，这会一定程度影响 MoE 的计算效率。我们进一步实验了在主要使用全局均衡的情况下，在训练过程中添加局部均衡（默认实现的 LBL，损失权重为全局 LBL 的 1%）限制对于模型性能和效率的影响。可以看到，添加局部均衡能提升模型的速度（每个更新步耗时从 1.64秒提升到1.59秒），同时模型的效果也几乎不受影响。

{{< figure src="https://qianwen-res.oss-cn-beijing.aliyuncs.com/assets/balance/efficiency.png#center" width="70%">}}

结论 Copy Chapter

我们回顾了目前 MoE 训练框架中均衡损失，发现目前的实现方式会将所有来自相同领域的局部输入都均匀分配，限制了专家的分化。通过轻量的通信将局部均衡放松为全局均衡，MoE 模型的性能和专家特异性都得到了显著的提升。我们认为这一进展解决了现有MoE训练中的一个关键问题，为MoE模型的优化提供了新的视角，并有助于构建更加可解释的模型。尽管我们的实验主要集中在基于语言的任务上，我们希望我们的工作能够为在不同领域训练更大规模、更有效的 MoE 模型提供帮助。

引用

如果你觉得我们的工作有用，欢迎引用！

@article{qiu2025demonsdetailimplementingload,
  title={Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models}, 
  author={
    Zihan Qiu and Zeyu Huang and Bo Zheng and Kaiyue Wen and Zekun Wang and Rui Men and Ivan Titov and Dayiheng Liu and Jingren Zhou and Junyang Lin
  },
  journal={arXiv preprint arXiv:2501.11873},
  year={2025}
}

File: content/blog/gspo/index.md Copy Chapter

title: "GSPO: Towards Scalable Reinforcement Learning for Language Models"
date: 2025-07-27T15:00:00+08:00
weight: 1

aliases: ["/first"]

tags: ["Research"]

draft: true

comments: false

disable_share: false

hide_meta: false

hide_summary: false # to hide summary in list

hide_footer: false

math: true

search_hidden: false # to hide from search page

show_reading_time: true
show_bread_crumbs: true
show_post_nav_links: false # the prev/next after the content
show_code_copy_buttons: true
show_word_count: true

use_hugo_toc: true

show_toc: true

toc_open: true # default expand all

cover:

image: "path"

# can also paste direct link from external site

# ex. https://i.ibb.co/K0HVPBd/paper-mod-profilemode.png

alt: ""

caption: ""

relative: true # To use relative path for cover image, used in hugo Page-bundles

responsive_images: true

hidden: false

header:

background: "" # background css value

background_image: ""

gradient: false

blur: false

{{< button href="https://huggingface.co/papers/2507.18071" label="PAPER" external=true >}}
{{< button href="https://discord.gg/yPEP2vHTu4" label="DISCORD" external=true >}}

Introduction Copy Chapter

Reinforcement Learning (RL) has emerged as a pivotal paradigm for scaling language models and enhancing their deep reasoning and problem-solving capabilities. To scale RL, the foremost prerequisite is maintaining stable and robust training dynamics. However, we observe that existing RL algorithms (such as GRPO) exhibit severe instability issues during long training and lead to irreversible model collapse, hindering further performance improvements with increased compute.

To enable successful RL scaling, we propose the Group Sequence Policy Optimization (GSPO) algorithm. Unlike previous RL algorithms, GSPO defines the importance ratio based on sequence likelihood and performs sequence-level clipping, rewarding, and optimization. Compared to GRPO, GSPO demonstrates remarkable advantages in the following aspects:

• Performant and Efficient: GSPO possesses significantly higher training efficiency and can achieve continuous performance improvements through increasing training compute;
• Notably Stable: GSPO maintains stable training processes and inherently resolves the stability challenges in the RL training of large Mixture-of-Experts (MoE) models;
• Infrastructure-Friendly: Due to sequence-level optimization, GSPO is fundamentally more tolerant to precision discrepancies, offering attractive potential for simplifying RL infrastructure.

These merits have contributed to the exceptional performance of the latest Qwen3 models (Instruct, Coder, Thinking).

Sequence-Level Optimization Objective Copy Chapter

Let $x$ be a query, $\pi_{\theta_\mathrm{old}}$ be the old policy that generates responses, $\{y_i\}_{i=1}^G$ be the sampled response group, $\widehat{A}_{i}$ be the group relative advantage of each response, and $\pi_\theta$ be the current policy to be optimized. GSPO adopts the following optimization objective:

{{< rawhtml >}}
$$
\mathcal{J}\text{GSPO} (\theta)
=,
\mathbb{E}{ x \sim \mathcal{D},, {y_i}{i=1}^G \sim \pi{\theta_\mathrm{old}}( \cdot | x) }
\left[
\frac{1}{G} \sum_{i=1}^{G}
\min \left( s_{i}(\theta) \widehat{A}{i}, , \mathrm{clip} \left( s{i}(\theta), 1 - {\varepsilon}, 1 + {\varepsilon} \right) \widehat{A}_{i} \right)
\right],
$$
{{< /rawhtml >}}

where

{{< rawhtml >}} $$ s_{i}(\theta)

\left( \frac{ \pi_{\theta} (y_i | x) }{ \pi_{\theta_\text{old}} (y_i | x)} \right)^{\frac{1}{|y_i|}}

\exp \left( \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \log \frac{ \pi_{\theta} (y_{i,t} | x, y_{i,<t}) }{ \pi_{\theta_\text{old}} (y_{i,t} | x,y_{i,<t})} \right).
$$
{{< /rawhtml >}}

Here, $s_i(\theta)$ is the importance ratio defined based on sequence likelihood in GSPO, where we perform length normalization to reduce variance and unify the numerical range of $s_i(\theta)$.

Training Efficiency and Performance Copy Chapter

We experiment with a cold-start model fine-tuned from Qwen3-30B-A3B-Base and report its training reward curves as well as performance curves on the AIME'24, LiveCodeBench, and CodeForces benchmarks. We compare against GRPO as the baseline. Note that GRPO necessitates the Routing Replay training strategy for the normal convergence of MoE RL (which we will discuss later), while GSPO has obviated the need for this strategy.

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/results.jpg#center" title="Experimental results">}}

As shown in the figure above, GSPO demonstrates significantly higher training efficiency than GRPO, achieving better performance under the same training cost. Particularly, we observe that GSPO can deliver continuous performance improvement through increasing the training compute, regularly updating the query set, and extending the generation length — this is exactly the scalability we expect from an algorithm. Ultimately, we successfully applied GSPO to the large-scale RL training of the latest Qwen3 models, further unleashing the potential of RL scaling!

An interesting observation is that the fraction of tokens clipped in GSPO is two orders of magnitude higher than that in GRPO (as shown in the figure below), while GSPO still achieves higher training efficiency. This further demonstrates that GRPO's token-level optimization objective is noisy and inefficient, while GSPO's sequence-level approach provides a more reliable and effective learning signal.

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/clipping.jpg" title="Fractions of clipped tokens">}}

Benefits for MoE RL and Infrastructure Copy Chapter

We found that when adopting the GRPO algorithm, the expert activation volatility of MoE models prevents RL training from converging properly. To address this challenge, we previously employed the Routing Replay training strategy, which caches the activated experts in $\pi_{\theta_\text{old}}$ and "replays" these routing patterns in $\pi_\theta$ when computing importance ratios. As shown in the figure below, Routing Replay is crucial for normal convergence of GRPO training on MoE models. However, the Routing Replay strategy incurs additional memory and communication overhead and may limit the actual capacity of MoE models.

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/routing_replay.jpg" title="Effect of Routing Replay in the GRPO training of MoE models">}}

The notable advantage of GSPO lies in completely eliminating the dependency on Routing Replay. The key insight is that GSPO only focuses on sequence-level likelihood (i.e., $\pi_\theta(y_i|x)$) and is not sensitive to individual token likelihood (i.e., $\pi_\theta(y_{i,t}|x,y_{i,<t})$). Therefore, it does not require infrastructure-heavy workarounds like Routing Replay, both simplifying and stabilizing the training process while allowing models to maximize their capacity.

Additionally, since GSPO uses only sequence-level rather than token-level likelihoods for optimization, intuitively the former is much more tolerant of precision discrepancies. Therefore, GSPO makes it possible to directly use likelihoods returned by inference engines for optimization, eliminating the need for recomputation with training engines. This is particularly beneficial in scenarios such as partial rollout, multi-turn RL, and training-inference disaggregated frameworks.

Conclusion Copy Chapter

We propose Group Sequence Policy Optimization (GSPO), a new RL algorithm for training language models. GSPO demonstrates notably superior training stability, efficiency, and performance compared to GRPO and exhibits particular efficacy for the large-scale RL training of MoE models, laying the foundation for the exceptional improvements in the latest Qwen3 models. With GSPO as our algorithmic cornerstone, we will continue to push the boundaries of RL scaling and look forward to the resulting fundamental advances in intelligence.

Citation Copy Chapter

If you find our work helpful, feel free to give us a citation.

@article{gspo,
  title={Group Sequence Policy Optimization}, 
  author={
    Chujie Zheng and Shixuan Liu and Mingze Li and Xiong-Hui Chen and Bowen Yu and 
    Chang Gao and Kai Dang and Yuqiong Liu and Rui Men and An Yang and Jingren Zhou and 
    Junyang Lin 
  },
  journal={arXiv preprint arXiv:2507.18071},
  year={2025}
}

File: content/blog/gspo/index.zh.md Copy Chapter

title: "GSPO：迈向持续拓展的语言模型强化学习"
date: 2025-07-27T15:00:00+08:00
weight: 1

aliases: ["/first"]

tags: ["Research"]

draft: true

comments: false

disable_share: false

hide_meta: false

hide_summary: false # to hide summary in list

hide_footer: false

math: true

search_hidden: false # to hide from search page

show_reading_time: true
show_bread_crumbs: true
show_post_nav_links: false # the prev/next after the content
show_code_copy_buttons: true
show_word_count: true

use_hugo_toc: true

show_toc: true

toc_open: true # default expand all

cover:

image: "path"

# can also paste direct link from external site

# ex. https://i.ibb.co/K0HVPBd/paper-mod-profilemode.png

alt: ""

caption: ""

relative: true # To use relative path for cover image, used in hugo Page-bundles

responsive_images: true

hidden: false

header:

background: "" # background css value

background_image: ""

gradient: false

blur: false

{{< button href="https://huggingface.co/papers/2507.18071" label="PAPER" external=true >}}
{{< button href="https://discord.gg/yPEP2vHTu4" label="DISCORD" external=true >}}

引言 Copy Chapter

强化学习 （Reinforcement Learning，RL）已成为拓展语言模型、增强其深度推理与问题求解能力的关键技术范式。为了持续拓展 RL，首要前提是确保稳定、鲁棒的训练过程。然而，我们观察到现有的 RL 算法（如 GRPO）在长期训练中会暴露出严重的不稳定性问题并招致不可逆转的模型崩溃，阻碍了通过增加计算以获得进一步的性能提升。

为了能够持续拓展 RL，我们提出了 Group Sequence Policy Optimization (GSPO) 算法。不同于过去的 RL 算法，GSPO 定义了序列级别的重要性比率，并在序列层面执行裁剪、奖励和优化。相较于 GRPO，GSPO 在以下方面展现出突出优势：

• 强大高效：GSPO 具备显著更高的训练效率，并且能够通过增加计算获得持续的性能提升；
• 稳定性出色：GSPO 能够保持稳定的训练过程，并且根本地解决了混合专家（Mixture-of-Experts，MoE）模型的 RL 训练稳定性问题；
• 基础设施友好：由于在序列层面执行优化，GSPO 原则上对精度容忍度更高，具有简化 RL 基础设施的诱人前景。

以上优点促成了最新的 Qwen3 模型（Instruct、Coder、Thinking）的卓越性能。

序列级别的优化目标 Copy Chapter

设 $x$ 为查询，$\pi_{\theta_\mathrm{old}}$ 为用于采样回复的策略，$\{y_i\}_{i=1}^G$ 为采样得到的回复组，$\widehat{A}_{i}$ 为各个回复的组内相对优势，$\pi_\theta$ 为需优化的当前策略。GSPO 采用以下优化目标：

{{< rawhtml >}} $$ \mathcal{J}_\text{GSPO} (\theta)

\mathbb{E}{ x \sim \mathcal{D},, {y_i}{i=1}^G \sim \pi_{\theta_\text{old}}( \cdot | x) }
\left[
\frac{1}{G} \sum_{i=1}^{G}
\min \left( s_{i}(\theta) \widehat{A}{i}, , \mathrm{clip} \left( s{i}(\theta), 1 - {\varepsilon}, 1 + {\varepsilon} \right) \widehat{A}_{i} \right)
\right],
$$
{{< /rawhtml >}}

其中

{{< rawhtml >}} $$ s_{i}(\theta)

\left( \frac{ \pi_{\theta} (y_i | x) }{ \pi_{\theta_\text{old}} (y_i | x)} \right)^{\frac{1}{|y_i|}}

\exp \left( \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \log \frac{ \pi_{\theta} (y_{i,t} | x, y_{i,<t}) }{ \pi_{\theta_\text{old}} (y_{i,t} | x,y_{i,<t})} \right).
$$
{{< /rawhtml >}}

这里的 $s_i(\theta)$ 即为 GSPO 基于序列似然定义的重要性比率，其中我们进行了长度归一化以降低方差并统一 $s_i(\theta)$ 的数值范围。

训练效率与性能 Copy Chapter

我们使用基于 Qwen3-30B-A3B-Base 微调得到的冷启动模型进行实验，并汇报其训练奖励曲线以及在 AIME'24、LiveCodeBench 和 CodeForces 等基准上的性能曲线。我们对比 GRPO 作为基线。注意 GRPO 必需采用 Routing Replay 训练策略才能正常收敛（我们将在后文讨论），而 GSPO 则无需该策略。

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/results.jpg#center" title="实验结果">}}

从上图可见，GSPO 表现出比 GRPO 显著更高的训练效率，即在同等计算开销下能够取得更优的性能。特别地，我们观察到 GSPO 可以通过增加算力来获得持续的性能提升——这正是我们所期待的算法的可拓展性。最终，我们成功地将 GSPO 应用于最新的 Qwen3 模型的大规模 RL 训练，进一步释放了 RL scaling 的潜能！

一个有趣的观察是，GSPO 所裁剪的 token 比例比 GRPO 要高上两个数量级（如下图所示），但却具有更高的训练效率。这进一步表明 GRPO 采用的 token 级别的优化目标是有噪和低效的，而 GSPO 的序列级别的优化目标则提供了更可靠、有效的学习信号。

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/clipping.jpg#center" title="被裁剪的 token 比例">}}

对 MoE RL 和基础设施的收益 Copy Chapter

我们发现，当采用 GRPO 算法时，MoE 模型的专家激活波动性会使得 RL 训练无法正常收敛。为了解决这一挑战，我们过去采用了**路由回放（Routing Replay）**训练策略，即缓存 $\pi_{\theta_\text{old}}$ 中激活的专家，并在计算重要性比率时在 $\pi_\theta$ 中“回放”这些路由模式。从下图可见，Routing Replay 对于 GRPO 训练 MoE 模型的正常收敛至关重要。然而，Routing Replay 的做法会产生额外的内存和通信开销，并可能限制 MoE 模型的实际可用容量。

{{< figure src="https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/routing_replay.jpg#center" title="Routing Replay 对 MoE 模型的 GRPO 训练的作用">}}

GSPO 的一大突出优势在于彻底消除了对 Routing Replay 的依赖。其核心洞见在于：GSPO 仅关注序列级别的似然（即 $\pi_\theta(y_i|x)$），而对个别 token 的似然（即 $\pi_\theta(y_{i,t}|x,y_{i,<t})$）不敏感。因此，其无需 Routing Replay 等对基础设施负担较大的手段，既简化和稳定了训练过程，又使得模型能够最大化地发挥容量与潜能。

此外，鉴于 GSPO 仅使用序列级别而非 token 级别的似然进行优化，直观上前者对精度差异的容忍度要高得多。因此，GSPO 使得直接使用推理引擎返回的似然进行优化成为可能，从而无需使用训练引擎进行重计算，这在 partial rollout、多轮 RL 以及训推分离框架等场景中特别有益。

结论 Copy Chapter

我们提出了 Group Sequence Policy Optimization (GSPO)，这是用于训练语言模型的全新 RL 算法。相较于 GRPO，GSPO 在训练稳定性、效率和性能方面展现出显著优势，并在 MoE 模型的大规模 RL 训练中表现出突出的功效。这些优点为最新 Qwen3 模型的卓越性能奠定了算法基础。以 GSPO 作为算法基石，我们将持续推动 RL scaling 的边界，并期待由此带来的智能进步。

引用 Copy Chapter

如果你觉得我们的工作有用，欢迎引用！

@article{gspo,
  title={Group Sequence Policy Optimization}, 
  author={
    Chujie Zheng and Shixuan Liu and Mingze Li and Xiong-Hui Chen and Bowen Yu and 
    Chang Gao and Kai Dang and Yuqiong Liu and Rui Men and An Yang and Jingren Zhou and 
    Junyang Lin 
  },
  journal={arXiv preprint arXiv:2507.18071},
  year={2025}
}