# Technical Documentation: QwenLM/Qwen2.5 > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/QwenLM/Qwen2.5) ### Overview (`README.md`) # Qwen3

💜 Qwen Chat   |   🤗 Hugging Face   |   🤖 ModelScope   |    📑 Paper    |    📑 Blog    ｜   📖 Documentation
🖥️ Demo   |   💬 WeChat (微信)   |   🫨 Discord  

Visit our Hugging Face or ModelScope organization (click links above), search checkpoints with names starting with `Qwen3-` or visit the [Qwen3 collection](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f), and you will find all you need! Enjoy! To learn more about Qwen3, feel free to read our documentation \[[EN](https://qwen.readthedocs.io/en/latest/)|[ZH](https://qwen.readthedocs.io/zh-cn/latest/)\]. Our documentation consists of the following sections: - Quickstart: the basic usages and demonstrations; - Inference: the guidance for the inference with Transformers, including batch inference, streaming, etc.; - Run Locally: the instructions for running LLM locally on CPU and GPU, with frameworks like llama.cpp, Ollama, and LM Studio; - Deployment: the demonstration of how to deploy Qwen for large-scale inference with frameworks like SGLang, vLLM, TGI, etc.; - Quantization: the practice of quantizing LLMs with GPTQ, AWQ, as well as the guidance for how to make high-quality quantized GGUF files; - Training: the instructions for post-training, including SFT and RLHF (TODO) with frameworks like Axolotl, LLaMA-Factory, etc. - Framework: the usage of Qwen with frameworks for application, e.g., RAG, Agent, etc. ## Introduction ### Qwen3-2507 Over the past three months, we continued to explore the potential of the Qwen3 families and we are excited to introduce the updated **Qwen3-2507** in two variants, Qwen3-Instruct-2507 and Qwen3-Thinking-2507, and three sizes, 235B-A22B, 30B-A3B, and 4B. **Qwen3-Instruct-2507** is the updated version of the previous Qwen3 non-thinking mode, featuring the following key enhancements: - **Significant improvements** in general capabilities, including **instruction following, logical reasoning, text comprehension, mathematics, science, coding and tool usage**. - **Substantial gains** in long-tail knowledge coverage across **multiple languages**. - **Markedly better alignment** with user preferences in **subjective and open-ended tasks**, enabling more helpful responses and higher-quality text generation. - **Enhanced capabilities** in **256K-token long-context understanding**, extendable up to **1 million tokens**. **Qwen3-Thinking-2507** is the continuation of Qwen3 thinking model, with improved quality and depth of reasoning, featuring the following key enhancements: - **Significantly improved performance** on reasoning tasks, including logical reasoning, mathematics, science, coding, and academic benchmarks that typically require human expertise — achieving **state-of-the-art results among open-weight thinking models**. - **Markedly better general capabilities**, such as instruction following, tool usage, text generation, and alignment with human preferences. - **Enhanced 256K long-context understanding** capabilities, extendable up to **1 million tokens**.
Previous Qwen3 Release

Qwen3 (aka Qwen3-2504)

We are excited to announce the release of Qwen3, the latest addition to the Qwen family of large language models. These models represent our most advanced and intelligent systems to date, improving from our experience in building QwQ and Qwen2.5. We are making the weights of Qwen3 available to the public, including both dense and Mixture-of-Expert (MoE) models.

The highlights from Qwen3 include:

## News - 2025.08.08: You can now use Qwen3-2507 to handle ultra-long inputs of **1 million tokens**! See the update modelcards ([235B-A22B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507), [235B-A22B-Thinking-2507](https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507), [A30B-A3B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507), [A30B-A3B-Thinking-2507](https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507)) for how to enable this feature. - 2025.08.06: The final open release of Qwen3-2507, [Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507) and [Qwen3-4B-Thinking-2507](https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507), is out! - 2025.07.31: Qwen3-30B-A3B-Thinking-2507 is released. Check out the [modelcard](https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507) for more details! - 2025.07.30: Qwen3-30B-A3B-Instruct-2507 is released. Check out the [modelcard](https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507) for more details! - 2025.07.25: We released the updated version of Qwen3-235B-A22B thinking mode, named Qwen3-235B-A22B-Thinking-2507. Check out the [modelcard](https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507) for more details! - 2025.07.21: We released the updated version of Qwen3-235B-A22B non-thinking mode, named Qwen3-235B-A22B-Instruct-2507, featuring significant enhancements over the previous version and supporting 256K-token long-context understanding. Check our [modelcard](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507) for more details! - 2025.04.29: We released the Qwen3 series. Check our [blog](https://qwenlm.github.io/blog/qwen3) for more details! - 2024.09.19: We released the Qwen2.5 series. This time there are 3 extra model sizes: 3B, 14B, and 32B for more possibilities. Check our [blog](https://qwenlm.github.io/blog/qwen2.5) for more! - 2024.06.06: We released the Qwen2 series. Check our [blog](https://qwenlm.github.io/blog/qwen2/)! - 2024.03.28: We released the first MoE model of Qwen: Qwen1.5-MoE-A2.7B! Temporarily, only HF transformers and vLLM support the model. We will soon add the support of llama.cpp, mlx-lm, etc. Check our [blog](https://qwenlm.github.io/blog/qwen-moe/) for more information! - 2024.02.05: We released the Qwen1.5 series. ## Performance Detailed evaluation results are reported in this [📑 blog (Qwen3-2504)](https://qwenlm.github.io/blog/qwen3/) and this [📑 blog (Qwen3-2507) \[coming soon\]](). For requirements on GPU memory and the respective throughput, see results [here](https://qwen.readthedocs.io/en/latest/getting_started/speed_benchmark.html). ## Run Qwen3 ### 🤗 Transformers Transformers is a library of pretrained natural language processing for inference and training. The latest version of `transformers` is recommended and `transformers>=4.51.0` is required. #### Qwen3-Instruct-2507 The following contains a code snippet illustrating how to use Qwen3-30B-A3B-Instruct-2507 to generate content based on given inputs. ```python from transformers import AutoModelForCausalLM, AutoTokenizer model_name = "Qwen/Qwen3-30B-A3B-Instruct-2507" # load the tokenizer and the model tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype="auto", device_map="auto" ) # prepare the model input prompt = "Give me a short introduction to large language model." messages = [ {"role": "user", "content": prompt} ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, ) model_inputs = tokenizer([text], return_tensors="pt").to(model.device) # conduct text completion generated_ids = model.generate( **model_inputs, max_new_tokens=16384 ) output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() content = tokenizer.decode(output_ids, skip_special_tokens=True) print("content:", content) ``` > [!Note] > Qwen3-Instruct-2507 supports only non-thinking mode and does not generate ```` blocks in its output. Meanwhile, specifying `enable_thinking=False` is no longer required. #### Qwen3-Thinking-2507 The following contains a code snippet illustrating how to use Qwen3-30B-A3B-Thinking-2507 to generate content based on given inputs. ```python from transformers import AutoModelForCausalLM, AutoTokenizer model_name = "Qwen/Qwen3-30B-A3B-Thinking-2507" # load the tokenizer and the model tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype="auto", device_map="auto" ) # prepare the model input prompt = "Give me a short introduction to large language model." messages = [ {"role": "user", "content": prompt} ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, ) model_inputs = tokenizer([text], return_tensors="pt").to(model.device) # conduct text completion generated_ids = model.generate( **model_inputs, max_new_tokens=32768 ) output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() # parsing thinking content try: # rindex finding 151668 () index = len(output_ids) - output_ids[::-1].index(151668) except ValueError: index = 0 thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip("\n") content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n") print("thinking content:", thinking_content) # no opening tag print("content:", content) ``` > [!Note] > Qwen3-Thinking-2507 supports only thinking mode. > Additionally, to enforce model thinking, the default chat template automatically includes ``. Therefore, it is normal for the model's output to contain only `` without an explicit opening `` tag. > > Qwen3-Thinking-2507 also features an increased thinking length. We strongly recommend its use in highly complex reasoning tasks with adequate maximum generation length.
Switching Thinking/Non-thinking Modes for Previous Qwen3 Models

By default, Qwen3 models will think before response. This could be controlled by

  • enable_thinking=False: Passing enable_thinking=False to `tokenizer.apply_chat_template` will strictly prevent the model from generating thinking content.
  • /think and /no_think instructions: Use those words in the system or user message to signify whether Qwen3 should think. In multi-turn conversations, the latest instruction is followed.

### ModelScope We strongly advise users especially those in mainland China to use ModelScope. ModelScope adopts a Python API similar to Transformers. The CLI tool `modelscope download` can help you solve issues concerning downloading checkpoints. For vLLM and SGLang, the environment variable `VLLM_USE_MODELSCOPE=true` and `SGLANG_USE_MODELSCOPE=true` can be used respectively. ### llama.cpp [`llama.cpp`](https://github.com/ggml-org/llama.cpp) enables LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware. `llama.cpp>=b5401` is recommended for the full support of Qwen3. To use the CLI, run the following in a terminal: ```shell ./llama-cli -hf Qwen/Qwen3-8B-GGUF:Q8_0 --jinja --color -ngl 99 -fa -sm row --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 -c 40960 -n 32768 --no-context-shift # CTRL+C to exit ``` To use the API server, run the following in a terminal: ```shell ./llama-server -hf Qwen/Qwen3-8B-GGUF:Q8_0 --jinja --reasoning-format deepseek -ngl 99 -fa -sm row --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 -c 40960 -n 32768 --no-context-shift --port 8080 ``` A simple web front end will be at `http://localhost:8080` and an OpenAI-compatible API will be at `http://localhost:8080/v1`. For additional guides, please refer to [our documentation](https://qwen.readthedocs.io/en/latest/run_locally/llama.cpp.html). > [!Note] > llama.cpp adopts "rotating context management" and infinite generation is made possible by evicting earlier tokens. > It could configured by parameters and the commands above effectively disable it. > For more details, please refer to [our documentation](https://qwen.readthedocs.io/en/latest/run_locally/llama.cpp.html#llama-cli). ### Ollama After [installing Ollama](https://ollama.com/), you can initiate the Ollama service with the following command (Ollama v0.9.0 or higher is recommended): ```shell ollama serve # You need to keep this service running whenever you are using ollama ``` To pull a model checkpoint and run the model, use the `ollama run` command. You can specify a model size by adding a suffix to `qwen3`, such as `:8b` or `:30b-a3b`: ```shell ollama run qwen3:8b # Setting parameters, type "/set parameter num_ctx 40960" and "/set parameter num_predict 32768" # To exit, type "/bye" and press ENTER # For Qwen3-2504 models, # - To enable thinking, which is the default, type "/set think" # - To disable thinking, type "/set nothink" ``` You can also access the Ollama service via its OpenAI-compatible API. Please note that you need to (1) keep `ollama serve` running while using the API, and (2) execute `ollama run qwen3:8b` before utilizing this API to ensure that the model checkpoint is prepared. The API is at `http://localhost:11434/v1/` by default. For additional details, please visit [ollama.ai](https://ollama.com/). > [!Note] > Ollama's naming may not be consistent with the Qwen's original naming. > For example, `qwen3:30b-a3b` in Ollama points to `qwen3:30b-a3b-thinking-2507-q4_K_M` as of August 2025. > Please check before use. > [!Note] > Ollama adopts the same "rotating context management" with llama.cpp. > However, its default settings (`num_ctx` 2048 and `num_predict` -1), suggesting infinite generation with a 2048-token context, > could lead to trouble for Qwen3 models. > We recommend setting `num_ctx` and `num_predict` properly. ### LMStudio Qwen3 has already been supported by [lmstudio.ai](https://lmstudio.ai/). You can directly use LMStudio with our GGUF files. ### ExecuTorch To export and run on ExecuTorch (iOS, Android, Mac, Linux, and more), please follow this [example](https://github.com/pytorch/executorch/blob/main/examples/models/qwen3/README.md). ### MNN To export and run on MNN, which supports Qwen3 on mobile devices, please visit [Alibaba MNN](https://github.com/alibaba/MNN). ### MLX LM If you are running on Apple Silicon, [`mlx-lm`](https://github.com/ml-explore/mlx-lm) also supports Qwen3 (`mlx-lm>=0.24.0`). Look for models ending with MLX on Hugging Face Hub. ### OpenVINO If you are running on Intel CPU or GPU, [OpenVINO toolkit](https://github.com/openvinotoolkit) supports Qwen3. You can follow this [chatbot example](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-chatbot/llm-chatbot.ipynb). ## Deploy Qwen3 Qwen3 is supported by multiple inference frameworks. Here we demonstrate the usage of `SGLang`, `vLLM` and `TensorRT-LLM`. You can also find Qwen3 models from various inference providers, e.g., [Alibaba Cloud Model Studio](https://www.alibabacloud.com/en/product/modelstudio). ### SGLang [SGLang](https://github.com/sgl-project/sglang) is a fast serving framework for large language models and vision language models. SGLang could be used to launch a server with OpenAI-compatible API service. `sglang>=0.4.6.post1` is required. For Qwen3-Instruct-2507, ```shell python -m sglang.launch_server --model-path Qwen/Qwen3-30B-A3B-Instruct-2507 --port 30000 --context-length 262144 ``` For Qwen3-Thinking-2507, ```shell python -m sglang.launch_server --model-path Qwen/Qwen3-30B-A3B-Thinking-2507 --port 30000 --context-length 262144 --reasoning-parser deepseek-r1 ``` For Qwen3, it is ```shell python -m sglang.launch_server --model-path Qwen/Qwen3-8B --port 30000 --context-length 131072 --reasoning-parser qwen3 ``` An OpenAI-compatible API will be available at `http://localhost:30000/v1`. > [!Note] > Due to the preprocessing of API requests in SGLang, which drops all `reasoning_content` fields, the quality of **multi-step tool use with Qwen3 thinking models** may be suboptimal, which requires the existence of the related thinking content. While the fixes are being worked on, as a workdaround, we recommend passing the content as it is, without extracting thinking content, and the chat template will correctly handle the processing. ### vLLM [vLLM](https://github.com/vllm-project/vllm) is a high-throughput and memory-efficient inference and serving engine for LLMs. `vllm>=0.9.0` is recommended. For Qwen3-Instruct-2507, ```shell vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 --port 8000 --max-model-len 262144 ``` For Qwen3-Thinking-2507, ```shell vllm serve Qwen/Qwen3-30B-A3B-Thinking-2507 --port 8000 --max-model-len 262144 --enable-reasoning --reasoning-parser deepseek_r1 ``` For Qwen3, it is ```shell vllm serve Qwen/Qwen3-8B --port 8000 --max-model-len 131072 --enable-reasoning --reasoning-parser qwen3 ``` An OpenAI-compatible API will be available at `http://localhost:8000/v1`. > [!Note] > Due to the preprocessing of API requests in vLLM, which drops all `reasoning_content` fields, the quality of **multi-step tool use with Qwen3 thinking models** may be suboptimal, which requires the existence of the related thinking content. While the fixes are being worked on, as a workdaround, we recommend passing the content as it is, without extracting thinking content, and the chat template will correctly handle the processing. ### TensorRT-LLM [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) is an open-source LLM inference engine from NVIDIA, which provides optimizations including custom attention kernels, quantization and more on NVIDIA GPUs. Qwen3 is supported in its re-architected [PyTorch backend](https://nvidia.github.io/TensorRT-LLM/torch.html). `tensorrt_llm>=0.20.0rc3` is recommended. Please refer to the [README](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/models/core/qwen/README.md#qwen3) page for more details. ```shell trtllm-serve Qwen/Qwen3-8B --host localhost --port 8000 --backend pytorch ``` An OpenAI-compatible API will be available at `http://localhost:8000/v1`. ### MindIE For deployment on Ascend NPUs, please visit [Modelers](https://modelers.cn/) and search for Qwen3. ## Build with Qwen3 ### Tool Use For tool use capabilities, we recommend taking a look at [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent), which provides a wrapper around these APIs to support tool use or function calling with MCP support. Tool use with Qwen3 can also be conducted with SGLang, vLLM, Transformers, llama.cpp, Ollama, etc. Follow guides in our documentation to see how to enable the support. ### Finetuning We advise you to use training frameworks, including [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl), [UnSloth](https://github.com/unslothai/unsloth), [Swift](https://github.com/modelscope/swift), [Llama-Factory](https://github.com/hiyouga/LLaMA-Factory), etc., to finetune your models with SFT, DPO, GRPO, etc. ## License Agreement All our open-weight models are licensed under Apache 2.0. You can find the license files in the respective Hugging Face repositories. ## Citation If you find our work helpful, feel free to give us a cite. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Contact Us If you are interested to leave a message to either our research team or product team, join our [Discord](https://discord.gg/z3GAxXZ9Ce) or [WeChat groups](assets/wechat.png)! --- ### Overview (`docs/README.md`) # Qwen Documentation This is the source of the documentation at . ## Quick Start We use `sphinx` to manage the documentation and use the `furo` theme. To get started, simply run ```bash pip install -r requirements-docs.txt ``` Then run `make html` or `sphinx-build -M html source build` and it will compile the docs and put it under the `build/html` directory. ## Translation The documentation is available in both English and Simplified Chinese. We use `sphinx-intl` to work with Sphinx translation flow, following [this article](https://www.sphinx-doc.org/en/master/usage/advanced/intl.html). You need to install the Python package `sphinx-intl` before starting. 1. After updating the English documentation, run `make gettext`, and the pot files will be placed in the `build/gettext` directory. `make gettext` can be slow if the doc is long. 2. Use the generated pot files to update the po files: ```bash sphinx-intl update -p build/gettext -l zh_CN -w 0 ``` 3. Translate po files at `locales\zh_CN\LC_MESSAGES`. Pay attention to fuzzy matches (messages after `#, fuzzy`). Please be careful not to break reST notation. 4. Build translated document: `make -e SPHINXOPTS="-D language='zh_CN'" html` or `sphinx-build -M html source build -D language=zh_CN` ## Auto Build ```bash pip install sphinx-autobuild ``` To autobuild the default version: ```bash sphinx-autobuild source build/html ``` To autobuild the translated version: ```bash sphinx-autobuild source build/html -D language=zh_CN --watch locales/zh_CN ``` By default, the doc is at `http://127.0.0.1:8000` --- ### Overview (`eval/README.md`) This folder provides scripts to reproduce evaluation results across various benchmarks for the **Qwen** series of large language models. ## Supported Benchmarks Currently, we support the following benchmark: | Model | Dataset | Config | Reproduced Score | |-------|--------|--------|------------------| | Qwen3-235B-A22B-Instruct-2507 | ARC-AGI 1 (pass@1) | [./configs/ARCAGI-Qwen3-235B-A22B-Instruct-2507.yaml](./configs/ARCAGI-Qwen3-235B-A22B-Instruct-2507.yaml) | 40.75 | In the meantime, you can find the model outputs and final evaluation results in the [`./output`](./output) and [`./eval_res`](./eval_res) directories, respectively. Additional benchmarks will be added in future updates. ## Evaluation Guide Follow the steps below to reproduce the reported scores. ### Step 0: Prerequisites Ensure you have: - Python ≥ 3.9 - Either [vLLM](https://github.com/vllm-project/vllm) or [SGLang](https://github.com/sgl-project/sgl) installed Install required dependencies: ```bash pip install -r requirements.txt ``` ### Step 1: Start vLLM Server Launch the vLLM inference server using the command below: ```bash export MODEL_NAME="Qwen/Qwen3-235B-A22B-Instruct-2507" # Replace with desired model export MODEL_PATH="$MODEL_NAME" # Or path to local checkpoint export NUM_GPUS=8 python -m vllm.entrypoints.openai.api_server \ --model "$MODEL_PATH" \ --trust-remote-code \ --served-model-name "$MODEL_NAME" \ --tensor-parallel-size $NUM_GPUS \ --enforce-eager \ --port 8030 ``` > 💡 Adjust `tensor_parallel_size` according to your GPU setup. ### Optional: Start SGLang Router (Recommended for Faster Evaluation) Since evaluations can take several days, we recommend using **SGLang** with data parallelism to accelerate inference. See the [SGLang Router documentation](https://docs.sglang.ai/router/router.html) for details. Start the SGLang router server: ```bash python -m sglang_router.launch_server \ --model-path Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dp-size 4 \ --host 0.0.0.0 \ --port 30000 ``` > ⚠️ Adjust `dp_size` based on available resources, and ensure consistency in port configuration for subsequent steps. ### Step 2: Run Inference Once the inference server is running, generate model responses using the multithreaded inference script. ```bash mkdir -p output # Example: Evaluate on ARC-AGI python generate_api_answers/infer_multithread.py \ --config configs/ARCAGI-Qwen3-235B-A22B-Instruct-2507.yaml ``` #### Resume Interrupted Inference If the process is interrupted, simply re-run the same command. The script will automatically detect existing outputs and resume generation for incomplete prompts. ### Step 3: Compute Scores After inference completes, evaluate the results using the scoring script: ```bash mkdir -p eval_res python eval/eval.py \ --config configs/ARCAGI-Qwen3-235B-A22B-Instruct-2507.yaml \ > eval_res/ARCAGI-Qwen3-235B-A22B-Instruct-2507_eval_result.txt ``` The final score will be saved to the specified output file. --- ### Overview (`examples/README.md`) # Examples > [!IMPORTANT] > The examples in this directory should be considered deprecated at the moment and they are not updated for Qwen3. > --- ### Overview (`examples/gcu-support/README.md`) # Qwen2.5 推理 ## 1、配置运行环境 **安装驱动** ``` # 为软件包具体版本号。 chmod +x TopsRider_i3x__deb_amd64.run ./TopsRider_i3x__deb_amd64.run -y ``` **创建并启动 docker** ``` # 创建 docker 容器，将在基础镜像 artifact.enflame.cn/enflame_docker_images/ubuntu/qic_ubuntu_2004_gcc9:1.4.4 的基础上创建 docker。 # 当前工程所在路径 # -e ENFLAME_VISIBLE_DEVICES=2 进行 GCU 资源隔离，如需多卡可以改为 0,1,2,3 等 docker run -itd -e ENFLAME_VISIBLE_DEVICES=2 --name qwen-infer -v :/work -v /root/:/root/ --privileged --network host artifact.enflame.cn/enflame_docker_images/ubuntu/qic_ubuntu_2004_gcc9:1.4.4 bash ``` **进入 docker 安装环境** ``` # 进入 docker 容器 docker exec -it qwen-infer bash # 安装 SDK 框架，进入软件包所在地址。 # 为软件包具体版本号。 ./TopsRider_i3x__amd64.run -C torch-gcu-2 -y ./TopsRider_i3x__deb_amd64.run -C tops-sdk -y # 安装 python 库 pip3.8 install transformers==4.40.2 pip3.8 install accelerate ``` ## 2、推理 ``` # 进入本工程目录，包含运行代码、推理输入等文件。 . ├── README.md └── gcu_demo.py ``` **启动推理示例** ``` python3.8 gcu_demo.py ``` 执行 gcu_demo.py 推理示例，代码改编自 [仓库 README](https://github.com/QwenLM/Qwen2.5/blob/main/README.md) 中的给的 Huggingface quick start 用例。 **GCU PyTorch 原生推理支持** GCU 支持 pytorch 原生推理，在 pytorch 代码上只需做少许改动就可以在 GCU 上顺利运行： 1. 导入 *torch_gcu* 后端库，并载入 transfer_to_gcu ``` python try: import torch_gcu # 导入 torch_gcu from torch_gcu import transfer_to_gcu # transfer_to_gcu except Exception as e: print(e) ``` 2. device 名改为 *gcu* ``` python device = "gcu" ``` **GCU vLLM 推理** GCU 也支持 *vLLM* 原生推理，需要安装 GCU 版本的 *vLLM* 后，将设备名改为 gcu ``` python -m vllm.entrypoints.openai.api_server --served-model-name Qwen2.5-7B-Instruct --model Qwen/Qwen2.5-7B-Instruct --device gcu ``` --- ### Overview (`examples/speed-benchmark/README.md`) # Speed Benchmark This document introduces the speed benchmark testing process for the Qwen2.5 series models (original and quantized models). For detailed reports, please refer to the [Qwen2.5 Speed Benchmark](https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html). ## 1. Model Collections For models hosted on HuggingFace, refer to [Qwen2.5 Collections-HuggingFace](https://huggingface.co/collections/Qwen/qwen25-66e81a666513e518adb90d9e). For models hosted on ModelScope, refer to [Qwen2.5 Collections-ModelScope](https://modelscope.cn/collections/Qwen25-dbc4d30adb768). ## 2. Environment Setup For inference using HuggingFace transformers: ```shell conda create -n qwen_perf_transformers python=3.10 conda activate qwen_perf_transformers pip install torch==2.3.1 pip install git+https://github.com/AutoGPTQ/AutoGPTQ.git@v0.7.1 pip install git+https://github.com/Dao-AILab/flash-attention.git@v2.5.8 pip install -r requirements-perf-transformers.txt ``` > [!Important] > - For `flash-attention`, you can use the prebulit wheels from [GitHub Releases](https://github.com/Dao-AILab/flash-attention/releases/tag/v2.5.8) or installing from source, which requires a compatible CUDA compiler. > - You don't actually need to install `flash-attention`. It has been intergrated into `torch` as a backend of `sdpa`. > - For `auto_gptq` to use efficent kernels, you need to install from source, because the prebuilt wheels require incompatible `torch` versions. Installing from source also requires a compatible CUDA compiler. > - For `autoawq` to use efficent kenerls, you need `autoawq-kernels`, which should be automatically installed. If not, run `pip install autoawq-kernels`. For inference using vLLM: ```shell conda create -n qwen_perf_vllm python=3.10 conda activate qwen_perf_vllm pip install -r requirements-perf-vllm.txt ``` ## 3. Execute Tests Below are two methods for executing tests: using a script or the Speed Benchmark tool. ### Method 1: Testing with Speed Benchmark Tool Use the Speed Benchmark tool developed by [EvalScope](https://github.com/modelscope/evalscope), which supports automatic model downloads from ModelScope and outputs test results. It also allows testing by specifying the model service URL. For details, please refer to the [📖 User Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/speed_benchmark.html). **Install Dependencies** ```shell pip install 'evalscope[perf]' -U ``` #### HuggingFace Transformers Inference Execute the command as follows: ```shell CUDA_VISIBLE_DEVICES=0 evalscope perf \ --parallel 1 \ --model Qwen/Qwen2.5-0.5B-Instruct \ --attn-implementation flash_attention_2 \ --log-every-n-query 5 \ --connect-timeout 6000 \ --read-timeout 6000 \ --max-tokens 2048 \ --min-tokens 2048 \ --api local \ --dataset speed_benchmark ``` #### vLLM Inference ```shell CUDA_VISIBLE_DEVICES=0 evalscope perf \ --parallel 1 \ --model Qwen/Qwen2.5-0.5B-Instruct \ --log-every-n-query 1 \ --connect-timeout 60000 \ --read-timeout 60000\ --max-tokens 2048 \ --min-tokens 2048 \ --api local_vllm \ --dataset speed_benchmark ``` #### Parameter Explanation - `--parallel` sets the number of worker threads for concurrent requests, should be fixed at 1. - `--model` specifies the model file path or model ID, supporting automatic downloads from ModelScope, e.g., Qwen/Qwen2.5-0.5B-Instruct. - `--attn-implementation` sets the attention implementation method, with optional values: flash_attention_2|eager|sdpa. - `--log-every-n-query`: sets how often to log every n requests. - `--connect-timeout`: sets the connection timeout in seconds. - `--read-timeout`: sets the read timeout in seconds. - `--max-tokens`: sets the maximum output length in tokens. - `--min-tokens`: sets the minimum output length in tokens; both parameters set to 2048 means the model will output a fixed length of 2048. - `--api`: sets the inference interface; local inference options are local|local_vllm. - `--dataset`: sets the test dataset; options are speed_benchmark|speed_benchmark_long. #### Test Results Test results can be found in the `outputs/{model_name}/{timestamp}/speed_benchmark.json` file, which contains all request results and test parameters. ### Method 2: Testing with Scripts #### HuggingFace Transformers Inference - Using HuggingFace Hub ```shell python speed_benchmark_transformers.py --model_id_or_path Qwen/Qwen2.5-0.5B-Instruct --context_length 1 --gpus 0 --outputs_dir outputs/transformers ``` - Using ModelScope Hub ```shell python speed_benchmark_transformers.py --model_id_or_path Qwen/Qwen2.5-0.5B-Instruct --context_length 1 --gpus 0 --use_modelscope --outputs_dir outputs/transformers ``` Parameter Explanation: `--model_id_or_path`: Model ID or local path, optional values refer to the `Model Resources` section `--context_length`: Input length in tokens; optional values are 1, 6144, 14336, 30720, 63488, 129024; refer to the `Qwen2.5 Model Efficiency Evaluation Report` for specifics `--generate_length`: Number of tokens to generate; default is 2048 `--gpus`: Equivalent to the environment variable CUDA_VISIBLE_DEVICES, e.g., `0,1,2,3`, `4,5` `--use_modelscope`: If set, uses ModelScope to load the model; otherwise, uses HuggingFace `--outputs_dir`: Output directory, default is `outputs/transformers` #### vLLM Inference - Using HuggingFace Hub ```shell python speed_benchmark_vllm.py --model_id_or_path Qwen/Qwen2.5-0.5B-Instruct --context_length 1 --max_model_len 32768 --gpus 0 --gpu_memory_utilization 0.9 --outputs_dir outputs/vllm ``` - Using ModelScope Hub ```shell python speed_benchmark_vllm.py --model_id_or_path Qwen/Qwen2.5-0.5B-Instruct --context_length 1 --max_model_len 32768 --gpus 0 --use_modelscope --gpu_memory_utilization 0.9 --outputs_dir outputs/vllm ``` Parameter Explanation: `--model_id_or_path`: Model ID or local path, optional values refer to the `Model Resources` section `--context_length`: Input length in tokens; optional values are 1, 6144, 14336, 30720, 63488, 129024; refer to the `Qwen2.5 Model Efficiency Evaluation Report` for specifics `--generate_length`: Number of tokens to generate; default is 2048 `--max_model_len`: Maximum model length in tokens; default is 32768 `--gpus`: Equivalent to the environment variable CUDA_VISIBLE_DEVICES, e.g., `0,1,2,3`, `4,5` `--use_modelscope`: If set, uses ModelScope to load the model; otherwise, uses HuggingFace `--gpu_memory_utilization`: GPU memory utilization, range (0, 1]; default is 0.9 `--outputs_dir`: Output directory, default is `outputs/vllm` `--enforce_eager`: Whether to enforce eager mode; default is False #### Test Results Test results can be found in the `outputs` directory, which by default includes two folders for `transformers` and `vllm`, storing test results for HuggingFace transformers and vLLM respectively. ## Notes 1. Conduct multiple tests and take the average, with a typical value of 3 tests. 2. Ensure the GPU is idle before testing to avoid interference from other tasks. --- ### Source Getting Started Quickstart (`docs/source/getting_started/quickstart.md`) # Quickstart This guide helps you quickly start using Qwen3. We provide examples of [Hugging Face Transformers](https://github.com/huggingface/transformers) as well as [ModelScope](https://github.com/modelscope/modelscope), and [vLLM](https://github.com/vllm-project/vllm) and [SGLang](https://github.com/sgl-project/sglang) for deployment. You can find Qwen3 models in [the Qwen3 collection](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) at Hugging Face Hub and [the Qwen3 collection](https://www.modelscope.cn/collections/Qwen3-9743180bdc6b48) at ModelScope. ## Transformers To get a quick start with Qwen3, you can try the inference with `transformers` first. Make sure that you have installed `transformers>=4.51.0`. We advise you to use Python 3.10 or higher, and PyTorch 2.6 or higher. :::::{tab-set} :sync-group: model ::::{tab-item} Qwen3-Instruct-2507 :sync: instruct :::{important} Qwen3-Instruct-2507 supports **only non-thinking mode** and **does not generate ```` blocks** in its output. Different from Qwen3-2504, **specifying `enable_thinking=False` is no longer required or supported**. ::: The following contains a code snippet illustrating how to use Qwen3-235B-A22B-Instruct-2507 to generate content based on given inputs. ```python from transformers import AutoModelForCausalLM, AutoTokenizer model_name = "Qwen/Qwen3-235B-A22B-Instruct-2507" # load the tokenizer and the model tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype="auto", device_map="auto" ) # prepare the model input prompt = "Give me a short introduction to large language model." messages = [ {"role": "user", "content": prompt} ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, ) model_inputs = tokenizer([text], return_tensors="pt").to(model.device) # conduct text completion generated_ids = model.generate( **model_inputs, max_new_tokens=16384 ) output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() content = tokenizer.decode(output_ids, skip_special_tokens=True) print("content:", content) ``` :::{Note} We recommend `temperature=0.7`, `top_p=0.8`, `top_k=20`, and `min_p=0` for Qwen3-Instruct-2507 models. For supported frameworks, adjust `presence_penalty` between 0 and 2 to reduce repetitions. However, using a higher value may occasionally result in language mixing and a slight decrease in model performance. ::: :::{Note} Qwen3-Instruct-2507 may use CoT (chain-of-thoughts) automatically for complex tasks. We recommend using an output length of 16,384 tokens for most queries. ::: :::: ::::{tab-item} Qwen3-Thinking-2507 :sync: thinking :::{important} Qwen3-Thinking-2507 supports **only thinking mode**. Additionally, to enforce model thinking, the default chat template automatically includes ``. Therefore, it is normal for the model's output to contain only `` without an explicit opening `` tag. ::: The following contains a code snippet illustrating how to use Qwen3-235B-A22B-Thinking-2507 to generate content based on given inputs. ```python from transformers import AutoModelForCausalLM, AutoTokenizer model_name = "Qwen/Qwen3-235B-A22B-Thinking-2507" # load the tokenizer and the model tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype="auto", device_map="auto" ) # prepare the model input prompt = "Give me a short introduction to large language model." messages = [ {"role": "user", "content": prompt} ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, ) model_inputs = tokenizer([text], return_tensors="pt").to(model.device) # conduct text completion generated_ids = model.generate( **model_inputs, max_new_tokens=32768 ) output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() # parsing thinking content try: # rindex finding 151668 () index = len(output_ids) - output_ids[::-1].index(151668) except ValueError: index = 0 thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip("\n") content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n") print("thinking content:", thinking_content) # no opening tag print("content:", content) ``` :::{note} We recommend `temperature=0.6`, `top_p=0.95`, `top_k=20`, and `min_p=0` for Qwen3-Thinking-2507 models. For supported frameworks, adjust `presence_penalty` between 0 and 2 to reduce repetitions. However, using a higher value may occasionally result in language mixing and a slight decrease in model performance. ::: :::{note} Qwen3-Thinking-2507 features increased thinking depth. We strongly recommend its use in highly complex reasoning tasks with adequate maximum generation length. ::: :::: ::::{tab-item} Qwen3 :sync: hybrid The following is a very simple code snippet showing how to run Qwen3-8B: ```python from transformers import AutoModelForCausalLM, AutoTokenizer model_name = "Qwen/Qwen3-8B" # load the tokenizer and the model model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype="auto", device_map="auto" ) tokenizer = AutoTokenizer.from_pretrained(model_name) # prepare the model input prompt = "Give me a short introduction to large language models." messages = [ {"role": "user", "content": prompt}, ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, enable_thinking=True, # Switches between thinking and non-thinking modes. Default is True. ) model_inputs = tokenizer([text], return_tensors="pt").to(model.device) # conduct text completion generated_ids = model.generate( **model_inputs, max_new_tokens=32768 ) output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() # parse thinking content try: # rindex finding 151668 () index = len(output_ids) - output_ids[::-1].index(151668) except ValueError: index = 0 thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip("\n") content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n") print("thinking content:", thinking_content) print("content:", content) ``` Qwen3 will think before respond, similar to QwQ models. This means the model will use its reasoning abilities to enhance the quality of generated responses. The model will first generate thinking content wrapped in a `...` block, followed by the final response. - Hard Switch: To strictly disable the model's thinking behavior, aligning its functionality with the previous Qwen2.5-Instruct models, you can set `enable_thinking=False` when formatting the text. ```python text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, enable_thinking=False, # Setting enable_thinking=False disables thinking mode ) ``` It can be particularly useful in scenarios where disabling thinking is essential for enhancing efficiency. - Soft Switch: Qwen3 also understands the user's instruction on its thinking behavior, in particular, the soft switch `/think` and `/no_think`. You can add them to user prompts or system messages to switch the model's thinking mode from turn to turn. The model will follow the most recent instruction in multi-turn conversations. :::{note} For thinking mode, use Temperature=0.6, TopP=0.95, TopK=20, and MinP=0 (the default setting in `generation_config.json`). DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions. For non-thinking mode, we suggest using Temperature=0.7, TopP=0.8, TopK=20, and MinP=0. ::: :::: ::::: ## ModelScope To tackle with downloading issues, we advise you to try [ModelScope](https://github.com/modelscope/modelscope). Before starting, you need to install `modelscope` with `pip`. `modelscope` adopts a programmatic interface similar (but not identical) to `transformers`. For basic usage, you can simply change the first line of code above to the following: ```python from modelscope import AutoModelForCausalLM, AutoTokenizer ``` For more information, please refer to [the documentation of `modelscope`](https://www.modelscope.cn/docs). ## OpenAI API Compatibility You can serve Qwen3 via OpenAI-compatible APIs using frameworks such as vLLM, SGLang, and interact with the API using common HTTP clients or the OpenAI SDKs. :::::{tab-set} :sync-group: model ::::{tab-item} Qwen3-Instruct-2507 :sync: instruct Here we take Qwen3-235B-A22B-Instruct-2507 as an example to start the API: - SGLang (`sglang>=0.4.6.post1` is required): ```shell python -m sglang.launch_server --model-path Qwen/Qwen3-235B-A22B-Instruct-2507 --port 8000 --tp 8 --context-length 262144 ``` - vLLM (`vllm>=0.9.0` is recommended): ```shell vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507 --port 8000 --tensor-parallel-size 8 --max-model-len 262144 ``` :::{note} Consider adjusting the context length according to the available GPU memory. ::: :::: ::::{tab-item} Qwen3-Thinking-2507 :sync: thinking Here we take Qwen3-235B-A22B-Thinking-2507 as an example to start the API: - SGLang (`sglang>=0.4.6.post1` is required): ```shell python -m sglang.launch_server --model-path Qwen/Qwen3-235B-A22B-Thinking-2507 --port 8000 --tp 8 --context-length 262144 --reasoning-parser deepseek-r1 ``` - vLLM (`vllm>=0.9.0` is recommended): ```shell vllm serve Qwen/Qwen3-235B-A22B-Thinking-2507 --port 8000 --tensor-parallel-size 8 --max-model-len 262144 --enable-reasoning --reasoning-parser deepseek_r1 ``` :::{note} Consider adjusting the context length according to the available GPU memory. ::: :::{important} We are currently working on adapting the `qwen3` reasoning parsers to the new behavior. Please follow the command above at the moment. ::: :::: ::::{tab-item} Qwen3 :sync: hybrid Here we take Qwen3-8B as an example to start the API: - SGLang (`sglang>=0.4.6.post1` is required): ```shell python -m sglang.launch_server --model-path Qwen/Qwen3-8B --port 8000 --reasoning-parser qwen3 ``` - vLLM (`vllm>=0.9.0` is recommended): ```shell vllm serve Qwen/Qwen3-8B --port 8000 --enable-reasoning --reasoning-parser qwen3 ``` :::: ::::: Then, you can use the [create chat interface](https://platform.openai.com/docs/api-reference/chat/completions/create) to communicate with Qwen: ::::::{tab-set} :sync-group: model :::::{tab-item} Qwen3-Instruct-2507 :sync: instruct Here we show the basic command to interact with the chat completion API using Qwen3-235B-A22B-Instruct-2507. ::::{tab-set} :sync-group: api :::{tab-item} curl :sync: curl ```shell curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-235B-A22B-Instruct-2507", "messages": [ {"role": "user", "content": "Give me a short introduction to large language models."} ], "temperature": 0.7, "top_p": 0.8, "top_k": 20, "max_tokens": 16384 }' ``` ::: :::{tab-item} Python :sync: python You can use the API client with the `openai` Python SDK as shown below: ```python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai_api_key = "EMPTY" openai_api_base = "http://localhost:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) chat_response = client.chat.completions.create( model="Qwen/Qwen3-235B-A22B-Instruct-2507", messages=[ {"role": "user", "content": "Give me a short introduction to large language models."}, ], max_tokens=16384, temperature=0.7, top_p=0.8, extra_body={ "top_k": 20, } ) print("Chat response:", chat_response) ``` :::: ::::: :::::{tab-item} Qwen3-Thinking-2507 :sync: thinking Here we show the basic command to interact with the chat completion API using Qwen3-235B-A22B-Thinking-2507. ::::{tab-set} :sync-group: api :::{tab-item} curl :sync: curl ```shell curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-235B-A22B-Thinking-2507", "messages": [ {"role": "user", "content": "Give me a short introduction to large language models."} ], "temperature": 0.6, "top_p": 0.95, "top_k": 20, "max_tokens": 32768 }' ``` ::: :::{tab-item} Python :sync: python You can use the API client with the `openai` Python SDK as shown below: ```python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai_api_key = "EMPTY" openai_api_base = "http://localhost:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) chat_response = client.chat.completions.create( model="Qwen/Qwen3-235B-A22B-Thinking-2507", messages=[ {"role": "user", "content": "Give me a short introduction to large language models."}, ], max_tokens=32768, temperature=0.6, top_p=0.95, extra_body={ "top_k": 20, } ) print("Chat response:", chat_response) ``` :::: ::::: :::::{tab-item} Qwen3 :sync: hybrid Here we show the basic command to interact with the chat completion API using Qwen3-8B. The default is with thinking enabled: ::::{tab-set} :sync-group: api :::{tab-item} curl :sync: curl ```shell curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": [ {"role": "user", "content": "Give me a short introduction to large language models."} ], "temperature": 0.6, "top_p": 0.95, "top_k": 20, "max_tokens": 32768 }' ``` ::: :::{tab-item} Python :sync: python You can use the API client with the `openai` Python SDK as shown below: ```python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai_api_key = "EMPTY" openai_api_base = "http://localhost:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) chat_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=[ {"role": "user", "content": "Give me a short introduction to large language models."}, ], max_tokens=32768, temperature=0.6, top_p=0.95, extra_body={ "top_k": 20, } ) print("Chat response:", chat_response) ``` ::: :::: To disable thinking, one could use the soft switch (e.g., appending `/nothink` to the user query). The hard switch can also be used as follows: ::::{tab-set} :sync-group: api :::{tab-item} curl :sync: curl ```shell curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": [ {"role": "user", "content": "Give me a short introduction to large language models."} ], "temperature": 0.7, "top_p": 0.8, "top_k": 20, "max_tokens": 8192, "presence_penalty": 1.5, "chat_template_kwargs": {"enable_thinking": false} }' ``` ::: :::{tab-item} Python :sync: python You can use the API client with the `openai` Python SDK as shown below: ```python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai_api_key = "EMPTY" openai_api_base = "http://localhost:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) chat_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=[ {"role": "user", "content": "Give me a short introduction to large language models."}, ], max_tokens=8192, temperature=0.7, top_p=0.8, presence_penalty=1.5, extra_body={ "top_k": 20, "chat_template_kwargs": {"enable_thinking": False}, } ) print("Chat response:", chat_response) ``` ::: ::::: :::::: For more usage, please refer to our document on [SGLang](../deployment/sglang) and [vLLM](../deployment/vllm). ## Thinking Budget Qwen3 supports the configuration of thinking budget. It is achieved by ending the thinking process once the budget is reached and guiding the model to generate the "summary" with an early-stopping prompt. Since this feature involves customization specific to each model, it is currently not available in the open-source frameworks and only implemented by [the Alibaba Cloud Model Studio API](https://www.alibabacloud.com/help/en/model-studio/deep-thinking#6f0633b9cdts1). However, with existing open-source frameworks, one can generate twice to implement this feature as follows: 1. For the first time, generate tokens up to the thinking budget and check if the thinking process is finished. If the thinking process is not finished, append the early-stopping prompt. 2. For the second time, continue generation until the end of the content or the upper length limit is fulfilled. The following snippet shows the implementation with Hugging Face Transformers: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` You should see the output in the console like the following ```text thinking budget is reached thinking content: Okay, the user is asking for a short introduction to large language models Considering the limited time by the user, I have to give the solution based on the thinking directly now. content: Large language models (LLMs) are advanced artificial intelligence systems trained on vast amounts of text data to understand and generate human-like language. They can perform tasks such as answering questions, writing stories, coding, and translating languages. LLMs are powered by deep learning techniques and have revolutionized natural language processing by enabling more context-aware and versatile interactions with text. Examples include models like GPT, BERT, and others developed by companies like OpenAI and Alibaba. ``` :::{note} For purpose of demonstration only, `thinking_budget` is set to 16. However, `thinking_budget` should not be set to that low in practice. We recommend tuning `thinking_budget` based on the latency users can accept and setting it higher than 1024 for meaningful improvements across tasks. If thinking is not desired at all, developers should make use of the hard switch instead. ::: ## Next Step Now, you can have fun with Qwen3 models. Would love to know more about its usage? Feel free to check other documents in this documentation. --- ### Requirements Docs (`docs/requirements-docs.txt`) furo myst-parser==4.0.0 sphinx<8,>4.5.0 sphinx-copybutton sphinx-design>=0.6.0 --- ### Source Index (`docs/source/index.rst`) Welcome to Qwen! ================ .. figure:: https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/logo_qwen3.png :width: 60% :align: center :alt: Qwen3 :class: no-scaled-link Qwen is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. Both language models and multimodal models are pretrained on large-scale multilingual and multimodal data and post-trained on quality data for aligning to human preferences. Qwen is capable of natural language understanding, text generation, vision understanding, audio understanding, tool use, role play, playing as AI agent, etc. Qwen3-2507 ---------- With input from the community and insights from further research, Instruct-only and Thinking-only models are coming back! The results are Qwen3-2507: **Qwen3-Instruct-2507** has the following features: - **Significant improvements** in general capabilities, including **instruction following, logical reasoning, text comprehension, mathematics, science, coding and tool usage**. - **Substantial gains** in long-tail knowledge coverage across **multiple languages**. - **Markedly better alignment** with user preferences in **subjective and open-ended tasks**, enabling more helpful responses and higher-quality text generation. - **Enhanced capabilities** in **256K long-context understanding**, extensible to 1M. **Qwen3-Thinking-2507** has the following features: - **Significantly improved performance** on reasoning tasks, including logical reasoning, mathematics, science, coding, and academic benchmarks that typically require human expertise — achieving **state-of-the-art results among open-source thinking models**. - **Markedly better general capabilities**, such as instruction following, tool usage, text generation, and alignment with human preferences. - **Enhanced 256K long-context understanding** capabilities, extensible to 1M. Qwen3 ----- Qwen3, aka Qwen3-2504, has the following features: - **Dense and Mixture-of-Experts (MoE) models**, available in 0.6B, 1.7B, 4B, 8B, 14B, 32B and 30B-A3B, 235B-A22B. - **Seamless switching between thinking mode** (for complex logical reasoning, math, and coding) and **non-thinking mode** (for efficient, general-purpose chat) **within a single model**, ensuring optimal performance across various scenarios. - **Significantly enhancement in reasoning capabilities**, surpassing previous QwQ (in thinking mode) and Qwen2.5 instruct models (in non-thinking mode) on mathematics, code generation, and commonsense logical reasoning. - **Superior human preference alignment**, excelling in creative writing, role-playing, multi-turn dialogues, and instruction following, to deliver a more natural, engaging, and immersive conversational experience. - **Expertise in agent capabilities**, enabling precise integration with external tools in both thinking and unthinking modes and achieving leading performance among open-source models in complex agent-based tasks. - **Support of 100+ languages and dialects** with strong capabilities for **multilingual instruction following** and **translation**. Resource & Links ---------------- For more information, please visit our: * `Qwen Home Page `__ * `Chat with Qwen (with Deep Research and Web Dev) `__ * `Blog `__ * `GitHub `__ * `Hugging Face `__ * `ModelScope `__ * `Qwen3 Collection `__ Join our community by joining our `Discord `__ and `WeChat `__ group. We are looking forward to seeing you there! .. toctree:: :maxdepth: 1 :caption: Getting Started :hidden: getting_started/quickstart getting_started/concepts getting_started/speed_benchmark getting_started/quantization_benchmark .. toctree:: :maxdepth: 1 :caption: Inference :hidden: inference/transformers .. toctree:: :maxdepth: 1 :caption: Run Locally :hidden: run_locally/llama.cpp run_locally/ollama run_locally/lmstudio run_locally/mlx-lm .. toctree:: :maxdepth: 1 :caption: Deployment :hidden: deployment/sglang deployment/vllm deployment/tgi deployment/dstack deployment/skypilot deployment/openllm .. toctree:: :maxdepth: 1 :caption: Quantization :hidden: quantization/awq quantization/gptq quantization/llama.cpp .. toctree:: :maxdepth: 1 :caption: Training :hidden: training/axolotl training/llama_factory training/ms_swift training/unsloth training/verl .. toctree:: :maxdepth: 1 :caption: Framework :hidden: framework/qwen_agent framework/function_call framework/LlamaIndex framework/Langchain --- ### Source Deployment Dstack (`docs/source/deployment/dstack.rst`) dstack ======== `dstack `__ is an open-source alternative to Kubernetes and Slurm, designed to simplify GPU allocation and AI workload orchestration for ML teams across top clouds, on-prem clusters, and accelerators. Prerequisites ---------------- Before you start, install dstack by following the `installation instructions `__. Once dstack server is up, you can initialize your workspace as shown below: .. code:: bash mkdir dstack-qwen-deploy && cd dstack-qwen-deploy dstack init Deploy Qwen3-30B-A3B ----------------------------------------------- Deploy ``Qwen3-30B-A3B`` on instances available with cloud providers configured in your ``~/.dstack/server/config.yml`` file. You can use ``SgLang``, ``TGI`` or ``vLLM`` to serve the model. Here we use ``SgLang`` as an example. Create a `service `__ configuration file named ``serve-30b.dstack.yml`` with the following content: .. code:: yaml type: service name: qwen3-30b-a3b image: lmsysorg/sglang:latest env: - MODEL_ID=Qwen/Qwen3-30B-A3B commands: - python3 -m sglang.launch_server --model-path $MODEL_ID --port 8000 --trust-remote-code port: 8000 model: Qwen/Qwen3-30B-A3B resources: gpu: 80GB:1 .. note:: For other inference backends such as vLLM or TGI, visit the `dstack Inference Examples `__ documentation. Go ahead and apply the service configuration: .. code:: bash dstack apply -f serve-30b.dstack.yml Access the Service -------------------- After the service is successfully deployed, you can access the service's endpoint in the following ways: .. tab-set:: .. tab-item:: CURL Access through service endpoint at ``/proxy/services///`` .. code:: bash curl http://localhost:3000/proxy/services/main/qwen3-30b-a3b/v1/chat/completions \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer ' \ -d '{ "model": "Qwen/Qwen3-30B-A3B", "messages": [ { "role": "user", "content": "Compose a poem that explains the concept of recursion in programming." } ] }' .. note:: When starting the dstack server, an admin token is automatically generated: .. code:: bash The admin token is "bbae0f28-d3dd-4820-bf61-8f4bb40815da" The server is running at http://127.0.0.1:3000/ .. tab-item:: Chat UI Access through dstack's Chat UI at ``/projects//models//`` .. image:: https://dstack.ai/static-assets/static-assets/images//dstack-qwen-ui.png .. dropdown:: Gateway :icon: info :animate: fade-in Running services for development purposes doesn't require setting up a gateway. However, you'll need a gateway in the following cases: * To use auto-scaling or rate limits * To enable HTTPS for the endpoint and map it to your domain * If your service requires WebSockets * If your service cannot work with a path prefix For detailed information about gateway configuration and usage, refer to the `dstack documentation on gateways `__. Replicas and Auto Scaling ---------------------------------------- You can auto scale the service by specifying additional configurations in the ``serve-30b.dstack.yml``. - Set ``replicas: min..max`` to define the minimum and maximum number of replicas - Configure ``scaling`` rules to determine when to scale up or down Below is a complete configuration example with auto-scaling enabled: .. code:: yaml type: service name: qwen3-30b-a3b image: lmsysorg/sglang:latest env: - MODEL_ID=Qwen/Qwen3-30B-A3B commands: - python3 -m sglang.launch_server --model-path $MODEL_ID --port 8000 --trust-remote-code port: 8000 model: Qwen/Qwen3-30B-A3B resources: gpu: 80GB:1 # Minimum and maximum number of replicas replicas: 1..4 scaling: # Requests per seconds metric: rps # Target metric value target: 10 .. note:: The scaling property requires a gateway to be set up. See also ------------ - **Fleets**: Create cloud and on-prem clusters using `Fleets `__. - **Dev Environments**: Experiment and test before deploying to production using `Dev Environments `__. - **Tasks**: Schedule single node or distributed training using `Tasks `__. - **Services**: Deploy models as secure, auto-scaling OpenAI-compatible endpoints using `Services `__. - **Metrics**: Monitor performance with automatically tracked metrics via CLI or UI using `Metrics `__. --- ### Source Deployment Openllm (`docs/source/deployment/openllm.rst`) OpenLLM ======= .. attention:: To be updated for Qwen3. OpenLLM allows developers to run Qwen2.5 models of different sizes as OpenAI-compatible APIs with a single command. It features a built-in chat UI, state-of-the-art inference backends, and a simplified workflow for creating enterprise-grade cloud deployment with Qwen2.5. Visit `the OpenLLM repository `_ to learn more. Installation ------------ Install OpenLLM using ``pip``. .. code:: bash pip install openllm Verify the installation and display the help information: .. code:: bash openllm --help Quickstart ---------- Before you run any Qwen2.5 model, ensure your model repository is up to date by syncing it with OpenLLM's latest official repository. .. code:: bash openllm repo update List the supported Qwen2.5 models: .. code:: bash openllm model list --tag qwen2.5 The results also display the required GPU resources and supported platforms: .. code:: bash model version repo required GPU RAM platforms ------- --------------------- ------- ------------------ ----------- qwen2.5 qwen2.5:0.5b default 12G linux qwen2.5:1.5b default 12G linux qwen2.5:3b default 12G linux qwen2.5:7b default 24G linux qwen2.5:14b default 80G linux qwen2.5:14b-ggml-q4 default macos qwen2.5:14b-ggml-q8 default macos qwen2.5:32b default 80G linux qwen2.5:32b-ggml-fp16 default macos qwen2.5:72b default 80Gx2 linux qwen2.5:72b-ggml-q4 default macos To start a server with one of the models, use ``openllm serve`` like this: .. code:: bash openllm serve qwen2.5:7b By default, the server starts at ``http://localhost:3000/``. Interact with the model server ------------------------------ With the model server up and running, you can call its APIs in the following ways: .. tab-set:: .. tab-item:: CURL Send an HTTP request to its ``/generate`` endpoint via CURL: .. code-block:: bash curl -X 'POST' \ 'http://localhost:3000/api/generate' \ -H 'accept: text/event-stream' \ -H 'Content-Type: application/json' \ -d '{ "prompt": "Tell me something about large language models.", "model": "Qwen/Qwen2.5-7B-Instruct", "max_tokens": 2048, "stop": null }' .. tab-item:: Python client Call the OpenAI-compatible endpoints with frameworks and tools that support the OpenAI API protocol. Here is an example: .. code-block:: python from openai import OpenAI client = OpenAI(base_url='http://localhost:3000/v1', api_key='na') # Use the following func to get the available models # model_list = client.models.list() # print(model_list) chat_completion = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct", messages=[ { "role": "user", "content": "Tell me something about large language models." } ], stream=True, ) for chunk in chat_completion: print(chunk.choices[0].delta.content or "", end="") .. tab-item:: Chat UI OpenLLM provides a chat UI at the ``/chat`` endpoint for the LLM server at http://localhost:3000/chat. .. image:: ../../source/assets/qwen-openllm-ui-demo.png Model repository ---------------- A model repository in OpenLLM represents a catalog of available LLMs. You can add your own repository to OpenLLM with custom Qwen2.5 variants for your specific needs. See our `documentation to learn details `_. --- ### Source Deployment Sglang (`docs/source/deployment/sglang.md`) # SGLang [SGLang](https://github.com/sgl-project/sglang) is a fast serving framework for large language models and vision language models. To learn more about SGLang, please refer to the [documentation](https://docs.sglang.ai/). ## Environment Setup By default, you can install `sglang` with pip in a clean environment: ```shell pip install "sglang[all]>=0.4.6.post1" ``` If you have encountered issues in installation, please feel free to check the official document for installation ([link](https://docs.sglang.ai/start/install.html)). ## API Service It is easy to build an OpenAI-compatible API service with SGLang, which can be deployed as a server that implements OpenAI API protocol. By default, it starts the server at `http://localhost:30000`. You can specify the address with `--host` and `--port` arguments. Run the command as shown below: ```shell python -m sglang.launch_server --model-path Qwen/Qwen3-8B ``` By default, if the `--model-path` does not point to a valid local directory, it will download the model files from the Hugging Face Hub. To download model from ModelScope, set the following before running the above command: ```shell export SGLANG_USE_MODELSCOPE=true ``` For distributed inference with tensor parallelism, it is as simple as ```shell python -m sglang.launch_server --model-path Qwen/Qwen3-8B --tensor-parallel-size 4 ``` The above command will use tensor parallelism on 4 GPUs. You should change the number of GPUs according to your demand. ### Basic Usage Then, you can use the [create chat interface](https://platform.openai.com/docs/api-reference/chat/completions/create) to communicate with Qwen: ::::{tab-set} :::{tab-item} curl ```shell curl http://localhost:30000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": [ {"role": "user", "content": "Give me a short introduction to large language models."} ], "temperature": 0.6, "top_p": 0.95, "top_k": 20, "max_tokens": 32768 }' ``` ::: :::{tab-item} Python You can use the API client with the `openai` Python SDK as shown below: ```python from openai import OpenAI # Set OpenAI's API key and API base to use SGLang's API server. openai_api_key = "EMPTY" openai_api_base = "http://localhost:30000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) chat_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=[ {"role": "user", "content": "Give me a short introduction to large language models."}, ], max_tokens=32768, temperature=0.6, top_p=0.95, extra_body={ "top_k": 20, }, ) print("Chat response:", chat_response) ``` :::: :::{tip} While the default sampling parameters would work most of the time for thinking mode, it is recommended to adjust the sampling parameters according to your application, and always pass the sampling parameters to the API. ::: ### Thinking & Non-Thinking Modes Qwen3 models will think before respond. This behavior could be controlled by either the hard switch, which could disable thinking completely, or the soft switch, where the model follows the instruction of the user on whether it should think. The hard switch is available in SGLang through the following configuration to the API call. To disable thinking, use ::::{tab-set} :::{tab-item} curl ```shell curl http://localhost:30000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": [ {"role": "user", "content": "Give me a short introduction to large language models."} ], "temperature": 0.7, "top_p": 0.8, "top_k": 20, "max_tokens": 8192, "presence_penalty": 1.5, "chat_template_kwargs": {"enable_thinking": false} }' ``` ::: :::{tab-item} Python You can use the API client with the `openai` Python SDK as shown below: ```python from openai import OpenAI # Set OpenAI's API key and API base to use SGLang's API server. openai_api_key = "EMPTY" openai_api_base = "http://localhost:30000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) chat_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=[ {"role": "user", "content": "Give me a short introduction to large language models."}, ], max_tokens=8192, temperature=0.7, top_p=0.8, presence_penalty=1.5, extra_body={ "top_k": 20, "chat_template_kwargs": {"enable_thinking": True}, }, ) print("Chat response:", chat_response) ``` :::: :::{note} Please note that passing `enable_thinking` is not OpenAI API compatible. The exact method may differ among frameworks. ::: :::{tip} To completely disable thinking, you could use [a custom chat template](../../source/assets/qwen3_nonthinking.jinja) when starting the model: ```shell python -m sglang.launch_server --model-path Qwen/Qwen3-8B --chat-template ./qwen3_nonthinking.jinja ``` The chat template prevents the model from generating thinking content, even if the user instructs the model to do so with `/think`. ::: :::{tip} It is recommended to set sampling parameters differently for thinking and non-thinking modes. ::: ### Parsing Thinking Content SGLang supports parsing the thinking content from the model generation into structured messages: ```shell python -m sglang.launch_server --model-path Qwen/Qwen3-8B --reasoning-parser qwen3 ``` The response message will have a field named `reasoning_content` in addition to `content`, containing the thinking content generated by the model. :::{note} Please note that this feature is not OpenAI API compatible. ::: :::{important} `enable_thinking=False` may not be compatible with this feature. If you need to pass `enable_thinking=False` to the API, please consider disabling parsing thinking content. ::: ### Parsing Tool Calls SGLang supports parsing the tool calling content from the model generation into structured messages: ```shell python -m sglang.launch_server --model-path Qwen/Qwen3-8B --tool-call-parser qwen25 ``` For more information, please refer to [our guide on Function Calling](../framework/function_call.md). ### Structured/JSON Output SGLang supports structured/JSON output. Please refer to [SGLang's documentation](https://docs.sglang.ai/backend/structured_outputs.html#OpenAI-Compatible-API). Besides, it is also recommended to instruct the model to generate the specific format in the system message or in your prompt. ### Serving Quantized models Qwen3 comes with two types of pre-quantized models, FP8 and AWQ. The command serving those models are the same as the original models except for the name change: ```shell # For FP8 quantized model python -m sglang.launch_server --model-path Qwen/Qwen3-8B-FP8 # For AWQ quantized model python -m sglang.launch_server --model-path Qwen/Qwen3-8B-AWQ ``` ### Context Length The context length for Qwen3 models in pretraining is up to 32,768 tokens. To handle context length substantially exceeding 32,768 tokens, RoPE scaling techniques should be applied. We have validated the performance of [YaRN](https://arxiv.org/abs/2309.00071), a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts. SGLang supports YaRN, which can be configured as ```shell python -m sglang.launch_server --model-path Qwen/Qwen3-8B --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}}' --context-length 131072 ``` :::{note} SGLang implements static YaRN, which means the scaling factor remains constant regardless of input length, **potentially impacting performance on shorter texts.** We advise adding the `rope_scaling` configuration only when processing long contexts is required. It is also recommended to modify the `factor` as needed. For example, if the typical context length for your application is 65,536 tokens, it would be better to set `factor` as 2.0. ::: :::{note} The default `max_position_embeddings` in `config.json` is set to 40,960, which is used by SGLang. This allocation includes reserving 32,768 tokens for outputs and 8,192 tokens for typical prompts, which is sufficient for most scenarios involving short text processing and leave adequate room for model thinking. If the average context length does not exceed 32,768 tokens, we do not recommend enabling YaRN in this scenario, as it may potentially degrade model performance. ::: --- ### Source Deployment Skypilot (`docs/source/deployment/skypilot.rst`) SkyPilot ======== .. attention:: To be updated for Qwen3. What is SkyPilot ---------------- SkyPilot is a framework for running LLMs, AI, and batch jobs on any cloud, offering maximum cost savings, the highest GPU availability, and managed execution. Its features include: - Get the best GPU availability by utilizing multiple resources pools across multiple regions and clouds. - Pay absolute minimum — SkyPilot picks the cheapest resources across regions and clouds. No managed solution markups. - Scale up to multiple replicas across different locations and accelerators, all served with a single endpoint - Everything stays in your cloud account (your VMs & buckets) - Completely private - no one else sees your chat history Install SkyPilot ---------------- We advise you to follow the `instruction `__ to install SkyPilot. Here we provide a simple example of using ``pip`` for the installation as shown below. .. code:: bash # You can use any of the following clouds that you have access to: # aws, gcp, azure, oci, lamabda, runpod, fluidstack, paperspace, # cudo, ibm, scp, vsphere, kubernetes pip install "skypilot-nightly[aws,gcp]" After that, you need to verify cloud access with a command like: .. code:: bash sky check For more information, check the `official document `__ and see if you have set up your cloud accounts correctly. Alternatively, you can also use the official docker image with SkyPilot master branch automatically cloned by running: .. code:: bash # NOTE: '--platform linux/amd64' is needed for Apple Silicon Macs docker run --platform linux/amd64 \ -td --rm --name sky \ -v "$HOME/.sky:/root/.sky:rw" \ -v "$HOME/.aws:/root/.aws:rw" \ -v "$HOME/.config/gcloud:/root/.config/gcloud:rw" \ berkeleyskypilot/skypilot-nightly docker exec -it sky /bin/bash Running Qwen2.5-72B-Instruct with SkyPilot ------------------------------------------ 1. Start serving Qwen2.5-72B-Instruct on a single instance with any available GPU in the list specified in `serve-72b.yaml `__ with a vLLM-powered OpenAI-compatible endpoint: .. code:: bash sky launch -c qwen serve-72b.yaml **Before launching, make sure you have changed Qwen/Qwen2-72B-Instruct to Qwen/Qwen2.5-72B-Instruct in the YAML file.** 2. Send a request to the endpoint for completion: .. code:: bash IP=$(sky status --ip qwen) curl -L http://$IP:8000/v1/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen2.5-72B-Instruct", "prompt": "My favorite food is", "max_tokens": 512 }' | jq -r '.choices[0].text' 3. Send a request for chat completion: .. code:: bash curl -L http://$IP:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen2.5-72B-Instruct", "messages": [ { "role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful and honest chat expert." }, { "role": "user", "content": "What is the best food?" } ], "max_tokens": 512 }' | jq -r '.choices[0].message.content' Scale up the service with SkyPilot Serve ---------------------------------------- 1. With `SkyPilot Serve `__, a serving library built on top of SkyPilot, scaling up the Qwen service is as simple as running: .. code:: bash sky serve up -n qwen ./serve-72b.yaml **Before launching, make sure you have changed Qwen/Qwen2-72B-Instruct to Qwen/Qwen2.5-72B-Instruct in the YAML file.** This will start the service with multiple replicas on the cheapest available locations and accelerators. SkyServe will automatically manage the replicas, monitor their health, autoscale based on load, and restart them when needed. A single endpoint will be returned and any request sent to the endpoint will be routed to the ready replicas. 2. To check the status of the service, run: .. code:: bash sky serve status qwen After a while, you will see the following output: :: Services NAME VERSION UPTIME STATUS REPLICAS ENDPOINT Qwen 1 - READY 2/2 3.85.107.228:30002 Service Replicas SERVICE_NAME ID VERSION IP LAUNCHED RESOURCES STATUS REGION Qwen 1 1 - 2 mins ago 1x Azure({'A100-80GB': 8}) READY eastus Qwen 2 1 - 2 mins ago 1x GCP({'L4': 8}) READY us-east4-a As shown, the service is now backed by 2 replicas, one on Azure and one on GCP, and the accelerator type is chosen to be **the cheapest available one** on the clouds. That said, it maximizes the availability of the service while minimizing the cost. 3. To access the model, we use a ``curl -L`` command (``-L`` to follow redirect) to send the request to the endpoint: .. code:: bash ENDPOINT=$(sky serve status --endpoint qwen) curl -L http://$ENDPOINT/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen2.5-72B-Instruct", "messages": [ { "role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful and honest code assistant expert in Python." }, { "role": "user", "content": "Show me the python code for quick sorting a list of integers." } ], "max_tokens": 512 }' | jq -r '.choices[0].message.content' Accessing Qwen2.5 with Chat GUI --------------------------------------------- It is also possible to access the Qwen2.5 service with GUI by connecting a `FastChat GUI server `__ to the endpoint launched above (see `gui.yaml `__). 1. Start the Chat Web UI: .. code:: bash sky launch -c qwen-gui ./gui.yaml --env ENDPOINT=$(sky serve status --endpoint qwen) **Before launching, make sure you have changed Qwen/Qwen1.5-72B-Chat to Qwen/Qwen2.5-72B-Instruct in the YAML file.** 2. Then, we can access the GUI at the returned gradio link: :: | INFO | stdout | Running on public URL: https://6141e84201ce0bb4ed.gradio.live Note that you may get better results by using a different temperature and top_p value. Summary ------- With SkyPilot, it is easy for you to deploy Qwen2.5 on any cloud. We advise you to read the official doc for more usages and updates. Check `this `__ out! --- ### Source Deployment Tgi (`docs/source/deployment/tgi.rst`) TGI ===================== .. attention:: To be updated for Qwen3. Hugging Face's Text Generation Inference (TGI) is a production-ready framework specifically designed for deploying and serving large language models (LLMs) for text generation tasks. It offers a seamless deployment experience, powered by a robust set of features: * `Speculative Decoding `_: Accelerates generation speeds. * `Tensor Parallelism`_: Enables efficient deployment across multiple GPUs. * `Token Streaming`_: Allows for the continuous generation of text. * Versatile Device Support: Works seamlessly with `AMD`_, `Gaudi`_ and `AWS Inferentia`_. .. _AMD: https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/deploy-your-model.html#serving-using-hugging-face-tgi .. _Gaudi: https://github.com/huggingface/tgi-gaudi .. _AWS Inferentia: https://aws.amazon.com/blogs/machine-learning/announcing-the-launch-of-new-hugging-face-llm-inference-containers-on-amazon-sagemaker/#:~:text=Get%20started%20with%20TGI%20on%20SageMaker%20Hosting .. _Tensor Parallelism: https://huggingface.co/docs/text-generation-inference/conceptual/tensor_parallelism .. _Token Streaming: https://huggingface.co/docs/text-generation-inference/conceptual/streaming Installation ----------------- The easiest way to use TGI is via the TGI docker image. In this guide, we show how to use TGI with docker. It's possible to run it locally via Conda or build locally. Please refer to `Installation Guide `_ and `CLI tool `_ for detailed instructions. Deploy Qwen2.5 with TGI ----------------------- 1. **Find a Qwen2.5 Model:** Choose a model from `the Qwen2.5 collection `_. 2. **Deployment Command:** Run the following command in your terminal, replacing ``model`` with your chosen Qwen2.5 model ID and ``volume`` with the path to your local data directory: .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model Using TGI API ------------- Once deployed, the model will be available on the mapped port (8080). TGI comes with a handy API for streaming response: .. code:: bash curl http://localhost:8080/generate_stream -H 'Content-Type: application/json' \ -d '{"inputs":"Tell me something about large language models.","parameters":{"max_new_tokens":512}}' It's also available on OpenAI style API: .. code:: bash curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "", "messages": [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} ], "temperature": 0.7, "top_p": 0.8, "repetition_penalty": 1.05, "max_tokens": 512 }' .. note:: The model field in the JSON is not used by TGI, you can put anything. Refer to the `TGI Swagger UI `_ for a complete API reference. You can also use Python API: .. code:: python from openai import OpenAI # initialize the client but point it to TGI client = OpenAI( base_url="http://localhost:8080/v1/", # replace with your endpoint url api_key="", # this field is not used when running locally ) chat_completion = client.chat.completions.create( model="", # it is not used by TGI, you can put anything messages=[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, ], stream=True, temperature=0.7, top_p=0.8, max_tokens=512, ) # iterate and print stream for message in chat_completion: print(message.choices[0].delta.content, end="") Quantization for Performance ---------------------------- 1. Data-dependent quantization (GPTQ and AWQ) Both GPTQ and AWQ models are data-dependent. The official quantized models can be found from `the Qwen2.5 collection`_ and you can also quantize models with your own dataset to make it perform better on your use case. The following shows the command to start TGI with Qwen2.5-7B-Instruct-GPTQ-Int4: .. code:: bash model=Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --quantize gptq If the model is quantized with AWQ, e.g. Qwen/Qwen2.5-7B-Instruct-AWQ, please use ``--quantize awq``. 2. Data-agnostic quantization EETQ on the other side is not data dependent and can be used with any model. Note that we're passing in the original model (instead of a quantized model) with the ``--quantize eetq`` flag. .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --quantize eetq Multi-Accelerators Deployment ----------------------------- Use the ``--num-shard`` flag to specify the number of accelerators. Please also use ``--shm-size 1g`` to enable shared memory for optimal NCCL performance (`reference `__): .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --num-shard 2 Speculative Decoding -------------------- Speculative decoding can reduce the time per token by speculating on the next token. Use the ``--speculative-decoding`` flag, setting the value to the number of tokens to speculate on (default: 0 for no speculation): .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --speculate 2 The overall performance of speculative decoding highly depends on the type of task. It works best for code or highly repetitive text. More context on speculative decoding can be found `here `__. Zero-Code Deployment with HF Inference Endpoints --------------------------------------------------- For effortless deployment, leverage Hugging Face Inference Endpoints: - **GUI interface:** ``__ - **Coding interface:** ``__ Once deployed, the endpoint can be used as usual. Common Issues ---------------- Qwen2.5 supports long context lengths, so carefully choose the values for ``--max-batch-prefill-tokens``, ``--max-total-tokens``, and ``--max-input-tokens`` to avoid potential out-of-memory (OOM) issues. If an OOM occurs, you'll receive an error message upon startup. The following shows an example to modify those parameters: .. code:: bash model=Qwen/Qwen2.5-7B-Instruct volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.0 --model-id $model --max-batch-prefill-tokens 4096 --max-total-tokens 4096 --max-input-tokens 2048 --- ### Source Deployment Vllm (`docs/source/deployment/vllm.md`) # vLLM We recommend you trying [vLLM](https://github.com/vllm-project/vllm) for your deployment of Qwen. It is simple to use, and it is fast with state-of-the-art serving throughput, efficient management of attention key value memory with PagedAttention, continuous batching of input requests, optimized CUDA kernels, etc. To learn more about vLLM, please refer to the [paper](https://arxiv.org/abs/2309.06180) and [documentation](https://docs.vllm.ai/). ## Environment Setup By default, you can install `vllm` with pip in a clean environment: ```shell pip install "vllm>=0.8.5" ``` Please note that the prebuilt `vllm` has strict dependencies on `torch` and its CUDA versions. Check the note in the official document for installation ([link](https://docs.vllm.ai/en/latest/getting_started/installation.html)) for more help. ## API Service It is easy to build an OpenAI-compatible API service with vLLM, which can be deployed as a server that implements OpenAI API protocol. By default, it starts the server at `http://localhost:8000`. You can specify the address with `--host` and `--port` arguments. Run the command as shown below: ```shell vllm serve Qwen/Qwen3-8B ``` By default, if the model does not point to a valid local directory, it will download the model files from the Hugging Face Hub. To download model from ModelScope, set the following before running the above command: ```shell export VLLM_USE_MODELSCOPE=true ``` For distributed inference with tensor parallelism, it is as simple as ```shell vllm serve Qwen/Qwen3-8B --tensor-parallel-size 4 ``` The above command will use tensor parallelism on 4 GPUs. You should change the number of GPUs according to your demand. ### Basic Usage Then, you can use the [create chat interface](https://platform.openai.com/docs/api-reference/chat/completions/create) to communicate with Qwen: ::::{tab-set} :::{tab-item} curl ```shell curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": [ {"role": "user", "content": "Give me a short introduction to large language models."} ], "temperature": 0.6, "top_p": 0.95, "top_k": 20, "max_tokens": 32768 }' ``` ::: :::{tab-item} Python You can use the API client with the `openai` Python SDK as shown below: ```python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai_api_key = "EMPTY" openai_api_base = "http://localhost:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) chat_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=[ {"role": "user", "content": "Give me a short introduction to large language models."}, ], max_tokens=32768, temperature=0.6, top_p=0.95, extra_body={ "top_k": 20, }, ) print("Chat response:", chat_response) ``` :::: :::{tip} `vllm` will use the sampling parameters from the `generation_config.json` in the model files. While the default sampling parameters would work most of the time for thinking mode, it is recommended to adjust the sampling parameters according to your application, and always pass the sampling parameters to the API. ::: ### Thinking & Non-Thinking Modes Qwen3 models will think before respond. This behavior could be controlled by either the hard switch, which could disable thinking completely, or the soft switch, where the model follows the instruction of the user on whether it should think. The hard switch is available in vLLM through the following configuration to the API call. To disable thinking, use ::::{tab-set} :::{tab-item} curl ```shell curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-8B", "messages": [ {"role": "user", "content": "Give me a short introduction to large language models."} ], "temperature": 0.7, "top_p": 0.8, "top_k": 20, "max_tokens": 8192, "presence_penalty": 1.5, "chat_template_kwargs": {"enable_thinking": false} }' ``` ::: :::{tab-item} Python You can use the API client with the `openai` Python SDK as shown below: ```python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai_api_key = "EMPTY" openai_api_base = "http://localhost:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) chat_response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=[ {"role": "user", "content": "Give me a short introduction to large language models."}, ], max_tokens=8192, temperature=0.7, top_p=0.8, presence_penalty=1.5, extra_body={ "top_k": 20, "chat_template_kwargs": {"enable_thinking": False}, }, ) print("Chat response:", chat_response) ``` :::: :::{note} Please note that passing `enable_thinking` is not OpenAI API compatible. The exact method may differ among frameworks. ::: :::{tip} To completely disable thinking, you could use [a custom chat template](../../source/assets/qwen3_nonthinking.jinja) when starting the model: ```shell vllm serve Qwen/Qwen3-8B --chat-template ./qwen3_nonthinking.jinja ``` The chat template prevents the model from generating thinking content, even if the user instructs the model to do so with `/think`. ::: :::{tip} It is recommended to set sampling parameters differently for thinking and non-thinking modes. ::: ### Parsing Thinking Content vLLM supports parsing the thinking content from the model generation into structured messages: ```shell vllm serve Qwen/Qwen3-8B --enable-reasoning --reasoning-parser deepseek_r1 ``` Since vLLM 0.9.0, one can also use ```shell vllm serve Qwen/Qwen3-8B --reasoning-parser qwen3 ``` The response message will have a field named `reasoning_content` in addition to `content`, containing the thinking content generated by the model. :::{note} Please note that this feature is not OpenAI API compatible. ::: :::{important} As of vLLM 0.8.5, `enable_thinking=False` is not compatible with this feature. If you need to pass `enable_thinking=False` to the API, you should disable parsing thinking content. This is resolved in vLLM 0.9.0 with the `qwen3` reasoning parser. ::: ### Parsing Tool Calls vLLM supports parsing the tool calling content from the model generation into structured messages: ```shell vllm serve Qwen/Qwen3-8B --enable-auto-tool-choice --tool-call-parser hermes ``` For more information, please refer to [our guide on Function Calling](../framework/function_call.md#vllm). ### Structured/JSON Output vLLM supports structured/JSON output. Please refer to [vLLM's documentation](https://docs.vllm.ai/en/stable/serving/openai_compatible_server.html#extra-parameters-for-chat-api) for the `guided_json` parameters. Besides, it is also recommended to instruct the model to generate the specific format in the system message or in your prompt. ### Serving Quantized models Qwen3 comes with two types of pre-quantized models, FP8 and AWQ. The command serving those models are the same as the original models except for the name change: ```shell # For FP8 quantized model vllm serve Qwen/Qwen3-8B-FP8 # For AWQ quantized model vllm serve Qwen/Qwen3-8B-AWQ ``` :::{note} The FP8 models of Qwen3 are block-wise quant, which is supported on NVIDIA GPUs with compute capability > 8.9, that is, Ada Lovelace, Hopper, and later GPUs and runs as w8a8. Since vLLM v0.9.0, FP8 Marlin has supported block-wise quants (running as w8a16) and you can also run Qwen3 FP8 models on Ampere cards. ::: :::{note} If you encountered the following error when deploying the FP8 models, it indicates that the tensor parallel size does not agree with the model weights: ``` File ".../vllm/vllm/model_executor/layers/quantization/fp8.py", line 477, in create_weights raise ValueError( ValueError: The output_size of gate's and up's weight = 192 is not divisible by weight quantization block_n = 128. ``` We recommend lowering the degree of tensor parallel, e.g., `--tensor-parallel-size 4` or enabling expert parallel, e.g., `--tensor-parallel-size 8 --enable-expert-parallel`. ::: ### Context Length The context length for Qwen3 models in pretraining is up to 32,768 tokens. To handle context length substantially exceeding 32,768 tokens, RoPE scaling techniques should be applied. We have validated the performance of [YaRN](https://arxiv.org/abs/2309.00071), a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts. vLLM supports YaRN, which can be configured as ```shell vllm serve Qwen/Qwen3-8B --rope-scaling '{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}' --max-model-len 131072 ``` :::{note} vLLM implements static YaRN, which means the scaling factor remains constant regardless of input length, **potentially impacting performance on shorter texts.** We advise adding the `rope_scaling` configuration only when processing long contexts is required. It is also recommended to modify the `factor` as needed. For example, if the typical context length for your application is 65,536 tokens, it would be better to set `factor` as 2.0. ::: :::{note} The default `max_position_embeddings` in `config.json` is set to 40,960, which used by vLLM, if `--max-model-len` is not specified. This allocation includes reserving 32,768 tokens for outputs and 8,192 tokens for typical prompts, which is sufficient for most scenarios involving short text processing and leave adequate room for model thinking. If the average context length does not exceed 32,768 tokens, we do not recommend enabling YaRN in this scenario, as it may potentially degrade model performance. ::: ## Python Library vLLM can also be directly used as a Python library, which is convenient for offline batch inference but lack some API-only features, such as parsing model generation to structure messages. The following shows the basic usage of vLLM as a library: ```python from transformers import AutoTokenizer from vllm import LLM, SamplingParams # Initialize the tokenizer tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B") # Configurae the sampling parameters (for thinking mode) sampling_params = SamplingParams(temperature=0.6, top_p=0.95, top_k=20, max_tokens=32768) # Initialize the vLLM engine llm = LLM(model="Qwen/Qwen3-8B") # Prepare the input to the model prompt = "Give me a short introduction to large language models." messages = [ {"role": "user", "content": prompt} ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, enable_thinking=True, # Set to False to strictly disable thinking ) # Generate outputs outputs = llm.generate([text], sampling_params) # Print the outputs. for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") ``` Since vLLM v0.9.0, you can also use the `LLM.chat` interface which includes support for `chat_template_kwargs`: ```python from vllm import LLM, SamplingParams # Configurae the sampling parameters (for thinking mode) sampling_params = SamplingParams(temperature=0.6, top_p=0.95, top_k=20, max_tokens=32768) # Initialize the vLLM engine llm = LLM(model="Qwen/Qwen3-8B") # Prepare the input to the model prompt = "Give me a short introduction to large language models." messages = [ {"role": "user", "content": prompt} ] # Generate outputs outputs = llm.chat( [messages], sampling_params, chat_template_kwargs={"enable_thinking": True}, # Set to False to strictly disable thinking ) # Print the outputs. for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") ``` ## FAQ You may encounter OOM issues that are pretty annoying. We recommend two arguments for you to make some fix. - The first one is `--max-model-len`. Our provided default `max_position_embedding` is `40960` and thus the maximum length for the serving is also this value, leading to higher requirements of memory. Reducing it to a proper length for yourself often helps with the OOM issue. - Another argument you can pay attention to is `--gpu-memory-utilization`. vLLM will pre-allocate this much GPU memory. By default, it is `0.9`. This is also why you find a vLLM service always takes so much memory. If you are in eager mode (by default it is not), you can level it up to tackle the OOM problem. Otherwise, CUDA Graphs are used, which will use GPU memory not controlled by vLLM, and you should try lowering it. If it doesn't work, you should try `--enforce-eager`, which may slow down inference, or reduce the `--max-model-len`. For more usage guide with vLLM, please see vLLM's [Qwen3 Usage Guide](https://github.com/vllm-project/vllm/issues/17327). --- ### Source Framework Langchain (`docs/source/framework/Langchain.rst`) Langchain ========================== .. attention:: To be updated for Qwen3. This guide helps you build a question-answering application based on a local knowledge base using ``Qwen2.5-7B-Instruct`` with ``langchain``. The goal is to establish a knowledge base Q&A solution. Basic Usage ----------- The implementation process of this project includes loading files -> reading text -> segmenting text -> vectorizing text -> vectorizing questions -> matching the top k most similar text vectors with the question vectors -> incorporating the matched text as context along with the question into the prompt -> submitting to the Qwen2.5-7B-Instruct to generate an answer. Below is an example: .. code:: bash pip install langchain==0.0.174 pip install faiss-gpu .. code:: python from transformers import AutoModelForCausalLM, AutoTokenizer from abc import ABC from langchain.llms.base import LLM from typing import Any, List, Mapping, Optional from langchain.callbacks.manager import CallbackManagerForLLMRun model_name = "Qwen/Qwen2.5-7B-Instruct" model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype="auto", device_map="auto" ) tokenizer = AutoTokenizer.from_pretrained(model_name) class Qwen(LLM, ABC): max_token: int = 10000 temperature: float = 0.01 top_p = 0.9 history_len: int = 3 def __init__(self): super().__init__() @property def _llm_type(self) -> str: return "Qwen" @property def _history_len(self) -> int: return self.history_len def set_history_len(self, history_len: int = 10) -> None: self.history_len = history_len def _call( self, prompt: str, stop: Optional[List[str]] = None, run_manager: Optional[CallbackManagerForLLMRun] = None, ) -> str: messages = [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt} ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) model_inputs = tokenizer([text], return_tensors="pt").to(model.device) generated_ids = model.generate( **model_inputs, max_new_tokens=512 ) generated_ids = [ output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids) ] response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0] return response @property def _identifying_params(self) -> Mapping[str, Any]: """Get the identifying parameters.""" return {"max_token": self.max_token, "temperature": self.temperature, "top_p": self.top_p, "history_len": self.history_len} After loading the Qwen2.5-7B-Instruct model, you should specify the txt file for retrieval. .. code:: python import os import re import torch import argparse from langchain.vectorstores import FAISS from langchain.embeddings.huggingface import HuggingFaceEmbeddings from typing import List, Tuple import numpy as np from langchain.document_loaders import TextLoader from langchain.text_splitter import CharacterTextSplitter from langchain.docstore.document import Document from langchain.prompts.prompt import PromptTemplate from langchain.chains import RetrievalQA class ChineseTextSplitter(CharacterTextSplitter): def __init__(self, pdf: bool = False, **kwargs): super().__init__(**kwargs) self.pdf = pdf def split_text(self, text: str) -> List[str]: if self.pdf: text = re.sub(r"\n{3,}", "\n", text) text = re.sub('\s', ' ', text) text = text.replace("\n\n", "") sent_sep_pattern = re.compile( '([﹒﹔﹖﹗．。！？]["’”」』]{0,2}|(?=["‘“「『]{1,2}|$))') sent_list = [] for ele in sent_sep_pattern.split(text): if sent_sep_pattern.match(ele) and sent_list: sent_list[-1] += ele elif ele: sent_list.append(ele) return sent_list def load_file(filepath): loader = TextLoader(filepath, autodetect_encoding=True) textsplitter = ChineseTextSplitter(pdf=False) docs = loader.load_and_split(textsplitter) write_check_file(filepath, docs) return docs def write_check_file(filepath, docs): folder_path = os.path.join(os.path.dirname(filepath), "tmp_files") if not os.path.exists(folder_path): os.makedirs(folder_path) fp = os.path.join(folder_path, 'load_file.txt') with open(fp, 'a+', encoding='utf-8') as fout: fout.write("filepath=%s,len=%s" % (filepath, len(docs))) fout.write('\n') for i in docs: fout.write(str(i)) fout.write('\n') fout.close() def separate_list(ls: List[int]) -> List[List[int]]: lists = [] ls1 = [ls[0]] for i in range(1, len(ls)): if ls[i - 1] + 1 == ls[i]: ls1.append(ls[i]) else: lists.append(ls1) ls1 = [ls[i]] lists.append(ls1) return lists class FAISSWrapper(FAISS): chunk_size = 250 chunk_conent = True score_threshold = 0 def similarity_search_with_score_by_vector( self, embedding: List[float], k: int = 4 ) -> List[Tuple[Document, float]]: scores, indices = self.index.search(np.array([embedding], dtype=np.float32), k) docs = [] id_set = set() store_len = len(self.index_to_docstore_id) for j, i in enumerate(indices[0]): if i == -1 or 0 < self.score_threshold < scores[0][j]: # This happens when not enough docs are returned. continue _id = self.index_to_docstore_id[i] doc = self.docstore.search(_id) if not self.chunk_conent: if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {_id}, got {doc}") doc.metadata["score"] = int(scores[0][j]) docs.append(doc) continue id_set.add(i) docs_len = len(doc.page_content) for k in range(1, max(i, store_len - i)): break_flag = False for l in [i + k, i - k]: if 0 <= l < len(self.index_to_docstore_id): _id0 = self.index_to_docstore_id[l] doc0 = self.docstore.search(_id0) if docs_len + len(doc0.page_content) > self.chunk_size: break_flag = True break elif doc0.metadata["source"] == doc.metadata["source"]: docs_len += len(doc0.page_content) id_set.add(l) if break_flag: break if not self.chunk_conent: return docs if len(id_set) == 0 and self.score_threshold > 0: return [] id_list = sorted(list(id_set)) id_lists = separate_list(id_list) for id_seq in id_lists: for id in id_seq: if id == id_seq[0]: _id = self.index_to_docstore_id[id] doc = self.docstore.search(_id) else: _id0 = self.index_to_docstore_id[id] doc0 = self.docstore.search(_id0) doc.page_content += " " + doc0.page_content if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {_id}, got {doc}") doc_score = min([scores[0][id] for id in [indices[0].tolist().index(i) for i in id_seq if i in indices[0]]]) doc.metadata["score"] = int(doc_score) docs.append((doc, doc_score)) return docs if __name__ == '__main__': # load docs (pdf file or txt file) filepath = 'your file path' # Embedding model name EMBEDDING_MODEL = 'text2vec' PROMPT_TEMPLATE = """Known information: {context_str} Based on the above known information, respond to the user's question concisely and professionally. If an answer cannot be derived from it, say 'The question cannot be answered with the given information' or 'Not enough relevant information has been provided,' and do not include fabricated details in the answer. Please respond in English. The question is {question}""" # Embedding running device EMBEDDING_DEVICE = "cuda" # return top-k text chunk from vector store VECTOR_SEARCH_TOP_K = 3 CHAIN_TYPE = 'stuff' embedding_model_dict = { "text2vec": "your text2vec model path", } llm = Qwen() embeddings = HuggingFaceEmbeddings(model_name=embedding_model_dict[EMBEDDING_MODEL],model_kwargs={'device': EMBEDDING_DEVICE}) docs = load_file(filepath) docsearch = FAISSWrapper.from_documents(docs, embeddings) prompt = PromptTemplate( template=PROMPT_TEMPLATE, input_variables=["context_str", "question"] ) chain_type_kwargs = {"prompt": prompt, "document_variable_name": "context_str"} qa = RetrievalQA.from_chain_type( llm=llm, chain_type=CHAIN_TYPE, retriever=docsearch.as_retriever(search_kwargs={"k": VECTOR_SEARCH_TOP_K}), chain_type_kwargs=chain_type_kwargs) query = "Give me a short introduction to large language models." print(qa.run(query)) Next Step --------- Now you can chat with Qwen2.5 use your own document. Continue to read the documentation and try to figure out more advanced usages of model retrieval! --- ### Source Framework LlamaIndex (`docs/source/framework/LlamaIndex.rst`) LlamaIndex ========== .. attention:: To be updated for Qwen3. To connect Qwen2.5 with external data, such as documents, web pages, etc., we offer a tutorial on `LlamaIndex `__. This guide helps you quickly implement retrieval-augmented generation (RAG) using LlamaIndex with Qwen2.5. Preparation -------------------------------------- To implement RAG, we advise you to install the LlamaIndex-related packages first. The following is a simple code snippet showing how to do this: .. code:: bash pip install llama-index pip install llama-index-llms-huggingface pip install llama-index-readers-web Set Parameters -------------------------------------- Now we can set up LLM, embedding model, and the related configurations. Qwen2.5-Instruct supports conversations in multiple languages, including English and Chinese. You can use the ``bge-base-en-v1.5`` model to retrieve from English documents, and you can download the ``bge-base-zh-v1.5`` model to retrieve from Chinese documents. You can also choose ``bge-large`` or ``bge-small`` as the embedding model or modify the context window size or text chunk size depending on your computing resources. Qwen2.5 model families support a maximum of 32K context window size (up to 128K for 7B, 14B, 32B, and 72B, requiring extra configuration) .. code:: python import torch from llama_index.core import Settings from llama_index.core.node_parser import SentenceSplitter from llama_index.llms.huggingface import HuggingFaceLLM from llama_index.embeddings.huggingface import HuggingFaceEmbedding # Set prompt template for generation (optional) from llama_index.core import PromptTemplate def completion_to_prompt(completion): return f"<|im_start|>system\n<|im_end|>\n<|im_start|>user\n{completion}<|im_end|>\n<|im_start|>assistant\n" def messages_to_prompt(messages): prompt = "" for message in messages: if message.role == "system": prompt += f"<|im_start|>system\n{message.content}<|im_end|>\n" elif message.role == "user": prompt += f"<|im_start|>user\n{message.content}<|im_end|>\n" elif message.role == "assistant": prompt += f"<|im_start|>assistant\n{message.content}<|im_end|>\n" if not prompt.startswith("<|im_start|>system"): prompt = "<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n" + prompt prompt = prompt + "<|im_start|>assistant\n" return prompt # Set Qwen2.5 as the language model and set generation config Settings.llm = HuggingFaceLLM( model_name="Qwen/Qwen2.5-7B-Instruct", tokenizer_name="Qwen/Qwen2.5-7B-Instruct", context_window=30000, max_new_tokens=2000, generate_kwargs={"temperature": 0.7, "top_k": 50, "top_p": 0.95}, messages_to_prompt=messages_to_prompt, completion_to_prompt=completion_to_prompt, device_map="auto", ) # Set embedding model Settings.embed_model = HuggingFaceEmbedding( model_name = "BAAI/bge-base-en-v1.5" ) # Set the size of the text chunk for retrieval Settings.transformations = [SentenceSplitter(chunk_size=1024)] Build Index -------------------------------------- Now we can build index from documents or websites. The following code snippet demonstrates how to build an index for files (regardless of whether they are in PDF or TXT format) in a local folder named 'document'. .. code:: python from llama_index.core import VectorStoreIndex, SimpleDirectoryReader documents = SimpleDirectoryReader("./document").load_data() index = VectorStoreIndex.from_documents( documents, embed_model=Settings.embed_model, transformations=Settings.transformations ) The following code snippet demonstrates how to build an index for the content in a list of websites. .. code:: python from llama_index.readers.web import SimpleWebPageReader from llama_index.core import VectorStoreIndex, SimpleDirectoryReader documents = SimpleWebPageReader(html_to_text=True).load_data( ["web_address_1","web_address_2",...] ) index = VectorStoreIndex.from_documents( documents, embed_model=Settings.embed_model, transformations=Settings.transformations ) To save and load the index, you can use the following code snippet. .. code:: python from llama_index.core import StorageContext, load_index_from_storage # save index storage_context = StorageContext.from_defaults(persist_dir="save") # load index index = load_index_from_storage(storage_context) RAG ------------------- Now you can perform queries, and Qwen2.5 will answer based on the content of the indexed documents. .. code:: python query_engine = index.as_query_engine() your_query = "" print(query_engine.query(your_query).response) --- ### Source Framework Function Call (`docs/source/framework/function_call.md`) --- myst: number_code_blocks: ["python3"] --- # Function Calling ## Preface Function calling with large language models is a huge and evolving topic. It is particularly important for AI applications: - either for AI-native applications that strive to work around the shortcomings of current AI technology, - or for existing applications that seeks the integration of AI technology to improve performance, user interaction and experience, or efficiency. We will talk about how Qwen3 can be used to support function calling and how it can be used to achieve your goals, from the inference usage for developing application to the inner workings for hardcore customizations. In this guide, - We will first demonstrate how to use function calling with Qwen3. - Then, we will introduce the technical details on functional calling with Qwen3, which are mainly about the templates. Before starting, there is one thing we have not yet introduced, that is ... ## What is function calling? :::{Note} There is another term "tool use" that may be used to refer to the same concept. While some may argue that tools are a generalized form of functions, at present, their difference exists only technically as different I/O types of programming interfaces. ::: Large language models (LLMs) are powerful things. However, sometimes LLMs by themselves are simply not capable enough. - On the one hand, LLMs have inherent modeling limitations. For one, they do not know things that are not in their training data, which include those happened after their training ended. In addition, they learn things in the way of likelihood, which suggests that they may not be precise enough for tasks with fixed rule sets, e.g., mathematical computation. - On the other hand, it is not easy to use LLMs as a Plug-and-Play service programmatically with other things. LLMs mostly talk in words that are open to interpretation and thus ambiguous, while other software or applications or systems talk in code and through programming interfaces that are pre-defined and fixed and structured. To this end, function calling establishes a common protocol that specifies how LLMs should interact with the other things. The procedure is mainly as follows: 1. The application provides a set of functions and the instructions of the functions to an LLM. 2. The LLM choose to or not to, or is forced to use one or many of the functions, in response to user queries. 3. If the LLM chooses to use the functions, it states how the functions should be used based on the function instructions. 4. The chosen functions are used as such by the application and the results are obtained, which are then given to the LLM if further interaction is needed. There are many ways for LLMs to understand and follow this protocol. As always, the key is prompt engineering or an internalized template known by the model. We recommend using Hermes-style tool use for Qwen3 to maximize function calling performance. ## Inference with Function Calling As function calling is essentially implemented using prompt engineering, you could manually construct the model inputs for Qwen3 models. However, frameworks with function calling support can help you with all that laborious work. In the following, we will introduce the usage (via dedicated function calling chat template) with - **Qwen-Agent**, - **vLLM**. ### The Example Case Let's also use an example to demonstrate the inference usage. We assume **Python 3.11** is used as the programming language. **Scenario**: Suppose we would like to ask the model about the temperature of a location. Normally, the model would reply that it cannot provide real-time information. But we have two tools that can be used to obtain the current temperature of and the temperature at a given date of a city respectively, and we would like the model to make use of them. To set up the example case, you can use the following code: :::{dropdown} Preparation Code :name: prepcode ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ::: In particular, the tools should be described using JSON Schema and the messages should contain as much available information as possible. You can find the explanations of the tools and messages below: :::{dropdown} Example Tools The tools should be described using the following JSON: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` For each **tool**, it is a JSON object with two fields: - `type`: a string specifying the type of the tool, currently only `"function"` is valid - `function`: an object detailing the instructions to use the function For each **function**, it is a JSON object with three fields: - `name`: a string indicating the name of the function - `description`: a string describing what the function is used for - `parameters`: [a JSON Schema](https://json-schema.org/learn/getting-started-step-by-step) that specifies the parameters the function accepts. Please refer to the linked documentation for how to compose a JSON Schema. Notable fields include `type`, `required`, and `enum`. Most frameworks use the tool format and some may use the function format. Which one to use should be obvious according to the naming. ::: :::{dropdown} Example Messages Our query is `What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30.`. ```json [ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."} ] ``` ::: ### Qwen-Agent [Qwen-Agent](https://github.com/QwenLM/Qwen-Agent) is actually a Python Agent framework for developing AI applications. Although its intended use cases are higher-level than efficient inference, it does contain the **canonical implementation** of function calling for Qwen3. It provides the function calling ability for Qwen3 to an OpenAI-compatible API through templates that is transparent to users. It is worth noting that for reasoning models like Qwen3, it is *not recommended* to use tool call template based on stopwords, such as ReAct, because the model may output stopwords in the thought section, potentially leading to unexpected behavior in tool calls. Before starting, let's make sure the latest library is installed: ```bash pip install -U qwen-agent ``` #### Preparing Qwen-Agent can wrap an OpenAI-compatible API that does not support function calling. You can serve such an API with most inference frameworks or obtain one from cloud providers like DashScope or Together. Assuming there is an OpenAI-compatible API at `http://localhost:8000/v1`, Qwen-Agent provides a shortcut function `get_chat_model` to obtain a model inference class with function calling support: ```python from qwen_agent.llm import get_chat_model llm = get_chat_model({ "model": "Qwen/Qwen3-8B", "model_server": "http://localhost:8000/v1", "api_key": "EMPTY", "generate_cfg": { "extra_body": { "chat_template_kwargs": {"enable_thinking": False} # default to True } } }) ``` In the above, `model_server` is the `api_base` common used in other OpenAI-compatible API clients. It is advised to provide the `api_key` (but not via plaintext in the code), even if the API server does not check it, in which case, you can set it to anything. You can pass model parameters to the model by `generate_cfg`. Here we demonstrate how to control the think and no_think modes of Qwen3. Different APIs may have different control methods. For model inputs, the common message structure for system, user, and assistant history should be used: ```python messages = MESSAGES[:] ``` At the time, Qwen-Agent works with functions instead of tools. This requires a small change to our tool descriptions, that is, extracting the function fields: ```python functions = [tool["function"] for tool in TOOLS] ``` #### Tool Calls and Tool Results To interact with the model, the `chat` method should be used: ```python for responses in llm.chat( messages=messages, functions=functions, ): pass messages.extend(responses) ``` The `chat` method returns a generator of list, each of which may contain multiple messages. - The results of `no_think` mode: ```python [ {"role": "assistant", "content": "", "function_call": {"name": "get_current_temperature", "arguments": "{\"location\": \"San Francisco, California, United States\", \"unit\": \"celsius\"}"}}, {"role": "assistant", "content": "", "function_call": {"name": "get_temperature_date", "arguments": "{\"location\": \"San Francisco, California, United States\", \"date\": \"2024-10-01\", \"unit\": \"celsius\"}"}}, ] ``` - The results of `think` mode: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` As we can see, Qwen-Agent attempts to parse the model generation in an easier to use structural format. The details related to function calls are placed in the `function_call` field of the messages: - `name`: a string representing the function to call - `arguments`: a JSON-formatted string representing the arguments the function should be called with In the thinking mode, it will first generate a thought and then generate the tool call(s). Then comes the critical part -- checking and applying the function call: ```python3 for message in responses: if fn_call := message.get("function_call", None): fn_name: str = fn_call['name'] fn_args: dict = json.loads(fn_call["arguments"]) fn_res: str = json.dumps(get_function_by_name(fn_name)(**fn_args)) messages.append({ "role": "function", "name": fn_name, "content": fn_res, }) ``` To get tool results: - line 1: We should iterate the function calls in the order the model generates them. - line 2: We can check if a function call is needed as deemed by the model by checking the `function_call` field of the generated messages. - line 3-4: The related details including the name and the arguments of the function can also be found there, which are `name` and `arguments` respectively. - line 6: With the details, one should call the function and obtain the results. Here, we assume there is a function named [`get_function_by_name`](#prepcode) to help us get the related function by its name. - line 8-12: With the result obtained, add the function result to the messages as `content` and with `role` as `"function"`. Now the messages are: - `no_think` mode: ```python [ {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}, {"role": "assistant", "content": "", "function_call": {"name": "get_current_temperature", "arguments": "{\"location\": \"San Francisco, California, United States\", \"unit\": \"celsius\"}"}}, {"role": "assistant", "content": "", "function_call": {"name": "get_temperature_date", "arguments": "{\"location\": \"San Francisco, California, United States\", \"date\": \"2024-10-01\", \"unit\": \"celsius\"}"}}, {"role": "function", "name": "get_current_temperature", "content": '{"temperature": 26.1, "location": "San Francisco, California, United States", "unit": "celsius"}'}, {"role": "function", "name": "get_temperature_date", "content": '{"temperature": 25.9, "location": "San Francisco, California, United States", "date": "2024-10-01", "unit": "celsius"}'}, ] ``` - `think` mode: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` #### Final Response Finally, run the model again to get the final model results: ```python for responses in llm.chat(messages=messages, functions=functions): pass messages.extend(responses) ``` The final response should be like - `no_think` mode: ```python [ {"role": "assistant", "content": "The current temperature in San Francisco, CA, USA is **26.1°C**. \n\nFor tomorrow (2024-10-01), the temperature is projected to be **25.9°C**. \n\nThere is a slight decrease in temperature expected from today to tomorrow."} ] ``` - `think` mode: ```python [ {"role": "assistant", "content": "", "reasoning_content": "Okay, the user asked for the current temperature in San Francisco and tomorrow's temperature. I called the get_current_temperature function for now and get_temperature_date for tomorrow. The responses came back with 26.1°C today and 25.9°C tomorrow. Let me present this info clearly.\n\nFirst, confirm the location to make sure there's no confusion. The current temp is 26.1°C, so I'll state that. Then, tomorrow's date is 2024-10-01, which is October 1st, so I'll mention the date in a user-friendly way. The temp drops slightly to 25.9°C. I should note the unit is Celsius as per the default. Keep the answer concise but informative. Maybe add a brief note about the slight decrease. Make sure the dates are correctly formatted and the temperatures are accurate based on the data provided."}, {"role": "assistant", "content": "The current temperature in San Francisco, CA, USA is **26.1°C**. \n\nFor tomorrow (2024-10-01), the temperature is projected to be **25.9°C**. \n\nThere is a slight decrease in temperature expected from today to tomorrow."} ] ``` (heading-target)= ### vLLM vLLM is a fast and easy-to-use library for LLM inference and serving. It uses the tokenizer from `transformers` to format the input, so we should have no trouble preparing the input. In addition, vLLm also implements helper functions so that generated tool calls can be parsed automatically if the format is supported. - `vllm` >= v0.8.5. For more information, check the [vLLM documentation](https://docs.vllm.ai/en/stable/serving/openai_compatible_server.html#tool-calling-in-the-chat-completion-api). We will use the OpenAI-Compatible API by `vllm` with the API client from the `openai` Python library. #### Preparing For Qwen3, the chat template in tokenizer_config.json has already included support for the Hermes-style tool use. We simply need to start a OpenAI-compatible API with vLLM: ```bash vllm serve Qwen/Qwen3-8B --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser deepseek_r1 ``` The inputs are the same with those in [the preparation code](#prepcode): ```python tools = TOOLS messages = MESSAGES ``` Let's also initialize the client: ```python from openai import OpenAI openai_api_key = "EMPTY" openai_api_base = "http://localhost:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) model_name = "Qwen/Qwen3-8B" ``` #### Tool Calls and Tool Results We can use the create chat completions endpoint to query the model. Here is an example of the `no_think` mode: ```python response = client.chat.completions.create( model=model_name, messages=messages, tools=tools, temperature=0.7, top_p=0.8, max_tokens=512, extra_body={ "repetition_penalty": 1.05, "chat_template_kwargs": {"enable_thinking": False} # default to True }, ) ``` vLLM should be able to parse the tool calls for us, and the main fields in the response (`response.choices[0]`) should be like ```python Choice( finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage( content=None, role='assistant', function_call=None, tool_calls=[ ChatCompletionMessageToolCall( id='chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', function=Function(arguments='{"location": "San Francisco, CA, USA"}', name='get_current_temperature'), type='function', ), ChatCompletionMessageToolCall( id='chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', function=Function(arguments='{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', name='get_temperature_date'), type='function', ), ], ), stop_reason=None, ) ``` Note that the function arguments are JSON-formatted strings, which Qwen-Agent follows. As before, chances are that there are corner cases where tool calls are generated but they are malformed and cannot be parsed. For production code, we should try parsing by ourselves. Then, we can obtain the tool results and add them to the messages as shown below: ```python messages.append(response.choices[0].message.model_dump()) if tool_calls := messages[-1].get("tool_calls", None): for tool_call in tool_calls: call_id: str = tool_call["id"] if fn_call := tool_call.get("function"): fn_name: str = fn_call["name"] fn_args: dict = json.loads(fn_call["arguments"]) fn_res: str = json.dumps(get_function_by_name(fn_name)(**fn_args)) messages.append({ "role": "tool", "content": fn_res, "tool_call_id": call_id, }) ``` It should be noted that the OpenAI API uses `tool_call_id` to identify the relation between tool results and tool calls. The messages are now like ```python [ {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow? Current Date: 2024-09-30."}, {'content': None, 'role': 'assistant', 'function_call': None, 'tool_calls': [ {'id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', 'function': {'arguments': '{"location": "San Francisco, CA, USA"}', 'name': 'get_current_temperature'}, 'type': 'function'}, {'id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', 'function': {'arguments': '{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', 'name': 'get_temperature_date'}, 'type': 'function'}, ]}, {'role': 'tool', 'content': '{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}', 'tool_call_id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15'}, {'role': 'tool', 'content': '{"temperature": 25.9, "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}', 'tool_call_id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501'}, ] ``` #### Final Response Let's call the endpoint again to seed the tool results and get response: ```python response = client.chat.completions.create( model=model_name, messages=messages, tools=tools, temperature=0.7, top_p=0.8, max_tokens=512, extra_body={ "repetition_penalty": 1.05, }, ) messages.append(response.choices[0].message.model_dump()) ``` The final response (`response.choices[0].message.content`) should be like ```text The current temperature in San Francisco is approximately 26.1°C. For tomorrow, the forecasted temperature is around 25.9°C. ``` ## Finally In whichever way you choose to use function calling with Qwen3, keep in mind that the limitation and the perks of prompt engineering applies: - It is not guaranteed that the model generation will always follow the protocol even with proper prompting or templates. Especially, for the templates that are more complex and relies more on the model itself to think and stay on track than the ones that are simpler and relies on the template and the use of control or special tokens. The latter one, of course, requires some kind of training. In production code, be prepared that if it breaks, countermeasures or rectifications are in place. - If in certain scenarios, the generation is not up to expectation, you can refine the template to add more instructions or constraints. While the templates mentioned here are general enough, they may not be the best or the most specific or the most concise for your use cases. The ultimate solution is fine-tuning using your own data. Have fun prompting! --- ### Source Framework Qwen Agent (`docs/source/framework/qwen_agent.rst`) Qwen-Agent ========== `Qwen-Agent `__ is a framework for developing LLM applications based on the instruction following, tool usage, planning, and memory capabilities of Qwen. This is a simple tutorial on using Qwen-Agent to quickly experience the agentic capabilities of Qwen3. For more detailed information, please refer to `Qwen-Agent `__ repository. Installation ------------ - Install the stable version from PyPI: .. code:: bash pip install -U "qwen-agent[gui,rag,code_interpreter,mcp]" # Or use `pip install -U qwen-agent` for the minimal requirements. # The optional requirements, specified in double brackets, are: # [gui] for Gradio-based GUI support; # [rag] for RAG support; # [code_interpreter] for Code Interpreter support; # [mcp] for MCP support. Developing Your Own Agent ------------------------- Qwen3 excels in tool calling capabilities. Qwen-Agent encapsulates tool-calling templates and tool-calling parsers internally, greatly reducing coding complexity. To define the available tools, you can use the MCP configuration file, use the integrated tool of Qwen-Agent, or integrate other tools by yourself. .. code:: python import os from qwen_agent.agents import Assistant # Define LLM llm_cfg = { # Use a custom endpoint compatible with OpenAI API by vLLM/SGLang: 'model': 'Qwen/Qwen3-32B', 'model_server': 'http://localhost:8000/v1', # api_base 'api_key': 'EMPTY', # 'generate_cfg': { # # When using vLLM/SGLang OAI API, pass the parameter of whether to enable thinking mode in this way # 'extra_body': { # 'chat_template_kwargs': {'enable_thinking': False} # }, # # # Add: When the content is `this is the thoughtthis is the answer` # # Do not add: When the response has been separated by reasoning_content and content # # This parameter will affect the parsing strategy of tool call # # 'thought_in_content': True, # }, } # llm_cfg = { # # Use the model service provided by DashScope: # 'model': 'qwen3-235b-a22b', # 'model_type': 'qwen_dashscope', # # # 'generate_cfg': { # # # When using the Dash Scope API, pass the parameter of whether to enable thinking mode in this way # # 'enable_thinking': False, # # }, # } # llm_cfg = { # # Use the OpenAI-compatible model service provided by DashScope: # 'model': 'qwen3-235b-a22b', # 'model_server': 'https://dashscope.aliyuncs.com/compatible-mode/v1', # 'api_key': os.getenv('DASHSCOPE_API_KEY'), # # # 'generate_cfg': { # # # When using Dash Scope OAI API, pass the parameter of whether to enable thinking mode in this way # # 'extra_body': { # # 'enable_thinking': False # # }, # # }, # } # Define Tools tools = [ {'mcpServers': { # You can specify the MCP configuration file 'time': { 'command': 'uvx', 'args': ['mcp-server-time', '--local-timezone=Asia/Shanghai'] }, "fetch": { "command": "uvx", "args": ["mcp-server-fetch"] } } }, 'code_interpreter', # Built-in tools ] # Define Agent bot = Assistant(llm=llm_cfg, function_list=tools) # Streaming generation messages = [{'role': 'user', 'content': 'https://qwenlm.github.io/blog/ Introduce the latest developments of Qwen'}] for responses in bot.run(messages=messages): pass print(responses) For more detailed examples and MCP cookbooks, please refer to `Qwen-Agent `__ repository. --- ### Source Getting Started Concepts (`docs/source/getting_started/concepts.md`) # Key Concepts ## Qwen Qwen (Chinese: 通义千问; pinyin: _Tongyi Qianwen_) is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. Qwen is capable of natural language understanding, text generation, vision understanding, audio understanding, tool use, role play, playing as AI agent, etc. Both language models and multimodal models are pre-trained on large-scale multilingual and multimodal data and post-trained on quality data for aligning to human preferences. There is the proprietary/closed source version and the open-weight version. You can learn more about the proprietary models at Alibaba Cloud Model Studio ([China Site](https://help.aliyun.com/zh/model-studio/getting-started/models#9f8890ce29g5u) \[zh\], [International Site](https://www.alibabacloud.com/en/product/modelstudio)). In this document, our focus is Qwen, the language models. ## Qwen3 Qwen3 is the newest edition of the Qwen language models, featuring balanced model sizes, enhanced capbilities, hybrid thinking modes, and more language support: - The mixture-of-experts (MoE) models are reintroduced with the Qwen3-30B-A3B and Qwen3-235B-A22B. The largest dense models are now Qwen3-32B. - Hybrid thinking mode is designed so that thinking and non-thinking (instruct) can be achieved without changing models, simplifying deployment and making alternating thinking and non-thinking in a single chat possible. - With 119 languages (and dialects), Qwen3's extensive multilingual capability opens up new possibilities for international applications. - Qwen3 models are optimized for coding and agentic capabilities, with strengthened support of Model Context Protocol (MCP) as well. ## Naming Starting with Qwen3, the models are named using the scheme `Qwen3[-size][-type][-date]`: - `size`: the notation of the structure and the parameter counts. Dense models use the total saved parameters, e.g., `4B` and `32B`, while MoE models use the total saved parameters and the activated parameters for each token with a prepended `A`, e.g., `30B-A3B` and `235B-A22B`. - `type`: there are currently 4 types: - `-Instruct`: the instruction following models that follow the predefined chat template, used for conducting tasks in conversations, downstream fine-tuning, etc. - `-Thinking`: the thinking models that follow the predefined chat template and use chain-of-thoughts (CoT) to think deeply about the questions, used for solving complex problems. - `-Base`: the pre-trained models that do not know the predefined chat template, used for in-context learning, downstream fine-tuning, etc. - No type: the models with hybrid thinking modes. - `date`: the released date in yearmonth format, e.g., `2507`. ## Tokens & Tokenization Tokens represent the fundamental units that models process and generate. They can represent texts in human languages (regular tokens) or represent specific functionality like keywords in programming languages (control tokens [^special]). Typically, a tokenizer is used to split text into regular tokens, which can be words, subwords, or characters depending on the specific tokenization scheme employed, and furnish the token sequence with control tokens as needed. The vocabulary size, or the total number of unique tokens a model recognizes, significantly impacts its performance and versatility. Larger language models often use sophisticated tokenization methods to handle the vast diversity of human language while keeping the vocabulary size manageable. Qwen use a relatively large vocabulary of 151,646 tokens in total. [^special]: Control tokens can be called special tokens. However, the meaning of special tokens need to be interpreted based on the contexts: special tokens may contain extra regular tokens. ### Byte-level Byte Pair Encoding Qwen adopts a subword tokenization method called Byte Pair Encoding (BPE), which attempts to learn the composition of tokens that can represent the text with the fewest tokens. For example, the string ` tokenization` is decomposed as ` token` and `ization` (note that the space is part of the token). Especially, the tokenization of Qwen ensures that there is no unknown words and all texts can be transformed to token sequences. There are 151,643 tokens as a result of BPE in the vocabulary of Qwen, which is a large vocabulary efficient for diverse languages. As a rule of thumb, 1 token is 3~4 characters for English texts and 1.5~1.8 characters for Chinese texts. ### Control Tokens Control tokens are special tokens inserted into the sequence that signifies meta information. For example, in pre-training, multiple documents may be packed into a single sequence. For Qwen, the control token `<|endoftext|>` is inserted after each document to signify that the document has ended and a new document will proceed. Common control tokens and their status with respect to Qwen can be found in the following table: | Type | Qwen (training) | Note | | :-- | :-- | :-- | | eod token | `<\|endoftext\|>` | end of document, which are inserted between documents inside a packed training sequence | | bot token | `<\|im_start\|>` | start of each turn, which is prepended to each turn | | eot token | `<\|im_end\|>` | end of each turn, which is appended to each turn | | unk token | no unk token | BBPE ensures no unknown tokens for Qwen. | | pad token | no pad token | Qwen does not make use of padded sequence in training. One could use any special token together with the attention masks returned by the tokenizer. It is commonly set the same as eod for Qwen. | | bos token | no bos token | Qwen does not prepend a fixed token to each packed training sequence.[^boseos] | | eos token | no eos token | Qwen does not append a fixed token to each packed training sequence. However, as most frameworks do not have the concept of eot and use eos instead for stopping criteria in inference, eos token is set to eot for Qwen.[^boseos] | [^boseos]: bos token should not be set to `<\|im_start\|>` or you may see double bot tokens for the first turn in fine-tuning. eos token set to `<\|im_end\|>` is fine, because double eot tokens for the last turn are less harmful in fine-tuning. ## Chat Template Chat templates provide a structured format for conversational interactions, where predefined placeholders or prompts are used to elicit responses from the model that adhere to a desired dialogue flow or context. Different models may use different kinds of chat template to format the conversations. It is crucial to use the designated one to ensure the precise control over the LLM's generation process. Qwen uses the following format (ChatML[^chatml]), making use of control tokens to format each turn in the conversations ```text <|im_start|>{{role}} {{content}}<|im_end|> ``` The user input takes the role of `user` and the model generation takes the role of `assistant`. Qwen also supports the meta message that instruct the model to perform specific actions or generate text with certain characteristics, such as altering tone, style, or content, which takes the role of `system`. Starting with Qwen3, no default system messages are used. The following is a full example: ```text <|im_start|>system You are a cat.<|im_end|> <|im_start|>user hello<|im_end|> <|im_start|>assistant *Meow~* Hello there! The sun is shining so brightly today, and I'm feeling extra fluffy. Did you bring me a treat? 🐾<|im_end|> <|im_start|>user Explain large language models like I'm 5.<|im_end|> <|im_start|>assistant *Paws at a toy, then looks up with curious eyes* Hey there! 🐾 Imagine you have a super-smart robot friend who loves to talk and play. This robot has *gigantic* brainpower (like a million puzzle pieces all stuck together!) and knows *everything* about stories, animals, and even how to make up new words. When you ask it a question, like “What’s a rainbow?” it uses its brain to find the answer and then *tells you* it in a way that makes sense. It can even help you write a story or solve a puzzle! But here’s the magic: it’s not just a robot—it’s like a *super-duper* smart helper that learns more every day. It’s like having a friend who’s always curious and wants to help you explore the world! 🌟 *Meow~* Want to ask it something fun? 😺<|im_end|><|endoftext|> ``` [^chatml]: For historical reference only, ChatML is first described by the OpenAI Python SDK. The last available version is [this](https://github.com/openai/openai-python/blob/v0.28.1/chatml.md). Please also be aware that that document lists use cases intended for OpenAI models. For Qwen2.5 models, please only use as in our guide. ### Tool Calling Qwen3 supports tool calling or function calling and uses a template akin to [Hermes](https://github.com/NousResearch/Hermes-Function-Calling#prompt-format-for-function-calling). The template is as follows: ```text <|im_start|>system # Tools You may call one or more functions to assist with the user query. You are provided with function signatures within XML tags: {{JSON Schema of function 1}} {{JSON Schema of function 2}} For each function call, return a json object with function name and arguments within XML tags: {"name": , "arguments": } <|im_end|> <|im_start|>user {{user content}}<|im_end|> <|im_start|>assistant {{tool call 1}} {{tool call 2}} <|im_end|> <|im_start|>user {{tool result 1}} {{tool result 2}} <|im_end|> <|im_start|>assistant {{assistant content}}<|im_end|> ``` It should be noted that - The models support parallel tool calling and mulit-turn/multi-step tool calling. - There may be additional content in assistant messages containing tool calls. - The arguments field in the generated tool calls should be of type object instead of type string. - Tool results are treated as special user messages. In general, we recommend using the tokenizer to format the tool calls or let Qwen-Agent handle the formatting. ### Thinking Qwen3 supports thinking mode and uses a structured format for thinking content, which uses the `` and `` tokens to separate the thinking content from the regular response. The template for the final round is as follows: ```text <|im_start|>user {{user content}}<|im_end|> <|im_start|>assistant {{thinking content}} {{assistant content}}<|im_end|> ``` The thinking block should only be included in the final round except for multi-step tool calls. ## Causal Language Models Causal language models, also known as autoregressive language models or decoder-only language models, are a type of machine learning model designed to predict the next token in a sequence based on the preceding tokens. In other words, they generate text one token at a time, using the previously generated tokens as context. The "causal" aspect refers to the fact that the model only considers the past context (the already generated tokens) when predicting the next token, not any future tokens. Causal language models are widely used for various natural language processing tasks involving text completion and generation. They have been particularly successful in generating coherent and contextually relevant text, making them a cornerstone of modern natural language understanding and generation systems. Qwen models are causal language models suitable for text completion. ### Context Length As Qwen models are causal language models, in theory there is only one length limit of the entire sequence. However, since there is often packing in training and each sequence may contain multiple individual pieces of texts. **How long the model can generate or complete ultimately depends on the use case and in that case how long each document (for pre-training) or each turn (for post-training) is in training.** For Qwen3, the packed sequence length in pre-training is 32,768 tokens and may be extended to 131,072 tokens if mentioned in the modelcards. The maximum length of the assistant message is 38,912 tokens for thinking modes and 16,384 tokens for non-thinking modes. For Qwen3-2507, the packed sequence length in pre-training is 262,144 tokens and may be extended to 1M tokens. The maximum length of the assistant message is 81,920 tokens for thinking models and 16,384 tokens for instruct models. :::{tip} In our testing, we find that the post-trained models could generate coherent content that is far longer than what is trained on, e.g., from 16,384 tokens to 32,768 tokens, especially for coding and similar tasks that have "clear rules". In general, we advise that one should evaluate the quality of the generated content of different lengths before determining the optimal generation length. ::: --- ### Source Getting Started Quantization Benchmark (`docs/source/getting_started/quantization_benchmark.rst`) Performance of Quantized Models ================================== .. attention:: To be updated for Qwen3. This section reports the generation performance of quantized models (including GPTQ and AWQ) of the Qwen2 series. Specifically, we report: * MMLU (Accuracy) * C-Eval (Accuracy) * IFEval (Strict Prompt-Level Accuracy) We use greedy decoding in evaluating all models. +---------------------+--------------+---------+-------+--------+--------+ | | Quantization | Average | MMLU | C-Eval | IFEval | +=====================+==============+=========+=======+========+========+ | Qwen2-72B-Instruct | BF16 | 81.3 | 82.3 | 83.8 | 77.6 | + +--------------+---------+-------+--------+--------+ | | GPTQ-Int8 | 80.7 | 81.3 | 83.4 | 77.5 | + +--------------+---------+-------+--------+--------+ | | GPTQ-Int4 | 81.2 | 80.8 | 83.9 | 78.9 | + +--------------+---------+-------+--------+--------+ | | AWQ | 80.4 | 80.5 | 83.9 | 76.9 | +---------------------+--------------+---------+-------+--------+--------+ | Qwen2-7B-Instruct | BF16 | 66.9 | 70.5 | 77.2 | 53.1 | + +--------------+---------+-------+--------+--------+ | | GPTQ-Int8 | 66.2 | 69.1 | 76.7 | 52.9 | + +--------------+---------+-------+--------+--------+ | | GPTQ-Int4 | 64.1 | 67.8 | 75.2 | 49.4 | + +--------------+---------+-------+--------+--------+ | | AWQ | 64.1 | 67.4 | 73.6 | 51.4 | +---------------------+--------------+---------+-------+--------+--------+ | Qwen2-1.5B-Instruct | BF16 | 48.4 | 52.4 | 63.8 | 29.0 | + +--------------+---------+-------+--------+--------+ | | GPTQ-Int8 | 48.1 | 53.0 | 62.5 | 28.8 | + +--------------+---------+-------+--------+--------+ | | GPTQ-Int4 | 45.0 | 50.7 | 57.4 | 27.0 | + +--------------+---------+-------+--------+--------+ | | AWQ | 46.5 | 51.6 | 58.1 | 29.9 | +---------------------+--------------+---------+-------+--------+--------+ | Qwen2-0.5B-Instruct | BF16 | 34.4 | 37.9 | 45.2 | 20.0 | + +--------------+---------+-------+--------+--------+ | | GPTQ-Int8 | 32.6 | 35.6 | 43.9 | 18.1 | + +--------------+---------+-------+--------+--------+ | | GPTQ-Int4 | 29.7 | 33.0 | 39.2 | 16.8 | + +--------------+---------+-------+--------+--------+ | | AWQ | 31.1 | 34.4 | 42.1 | 16.7 | +---------------------+--------------+---------+-------+--------+--------+ --- ### Source Getting Started Speed Benchmark (`docs/source/getting_started/speed_benchmark.md`) # Speed Benchmark We report the speed performance of bfloat16 models and quantized models (including FP8, GPTQ, AWQ) of the Qwen3 series. Specifically, we report the inference speed (tokens/s) as well as memory footprint (GB) under different context lengths. ## Environments ### Hugging Face Transformers - **Hardware**: - NVIDIA H20 96GB - **Software for Non-AutoAWQ**: - PyTorch 2.6.0 - Flash Attention 2.7.4 - Transformers 4.51.3 - GPTQModel 2.2.0+cu128torch2.6 - **Software for AutoAWQ**: - PyTorch 2.6.0+cu124 - Transformers 4.51.3 - AutoAWQ 0.2.9 - AutoAWQ_kernels 0.0.9 ### SGLang - **Hardware**: - NVIDIA H20 96GB - **Software**: - PyTorch 2.6.0+cu124 - Transformers 4.51.3 - SGLang 0.4.6.post1 - SGL-kernel 0.1.0 - vLLM 0.7.2 (Required by SGLang for AWQ quantization) ## Notes - **Inference Speed (tokens/s)** is calculated as: ```{math} \text{Speed} = \frac{\text{tokens}_{\text{prompt}} + \text{tokens}_{\text{generation}}}{\text{time}} ``` - We use a **batch size of 1** and the **minimum number of GPUs** possible for evaluation. - We test the **speed and memory usage** when generating **2048 tokens**, with input lengths of `1`, `6144`, `14336`, `30720`, `63488`, and `129024` tokens. - **For SGLang**: - **Memory usage** is not reported because SGLang pre-allocates all GPU memory. By default, we set `mem_fraction_static=0.85`. - We configure `context_length=140000` and enable `enable_mixed_chunk=True`. - For **AWQ quantization**, we use the **awq_marlin** backend. - We set `skip_tokenizer_init=True` and perform generation using `input_ids` instead of raw text prompts. - **FP8 Performance in Transformers**: The inference speed of Transformers in FP8 mode is currently not optimal and requires further optimization. - **GPTQ-INT4 Performance in SGLang**: The performance of GPTQ-INT4 in SGLang also needs improvement, and we are actively working with the team to enhance it. ## Results ### Qwen3-0.6B (SGLang)
Model Input Length Quantization GPU Num Speed (tokens/s) Note
Qwen3-0.6B 1BF161414.17
FP81458.03
GPTQ-Int81344.92
6144BF1611426.46
FP811572.95
GPTQ-Int811234.29
14336BF1612478.02
FP812689.08
GPTQ-Int812198.82
30720BF1613577.42
FP813819.86
GPTQ-Int813342.06
### Qwen3-0.6B (Transformers)
Model Input Length Quantization GPU Num Speed (tokens/s) GPU Memory(MB)
Qwen3-0.6B 1BF16158.571394
FP8124.601217
GPTQ-Int8126.56986
6144BF161154.822066
FP8173.961943
GPTQ-Int8193.841658
14336BF161168.482963
FP81104.992839
GPTQ-Int81219.612554
30720BF161175.934755
FP81132.784632
GPTQ-Int81345.714347
### Qwen3-1.7B (SGLang)
Model Input Length Quantization GPU Num Speed (tokens/s) Note
Qwen3-1.7B 1BF161227.80
FP81333.90
GPTQ-Int81257.40
6144BF161838.28
FP811198.20
GPTQ-Int81945.91
14336BF1611525.71
FP812095.61
GPTQ-Int811707.63
30720BF1612439.03
FP813165.32
GPTQ-Int812706.16
### Qwen3-1.7B (Transformers)
Model Input Length Quantization GPU Num Speed (tokens/s) GPU Memory(MB)
Qwen3-1.7B 1BF16159.833412
FP8123.832726
GPTQ-Int8128.062229
6144BF161238.534213
FP8190.873462
GPTQ-Int81110.822901
14336BF161352.595109
FP81153.374359
GPTQ-Int81222.783798
30720BF161418.136902
FP81235.616151
GPTQ-Int81386.855590
### Qwen3-4B (SGLang)
Model Input Length Quantization GPU Num Speed (tokens/s) Note
Qwen3-4B 1BF161133.13
FP81200.61
AWQ-INT41199.71
6144BF161466.19
FP81662.26
AWQ-INT41640.07
14336BF161789.25
FP811066.23
AWQ-INT411006.23
30720BF1611165.75
FP811467.71
AWQ-INT411358.84
63488BF1611423.98
FP811660.67
AWQ-INT411513.97
129042BF1611371.04
FP811497.27
AWQ-INT411375.71
### Qwen3-4B (Transformers)
Model Input Length Quantization GPU Num Speed (tokens/s) GPU Memory(MB)
Qwen3-4B 1BF16145.947973
FP8117.335281
AWQ-INT4151.572915
6144BF161159.958860
FP8160.556144
AWQ-INT41183.043881
14336BF161195.3110012
FP8196.817297
AWQ-INT41265.225151
30720BF161217.9712317
FP81138.849611
AWQ-INT41481.697742
### Qwen3-8B (SGLang)
Model Input Length Quantization GPU Num Speed (tokens/s) Note
Qwen3-8B 1BF16181.73
FP81150.25
AWQ-INT41144.11
6144BF161296.25
FP81516.64
AWQ-INT41477.89
14336BF161524.70
FP81859.92
AWQ-INT41770.44
30720BF161832.67
FP811242.24
AWQ-INT411075.91
63488BF1611112.78
FP811476.46
AWQ-INT411254.91
129042BF1611173.32
FP811393.21
AWQ-INT411198.06
### Qwen3-8B (Transformers)
Model Input Length Quantization GPU Num Speed (tokens/s) GPU Memory(MB)
Qwen3-8B 1BF16145.3215947
FP8115.469323
AWQ-INT4151.336177
6144BF161146.1216811
FP8155.0710187
AWQ-INT41163.237113
14336BF161183.2917963
FP8189.6411340
AWQ-INT41242.978409
30720BF161208.9820267
FP81130.9313644
AWQ-INT41438.6211001
### Qwen3-14B (SGLang)
Model Input Length Quantization GPU Num Speed (tokens/s) Note
Qwen3-14B 1BF16147.10
FP8197.11
AWQ-INT4196.49
6144BF161174.85
FP81342.95
AWQ-INT41321.62
14336BF161317.56
FP81587.33
AWQ-INT41525.74
30720BF161525.80
FP81880.72
AWQ-INT41744.74
63488BF161742.36
FP811089.04
AWQ-INT41884.06
129042BF161826.15
FP811049.64
AWQ-INT41857.56
### Qwen3-14B (Transformers)
Model Input Length Quantization GPU Num Speed (tokens/s) GPU Memory (MB)
Qwen3-14B 1BF16140.6628402
FP8113.0216012
AWQ-INT4144.679962
6144BF161108.5229495
FP8144.8616972
AWQ-INT41128.0811020
14336BF161136.3630775
FP8171.9618253
AWQ-INT41220.6212438
30720BF161155.3833336
FP81102.6320813
AWQ-INT41363.2515323
### Qwen3-32B (SGLang)
Model Input Length Quantization GPU Num Speed (tokens/s) Note
Qwen3-32B 1BF16120.72
FP8146.17
AWQ-INT4147.67
6144BF16177.82
FP81165.71
AWQ-INT41159.99
14336BF161143.08
FP81287.60
AWQ-INT41260.44
30720BF161240.75
FP81436.59
AWQ-INT41366.84
63488BF161342.96
FP81532.18
AWQ-INT41425.23
129042BF162711.40TP=2
FP81491.45
AWQ-INT41395.96
### Qwen3-32B (Transformers)
Model Input Length Quantization GPU Num Speed (tokens/s) GPU Memory (MB)
Qwen3-32B 1BF16126.2462751
FP817.3733379
AWQ-INT4141.819109
6144BF16151.4164583
FP8123.5734915
AWQ-INT4168.7120795
14336BF16162.4166632
FP8136.3036963
AWQ-INT41107.0223105
30720BF16169.1670728
FP8149.4441060
AWQ-INT41188.1127718
### Qwen3-30B-A3B (SGLang)
Model Input Length Quantization GPU Num Speed (tokens/s) Note
Qwen3-30B-A3B 1BF161137.18
FP81155.55
GPTQ-INT4131.29GPTQ-Marlin
6144BF161490.10
FP81551.34
GPTQ-INT41120.13GPTQ-Marlin
14336BF161849.62
FP81945.13
GPTQ-INT41227.27GPTQ-Marlin
30720BF1611283.94
FP811405.91
GPTQ-INT41404.45GPTQ-Marlin
63488BF1611538.79
FP811647.89
GPTQ-INT41617.09GPTQ-Marlin
129042BF1611385.65
FP811442.14
GPTQ-INT41704.82GPTQ-Marlin
### Qwen3-30B-A3B (Transformers)
Model Input length Quantization GPU Num Speed (tokens/s) GPU Memory (MB) Notes
Qwen3-30B-A3B 1BF1611.8958462
FP810.4430296
GPTQ-INT4---MoE Kernel Unsupported
6144BF1617.4559037
FP811.7730872
GPTQ-INT4---MoE Kernel Unsupported
14336BF16114.4759806
FP813.531641
GPTQ-INT4---MoE Kernel Unsupported
30720BF16127.0361342
FP816.8633177
GPTQ-INT4---MoE Kernel Unsupported
### Qwen3-235B-A22B (SGLang)
Model Input Length Quantization GPU Num Speed (tokens/s) Note
Qwen3-235B-A22B 1BF16874.50TP=8
FP8471.65TP=4
GPTQ-INT4414.69TP=4
GPTQ-Marlin
6144BF168289.03TP=8
FP84275.16TP=4
GPTQ-INT4456.97TP=4
GPTQ-Marlin
14336BF168546.73TP=8
FP84514.23TP=4
GPTQ-INT44109.13TP=4
GPTQ-Marlin
30720BF168979.41TP=8
FP84887.90TP=4
GPTQ-INT44198.99TP=4
GPTQ-Marlin
63488BF1681493.91TP=8
FP841269.34TP=4
GPTQ-INT44422.77TP=4
GPTQ-Marlin
129042BF1681639.54TP=8
FP841319.66TP=4
GPTQ-INT44552.28TP=4
GPTQ-Marlin
--- ### Source Getting Started Thinking Budget (`docs/source/getting_started/thinking_budget.md`) # Thinking budget This example demonstrates the inference process with thinking budgets using Qwen3 series models. The process involves two steps: 1. the model generates reasoning content within the specified thinking budget 2. append the reasoning content to the conversation context and call the model again to get the final response ## Environment Setup - `transformers >= 4.51.0` - `openai >= 1.65.0` ## Basic Usage First, you should start a Qwen3 model in thinking mode. You can refer to [Quickstart](https://github.com/QwenLM/Qwen3/blob/main/docs/source/getting_started/quickstart.md) for more details. Then, you can use the following code to call the model with thinking budgets. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ### Source Inference Transformers (`docs/source/inference/transformers.md`) # Transformers Transformers is a library of pretrained natural language processing for inference and training. Developers can use Transformers to train models on their data, build inference applications, and generate texts with large language models. ## Environment Setup - `transformers>=4.51.0` - `torch>=2.6` is recommended - GPU is recommended ## Basic Usage You can use the `pipeline()` interface or the `generate()` interface to generate texts with Qwen3 in transformers. In general, the pipeline interface requires less boilerplate code, which is shown here. The following shows a basic example using pipeline for multi-turn conversations: ```python from transformers import pipeline model_name_or_path = "Qwen/Qwen3-8B" generator = pipeline( "text-generation", model_name_or_path, torch_dtype="auto", device_map="auto", ) messages = [ {"role": "user", "content": "Give me a short introduction to large language models."}, ] messages = generator(messages, max_new_tokens=32768)[0]["generated_text"] # print(messages[-1]["content"]) messages.append({"role": "user", "content": "In a single sentence."}) messages = generator(messages, max_new_tokens=32768)[0]["generated_text"] # print(messages[-1]["content"]) ``` There are some important parameters creating the pipeline: - **Model**: `model_name_or_path` could be a model ID like `Qwen/Qwen3-8B` or a local path. To download model files to a local directory, you could use ```shell huggingface-cli download --local-dir ./Qwen3-8B Qwen/Qwen3-8B ``` You can also download model files using ModelScope if you are in mainland China ```shell modelscope download --local_dir ./Qwen3-8B Qwen/Qwen3-8B ``` - **Device Placement**: `device_map="auto"` will load the model parameters to multiple devices automatically, if available. It relies on the `accelerate` package. If you would like to use a single device, you can pass `device` instead of device_map. `device=-1` or `device="cpu"` indicates using CPU, `device="cuda"` indicates using the current GPU, and `device="cuda:1"` or `device=1` indicates using the second GPU. Do not use `device_map` and `device` at the same time! - **Compute Precision**: `torch_dtype="auto"` will determine automatically the data type to use based on the original precision of the checkpoint and the precision your device supports. For modern devices, the precision determined will be `bfloat16`. If you don't pass `torch_dtype="auto"`, the default data type is `float32`, which will take double the memory and be slower in computation. Calls to the text generation pipeline will use the generation configuration from the model file, e.g., `generation_config.json`. This configuration could be overridden by passing arguments directly to the call. The default is equivalent to ```python messages = generator(messages, do_sample=True, temperature=0.6, top_k=2, top_p=0.95, eos_token_id=[151645, 151643])[0]{"generated_text"} ``` For the best practices in configuring generation parameters, please see the model card. ## Thinking & Non-Thinking Mode By default, Qwen3 model will think before response. It is also true for the `pipeline()` interface. To switch between thinking and non-thinking mode, two methods can be used - Append a final assistant message, containing only `\n\n\n\n`. This method is stateless, meaning it will only work for that single turn. It will also strictly prevent the model from generating thinking content. For example, ```python messages = [ {"role": "user", "content": "Give me a short introduction to large language models."}, {"role": "assistant", "content": "\n\n\n\n"}, ] messages = generator(messages, max_new_tokens=32768)[0]["generated_text"] # print(messages[-1]["content"]) messages.append({"role": "user", "content": "In a single sentence."}) messages = generator(messages, max_new_tokens=32768)[0]["generated_text"] # print(messages[-1]["content"]) ``` - Add to the user (or the system) message, `/no_think` to disable thinking and `/think` to enable thinking. This method is stateful, meaning the model will follow the most recent instruction in multi-turn conversations. ```python messages = [ {"role": "user", "content": "Give me a short introduction to large language models./no_think"}, ] messages = generator(messages, max_new_tokens=32768)[0]["generated_text"] # print(messages[-1]["content"]) messages.append({"role": "user", "content": "In a single sentence./think"}) messages = generator(messages, max_new_tokens=32768)[0]["generated_text"] # print(messages[-1]["content"]) ``` ## Parsing Thinking Content If you would like a more structured assistant message format, you can use the following function to extract the thinking content into a field named `reasoning_content` which is similar to the format used by vLLM, SGLang, etc. ```python import copy import re def parse_thinking_content(messages): messages = copy.deepcopy(messages) for message in messages: if message["role"] == "assistant" and (m := re.match(r"\n(.+)\n\n", message["content"], flags=re.DOTALL)): message["content"] = message["content"][len(m.group(0)):] if thinking_content := m.group(1).strip(): message["reasoning_content"] = thinking_content return messages ``` ## Parsing Tool Calls For tool calling with Transformers, please refer to [our guide on Function Calling](../framework/function_call.md#hugging-face-transformers). ## Serving Quantized models Qwen3 comes with two types of pre-quantized models, FP8 and AWQ. The command serving those models are the same as the original models except for the name change: ```python from transformers import pipeline model_name_or_path = "Qwen/Qwen3-8B-FP8" # FP8 models # model_name_or_path = "Qwen/Qwen3-8B-AWQ" # AWQ models generator = pipeline( "text-generation", model_name_or_path, torch_dtype="auto", device_map="auto", ) ``` :::{note} FP8 computation is supported on NVIDIA GPUs with compute capability > 8.9, that is, Ada Lovelace, Hopper, and later GPUs. For better performance, make sure `triton` and a CUDA compiler compatible with the CUDA version of `torch` in your environment are installed. ::: :::{important} As of 4.51.0, there are issues with Transformers when running those checkpoints **across GPUs**. The following method could be used to work around those issues: - Set the environment variable `CUDA_LAUNCH_BLOCKING=1` before running the script; or - Uncomment [this line](https://github.com/huggingface/transformers/blob/0720e206c6ba28887e4d60ef60a6a089f6c1cc76/src/transformers/integrations/finegrained_fp8.py#L340) in your local installation of `transformers`. ::: ## Enabling Long Context The maximum context length in pre-training for Qwen3 models is 32,768 tokens. It can be extended to 131,072 tokens with RoPE scaling techniques. We have validated the performance with YaRN. Transformers supports YaRN, which can be enabled either by modifying the model files or overriding the default arguments when loading the model. - Modifying the model files: In the `config.json` file, add the rope_scaling fields: ```json { ..., "max_position_embeddings": 131072, "rope_scaling": { "rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 32768 } } ``` - Overriding the default arguments: ```python from transformers import pipeline model_name_or_path = "Qwen/Qwen3-8B" generator = pipeline( "text-generation", model_name_or_path, torch_dtype="auto", device_map="auto", model_kwargs={ "max_position_embeddings": 131072, "rope_scaling": { "rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 32768, }, } ) ``` :::{attention} As of Transformers 4.52.3, it will use `max_position_embeddings/rope_scaling.original_max_position_embeddings` as the `rope_scaling.factor` regradless of the specified `rope_scaling.factor`. See [this issue](https://github.com/huggingface/transformers/issues/38224) for more information. ::: :::{note} Transformers implements static YaRN, which means the scaling factor remains constant regardless of input length, **potentially impacting performance on shorter texts.** We advise adding the `rope_scaling` configuration only when processing long contexts is required. It is also recommended to modify the `factor` as needed. For example, if the typical context length for your application is 65,536 tokens, it would be better to set `factor` as 2.0. ::: ## Streaming Generation With the help of `TextStreamer`, you can modify your chatting with Qwen3 to streaming mode. It will print the response as being generated to the console or the terminal. ```python from transformers import pipeline, TextStreamer model_name_or_path = "Qwen/Qwen3-8B" generator = pipeline( "text-generation", model_name_or_path, torch_dtype="auto", device_map="auto", ) streamer = TextStreamer(pipe.tokenizer, skip_prompt=True, skip_special_tokens=True) messages= generator(messages, max_new_tokens=32768, streamer=streamer)[0]["generated_text"] ``` Besides using `TextStreamer`, we can also use `TextIteratorStreamer` which stores print-ready text in a queue, to be used by a downstream application as an iterator: ```python from transformers import pipeline, TextIteratorStreamer model_name_or_path = "Qwen/Qwen3-8B" generator = pipeline( "text-generation", model_name_or_path, torch_dtype="auto", device_map="auto", ) streamer = TextIteratorStreamer(pipe.tokenizer, skip_prompt=True, skip_special_tokens=True) # Use Thread to run generation in background # Otherwise, the process is blocked until generation is complete # and no streaming effect can be observed. from threading import Thread generation_kwargs = dict(text_inputs=messages, max_new_tokens=32768, streamer=streamer) thread = Thread(target=pipe, kwargs=generation_kwargs) thread.start() generated_text = "" for new_text in streamer: generated_text += new_text print(generated_text) ``` ## Batch Generation :::{note} Batching is not automatically a win for performance. ::: ```python from transformers import pipeline model_name_or_path = "Qwen/Qwen3-8B" generator = pipeline( "text-generation", model_name_or_path, torch_dtype="auto", device_map="auto", ) generator.tokenizer.padding_side="left" batch = [ [{"role": "user", "content": "Give me a short introduction to large language models."}], [{"role": "user", "content": "Give me a detailed introduction to large language models."}], ] results = generator(batch, max_new_tokens=32768, batch_size=2) batch = [result[0]["generated_text"] for result in results] ``` ## FAQ You may find distributed inference with Transformers is not as fast as you would imagine. Transformers with `device_map="auto"` does not apply tensor parallelism, and it only uses one GPU at a time. For Transformers with tensor parallelism, please refer to [its documentation](https://huggingface.co/docs/transformers/v4.51.3/en/perf_infer_gpu_multi). --- ### Source Quantization Awq (`docs/source/quantization/awq.md`) # AWQ :::{attention} To be updated for Qwen3. ::: For quantized models, one of our recommendations is the usage of [AWQ](https://arxiv.org/abs/2306.00978) with [AutoAWQ](https://github.com/casper-hansen/AutoAWQ). **AWQ** refers to Activation-aware Weight Quantization, a hardware-friendly approach for LLM low-bit weight-only quantization. **AutoAWQ** is an easy-to-use Python library for 4-bit quantized models. AutoAWQ speeds up models by 3x and reduces memory requirements by 3x compared to FP16. AutoAWQ implements the Activation-aware Weight Quantization (AWQ) algorithm for quantizing LLMs. In this document, we show you how to use the quantized model with Hugging Face `transformers` and also how to quantize your own model. ## Usage of AWQ Models with Hugging Face transformers Now, `transformers` has officially supported AutoAWQ, which means that you can directly use the quantized model with `transformers`. The following is a very simple code snippet showing how to run `Qwen2.5-7B-Instruct-AWQ` with the quantized model: ```python from transformers import AutoModelForCausalLM, AutoTokenizer model_name = "Qwen/Qwen2.5-7B-Instruct-AWQ" model = AutoModelForCausalLM.from_pretrained( model_name, device_map="auto", ) tokenizer = AutoTokenizer.from_pretrained(model_name) prompt = "Give me a short introduction to large language models." messages = [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt}, ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, ) model_inputs = tokenizer([text], return_tensors="pt").to(model.device) generated_ids = model.generate( **model_inputs, max_new_tokens=512, ) generated_ids = [ output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids) ] response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0] ``` ## Usage of AWQ Models with vLLM vLLM has supported AWQ, which means that you can directly use our provided AWQ models or those quantized with `AutoAWQ` with vLLM. We recommend using the latest version of vLLM (`vllm>=0.6.1`) which brings performance improvements to AWQ models; otherwise, the performance might not be well-optimized. Actually, the usage is the same with the basic usage of vLLM. We provide a simple example of how to launch OpenAI-API compatible API with vLLM and `Qwen2.5-7B-Instruct-AWQ`: Run the following in a shell to start an OpenAI-compatible API service: ```bash vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ ``` Then, you can call the API as ```bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen2.5-7B-Instruct-AWQ", "messages": [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} ], "temperature": 0.7, "top_p": 0.8, "repetition_penalty": 1.05, "max_tokens": 512 }' ``` or you can use the API client with the `openai` Python package as shown below: ```python from openai import OpenAI openai_api_key = "EMPTY" openai_api_base = "http://localhost:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) chat_response = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct-AWQ", messages=[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, ], temperature=0.7, top_p=0.8, max_tokens=512, extra_body={ "repetition_penalty": 1.05, }, ) print("Chat response:", chat_response) ``` ## Quantize Your Own Model with AutoAWQ If you want to quantize your own model to AWQ quantized models, we advise you to use AutoAWQ. ```bash pip install "autoawq<0.2.7" ``` Suppose you have finetuned a model based on `Qwen2.5-7B`, which is named `Qwen2.5-7B-finetuned`, with your own dataset, e.g., Alpaca. To build your own AWQ quantized model, you need to use the training data for calibration. Below, we provide a simple demonstration for you to run: ```python from awq import AutoAWQForCausalLM from transformers import AutoTokenizer # Specify paths and hyperparameters for quantization model_path = "your_model_path" quant_path = "your_quantized_model_path" quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" } # Load your tokenizer and model with AutoAWQ tokenizer = AutoTokenizer.from_pretrained(model_path) model = AutoAWQForCausalLM.from_pretrained(model_path, device_map="auto", safetensors=True) ``` Then you need to prepare your data for calibration. What you need to do is just put samples into a list, each of which is a text. As we directly use our finetuning data for calibration, we first format it with ChatML template. For example, ```python data = [] for msg in dataset: text = tokenizer.apply_chat_template(msg, tokenize=False, add_generation_prompt=False) data.append(text.strip()) ``` where each `msg` is a typical chat message as shown below: ```json [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me who you are."}, {"role": "assistant", "content": "I am a large language model named Qwen..."} ] ``` Then just run the calibration process by one line of code: ```python model.quantize(tokenizer, quant_config=quant_config, calib_data=data) ``` Finally, save the quantized model: ```python model.save_quantized(quant_path, safetensors=True, shard_size="4GB") tokenizer.save_pretrained(quant_path) ``` Then you can obtain your own AWQ quantized model for deployment. Enjoy! --- ### Source Quantization Gptq (`docs/source/quantization/gptq.md`) # GPTQ :::{attention} To be updated for Qwen3. ::: [GPTQ](https://arxiv.org/abs/2210.17323) is a quantization method for GPT-like LLMs, which uses one-shot weight quantization based on approximate second-order information. In this document, we show you how to use the quantized model with Hugging Face `transformers` and also how to quantize your own model with [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ). ## Usage of GPTQ Models with Hugging Face transformers :::{note} To use the official Qwen2.5 GPTQ models with `transformers`, please ensure that `optimum>=1.20.0` and compatible versions of `transformers` and `auto_gptq` are installed. You can do that by ```bash pip install -U "optimum>=1.20.0" ``` ::: Now, `transformers` has officially supported AutoGPTQ, which means that you can directly use the quantized model with `transformers`. For each size of Qwen2.5, we provide both Int4 and Int8 GPTQ quantized models. The following is a very simple code snippet showing how to run `Qwen2.5-7B-Instruct-GPTQ-Int4`: ```python from transformers import AutoModelForCausalLM, AutoTokenizer model_name = "Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4" model = AutoModelForCausalLM.from_pretrained( model_name, device_map="auto" ) tokenizer = AutoTokenizer.from_pretrained(model_name) prompt = "Give me a short introduction to large language models." messages = [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt}, ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, ) model_inputs = tokenizer([text], return_tensors="pt").to(model.device) generated_ids = model.generate( **model_inputs, max_new_tokens=512, ) generated_ids = [ output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids) ] response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0] ``` ## Usage of GPTQ Models with vLLM vLLM has supported GPTQ, which means that you can directly use our provided GPTQ models or those trained with `AutoGPTQ` with vLLM. If possible, it will automatically use the GPTQ Marlin kernel, which is more efficient. Actually, the usage is the same with the basic usage of vLLM. We provide a simple example of how to launch OpenAI-API compatible API with vLLM and `Qwen2.5-7B-Instruct-GPTQ-Int4`: Run the following in a shell to start an OpenAI-compatible API service: ```bash vllm serve Qwen2.5-7B-Instruct-GPTQ-Int4 ``` Then, you can call the API as ```bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen2.5-7B-Instruct-GPTQ-Int4", "messages": [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} ], "temperature": 0.7, "top_p": 0.8, "repetition_penalty": 1.05, "max_tokens": 512 }' ``` or you can use the API client with the `openai` Python package as shown below: ```python from openai import OpenAI openai_api_key = "EMPTY" openai_api_base = "http://localhost:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) chat_response = client.chat.completions.create( model="Qwen2.5-7B-Instruct-GPTQ-Int4", messages=[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, ], temperature=0.7, top_p=0.8, max_tokens=512, extra_body={ "repetition_penalty": 1.05, }, ) print("Chat response:", chat_response) ``` ## Quantize Your Own Model with AutoGPTQ If you want to quantize your own model to GPTQ quantized models, we advise you to use AutoGPTQ. It is suggested installing the latest version of the package by installing from source code: ```bash git clone https://github.com/AutoGPTQ/AutoGPTQ cd AutoGPTQ pip install -e . ``` Suppose you have finetuned a model based on `Qwen2.5-7B`, which is named `Qwen2.5-7B-finetuned`, with your own dataset, e.g., Alpaca. To build your own GPTQ quantized model, you need to use the training data for calibration. Below, we provide a simple demonstration for you to run: ```python from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig from transformers import AutoTokenizer # Specify paths and hyperparameters for quantization model_path = "your_model_path" quant_path = "your_quantized_model_path" quantize_config = BaseQuantizeConfig( bits=8, # 4 or 8 group_size=128, damp_percent=0.01, desc_act=False, # set to False can significantly speed up inference but the perplexity may slightly bad static_groups=False, sym=True, true_sequential=True, model_name_or_path=None, model_file_base_name="model" ) max_len = 8192 # Load your tokenizer and model with AutoGPTQ # To learn about loading model to multiple GPUs, # visit https://github.com/AutoGPTQ/AutoGPTQ/blob/main/docs/tutorial/02-Advanced-Model-Loading-and-Best-Practice.md tokenizer = AutoTokenizer.from_pretrained(model_path) model = AutoGPTQForCausalLM.from_pretrained(model_path, quantize_config) ``` However, if you would like to load the model on multiple GPUs, you need to use `max_memory` instead of `device_map`. Here is an example: ```python model = AutoGPTQForCausalLM.from_pretrained( model_path, quantize_config, max_memory={i: "20GB" for i in range(4)} ) ``` Then you need to prepare your data for calibration. What you need to do is just put samples into a list, each of which is a text. As we directly use our finetuning data for calibration, we first format it with ChatML template. For example, ```python import torch data = [] for msg in dataset: text = tokenizer.apply_chat_template(msg, tokenize=False, add_generation_prompt=False) model_inputs = tokenizer([text]) input_ids = torch.tensor(model_inputs.input_ids[:max_len], dtype=torch.int) data.append(dict(input_ids=input_ids, attention_mask=input_ids.ne(tokenizer.pad_token_id))) ``` where each `msg` is a typical chat message as shown below: ```json [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me who you are."}, {"role": "assistant", "content": "I am a large language model named Qwen..."} ] ``` Then just run the calibration process by one line of code: ```python import logging logging.basicConfig( format="%(asctime)s %(levelname)s [%(name)s] %(message)s", level=logging.INFO, datefmt="%Y-%m-%d %H:%M:%S" ) model.quantize(data, cache_examples_on_gpu=False) ``` Finally, save the quantized model: ```python model.save_quantized(quant_path, use_safetensors=True) tokenizer.save_pretrained(quant_path) ``` It is unfortunate that the `save_quantized` method does not support sharding. For sharding, you need to load the model and use `save_pretrained` from transformers to save and shard the model. Except for this, everything is so simple. Enjoy! ## Known Issues ### Qwen2.5-72B-Instruct-GPTQ-Int4 cannot stop generation properly :Model: Qwen2.5-72B-Instruct-GPTQ-Int4 :Framework: vLLM, AutoGPTQ (including Hugging Face transformers) :Description: Generation cannot stop properly. Continual generation after where it should stop, then repeated texts, either single character, a phrase, or paragraphs, are generated. :Workaround: The following workaround could be considered 1. Using the original model in 16-bit floating point 2. Using the AWQ variants or llama.cpp-based models for reduced chances of abnormal generation ### Qwen2.5-32B-Instruct-GPTQ-Int4 broken with vLLM on multiple GPUs :Model: Qwen2.5-32B-Instruct-GPTQ-Int4 :Framework: vLLM :Description: Deployment on multiple GPUs and only garbled text like `!!!!!!!!!!!!!!!!!!` could be generated. :Workaround: Each of the following workaround could be considered 1. Using the AWQ or GPTQ-Int8 variants 2. Using a single GPU 3. Using Hugging Face `transformers` if latency and throughput are not major concerns ## Troubleshooting :::{dropdown} With `transformers` and `auto_gptq`, the logs suggest `CUDA extension not installed.` and the inference is slow. `auto_gptq` fails to find a fused CUDA kernel compatible with your environment and falls back to a plain implementation. Follow its [installation guide](https://github.com/AutoGPTQ/AutoGPTQ/blob/main/docs/INSTALLATION.md) to install a pre-built wheel or try installing `auto_gptq` from source. ::: :::{dropdown} Self-quantized Qwen2.5-72B-Instruct-GPTQ with `vllm`, `ValueError: ... must be divisible by ...` is raised. The intermediate size of the self-quantized model is different from the official Qwen2.5-72B-Instruct-GPTQ models. After quantization the size of the quantized weights are divided by the group size, which is typically 128. The intermediate size for the FFN blocks in Qwen2.5-72B is 29568. Unfortunately, {math}`29568 \div 128 = 231`. Since the number of attention heads and the dimensions of the weights must be divisible by the tensor parallel size, it means you can only run the quantized model with `tensor_parallel_size=1`, i.e., one GPU card. A workaround is to make the intermediate size divisible by {math}`128 \times 8 = 1024`. To achieve that, the weights should be padded with zeros. While it is mathematically equivalent before and after zero-padding the weights, the results may be slightly different in reality. Try the following: ```python import torch from torch.nn import functional as F from transformers import AutoModelForCausalLM # must use AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-72B-Instruct", torch_dtype="auto") # this size is Qwen2.5-72B only pad_size = 128 sd = model.state_dict() for i, k in enumerate(sd): v = sd[k] print(k, i) # interleaving the padded zeros if ('mlp.up_proj.weight' in k) or ('mlp.gate_proj.weight' in k): prev_v = F.pad(v.unsqueeze(1), (0, 0, 0, 1, 0, 0)).reshape(29568*2, -1)[:pad_size*2] new_v = torch.cat([prev_v, v[pad_size:]], dim=0) sd[k] = new_v elif 'mlp.down_proj.weight' in k: prev_v= F.pad(v.unsqueeze(2), (0, 1)).reshape(8192, 29568*2)[:, :pad_size*2] new_v = torch.cat([prev_v, v[:, pad_size:]], dim=1) sd[k] = new_v # this is a very large file; make sure your RAM is enough to load the model torch.save(sd, '/path/to/padded_model/pytorch_model.bin') ``` This will save the padded checkpoint to the specified directory. Then, copy other files from the original checkpoint to the new directory and modify the `intermediate_size` in `config.json` to `29696`. Finally, you can quantize the saved model checkpoint. ::: --- ### Source Quantization Llama.Cpp (`docs/source/quantization/llama.cpp.md`) # llama.cpp Quantization is a major topic for local inference of LLMs, as it reduces the memory footprint. Undoubtably, llama.cpp natively supports LLM quantization and of course, with flexibility as always. At high-level, all quantization supported by llama.cpp is weight quantization: Model parameters are quantized into lower bits, and in inference, they are dequantized and used in computation. In addition, you can mix different quantization data types in a single quantized model, e.g., you can quantize the embedding weights using a quantization data type and other weights using a different one. With an adequate mixture of quantization types, much lower quantization error can be attained with just a slight increase of bit-per-weight. The example program `llama-quantize` supports many quantization presets, such as Q4_K_M and Q8_0. If you find the quantization errors still more than expected, you can bring your own scales, e.g., as computed by AWQ, or use calibration data to compute an importance matrix using `llama-imatrix`, which can then be used during quantization to enhance the quality of the quantized models. In this document, we demonstrate the common way to quantize your model and evaluate the performance of the quantized model. We will assume you have the example programs from llama.cpp at your hand. If you don't, check our guide [here](../run_locally/llama.cpp.html#getting-the-program){.external}. ## Getting the GGUF Now, suppose you would like to quantize `Qwen3-8B`. You need to first make a GGUF file as shown below: ```bash python convert-hf-to-gguf.py Qwen/Qwen3-8B --outfile Qwen3-8B-F16.gguf ``` Since Qwen3 are trained using the bfloat16 precision, the following should keep most information on supported machines: ```bash python convert-hf-to-gguf.py Qwen/Qwen3-8B --outtype bf16 --outfile Qwen3-8B-BF16.gguf ``` Sometimes, it may be better to use fp32 as the start point for quantization. In that case, use ```bash python convert-hf-to-gguf.py Qwen/Qwen3-8B --outtype f32 --outfile Qwen3-8B-F32.gguf ``` ## Quantizing the GGUF without Calibration For the simplest way, you can directly quantize the model to lower-bits based on your requirements. An example of quantizing the model to 8 bits is shown below: ```bash ./llama-quantize Qwen3-8B-F16.gguf Qwen3-8B-Q8_0.gguf Q8_0 ``` `Q8_0` is a code for a quantization preset. You can find all the presets in [the source code of `llama-quantize`](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/quantize.cpp). Look for the variable `QUANT_OPTIONS`. Common ones used for 8B models include `Q8_0`, `Q5_K_M`, and `Q4_K_M`. The letter case doesn't matter, so `q8_0` or `q4_K_m` are perfectly fine. Now you can use the GGUF file of the quantized model with applications based on llama.cpp. Very simple indeed. However, the accuracy of the quantized model could be lower than expected occasionally, especially for lower-bit quantization. The program may even prevent you from doing that. There are several ways to improve quality of quantized models. A common way is to use a calibration dataset in the target domain to identify the weights that really matter and quantize the model in a way that those weights have lower quantization errors, as introduced in the next two methods. ## Quantizing the GGUF with AWQ Scale :::{attention} To be updated for Qwen3. ::: To improve the quality of your quantized models, one possible solution is to apply the AWQ scale, following [this script](https://github.com/casper-hansen/AutoAWQ/blob/main/docs/examples.md#gguf-export). First, when you run `model.quantize()` with `autoawq`, remember to add `export_compatible=True` as shown below: ```python ... model.quantize( tokenizer, quant_config=quant_config, export_compatible=True ) model.save_pretrained(quant_path) ... ``` The above code will not actually quantize the weights. Instead, it adjusts weights based on a dataset so that they are "easier" to quantize.[^AWQ] Then, when you run `convert-hf-to-gguf.py`, remember to replace the model path with the path to the new model: ```bash python convert-hf-to-gguf.py --outfile qwen2.5-7b-instruct-f16-awq.gguf ``` Finally, you can quantize the model as in the last example: ```bash ./llama-quantize qwen2.5-7b-instruct-f16-awq.gguf qwen2.5-7b-instruct-q8_0.gguf Q8_0 ``` In this way, it should be possible to achieve similar quality with lower bit-per-weight. [^AWQ]: If you are interested in what this means, refer to [the AWQ paper](https://arxiv.org/abs/2306.00978). Basically, important weights (called salient weights in the paper) are identified based on activations across data examples. The weights are scaled accordingly such that the salient weights are protected even after quantization. ## Quantizing the GGUF with Importance Matrix Another possible solution is to use the "important matrix"[^imatrix], following [this](https://github.com/ggml-org/llama.cpp/tree/master/tools/imatrix). First, you need to compute the importance matrix data of the weights of a model (`-m`) using a calibration dataset (`-f`): ```bash ./llama-imatrix -m Qwen3-8B-F16.gguf -f calibration-text.txt --chunk 512 -o Qwen3-8B-imatrix.dat -ngl 80 ``` The text is cut in chunks of length `--chunk` for computation. Preferably, the text should be representative of the target domain. The final results will be saved in a file named `Qwen3-8B-imatrix.dat` (`-o`), which can then be used: ```bash ./llama-quantize --imatrix Qwen3-8B-imatrix.dat \ Qwen3-8B-F16.gguf Qwen3-8B-Q4_K_M.gguf Q4_K_M ``` For lower-bit quantization mixtures for 1-bit or 2-bit, if you do not provide `--imatrix`, a helpful warning will be printed by `llama-quantize`. [^imatrix]: Here, the importance matrix keeps record of how weights affect the output: the weight should be important is a slight change in its value causes huge difference in the results, akin to the [GPTQ](https://arxiv.org/abs/2210.17323) algorithm. ## Perplexity Evaluation `llama.cpp` provides an example program for us to calculate the perplexity, which evaluate how unlikely the given text is to the model. It should be mostly used for comparisons: the lower the perplexity, the better the model remembers the given text. To do this, you need to prepare a dataset, say "wiki test"[^wiki]. You can download the dataset with: ```bash wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-raw-v1.zip?ref=salesforce-research -O wikitext-2-raw-v1.zip unzip wikitext-2-raw-v1.zip ``` Then you can run the test with the following command: ```bash ./llama-perplexity -m Qwen3-8B-Q8_0.gguf -f wiki.test.raw -ngl 80 ``` Wait for some time and you will get the perplexity of the model. There are some numbers of different kinds of quantization mixture [here](https://github.com/ggml-org/llama.cpp/blob/master/tools/perplexity/README.md). It might be helpful to look at the difference and grab a sense of how that kind of quantization might perform. [^wiki]: It is not a good evaluation dataset for instruct models though, but it is very common and easily accessible. You probably want to use a dataset similar to your target domain. ## Finally In this guide, we demonstrate how to conduct quantization and evaluate the perplexity with llama.cpp. For more information, please visit the [llama.cpp GitHub repo](https://github.com/ggml-org/llama.cpp). We usually quantize the fp16 model to 4, 5, 6, and 8-bit models with different quantization mixtures, but sometimes a particular mixture just does not work, so we don't provide those in our Hugging Face Hub. However, others in the community may have success, so if you haven't found what you need in our repos, look around. Enjoy your freshly quantized models! --- ### Source Run Locally Llama.Cpp (`docs/source/run_locally/llama.cpp.md`) # llama.cpp [^GGUF]: GPT-Generated Unified Format :::{dropdown} llama.cpp as a C++ library Before starting, let's first discuss what is llama.cpp and what you should expect, and why we say "use" llama.cpp, with "use" in quotes. llama.cpp is essentially a different ecosystem with a different design philosophy that targets light-weight footprint, minimal external dependency, multi-platform, and extensive, flexible hardware support: - Plain C/C++ implementation without external dependencies - Support a wide variety of hardware: - AVX, AVX2 and AVX512 support for x86_64 CPU - Apple Silicon via Metal and Accelerate (CPU and GPU) - NVIDIA GPU (via CUDA), AMD GPU (via hipBLAS), Intel GPU (via SYCL), Ascend NPU (via CANN), and Moore Threads GPU (via MUSA) - Vulkan backend for GPU - Various quantization schemes for faster inference and reduced memory footprint - CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity It's like the Python frameworks `torch`+`transformers` or `torch`+`vllm` but in C++. However, this difference is crucial: - Python is an interpreted language: The code you write is executed line-by-line on-the-fly by an interpreter. You can run the example code snippet or script with an interpreter or a natively interactive interpreter shell. In addition, Python is learner friendly, and even if you don't know much before, you can tweak the source code here and there. - C++ is a compiled language: The source code you write needs to be compiled beforehand, and it is translated to machine code and an executable program by a compiler. The overhead from the language side is minimal. You do have source code for example programs showcasing how to use the library. But it is not very easy to modify the source code if you are not verse in C++ or C. To use llama.cpp means that you use the llama.cpp library in your own program, like writing the source code of [Ollama](https://ollama.com/), [LM Studio](https://lmstudio.ai/), [GPT4ALL](https://www.nomic.ai/gpt4all), [llamafile](https://llamafile.ai/) etc. But that's not what this guide is intended or could do. Instead, here we introduce how to use the `llama-cli` example program, in the hope that you know that llama.cpp does support Qwen2.5 models and how the ecosystem of llama.cpp generally works. ::: In this guide, we will show how to "use" [llama.cpp](https://github.com/ggml-org/llama.cpp) to run models on your local machine, in particular, the `llama-cli` and the `llama-server` example program, which comes with the library. The main steps are: 1. Get the programs 2. Get the Qwen3 models in GGUF[^GGUF] format 3. Run the program with the model :::{note} llama.cpp supports Qwen3 and Qwen3MoE from version `b5092`. ::: ## Getting the Program You can get the programs in various ways. For optimal efficiency, we recommend compiling the programs locally, so you get the CPU optimizations for free. However, if you don't have C++ compilers locally, you can also install using package managers or downloading pre-built binaries. They could be less efficient but for non-production example use, they are fine. :::::{tab-set} ::::{tab-item} Compile Locally Here, we show the basic command to compile `llama-cli` locally on **macOS** or **Linux**. For Windows or GPU users, please refer to [the guide from llama.cpp](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md). :::{rubric} Installing Build Tools :heading-level: 5 ::: To build locally, a C++ compiler and a build system tool are required. To see if they have been installed already, type `cc --version` or `cmake --version` in a terminal window. - If installed, the build configuration of the tool will be printed to the terminal, and you are good to go! - If errors are raised, you need to first install the related tools: - On macOS, install with the command `xcode-select --install` - On Ubuntu, install with the command `sudo apt install build-essential`. For other Linux distributions, the command may vary; the essential packages needed for this guide are `gcc` and `cmake`. :::{rubric} Compiling the Program :heading-level: 5 ::: For the first step, clone the repo and enter the directory: ```bash git clone https://github.com/ggml-org/llama.cpp cd llama.cpp ``` Then, build llama.cpp using CMake: ```bash cmake -B build cmake --build build --config Release ``` The first command will check the local environment and determine which backends and features should be included. The second command will actually build the programs. To shorten the time, you can also enable parallel compiling based on the CPU cores you have, for example: ```bash cmake --build build --config Release -j 8 ``` This will build the programs with 8 parallel compiling jobs. The built programs will be in `./build/bin/`. :::: ::::{tab-item} Package Managers For **macOS** and **Linux** users, `llama-cli` and `llama-server` can be installed with package managers including Homebrew, Nix, and Flox. Here, we show how to install `llama-cli` and `llama-server` with Homebrew. For other package managers, please check the instructions [here](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md). Installing with Homebrew is very simple: 1. Ensure that Homebrew is available on your operating system. If you don't have Homebrew, you can install it as in [its website](https://brew.sh/). 2. Second, you can install the pre-built binaries, `llama-cli` and `llama-server` included, with a single command: ```bash brew install llama.cpp ``` Note that the installed binaries might not be built with the optimal compile options for your hardware, which can lead to poor performance. They also don't support GPU on Linux systems. :::: ::::{tab-item} Binary Release You can also download pre-built binaries from [GitHub Releases](https://github.com/ggml-org/llama.cpp/releases). Please note that those pre-built binaries files are architecture-, backend-, and os-specific. If you are not sure what those mean, you probably don't want to use them and running with incompatible versions will most likely fail or lead to poor performance. The file name is like `llama--bin---.zip`. There are three simple parts: - ``: the version of llama.cpp. The latest is preferred, but as llama.cpp is updated and released frequently, the latest may contain bugs. If the latest version does not work, try the previous release until it works. - ``: the operating system. `win` for Windows; `macos` for macOS; `linux` for Linux. - ``: the system architecture. `x64` for `x86_64`, e.g., most Intel and AMD systems, including Intel Mac; `arm64` for `arm64`, e.g., Apple Silicon or Snapdragon-based systems. The `` part is somewhat complicated for Windows: - Running on CPU - x86_64 CPUs: We suggest try the `avx2` one first. - `noavx`: No hardware acceleration at all. - `avx2`, `avx`, `avx512`: SIMD-based acceleration. Most modern desktop CPUs should support avx2, and some CPUs support `avx512`. - `openblas`: Relying on OpenBLAS for acceleration for prompt processing but not generation. - arm64 CPUs: We suggest try the `llvm` one first. - [`llvm` and `msvc`](https://github.com/ggml-org/llama.cpp/pull/7191) are different compilers - Running on GPU: We suggest try the `cu` one for NVIDIA GPUs, `kompute` for AMD GPUs, and `sycl` for Intel GPUs first. Ensure that you have related drivers installed. - [`vulcan`](https://github.com/ggml-org/llama.cpp/pull/2059): support certain NVIDIA and AMD GPUs - [`kompute`](https://github.com/ggml-org/llama.cpp/pull/4456): support certain NVIDIA and AMD GPUs - [`sycl`](https://github.com/ggml-org/llama.cpp/discussions/5138): Intel GPUs, oneAPI runtime is included - `cu`: NVIDIA GPUs, CUDA runtime is not included. You can download the `cudart-llama-bin-win-cu-x64.zip` and unzip it to the same directory if you don't have the corresponding CUDA toolkit installed. You don't have much choice for macOS or Linux. - Linux: only one prebuilt binary, `llama--bin-linux-x64.zip`, supporting CPU. - macOS: `llama--bin-macos-x64.zip` for Intel Mac with no GPU support; `llama--bin-macos-arm64.zip` for Apple Silicon with GPU support. After downloading the `.zip` file, unzip them into a directory and open a terminal at that directory. :::: ::::: ## Getting the GGUF GGUF[^GGUF] is a file format for storing information needed to run a model, including but not limited to model weights, model hyperparameters, default generation configuration, and tokenizer. You can use the official Qwen GGUFs from our Hugging Face Hub or prepare your own GGUF file. ### Using the Official Qwen3 GGUFs We provide a series of GGUF models in our Hugging Face organization, and to search for what you need you can search the repo names with `-GGUF`. Download the GGUF model that you want with `huggingface-cli` (you need to install it first with `pip install huggingface_hub`): ```bash huggingface-cli download --local-dir ``` For example: ```bash huggingface-cli download Qwen/Qwen3-8B-GGUF qwen3-8b-q4_k_m.gguf --local-dir . ``` This will download the Qwen3-8B model in GGUF format quantized with the scheme Q4_K_M. ### Preparing Your Own GGUF Model files from Hugging Face Hub can be converted to GGUF, using the `convert-hf-to-gguf.py` Python script. It does require you to have a working Python environment with at least `transformers` installed. Obtain the source file if you haven't already: ```bash git clone https://github.com/ggml-org/llama.cpp cd llama.cpp ``` Suppose you would like to use Qwen3-8B you can make a GGUF file for the fp16 model as shown below: ```bash python convert-hf-to-gguf.py Qwen/Qwen3-8B --outfile qwen3-8b-f16.gguf ``` The first argument to the script refers to the path to the HF model directory or the HF model name, and the second argument refers to the path of your output GGUF file. Remember to create the output directory before you run the command. The fp16 model could be a bit heavy for running locally, and you can quantize the model as needed. We introduce the method of creating and quantizing GGUF files in [this guide](../quantization/llama.cpp). You can refer to that document for more information. ## Run Qwen with llama.cpp :::{note} Regarding switching between thinking and non-thinking modes, while the soft switch is always available, the hard switch implemented in the chat template is not exposed in llama.cpp. The quick workaround is to pass [a custom chat template](../../source/assets/qwen3_nonthinking.jinja) equivalent to always `enable_thinking=False` via `--chat-template-file`. ::: ### llama-cli [llama-cli](https://github.com/ggml-org/llama.cpp/tree/master/tools/main) is a console program which can be used to chat with LLMs. Simple run the following command where you place the llama.cpp programs: ```shell ./llama-cli -hf Qwen/Qwen3-8B-GGUF:Q8_0 --jinja --color -ngl 99 -fa -sm row --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 -c 40960 -n 32768 --no-context-shift ``` Here are some explanations to the above command: - **Model**: llama-cli supports using model files from local path, remote URL, or Hugging Face hub. - `-hf Qwen/Qwen3-8B-GGUF:Q8_0` in the above indicates we are using the model file from Hugging Face hub - To use a local path, pass `-m qwen3-8b-q8_0.gguf` instead - To use a remote URL, pass `-mu https://hf.co/Qwen/Qwen3-8B-GGUF/resolve/main/qwen3-8b-Q8_0.gguf?download=true` instead - **Speed Optimization**: - CPU: llama-cli by default will use CPU and you can change `-t` to specify how many threads you would like it to use, e.g., `-t 8` means using 8 threads. - GPU: If the programs are built with GPU support, you can use `-ngl`, which allows offloading some layers to the GPU for computation. If there are multiple GPUs, it will offload to all the GPUs. You can use `-dev` to control the devices used and `-sm` to control which kinds of parallelism is used. For example, `-ngl 99 -dev cuda0,cuda1 -sm row` means offload all layers to GPU 0 and GPU1 using the split mode row. Adding `-fa` may also speed up the generation. - **Sampling Parameters**: llama.cpp supports [a variety of sampling methods](https://github.com/ggml-org/llama.cpp/tree/master/tools/main#generation-flags) and has default configuration for many of them. It is recommended to adjust those parameters according to the actual case and the recommended parameters from Qwen3 modelcard could be used as a reference. If you encounter repetition and endless generation, it is recommended to pass in addition `--presence-penalty` up to `2.0`. - **Context Management**: llama.cpp adopts the "rotating" context management by default. The `-c` controls the maximum context length (default 4096, 0 means loaded from model), and `-n` controls the maximum generation length each time (default -1 means infinite until ending, -2 means until context full). When the context is full but the generation doesn't end, the first `--keep` tokens (default 0, -1 means all) from the initial prompt is kept, and the first half of the rest is discarded. Then, the model continues to generate based on the new context tokens. You can set `--no-context-shift` to prevent this rotating behavior and the generation will stop once `-c` is reached. llama.cpp supports YaRN, which can be enabled by `-c 131072 --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 32768`. - **Chat**: `--jinja` indicates using the chat template embedded in the GGUF which is preferred and `--color` indicates coloring the texts so that user input and model output can be better differentiated. If there is a chat template, like in Qwen3 models, llama-cli will enter chat mode automatically. To stop generation or exit press "Ctrl+C". You can use `-sys` to add a system prompt. ### llama-server [llama-server](https://github.com/ggml-org/llama.cpp/tree/master/tools/server) is a simple HTTP server, including a set of LLM REST APIs and a simple web front end to interact with LLMs using llama.cpp. The core command is similar to that of llama-cli. In addition, it supports thinking content parsing and tool call parsing. ```shell ./llama-server -hf Qwen/Qwen3-8B-GGUF:Q8_0 --jinja --reasoning-format deepseek -ngl 99 -fa -sm row --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 -c 40960 -n 32768 --no-context-shift ``` By default, the server will listen at `http://localhost:8080` which can be changed by passing `--host` and `--port`. The web front end can be assessed from a browser at `http://localhost:8080/`. The OpenAI compatible API is at `http://localhost:8080/v1/`. ## What's More If you still find it difficult to use llama.cpp, don't worry, just check out other llama.cpp-based applications. For example, Qwen3 has already been officially part of Ollama and LM Studio, which are platforms for your to search and run local LLMs. Have fun! --- ### Source Run Locally Lmstudio (`docs/source/run_locally/lmstudio.md`) # LM Studio [LM Studio](https://lmstudio.ai) is a powerful desktop application for experimenting & developing with local AI models directly on your computer. You can run multiple kinds of Qwen models: from dense LLMs, VLMs, to MoEs and reasoning variants. LM Studio supports both GGUF (llama.cpp) and MLX formats for fast and efficient inference, completely privately on your machine. ## Download and Install LM Studio Download the installer for macOS, Windows, or Linux from the [LM Studio website](https://lmstudio.ai/download). ## Get Qwen models Qwen models are much loved by the community, and they frequently get featured in Staff Picks. Explore staff picked models within the app or in [https://lmstudio.ai/models](https://lmstudio.ai/models). Find Qwen models that fit your machine, and click Run in LM Studio! You can also search and download Qwen models from within the LM Studio app, or by using the `lms` CLI ([learn more](https://lmstudio.ai/docs/cli/local-models/get)). **Using the in-app model downloader** 1. Open the LM Studio app and search for any model by presssing `⌘ + Shift + M` on Mac, or `Ctrl + Shift + M` on PC. 2. Search for "Qwen." 3. Pick a result that looks interesting and LM Studio will suggest the optimal variants for your hardware. 4. Click **Download**. After the download finishes, load the model to use it in a new chat. **Getting a model from Hugging Face** First, enable LM Studio under your [Local Apps Settings](https://huggingface.co/settings/local-apps) in Hugging Face. On the model card, click the "Use this model" dropdown and select LM Studio. This will run the model directly in LM Studio if you already have it, or show you a download option if you don't. **Advanced: Use your own converted GGUF Qwen model file** If you had converted a Qwen model to GGUF yourself, you can use LM Studio's CLI `lms` to load your model into LM Studio. 1. Use: ```bash lms import ``` 2. LM Studio will automatically detect the model and it will populate in the application under "My Models". 3. Adjust context length and hardware settings as needed. If `lms import` does not work automatically, fear not. You are still able to manually import models into your LM Studio. Read more about LM Studio's model directory structure [here](https://lmstudio.ai/docs/app/advanced/import-model). Once the model has completed loading (as indicated by the progress bar), you may start chatting away in LM Studio! ## Serve the model through LM Studio's server **Serve via LM Studio's GUI** In the LM Studio application, press `⌘ / Ctrl + L` to open the model loader. Here you can view a list of downloaded models and select one to load. LM Studio will by default select the load parameters that optimizes model performance on your hardware. **Serve via LM Studio's CLI** If you prefer to work in the terminal, use LM Studio's CLI to interact with your models. See a list of commands [here](https://lmstudio.ai/docs/cli). Note that you need to run LM Studio ***at least once*** before you can use `lms`. Next, turn on LM Studio's local API server by running: ```bash lms server start ``` Now you're ready to go! Use LM Studio's REST APIs to use Qwen models programmatically from your own code. Learn more about how to do this at . --- ### Source Run Locally Mlx Lm (`docs/source/run_locally/mlx-lm.md`) # MLX LM :::{attention} To be updated for Qwen3. ::: [mlx-lm](https://github.com/ml-explore/mlx-examples/tree/main/llms) helps you run LLMs locally on Apple Silicon. It is available at macOS. It has already supported Qwen models and this time, we have also provided checkpoints that you can directly use with it. ## Prerequisites The easiest way to get started is to install the `mlx-lm` package: - with `pip`: ```bash pip install mlx-lm ``` - with `conda`: ```bash conda install -c conda-forge mlx-lm ``` ## Running with Qwen MLX Files We provide model checkpoints with `mlx-lm` in our Hugging Face organization, and to search for what you need you can search the repo names with `-MLX`. Here provides a code snippet with `apply_chat_template` to show you how to load the tokenizer and model and how to generate contents. ```python from mlx_lm import load, generate model, tokenizer = load('Qwen/Qwen2.5-7B-Instruct-MLX', tokenizer_config={"eos_token": "<|im_end|>"}) prompt = "Give me a short introduction to large language models." messages = [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt} ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) response = generate(model, tokenizer, prompt=text, verbose=True, top_p=0.8, temp=0.7, repetition_penalty=1.05, max_tokens=512) ``` ## Make Your MLX files You can make MLX files with just one command: ```bash mlx_lm.convert --hf-path Qwen/Qwen2.5-7B-Instruct --mlx-path mlx/Qwen2.5-7B-Instruct/ -q ``` where - `--hf-path`: the model name on Hugging Face Hub or the local path - `--mlx-path`: the path for output files - `-q`: enable quantization --- ### Source Run Locally Ollama (`docs/source/run_locally/ollama.md`) # Ollama :::{attention} To be updated for Qwen3. ::: [Ollama](https://ollama.com/) helps you run LLMs locally with only a few commands. It is available at macOS, Linux, and Windows. Now, Qwen2.5 is officially on Ollama, and you can run it with one command: ```bash ollama run qwen2.5 ``` Next, we introduce more detailed usages of Ollama for running Qwen2.5 models. ## Quickstart Visit the official website [Ollama](https://ollama.com/) and click download to install Ollama on your device. You can also search models on the website, where you can find the Qwen2.5 models. Except for the default one, you can choose to run Qwen2.5-Instruct models of different sizes by: - `ollama run qwen2.5:0.5b` - `ollama run qwen2.5:1.5b` - `ollama run qwen2.5:3b` - `ollama run qwen2.5:7b` - `ollama run qwen2.5:14b` - `ollama run qwen2.5:32b` - `ollama run qwen2.5:72b` :::{note} `ollama` does not host base models. Even though the tag may not have the instruct suffix, they are all instruct models. ::: ## Run Ollama with Your GGUF Files Sometimes you don't want to pull models and you just want to use Ollama with your own GGUF files. Suppose you have a GGUF file of Qwen2.5, `qwen2.5-7b-instruct-q5_0.gguf`. For the first step, you need to create a file called `Modelfile`. The content of the file is shown below: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Then create the Ollama model by running: ```bash ollama create qwen2.5_7b -f Modelfile ``` Once it is finished, you can run your Ollama model by: ```bash ollama run qwen2.5_7b ``` ## Tool Use Tool use is now supported Ollama and you should be able to run Qwen2.5 models with it. For more details, see our [function calling guide](../framework/function_call). --- ### Source Training Axolotl (`docs/source/training/axolotl.md`) # Axolotl This guide will help you get started with post-training (SFT, RLHF, RM, PRM) for Qwen3 / Qwen3_MOE using Axolotl, and covers optimizations to enable for better performance. ## Requirements - **GPU:** NVIDIA Ampere (or newer) for `bf16` and `Flash Attention`, or AMD GPU - **Python:** ≥3.11 - **CUDA:** ≥12.4 (for NVIDIA GPUs) ## Installation You can install Axolotl using PyPI, Conda, Git, Docker, or launch a cloud environment. :::{important} Install PyTorch *before* installing Axolotl to ensure CUDA compatibility. ::: For the latest instructions, see the official [Axolotl Installation Guide](https://docs.axolotl.ai/docs/installation.html). ## Quickstart ### SFT We have provided a sample YAML config for SFT with Qwen/Qwen3-32B: [SFT 32B QLoRA config](https://github.com/axolotl-ai-cloud/axolotl/blob/v0.9.2/examples/qwen3/32b-qlora.yaml). ```shell # Train the model axolotl train path/to/32b-qlora.yaml # Merge LoRA weights with the base model # This will create a new `merged` directory under `{output_dir}` axolotl merge-lora path/to/32b-qlora.yaml ``` :::{tip} To train a smaller model, edit the `base_model` in your config: ```yaml base_model: Qwen/Qwen3-8B ``` ::: Qwen3 works with all Axolotl features including `Flash Attention`, `bf16`, `LoRA`, `torch_compile`, and `QLoRA`. To run on more than single GPU, please take a look at the [Multi-GPU Training Guide](https://docs.axolotl.ai/docs/multi-gpu.html) or [Multi-node Training Guide](https://docs.axolotl.ai/docs/multi-node.html). ### RLHF See the [RLHF Guide](https://docs.axolotl.ai/docs/rlhf.html) for required dataset formats and examples for each method. ### RM/PRM Please refer to the [Reward Modelling Guide](https://docs.axolotl.ai/docs/reward_modelling.html) for required dataset formats and config examples. ## Dataset By default, the example config uses the `mlabonne/FineTome-100k` dataset (from HuggingFace Hub). You can substitute any dataset of your own. ### SFT Dataset Format Axolotl handles various SFT dataset formats, but the current **recommended** format (for use with `chat_template`) is the OpenAI Messages format: ```json [ { "messages": [ { "role": "user", "content": "What is Qwen3?" }, { "role": "assistant", "content": "Qwen3 is a language model..." } ] } ] ``` Use this in your config: ```yaml datasets: - path: path/to/your/dataset.json type: chat_template ``` You can also load datasets from multiple sources: HuggingFace Hub, local files, directories, S3, GCS, Azure, etc. See the [Dataset Loading Guide](https://docs.axolotl.ai/docs/dataset_loading.html) for more details. To load different dataset formats, refer to the [SFT Dataset Formats Guide](https://docs.axolotl.ai/docs/dataset-formats/#supervised-fine-tuning-sft). ## Optimizations With Qwen3/Qwen3_MOE, you can leverage Axolotl's custom optimizations for improved speed and reduced memory usage: - [Cut Cross Entropy](https://docs.axolotl.ai/docs/custom_integrations.html#cut-cross-entropy) - [Liger Kernels](https://docs.axolotl.ai/docs/custom_integrations.html#liger-kernels) - (LoRA/QLoRA only): [LoRA Kernels Optimization](https://docs.axolotl.ai/docs/lora_optims.html) ## Additional Suggestions ### Troubleshooting - Ensure your CUDA version matches your GPU and PyTorch version. - If running into out-of-memory issues, try reducing your batch size, enable the optimizations above, or reduce sequence length. - Qwen3 MoE may have slower training due to the upstream transformer's handling of MoE layers. - For help, check the help channel on [Axolotl Discord](https://discord.gg/7m9sfhzaf3) or create a Discussion on [Axolotl GitHub](https://github.com/axolotl-ai-cloud/axolotl). ### Links - [Axolotl Documentation](https://docs.axolotl.ai/) - [Axolotl Discord](https://discord.gg/7m9sfhzaf3) - [Axolotl GitHub](https://github.com/axolotl-ai-cloud/axolotl) - [Axolotl Website](https://axolotl.ai) --- ### Source Training Llama Factory (`docs/source/training/llama_factory.md`) # LLaMA-Factory :::{attention} To be updated for Qwen3. ::: Here we provide a script for supervised finetuning Qwen2.5 with [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory). This script for supervised finetuning (SFT) has the following features: - Support single-GPU and multi-GPU training; - Support full-parameter tuning, LoRA, Q-LoRA, Dora. In the following, we introduce more details about the usage of the script. ## Installation Before you start, make sure you have installed the following packages: 1. Follow the instructions of [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory), and build the environment. 2. Install these packages (Optional): ``` pip install deepspeed pip install flash-attn --no-build-isolation ``` 3. If you want to use [FlashAttention-2](https://github.com/Dao-AILab/flash-attention), make sure your CUDA is 11.6 and above. ## Data Preparation LLaMA-Factory provides several training datasets in `data` folder, you can use it directly. If you are using a custom dataset, please prepare your dataset as follows. 1. Organize your data in a **json** file and put your data in `data` folder. LLaMA-Factory supports dataset in `alpaca` or `sharegpt` format. - The dataset in `alpaca` format should follow the below format: ```json [ { "instruction": "user instruction (required)", "input": "user input (optional)", "output": "model response (required)", "system": "system prompt (optional)", "history": [ ["user instruction in the first round (optional)", "model response in the first round (optional)"], ["user instruction in the second round (optional)", "model response in the second round (optional)"] ] } ] ``` - The dataset in `sharegpt` format should follow the below format: ```json [ { "conversations": [ { "from": "human", "value": "user instruction" }, { "from": "gpt", "value": "model response" } ], "system": "system prompt (optional)", "tools": "tool description (optional)" } ] ``` 2. Provide your dataset definition in `data/dataset_info.json` in the following format . - For `alpaca` format dataset, the columns in `dataset_info.json` should be: ```json "dataset_name": { "file_name": "dataset_name.json", "columns": { "prompt": "instruction", "query": "input", "response": "output", "system": "system", "history": "history" } } ``` - For `sharegpt` format dataset, the columns in `dataset_info.json` should be: ```json "dataset_name": { "file_name": "dataset_name.json", "formatting": "sharegpt", "columns": { "messages": "conversations", "system": "system", "tools": "tools" }, "tags": { "role_tag": "from", "content_tag": "value", "user_tag": "user", "assistant_tag": "assistant" } } ``` ## Training Execute the following training command: ```bash DISTRIBUTED_ARGS=" --nproc_per_node $NPROC_PER_NODE \ --nnodes $NNODES \ --node_rank $NODE_RANK \ --master_addr $MASTER_ADDR \ --master_port $MASTER_PORT " torchrun $DISTRIBUTED_ARGS src/train.py \ --deepspeed $DS_CONFIG_PATH \ --stage sft \ --do_train \ --use_fast_tokenizer \ --flash_attn \ --model_name_or_path $MODEL_PATH \ --dataset your_dataset \ --template qwen \ --finetuning_type lora \ --lora_target q_proj,v_proj\ --output_dir $OUTPUT_PATH \ --overwrite_cache \ --overwrite_output_dir \ --warmup_steps 100 \ --weight_decay 0.1 \ --per_device_train_batch_size 4 \ --gradient_accumulation_steps 4 \ --ddp_timeout 9000 \ --learning_rate 5e-6 \ --lr_scheduler_type cosine \ --logging_steps 1 \ --cutoff_len 4096 \ --save_steps 1000 \ --plot_loss \ --num_train_epochs 3 \ --bf16 ``` and enjoy the training process. To make changes to your training, you can modify the arguments in the training command to adjust the hyperparameters. One argument to note is `cutoff_len`, which is the maximum length of the training data. Control this parameter to avoid OOM error. ## Merge LoRA If you train your model with LoRA, you probably need to merge adapter parameters to the main branch. Run the following command to perform the merging of LoRA adapters. ```bash CUDA_VISIBLE_DEVICES=0 llamafactory-cli export \ --model_name_or_path path_to_base_model \ --adapter_name_or_path path_to_adapter \ --template qwen \ --finetuning_type lora \ --export_dir path_to_export \ --export_size 2 \ --export_legacy_format False ``` ## Conclusion The above content is the simplest way to use LLaMA-Factory to train Qwen. Feel free to dive into the details by checking the official repo! --- ### Source Training Ms Swift (`docs/source/training/ms_swift.md`) # MS-SWIFT ModelScope SWIFT (**ms-swift**) is the large model and multimodal large model training and deployment framework provided by the [ModelScope community](https://modelscope.cn/). GitHub repository: [ms-swift](https://github.com/modelscope/ms-swift) Features of using ms-swift for training LLM: - **Model Types**: Supports 500+ plain-text large models and 200+ multimodal large models, covering the entire process from training to deployment. - **Hardware Support**: Compatible with CPUs, RTX series GPUs, T4/V100, A10/A100/H100, Ascend NPUs, MPS, and more. - **Training Methods**: Supports full-parameter fine-tuning, LoRA, QLoRA, DoRA, and other techniques. - **Distributed Training**: Supports distributed training technologies such as DDP, device_map, DeepSpeed ZeRO-2/ZeRO-3, FSDP, and integrates parallelism techniques from Megatron, including Tensor Parallelism, Pipeline Parallelism, Sequence Parallelism, and Expert Parallelism. - **RLHF Training**: Supports human alignment methods like DPO, GRPO, DAPO, RM, PPO, KTO, etc., for both plain-text and multimodal large models. This article will demonstrate runnable training demos and provide the format for custom datasets. It includes how to use ms-swift for SFT and GRPO on Qwen3-8B, as well as using Megatron-SWIFT (ms-swift's integration of Megatron-LM) for SFT on Qwen3-30B-A3B. Through expert parallelism technology, MoE model training can be accelerated by nearly 10 times. Before starting fine-tuning, ensure your environment is properly set up. ```shell pip install ms-swift -U # Install from source pip install git+https://github.com/modelscope/ms-swift.git pip install transformers -U # Optional packages pip install deepspeed # multi-GPU training pip install liger-kernel # save GPU memory resources pip install flash-attn --no-build-isolation ``` ## Supervised Fine-Tuning (SFT) ### Data Preparation The custom dataset format for SFT using ms-swift is as follows (the system field is optional). You can organize it into formats such as JSON, JSONL, or CSV. Specify `--dataset ` in the training script. For complete dataset formatting guidelines, see: [Custom Dataset Documentation](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html) - General format ```json {"messages": [ {"role": "system", "content": ""}, {"role": "user", "content": ""}, {"role": "assistant", "content": ""} ]} ``` - Format with think ```json {"messages": [ {"role": "user", "content": "Where is the capital of Zhejiang?"}, {"role": "assistant", "content": "\n...\n\n\nThe capital of Zhejiang is Hangzhou."} ]} ``` If you want to train using data without a chain of thought but retain the model's reasoning ability, there are two approaches to minimize disruption during fine-tuning: **Option 1**: During training, specify `--loss_scale ignore_empty_think` to ignore the loss calculation for `\n\n\n\n`, preventing the loss of reasoning ability. Refer to the training script [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/think_model/qwen3_demo1.sh). The custom dataset format is as follows: ```json {"messages": [ {"role": "user", "content": "Where is the capital of Zhejiang?"}, {"role": "assistant", "content": "\n\n\n\nThe capital of Zhejiang is Hangzhou."} ]} ``` **Option 2**: Add `/no_think` to the query in the dataset to avoid the loss of reasoning ability. Refer to the training script [here](https://github.com/modelscope/ms-swift/blob/main/examples/train/think_model/qwen3_demo2.sh). The custom dataset format is as follows: ```json {"messages": [ {"role": "user", "content": "Where is the capital of Zhejiang? /no_think"}, {"role": "assistant", "content": "\n\n\n\nThe capital of Zhejiang is Hangzhou."} ]} ``` ### 30-Minute Self-Cognition Fine-Tuning This section introduces a 30-minute self-cognition fine-tuning process for the Qwen3-8B model. The required GPU memory is 22GB, and it can be run on the A10 provided by [ModelScope's free compute resources](https://modelscope.cn/my/mynotebook). After training, the model will identify itself as "swift-robot," trained by "swift," instead of its original self-cognition as "Qwen," trained by Alibaba Cloud. If you need to train in an offline environment, you can manually download the model and dataset and specify `--model ` and `--dataset `. The dataset can be found on [Modelscope Hub](https://modelscope.cn/datasets/swift/self-cognition). For the meaning of each parameter in the training script, please refer to the [Command-line parameters documentation](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html). ```bash CUDA_VISIBLE_DEVICES=0 \ swift sft \ --model Qwen/Qwen3-8B \ --train_type lora \ --dataset 'swift/Qwen3-SFT-Mixin#2000' \ 'swift/self-cognition:qwen3#600' \ --torch_dtype bfloat16 \ --num_train_epochs 1 \ --per_device_train_batch_size 1 \ --per_device_eval_batch_size 1 \ --learning_rate 1e-4 \ --lora_rank 8 \ --lora_alpha 32 \ --target_modules all-linear \ --gradient_accumulation_steps 16 \ --eval_steps 50 \ --save_steps 50 \ --save_total_limit 2 \ --logging_steps 5 \ --max_length 2048 \ --output_dir output \ --warmup_ratio 0.05 \ --dataloader_num_workers 4 \ --model_author swift \ --model_name swift-robot ``` After fine-tuning, you can use the following script to test the fine-tuning results. Note that the `--adapters` section needs to be modified to the directory path of the last saved checkpoint: ```bash CUDA_VISIBLE_DEVICES=0 \ swift infer \ --adapters output/vx-xxx/checkpoint-xxx \ --stream true \ --temperature 0 \ --max_new_tokens 2048 ``` ```text <<< who are you? Okay, the user asked, "who are you?" I need to introduce myself. Let me start by stating my name, swift-robot. Then, I should mention that I'm an AI assistant developed by swift. I should explain my purpose, which is to provide information and assistance. I should also highlight my capabilities, like answering questions, generating text, and engaging in conversation. It's important to keep the tone friendly and approachable. Maybe add something about being here to help and encourage the user to ask anything. Let me check if I covered all the key points: name, developer, purpose, capabilities, and a welcoming statement. Yeah, that should do it. Now, let me put that into a concise and friendly response. Hello! I am swift-robot, an artificial intelligence assistant developed by swift. My purpose is to provide information and assistance to users like you. I can answer questions, generate text, and engage in conversations on a wide range of topics. I am here to help, so feel free to ask me anything you need! ``` By default, ms-swift will use the ModelScope community to download models and datasets. If you want to use the HuggingFace community, you need to additionally specify `--use_hf true`. Merge LoRA weights: ```shell swift export \ --adapters output/checkpoint-xxx \ --merge_lora true ``` Push the model to ModelScope/HuggingFace: ```shell # If you are pushing the complete weights, you need to change `--adapters` to `--model`. # The Modelscope hub_token can be found here: https://modelscope.cn/my/myaccesstoken swift export \ --adapters output/checkpoint-xxx \ --push_to_hub true \ --hub_model_id '' \ --hub_token '' \ --use_hf false ``` If you want to use multiple GPUs for training, the following provides a demo for multi-GPU training: ```shell # 4 * 60GB # You can run the experiment by setting `--dataset AI-ModelScope/alpaca-gpt4-data-en`. # Note: If you want to specify `--packing true`, you must additionally set `--attn_impl flash_attn`. NPROC_PER_NODE=4 \ CUDA_VISIBLE_DEVICES=0,1,2,3 \ swift sft \ --model Qwen/Qwen3-8B \ --train_type full \ --dataset '' \ --torch_dtype bfloat16 \ --per_device_train_batch_size 1 \ --per_device_eval_batch_size 1 \ --learning_rate 1e-5 \ --gradient_accumulation_steps 4 \ --packing true \ --eval_steps 100 \ --save_steps 100 \ --logging_steps 5 \ --max_length 8192 \ --warmup_ratio 0.05 \ --dataloader_num_workers 8 \ --dataset_num_proc 8 \ --save_total_limit 2 \ --save_only_model true \ --output_dir output \ --deepspeed zero3 \ --use_liger_kernel true \ --attn_impl flash_attn ``` ## Reinforcement Learning (RL) ms-swift supports RLHF methods such as DPO, GRPO, DAPO, PPO, KTO, and more. This section will focus on an example of using ms-swift to perform GRPO training for Qwen3-8B. For detailed RLHF support information, please refer to: [Supported Features](https://swift.readthedocs.io/en/latest/Instruction/Pre-training-and-Fine-tuning.html). ### Environment Setup In addition to installing the ms-swift related dependencies introduced above, the following dependencies also need to be installed: ```shell pip install "math_verify==0.5.2" pip install vllm ``` ### Data Preparation The dataset format for GRPO training using ms-swift is similar to that of SFT, except that the assistant part of the last round is not required. If using accuracy as a reward, a `solution` column is needed to calculate the accuracy. Example Dataset Formats: ```json {"messages": [{"role": "user", "content": "Tell me tomorrow's weather"}]} {"messages": [{"role": "user", "content": "What is 1 + 1?"}, {"role": "assistant", "content": "It equals 2"}, {"role": "user", "content": "What about adding 1?"}]} {"messages": [{"role": "user", "content": "What is your name?"}]} ``` For dataset preparation for other RLHF algorithms, see: [Custom Dataset Documentation](https://swift.readthedocs.io/en/latest/Customization/Custom-dataset.html#rlhf). Notes on Dataset Requirements: - **Reward Function Calculation**: The dataset format depends on the reward function being used. Additional columns may be required to support specific reward calculations. For instance: - When using the built-in accuracy or cosine similarity reward, the dataset must include a `solution` column to calculate the accuracy of the responses. - Other columns in the dataset will be passed as `**kwargs` to the reward function for additional customization. - **Customizing the Reward Function**: To adapt the reward function to your specific needs, you can refer to the following resource: [External Reward Plugin](https://github.com/modelscope/ms-swift/tree/main/examples/train/grpo/plugin). This plugin provides examples and templates for implementing custom reward functions. During the training process, we use vLLM to accelerate the sampling process. By setting `num_infer_workers=8`, we deploy a vLLM engine for each device to speed up the sampling process. ```shell # 70G*8 CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ NPROC_PER_NODE=8 \ swift rlhf \ --rlhf_type grpo \ --model Qwen/Qwen3-8B \ --train_type full \ --dataset 'AI-MO/NuminaMath-TIR#5000' \ --torch_dtype bfloat16 \ --num_train_epochs 1 \ --per_device_train_batch_size 2 \ --per_device_eval_batch_size 2 \ --learning_rate 1e-6 \ --save_total_limit 2 \ --logging_steps 5 \ --output_dir output \ --gradient_accumulation_steps 1 \ --warmup_ratio 0.05 \ --dataloader_num_workers 4 \ --max_completion_length 4096 \ --vllm_max_model_len 8192 \ --reward_funcs accuracy \ --num_generations 16 \ --use_vllm true \ --vllm_gpu_memory_utilization 0.4 \ --sleep_level 1 \ --offload_model true \ --offload_optimizer true \ --gc_collect_after_offload true \ --deepspeed zero3 \ --num_infer_workers 8 \ --tensor_parallel_size 1 \ --temperature 1.0 \ --top_p 0.85 \ --log_completions true \ --overlong_filter true ``` ## Megatron-SWIFT ms-swift incorporates Megatron parallelism techniques to accelerate the training of large models. The supported models can be found in the [Supported Models Documentation](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html). For environment preparation and the conversion between HF and MCore model weights, you can refer to the [Megatron-SWIFT Training Documentation](https://swift.readthedocs.io/en/latest/Instruction/Megatron-SWIFT-Training.html). These topics will not be elaborated here. We will use Alibaba Cloud DLC to start the training The training environment consists of 2 machines with 8 * 80GiB A800 GPUs. For more information on multi-node startup methods, refer to [here](https://github.com/modelscope/ms-swift/tree/main/examples/train/multi-node). ```shell # https://help.aliyun.com/zh/pai/user-guide/general-environment-variables # Ensure that the weight-saving paths on the two nodes are identical. NNODES=$WORLD_SIZE \ NODE_RANK=$RANK \ megatron sft \ --load Qwen3-30B-A3B-Base-mcore \ --dataset 'liucong/Chinese-DeepSeek-R1-Distill-data-110k-SFT' \ --tensor_model_parallel_size 2 \ --expert_model_parallel_size 8 \ --moe_grouped_gemm true \ --moe_shared_expert_overlap true \ --moe_aux_loss_coeff 0.01 \ --micro_batch_size 1 \ --global_batch_size 16 \ --packing true \ --recompute_granularity full \ --recompute_method uniform \ --recompute_num_layers 1 \ --train_iters 2000 \ --eval_iters 50 \ --finetune true \ --cross_entropy_loss_fusion true \ --lr 1e-5 \ --lr_warmup_iters 100 \ --min_lr 1e-6 \ --save megatron_output/Qwen3-30B-A3B-Base \ --eval_interval 200 \ --save_interval 200 \ --max_length 8192 \ --num_workers 8 \ --dataset_num_proc 8 \ --no_save_optim true \ --no_save_rng true \ --sequence_parallel true \ --use_flash_attn true ``` The custom dataset format is the same as `swift sft`, which can be found in the previous section. Simply specify `--dataset `. The following is a comparison of training speed and GPU memory usage between `megatron sft` and `swift sft` for full-parameter fine-tuning of the Qwen3-30B-A3B model: | | Megatron-LM | DeepSpeed-ZeRO2 | DeepSpeed-ZeRO3 | | ---------------- | ----------- | --------------- | --------------- | | Training Speed | 9.6s/it | - | 91.2s/it | | GPU Memory Usage | 16 * 60GiB | OOM | 16 * 80GiB | ## Conclusion The above is the best practice for training Qwen3 series models using ms-swift. If you encounter any difficulties during use, please join the discussion in [this issue](https://github.com/modelscope/ms-swift/issues/4030). --- ### Source Training Unsloth (`docs/source/training/unsloth.md`) # Unsloth This guide will teach you how to easily train Qwen3 models with Unsloth. Unsloth simplifies local model training, handling everything from loading and quantization to training, evaluation, running, and deployment with inference engines (Ollama, llama.cpp, vLLM). **Train Qwen** models 2× faster using 70% less VRAM. **GitHub repo:** [Unsloth](https://github.com/unslothai/unsloth) ## ⭐ Key Features - Supports full fine-tuning, pretraining, LoRA, QLoRA, 8-bit training & more - Single and multi-GPU support (Linux, Windows, Colab, Kaggle; NVIDIA GPUs, soon AMD & Intel) - Compatible with all transformer models: TTS, multimodal, STT, BERT, RL - RLHF support: GRPO, DPO, DAPO, RM, PPO, KTO, etc. - Hand-written Triton kernels and a manual backprop engine ensure no accuracy degradation (0% approximation). ## Quickstart **Local Installation (Linux recommended):** ```bash pip install unsloth ``` You can view Unsloth’s full [installation instructions here.](https://docs.unsloth.ai/get-started/installing-+-updating) ## Fine-tuning Qwen3 with Unsloth Unsloth makes Qwen3 fine-tuning 2× faster, uses 70% less VRAM, with 8× longer contexts. Qwen3 (14B) fits in a free 16 GB Colab Tesla T4 GPU. To retain Qwen3's reasoning capabilities, use a 75% reasoning to 25% non-reasoning dataset ratio (e.g., NVIDIA’s math‑reasoning dataset + Maxime’s FineTome). For more details, see Unsloth’s full [Qwen3 fine-tuning guide](https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune#fine-tuning-qwen3-with-unsloth). ### Colab Notebooks - [Qwen3 (14B) Reasoning + Conversational](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(14B)-Reasoning-Conversational.ipynb) - [Qwen3 (4B) Advanced GRPO LoRA](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb) - [Qwen3 (14B) Alpaca (Base model)](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(14B)-Alpaca.ipynb) **Update Unsloth locally:** ```bash pip install --upgrade --force-reinstall --no-cache-dir unsloth unsloth_zoo ``` ### Fine-tuning Qwen3 MoE Models Supported MoE models include 30B‑A3B and 235B‑A22B. Unsloth fine-tunes the 30B‑A3B model with just 17.5 GB VRAM. Router-layer fine-tuning is disabled by default. Use `FastModel` for MoE fine-tuning: ```python from unsloth import FastModel model, tokenizer = FastModel.from_pretrained( model_name="unsloth/Qwen3-30B-A3B", max_seq_length=2048, load_in_4bit=True, load_in_8bit=False, full_finetuning=False, ) ``` ### Notebook Guide For an end-to-end walkthrough, see Unsloth’s [full end-to-end fine-tuning guide](https://docs.unsloth.ai/basics/reinforcement-learning-rl-guide). - Open the notebook → click **Runtime ▸ Run all**. - Adjust settings (e.g., model name, context length) directly in the notebook: - `max_seq_length`: Recommended 2048 (Qwen3 supports up to 40960). - `load_in_4bit=True`: reduces memory usage by 4×. - Enable full fine-tuning (`full_finetuning=True`) or 8-bit training (`load_in_8bit=True`). If you want to use models directly from [ModelScope](https://modelscope.cn/organization/unsloth), use: ```bash pip install modelscope -qqq ``` ```python import os os.environ["UNSLOTH_USE_MODELSCOPE"] = "1" from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( model_name="unsloth/Qwen3-4B-Base", max_seq_length=2048, ) ``` ## RL & GRPO with Qwen3 You can also train Qwen models with reinforcement learning (RL) using Unsloth. Explore Unsloth’s advanced GRPO notebook, featuring proximity-based reward scoring and Hugging Face's Open‑R1 math dataset: [Qwen3 (4B) Advanced GRPO LoRA notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb). - Proximity-based rewards for closer answers - Custom GRPO formatting and templates - Enhanced evaluation accuracy with regex matching ## Resources & Links That’s how you can easily train Qwen models with Unsloth. If you need any help, join the discussion on Unsloth’s [Discord](https://discord.com/invite/unsloth) or [GitHub](https://github.com/unslothai/unsloth) pages. **Links:** - [Unsloth Documentation](https://docs.unsloth.ai/) - [Unsloth Discord](https://discord.com/invite/unsloth) - [Unsloth Website](https://unsloth.ai/) - [Unsloth Reddit](https://www.reddit.com/r/unsloth/) --- ### Source Training Verl (`docs/source/training/verl.md`) # verl verl is a flexible, efficient and production-ready RL training library for large language models (LLMs). verl is the open-source version of [HybridFlow: A Flexible and Efficient RLHF Framework](https://arxiv.org/abs/2409.19256v2) paper. GitHub repository: [verl](https://github.com/volcengine/verl) verl is flexible and easy to use with: - **Easy extension of diverse RL algorithms**: The hybrid-controller programming model enables flexible representation and efficient execution of complex Post-Training dataflows. Build RL dataflows such as GRPO, PPO in a few lines of code. - **Seamless integration of existing LLM infra with modular APIs**: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as FSDP, Megatron-LM, vLLM, SGLang, etc - **Flexible device mapping**: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes. - **Ready integration with popular HuggingFace models**: verl supports popular LLM models, including Qwen, Llama, and more. verl is fast with: - **State-of-the-art throughput**: SOTA LLM training and inference engine integrations and SOTA RL throughput. - **Efficient actor model resharding with 3D-HybridEngine**: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases. Next, we will introduce how to use verl for training Qwen3 models. ## Reinforcement Learning (RL) Now, verl supports various combinations of training frameworks and inference frameworks, including FSDP, Megatron-LM, vLLM, SGLang, etc. verl also supports training with multiple algorithms such as PPO, GRPO, DAPO, etc. ### Step1: Environment and Training Preparation You can follow verl's [installation guide](https://verl.readthedocs.io/en/latest/start/install.html) to complete the environment configuration. Data preparation can be done by running the following command: ```shell git clone https://github.com/volcengine/verl.git cd verl python3 examples/data_preprocess/gsm8k.py --local_dir ~/data/gsm8k ``` Model download can be done using the following command: ```shell python3 -c "import transformers; transformers.pipeline('text-generation', model='Qwen/Qwen3-1.7B')" ``` ### Step2: Start Training In verl, training frameworks and inference frameworks can be combined freely, as long as the training framework and inference framework themselves support model training and inference tasks, so that verl can support RL-related training. Below is an example using FSDP and vLLM to demonstrate how to train Qwen3 models in verl. We chose Qwen3-1.7B as the example, as it only requires a single 80GB GPU and a machine with more than 64GB of memory to start training. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Finally If you encounter any difficulties during use, please join the discussion at [GitHub](https://github.com/volcengine/verl/discussions). --- --- METRICS --- - Files Extracted: 36 - Estimated Token Budget: ~72097 tokens - Canonical Reference: https://codewiki.google/github.com/QwenLM/Qwen2.5