LlamaFactory Documentation

File: README.md Copy Chapter

Used by [Amazon](https://aws.amazon.com/cn/blogs/machine-learning/how-apoidea-group-enhances-visual-information-extraction-from-banking-documents-with-multimodal-models-using-llama-factory-on-amazon-sagemaker-hyperpod/), [NVIDIA](https://build.nvidia.com/spark/llama-factory), [Aliyun](https://help.aliyun.com/zh/pai/use-cases/fine-tune-a-llama-3-model-with-llama-factory), etc. Copy Section

https://github.com/user-attachments/assets/9b7033e8-f08a-4c3f-bd33-547896664e6e

👋 Join our WeChat and NPU user groups.

[ English | 中文 ]

Fine-tuning a large language model can be easy as...

https://github.com/user-attachments/assets/3991a3a8-4276-4d30-9cab-4cb0c4b9b99e

Start local training:

• Please refer to usage

Start cloud training:

• Colab (free): https://colab.research.google.com/drive/1eRTPn37ltBbYsISy9Aw2NuI2Aq5CQrD9?usp=sharing
• PAI-DSW (free trial): https://gallery.pai-ml.com/#/preview/deepLearning/nlp/llama_factory
• AMD GPU Cloud (free credits): https://github.com/AMD-AIM/AMD_Developers_Notebooks/blob/main/en/AMD_developer_LLaMAFactory_note_en.md

Read technical notes:

• Documentation (WIP): https://llamafactory.readthedocs.io/en/latest/
• Documentation (AMD GPU): https://rocm.docs.amd.com/projects/ai-developer-hub/en/latest/notebooks/fine_tune/llama_factory_llama3.html
• Documentation (ASCEND NPU): https://llamafactory.readthedocs.io/en/latest/multibackend/npu/index.html
• Official Blog: https://blog.llamafactory.net/en/

[!NOTE]
Except for the above links, all other websites are unauthorized third-party websites. Please carefully use them.

Table of Contents Copy Chapter

• Features
• Blogs
• Changelog
• Supported Models
• Supported Training Approaches
• Provided Datasets
• Requirement
• Getting Started
  • Installation
  • Data Preparation
  • Quickstart
  • Fine-Tuning with LLaMA Board GUI
  • Build Docker
  • Deploy with OpenAI-style API and vLLM
  • Download from ModelScope Hub
  • Download from Modelers Hub
  • Use W&B Logger
  • Use SwanLab Logger
• Projects using LlamaFactory
• License
• Citation
• Acknowledgement

Features Copy Chapter

• Various models: LLaMA, LLaVA, Mistral, Mixtral-MoE, Qwen3, Qwen3-VL, DeepSeek, Gemma, GLM, Phi, etc.
• Integrated methods: (Continuous) pre-training, (multimodal) supervised fine-tuning, reward modeling, PPO, DPO, KTO, ORPO, etc.
• Scalable resources: 16-bit full-tuning, freeze-tuning, LoRA and 2/3/4/5/6/8-bit QLoRA via AQLM/AWQ/GPTQ/LLM.int8/HQQ/EETQ.
• Advanced algorithms: GaLore, BAdam, APOLLO, Adam-mini, Muon, OFT, DoRA, LongLoRA, LLaMA Pro, Mixture-of-Depths, LoRA+, LoftQ and PiSSA.
• Practical tricks: FlashAttention-2, Unsloth, Liger Kernel, KTransformers, RoPE scaling, NEFTune and rsLoRA.
• Wide tasks: Multi-turn dialogue, tool using, image understanding, visual grounding, video recognition, audio understanding, etc.
• Experiment monitors: LlamaBoard, TensorBoard, Wandb, MLflow, SwanLab, etc.
• Faster inference: OpenAI-style API, Gradio UI and CLI with vLLM worker or SGLang worker.

Day-N Support for Fine-Tuning Cutting-Edge Models Copy Section

Blogs Copy Chapter

[!TIP]
Now we have a dedicated blog for LlamaFactory!

Website: https://blog.llamafactory.net/en/

• 💡 KTransformers Fine-Tuning × LlamaFactory: Fine-tuning 1000 Billion models with 2 4090-GPU + CPU (English)
• 💡 Easy Dataset × LlamaFactory: Enabling LLMs to Efficiently Learn Domain Knowledge (English)
• 💡 DataFlow × LlamaFactory: Producing High-Quality Data for LLM Training with a Data Preparation Pipeline (English) | 中文
• 💡 DataFlex × LlamaFactory: A Data-Centric Dynamic Training System Built on LlamaFactory (English) | 中文
• A One-Stop Code-Free Model Reinforcement Learning and Deployment Platform based on LlamaFactory and EasyR1 (Chinese)
• How Apoidea Group enhances visual information extraction from banking documents with multimodal models using LlamaFactory on Amazon SageMaker HyperPod (English)

All Blogs

• LlamaFactory: Fine-tuning the DeepSeek-R1-Distill-Qwen-7B Model for News Classifier (Chinese)
• A One-Stop Code-Free Model Fine-Tuning & Deployment Platform based on SageMaker and LlamaFactory (Chinese)
• LlamaFactory Multi-Modal Fine-Tuning Practice: Fine-Tuning Qwen2-VL for Personal Tourist Guide (Chinese)
• LlamaFactory: Fine-tuning Llama3 for Role-Playing (Chinese)

Changelog Copy Chapter

[25/10/26] We support Megatron-core training backend with mcore_adapter. See PR #9237 to get started.

[25/08/22] We supported OFT and OFTv2. See examples for usage.

[25/08/20] We supported fine-tuning the Intern-S1-mini models. See PR #8976 to get started.

[25/08/06] We supported fine-tuning the GPT-OSS models. See PR #8826 to get started.

Full Changelog

[25/07/02] We supported fine-tuning the GLM-4.1V-9B-Thinking model.

[25/04/28] We supported fine-tuning the Qwen3 model family.

[25/04/21] We supported the Muon optimizer. See examples for usage. Thank @tianshijing's PR.

[25/04/16] We supported fine-tuning the InternVL3 model. See PR #7258 to get started.

[25/04/14] We supported fine-tuning the GLM-Z1 and Kimi-VL models.

[25/04/06] We supported fine-tuning the Llama 4 model. See PR #7611 to get started.

[25/03/31] We supported fine-tuning the Qwen2.5 Omni model. See PR #7537 to get started.

[25/03/15] We supported SGLang as inference backend. Try infer_backend: sglang to accelerate inference.

[25/03/12] We supported fine-tuning the Gemma 3 model.

[25/02/24] Announcing EasyR1, an efficient, scalable and multi-modality RL training framework for efficient GRPO training.

[25/02/11] We supported saving the Ollama modelfile when exporting the model checkpoints. See examples for usage.

[25/02/05] We supported fine-tuning the Qwen2-Audio and MiniCPM-o-2.6 on audio understanding tasks.

[25/01/31] We supported fine-tuning the DeepSeek-R1 and Qwen2.5-VL models.

[25/01/15] We supported APOLLO optimizer. See examples for usage.

[25/01/14] We supported fine-tuning the MiniCPM-o-2.6 and MiniCPM-V-2.6 models. Thank @BUAADreamer's PR.

[25/01/14] We supported fine-tuning the InternLM 3 models. Thank @hhaAndroid's PR.

[25/01/10] We supported fine-tuning the Phi-4 model.

[24/12/21] We supported using SwanLab for experiment tracking and visualization. See this section for details.

[24/11/27] We supported fine-tuning the Skywork-o1 model and the OpenO1 dataset.

[24/10/09] We supported downloading pre-trained models and datasets from the Modelers Hub. See this tutorial for usage.

[24/09/19] We supported fine-tuning the Qwen2.5 models.

[24/08/30] We supported fine-tuning the Qwen2-VL models. Thank @simonJJJ's PR.

[24/08/27] We supported Liger Kernel. Try enable_liger_kernel: true for efficient training.

[24/08/09] We supported Adam-mini optimizer. See examples for usage. Thank @relic-yuexi's PR.

[24/07/04] We supported contamination-free packed training. Use neat_packing: true to activate it. Thank @chuan298's PR.

[24/06/16] We supported PiSSA algorithm. See examples for usage.

[24/06/07] We supported fine-tuning the Qwen2 and GLM-4 models.

[24/05/26] We supported SimPO algorithm for preference learning. See examples for usage.

[24/05/20] We supported fine-tuning the PaliGemma series models. Note that the PaliGemma models are pre-trained models, you need to fine-tune them with paligemma template for chat completion.

[24/05/18] We supported KTO algorithm for preference learning. See examples for usage.

[24/05/14] We supported training and inference on the Ascend NPU devices. Check installation section for details.

[24/04/26] We supported fine-tuning the LLaVA-1.5 multimodal LLMs. See examples for usage.

[24/04/22] We provided a Colab notebook for fine-tuning the Llama-3 model on a free T4 GPU. Two Llama-3-derived models fine-tuned using LlamaFactory are available at Hugging Face, check Llama3-8B-Chinese-Chat and Llama3-Chinese for details.

[24/04/21] We supported Mixture-of-Depths according to AstraMindAI's implementation. See examples for usage.

[24/04/16] We supported BAdam optimizer. See examples for usage.

[24/04/16] We supported unsloth's long-sequence training (Llama-2-7B-56k within 24GB). It achieves 117% speed and 50% memory compared with FlashAttention-2, more benchmarks can be found in this page.

[24/03/31] We supported ORPO. See examples for usage.

[24/03/21] Our paper "LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models" is available at arXiv!

[24/03/20] We supported FSDP+QLoRA that fine-tunes a 70B model on 2x24GB GPUs. See examples for usage.

[24/03/13] We supported LoRA+. See examples for usage.

[24/03/07] We supported GaLore optimizer. See examples for usage.

[24/03/07] We integrated vLLM for faster and concurrent inference. Try infer_backend: vllm to enjoy 270% inference speed.

[24/02/28] We supported weight-decomposed LoRA (DoRA). Try use_dora: true to activate DoRA training.

[24/02/15] We supported block expansion proposed by LLaMA Pro. See examples for usage.

[24/02/05] Qwen1.5 (Qwen2 beta version) series models are supported in LlamaFactory. Check this blog post for details.

[24/01/18] We supported agent tuning for most models, equipping model with tool using abilities by fine-tuning with dataset: glaive_toolcall_en.

[23/12/23] We supported unsloth's implementation to boost LoRA tuning for the LLaMA, Mistral and Yi models. Try use_unsloth: true argument to activate unsloth patch. It achieves 170% speed in our benchmark, check this page for details.

[23/12/12] We supported fine-tuning the latest MoE model Mixtral 8x7B in our framework. See hardware requirement here.

[23/12/01] We supported downloading pre-trained models and datasets from the ModelScope Hub. See this tutorial for usage.

[23/10/21] We supported NEFTune trick for fine-tuning. Try neftune_noise_alpha: 5 argument to activate NEFTune.

[23/09/27] We supported $S^2$-Attn proposed by LongLoRA for the LLaMA models. Try shift_attn: true argument to enable shift short attention.

[23/09/23] We integrated MMLU, C-Eval and CMMLU benchmarks in this repo. See examples for usage.

[23/09/10] We supported FlashAttention-2. Try flash_attn: fa2 argument to enable FlashAttention-2 if you are using RTX4090, A100 or H100 GPUs.

[23/08/12] We supported RoPE scaling to extend the context length of the LLaMA models. Try rope_scaling: linear argument in training and rope_scaling: dynamic argument at inference to extrapolate the position embeddings.

[23/08/11] We supported DPO training for instruction-tuned models. See examples for usage.

[23/07/31] We supported dataset streaming. Try streaming: true and max_steps: 10000 arguments to load your dataset in streaming mode.

[23/07/29] We released two instruction-tuned 13B models at Hugging Face. See these Hugging Face Repos (LLaMA-2 / Baichuan) for details.

[23/07/18] We developed an all-in-one Web UI for training, evaluation and inference. Try train_web.py to fine-tune models in your Web browser. Thank @KanadeSiina and @codemayq for their efforts in the development.

[23/07/09] We released FastEdit ⚡🩹, an easy-to-use package for editing the factual knowledge of large language models efficiently. Please follow FastEdit if you are interested.

[23/06/29] We provided a reproducible example of training a chat model using instruction-following datasets, see Baichuan-7B-sft for details.

[23/06/22] We aligned the demo API with the OpenAI's format where you can insert the fine-tuned model in arbitrary ChatGPT-based applications.

[23/06/03] We supported quantized training and inference (aka QLoRA). See examples for usage.

[!TIP]
If you cannot use the latest feature, please pull the latest code and install LlamaFactory again.

Supported Models Copy Chapter

[!NOTE]
For the "base" models, the template argument can be chosen from default, alpaca, vicuna etc. But make sure to use the corresponding template for the "instruct/chat" models.

If the model has both reasoning and non-reasoning versions, please use the _nothink suffix to distinguish between them. For example, qwen3 and qwen3_nothink.

Remember to use the SAME template in training and inference.

*: You should install the transformers from main branch and use DISABLE_VERSION_CHECK=1 to skip version check.

**: You need to install a specific version of transformers to use the corresponding model.

Please refer to constants.py for a full list of models we supported.

You also can add a custom chat template to template.py.

Supported Training Approaches Copy Chapter

[!TIP]
The implementation details of PPO can be found in this blog.

Provided Datasets Copy Chapter

Pre-training datasets

• Wiki Demo (en)
• RefinedWeb (en)
• RedPajama V2 (en)
• Wikipedia (en)
• Wikipedia (zh)
• Pile (en)
• SkyPile (zh)
• FineWeb (en)
• FineWeb-Edu (en)
• CCI3-HQ (zh)
• CCI3-Data (zh)
• CCI4.0-M2-Base-v1 (en&zh)
• CCI4.0-M2-CoT-v1 (en&zh)
• CCI4.0-M2-Extra-v1 (en&zh)
• The Stack (en)
• StarCoder (en)

Supervised fine-tuning datasets

• Identity (en&zh)
• Stanford Alpaca (en)
• Stanford Alpaca (zh)
• Alpaca GPT4 (en&zh)
• Glaive Function Calling V2 (en&zh)
• LIMA (en)
• Guanaco Dataset (multilingual)
• BELLE 2M (zh)
• BELLE 1M (zh)
• BELLE 0.5M (zh)
• BELLE Dialogue 0.4M (zh)
• BELLE School Math 0.25M (zh)
• BELLE Multiturn Chat 0.8M (zh)
• UltraChat (en)
• OpenPlatypus (en)
• CodeAlpaca 20k (en)
• Alpaca CoT (multilingual)
• OpenOrca (en)
• SlimOrca (en)
• MathInstruct (en)
• Firefly 1.1M (zh)
• Wiki QA (en)
• Web QA (zh)
• WebNovel (zh)
• Nectar (en)
• deepctrl (en&zh)
• Advertise Generating (zh)
• ShareGPT Hyperfiltered (en)
• ShareGPT4 (en&zh)
• UltraChat 200k (en)
• Infinity Instruct (zh)
• AgentInstruct (en)
• LMSYS Chat 1M (en)
• Evol Instruct V2 (en)
• Cosmopedia (en)
• STEM (zh)
• Ruozhiba (zh)
• Neo-sft (zh)
• Magpie-Pro-300K-Filtered (en)
• Magpie-ultra-v0.1 (en)
• WebInstructSub (en)
• OpenO1-SFT (en&zh)
• Open-Thoughts (en)
• Open-R1-Math (en)
• Chinese-DeepSeek-R1-Distill (zh)
• LLaVA mixed (en&zh)
• Pokemon-gpt4o-captions (en&zh)
• DLR-Web (en)
• Open Assistant (de)
• Dolly 15k (de)
• Alpaca GPT4 (de)
• OpenSchnabeltier (de)
• Evol Instruct (de)
• Dolphin (de)
• Booksum (de)
• Airoboros (de)
• Ultrachat (de)

Preference datasets

• DPO mixed (en&zh)
• UltraFeedback (en)
• COIG-P (zh)
• RLHF-V (en)
• VLFeedback (en)
• RLAIF-V (en)
• Orca DPO Pairs (en)
• HH-RLHF (en)
• Nectar (en)
• Orca DPO (de)
• KTO mixed (en)

Some datasets require confirmation before using them, so we recommend logging in with your Hugging Face account using these commands.

pip install "huggingface_hub<1.0.0"
huggingface-cli login

Requirement Copy Chapter

Hardware Requirement Copy Section

* estimated

Getting Started Copy Chapter

Installation Copy Section

[!IMPORTANT]
Installation is mandatory.

Install from Source

git clone --depth 1 https://github.com/hiyouga/LlamaFactory.git
cd LlamaFactory
pip install -e .
pip install -r requirements/metrics.txt

Optional dependencies available: metrics, deepspeed. Install with: pip install -e . && pip install -r requirements/metrics.txt -r requirements/deepspeed.txt

Additional dependencies for specific features are available in examples/requirements/.

Install from Docker Image

docker run -it --rm --gpus=all --ipc=host hiyouga/llamafactory:latest

This image is built on Ubuntu 22.04 (x86_64), CUDA 12.4, Python 3.11, PyTorch 2.6.0, and Flash-attn 2.7.4.

Find the pre-built images: https://hub.docker.com/r/hiyouga/llamafactory/tags

Please refer to build docker to build the image yourself.

Setting up a virtual environment with uv

Create an isolated Python environment with uv:

uv run llamafactory-cli webui

For Windows users

Install PyTorch

You need to manually install the GPU version of PyTorch on the Windows platform. Please refer to the official website and the following command to install PyTorch with CUDA support:

pip uninstall torch torchvision torchaudio
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
python -c "import torch; print(torch.cuda.is_available())"

If you see True then you have successfully installed PyTorch with CUDA support.

Try dataloader_num_workers: 0 if you encounter Can't pickle local object error.

Install BitsAndBytes

To enable Quantized LoRA (QLoRA) on Windows, you need to install bitsandbytes.

For most users, it is recommended to install the latest official release:

pip install bitsandbytes

If you are using uv to manage your virtual environment, it is recommended to install bitsandbytes after installing the GPU-enabled version of PyTorch:

uv pip install bitsandbytes --no-deps

[!IMPORTANT]
Pay attention to the CUDA Toolkit version when installing bitsandbytes. Official bitsandbytes releases are built for specific CUDA Toolkit versions. On Windows x86-64, separate builds are currently provided for CUDA 11.8–12.6 and CUDA 12.8–12.9. Support for NVIDIA RTX 50 Series GPUs (e.g., RTX 5060 Ti, sm_120) requires the CUDA 12.8–12.9 builds.

If your environment uses an older CUDA version, or you need compatibility with older Windows / PyTorch combinations, you can install the third-party precompiled Windows wheel:

pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.2.post2-py3-none-win_amd64.whl

Install Flash Attention-2

To enable FlashAttention-2 on the Windows platform, please use the script from flash-attention-windows-wheel to compile and install it by yourself.

For Ascend NPU users

To install LlamaFactory on Ascend NPU devices, please use Python 3.12 and install the extra dependencies with pip install -r requirements/npu.txt. Additionally, you need to install the Ascend CANN Toolkit and Kernels. Please follow the installation tutorial.

You can also download the pre-built Docker images:

# Docker Hub
docker pull hiyouga/llamafactory:latest-910b-ubuntu
docker pull hiyouga/llamafactory:latest-a3-ubuntu
docker pull hiyouga/llamafactory:latest-910b-openeuler
docker pull hiyouga/llamafactory:latest-a3-openeuler

# quay.io
docker pull quay.io/ascend/llamafactory:latest-910b-ubuntu
docker pull quay.io/ascend/llamafactory:latest-a3-ubuntu
docker pull quay.io/ascend/llamafactory:latest-910b-openeuler
docker pull quay.io/ascend/llamafactory:latest-a3-openeuler

Install BitsAndBytes

To use QLoRA based on bitsandbytes on Ascend NPU, please follow these 3 steps:

1. Manually compile bitsandbytes: Refer to the installation documentation for the NPU version of bitsandbytes to complete the compilation and installation. The compilation requires a cmake version of at least 3.22.1 and a g++ version of at least 12.x.

# Install bitsandbytes from source
# Clone bitsandbytes repo, Ascend NPU backend is currently enabled on multi-backend-refactor branch
git clone -b multi-backend-refactor https://github.com/bitsandbytes-foundation/bitsandbytes.git
cd bitsandbytes/

# Install dependencies
pip install -r requirements-dev.txt

# Install the dependencies for the compilation tools. Note that the commands for this step may vary depending on the operating system. The following are provided for reference
apt-get install -y build-essential cmake

# Compile & install  
cmake -DCOMPUTE_BACKEND=npu -S .
make
pip install .

2. Install transformers from the main branch.

git clone -b main https://github.com/huggingface/transformers.git
cd transformers
pip install .

3. Set double_quantization: false in the configuration. You can refer to the example.

Data Preparation Copy Section

Please refer to data/README.md for checking the details about the format of dataset files. You can use datasets on HuggingFace / ModelScope / Modelers hub, load the dataset in local disk, or specify a path to s3/gcs cloud storage.

[!NOTE]
Please update data/dataset_info.json to use your custom dataset.

You can also use Easy Dataset, DataFlow and GraphGen to create synthetic data for fine-tuning.

Quickstart Copy Section

Use the following 3 commands to run LoRA fine-tuning, inference and merging of the Qwen3-4B-Instruct model, respectively.

llamafactory-cli train examples/train_lora/qwen3_lora_sft.yaml
llamafactory-cli chat examples/inference/qwen3_lora_sft.yaml
llamafactory-cli export examples/merge_lora/qwen3_lora_sft.yaml

See examples/README.md for advanced usage (including distributed training).

[!TIP]
Use llamafactory-cli help to show help information.

Read FAQs first if you encounter any problems.

Fine-Tuning with LLaMA Board GUI (powered by [Gradio](https://github.com/gradio-app/gradio)) Copy Section

llamafactory-cli webui

Build Docker Copy Section

For CUDA users:

cd docker/docker-cuda/
docker compose up -d
docker compose exec llamafactory bash

For Ascend NPU users:

cd docker/docker-npu/

# A2 with Ubuntu
docker compose --profile a2-ubuntu up -d
docker compose --profile a2-ubuntu exec llamafactory-a2-ubuntu bash

# A3 with Ubuntu
docker compose --profile a3-ubuntu up -d
docker compose --profile a3-ubuntu exec llamafactory-a3-ubuntu bash

# A2 with openEuler
docker compose --profile a2-openeuler up -d
docker compose --profile a2-openeuler exec llamafactory-a2-openeuler bash

# A3 with openEuler
docker compose --profile a3-openeuler up -d
docker compose --profile a3-openeuler exec llamafactory-a3-openeuler bash

For AMD ROCm users:

cd docker/docker-rocm/
docker compose up -d
docker compose exec llamafactory bash

Build without Docker Compose

For CUDA users:

docker build -f ./docker/docker-cuda/Dockerfile \
    --build-arg PIP_INDEX=https://pypi.org/simple \
    -t llamafactory:latest .

docker run -dit --ipc=host --gpus=all \
    -p 7860:7860 \
    -p 8000:8000 \
    --name llamafactory \
    llamafactory:latest

docker exec -it llamafactory bash

For Ascend NPU users:

docker build -f ./docker/docker-npu/Dockerfile \
    --build-arg PIP_INDEX=https://pypi.org/simple \
    -t llamafactory:latest .

docker run -dit --ipc=host \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -p 7860:7860 \
    -p 8000:8000 \
    --device /dev/davinci0 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    --name llamafactory \
    llamafactory:latest

docker exec -it llamafactory bash

For AMD ROCm users:

docker build -f ./docker/docker-rocm/Dockerfile \
    --build-arg PIP_INDEX=https://pypi.org/simple \
    -t llamafactory:latest .

docker run -dit --ipc=host \
    -p 7860:7860 \
    -p 8000:8000 \
    --device /dev/kfd \
    --device /dev/dri \
    --name llamafactory \
    llamafactory:latest

docker exec -it llamafactory bash

Use Docker volumes

You can uncomment VOLUME [ "/root/.cache/huggingface", "/app/shared_data", "/app/output" ] in the Dockerfile to use data volumes.

When building the Docker image, use -v ./hf_cache:/root/.cache/huggingface argument to mount the local directory to the container. The following data volumes are available.

• hf_cache: Utilize Hugging Face cache on the host machine.
• shared_data: The directionary to store datasets on the host machine.
• output: Set export dir to this location so that the merged result can be accessed directly on the host machine.

Deploy with OpenAI-style API and vLLM Copy Section

API_PORT=8000 llamafactory-cli api examples/inference/qwen3.yaml infer_backend=vllm vllm_enforce_eager=true

[!TIP]
Visit this page for API document.

Examples: Image understanding | Function calling

Download from ModelScope Hub Copy Section

If you have trouble with downloading models and datasets from Hugging Face, you can use ModelScope.

export USE_MODELSCOPE_HUB=1 # `set USE_MODELSCOPE_HUB=1` for Windows

Train the model by specifying a model ID of the ModelScope Hub as the model_name_or_path. You can find a full list of model IDs at ModelScope Hub, e.g., LLM-Research/Meta-Llama-3-8B-Instruct.

Download from Modelers Hub Copy Section

You can also use Modelers Hub to download models and datasets.

export USE_OPENMIND_HUB=1 # `set USE_OPENMIND_HUB=1` for Windows

Train the model by specifying a model ID of the Modelers Hub as the model_name_or_path. You can find a full list of model IDs at Modelers Hub, e.g., TeleAI/TeleChat-7B-pt.

Use W&B Logger Copy Section

To use Weights & Biases for logging experimental results, you need to add the following arguments to yaml files.

report_to: wandb
run_name: test_run # optional

Set WANDB_API_KEY to your key when launching training tasks to log in with your W&B account.

Use SwanLab Logger Copy Section

To use SwanLab for logging experimental results, you need to add the following arguments to yaml files.

use_swanlab: true
swanlab_run_name: test_run # optional

When launching training tasks, you can log in to SwanLab in three ways:

1. Add swanlab_api_key=<your_api_key> to the yaml file, and set it to your API key.
2. Set the environment variable SWANLAB_API_KEY to your API key.
3. Use the swanlab login command to complete the login.

Projects using LlamaFactory Copy Chapter

If you have a project that should be incorporated, please contact via email or create a pull request.

Click to show

1. Wang et al. ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generation. 2023. [arxiv]
2. Yu et al. Open, Closed, or Small Language Models for Text Classification? 2023. [arxiv]
3. Wang et al. UbiPhysio: Support Daily Functioning, Fitness, and Rehabilitation with Action Understanding and Feedback in Natural Language. 2023. [arxiv]
4. Luceri et al. Leveraging Large Language Models to Detect Influence Campaigns in Social Media. 2023. [arxiv]
5. Zhang et al. Alleviating Hallucinations of Large Language Models through Induced Hallucinations. 2023. [arxiv]
6. Wang et al. Know Your Needs Better: Towards Structured Understanding of Marketer Demands with Analogical Reasoning Augmented LLMs. KDD 2024. [arxiv]
7. Wang et al. CANDLE: Iterative Conceptualization and Instantiation Distillation from Large Language Models for Commonsense Reasoning. ACL 2024. [arxiv]
8. Choi et al. FACT-GPT: Fact-Checking Augmentation via Claim Matching with LLMs. 2024. [arxiv]
9. Zhang et al. AutoMathText: Autonomous Data Selection with Language Models for Mathematical Texts. 2024. [arxiv]
10. Lyu et al. KnowTuning: Knowledge-aware Fine-tuning for Large Language Models. 2024. [arxiv]
11. Yang et al. LaCo: Large Language Model Pruning via Layer Collapse. 2024. [arxiv]
12. Bhardwaj et al. Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic. 2024. [arxiv]
13. Yang et al. Enhancing Empathetic Response Generation by Augmenting LLMs with Small-scale Empathetic Models. 2024. [arxiv]
14. Yi et al. Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding. ACL 2024 Findings. [arxiv]
15. Cao et al. Head-wise Shareable Attention for Large Language Models. 2024. [arxiv]
16. Zhang et al. Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages. 2024. [arxiv]
17. Kim et al. Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models. 2024. [arxiv]
18. Yu et al. KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Models. ACL 2024. [arxiv]
19. Huang et al. Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning. 2024. [arxiv]
20. Duan et al. Negating Negatives: Alignment without Human Positive Samples via Distributional Dispreference Optimization. 2024. [arxiv]
21. Xie and Schwertfeger. Empowering Robotics with Large Language Models: osmAG Map Comprehension with LLMs. 2024. [arxiv]
22. Wu et al. Large Language Models are Parallel Multilingual Learners. 2024. [arxiv]
23. Zhang et al. EDT: Improving Large Language Models' Generation by Entropy-based Dynamic Temperature Sampling. 2024. [arxiv]
24. Weller et al. FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions. 2024. [arxiv]
25. Hongbin Na. CBT-LLM: A Chinese Large Language Model for Cognitive Behavioral Therapy-based Mental Health Question Answering. COLING 2024. [arxiv]
26. Zan et al. CodeS: Natural Language to Code Repository via Multi-Layer Sketch. 2024. [arxiv]
27. Liu et al. Extensive Self-Contrast Enables Feedback-Free Language Model Alignment. 2024. [arxiv]
28. Luo et al. BAdam: A Memory Efficient Full Parameter Training Method for Large Language Models. 2024. [arxiv]
29. Du et al. Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model. 2024. [arxiv]
30. Ma et al. Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation. ICML 2024. [arxiv]
31. Liu et al. Dynamic Generation of Personalities with Large Language Models. 2024. [arxiv]
32. Shang et al. How Far Have We Gone in Stripped Binary Code Understanding Using Large Language Models. 2024. [arxiv]
33. Huang et al. LLMTune: Accelerate Database Knob Tuning with Large Language Models. 2024. [arxiv]
34. Deng et al. Text-Tuple-Table: Towards Information Integration in Text-to-Table Generation via Global Tuple Extraction. 2024. [arxiv]
35. Acikgoz et al. Hippocrates: An Open-Source Framework for Advancing Large Language Models in Healthcare. 2024. [arxiv]
36. Zhang et al. Small Language Models Need Strong Verifiers to Self-Correct Reasoning. ACL 2024 Findings. [arxiv]
37. Zhou et al. FREB-TQA: A Fine-Grained Robustness Evaluation Benchmark for Table Question Answering. NAACL 2024. [arxiv]
38. Xu et al. Large Language Models for Cyber Security: A Systematic Literature Review. 2024. [arxiv]
39. Dammu et al. "They are uncultured": Unveiling Covert Harms and Social Threats in LLM Generated Conversations. 2024. [arxiv]
40. Yi et al. A safety realignment framework via subspace-oriented model fusion for large language models. 2024. [arxiv]
41. Lou et al. SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling. 2024. [arxiv]
42. Zhang et al. Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners. 2024. [arxiv]
43. Zhang et al. TS-Align: A Teacher-Student Collaborative Framework for Scalable Iterative Finetuning of Large Language Models. 2024. [arxiv]
44. Zihong Chen. Sentence Segmentation and Sentence Punctuation Based on XunziALLM. 2024. [paper]
45. Gao et al. The Best of Both Worlds: Toward an Honest and Helpful Large Language Model. 2024. [arxiv]
46. Wang and Song. MARS: Benchmarking the Metaphysical Reasoning Abilities of Language Models with a Multi-task Evaluation Dataset. 2024. [arxiv]
47. Hu et al. Computational Limits of Low-Rank Adaptation (LoRA) for Transformer-Based Models. 2024. [arxiv]
48. Ge et al. Time Sensitive Knowledge Editing through Efficient Finetuning. ACL 2024. [arxiv]
49. Tan et al. Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions. 2024. [arxiv]
50. Song et al. Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters. 2024. [arxiv]
51. Gu et al. RWKV-CLIP: A Robust Vision-Language Representation Learner. 2024. [arxiv]
52. Chen et al. Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees. 2024. [arxiv]
53. Zhu et al. Are Large Language Models Good Statisticians?. 2024. [arxiv]
54. Li et al. Know the Unknown: An Uncertainty-Sensitive Method for LLM Instruction Tuning. 2024. [arxiv]
55. Ding et al. IntentionQA: A Benchmark for Evaluating Purchase Intention Comprehension Abilities of Language Models in E-commerce. 2024. [arxiv]
56. He et al. COMMUNITY-CROSS-INSTRUCT: Unsupervised Instruction Generation for Aligning Large Language Models to Online Communities. 2024. [arxiv]
57. Lin et al. FVEL: Interactive Formal Verification Environment with Large Language Models via Theorem Proving. 2024. [arxiv]
58. Treutlein et al. Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data. 2024. [arxiv]
59. Feng et al. SS-Bench: A Benchmark for Social Story Generation and Evaluation. 2024. [arxiv]
60. Feng et al. Self-Constructed Context Decompilation with Fined-grained Alignment Enhancement. 2024. [arxiv]
61. Liu et al. Large Language Models for Cuffless Blood Pressure Measurement From Wearable Biosignals. 2024. [arxiv]
62. Iyer et al. Exploring Very Low-Resource Translation with LLMs: The University of Edinburgh's Submission to AmericasNLP 2024 Translation Task. AmericasNLP 2024. [paper]
63. Li et al. Calibrating LLMs with Preference Optimization on Thought Trees for Generating Rationale in Science Question Scoring. 2024. [arxiv]
64. Yang et al. Financial Knowledge Large Language Model. 2024. [arxiv]
65. Lin et al. DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging. 2024. [arxiv]
66. Bako et al. Evaluating the Semantic Profiling Abilities of LLMs for Natural Language Utterances in Data Visualization. 2024. [arxiv]
67. Huang et al. RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization. 2024. [arxiv]
68. Jiang et al. LLM-Collaboration on Automatic Science Journalism for the General Audience. 2024. [arxiv]
69. Inouye et al. Applied Auto-tuning on LoRA Hyperparameters. 2024. [paper]
70. Qi et al. Research on Tibetan Tourism Viewpoints information generation system based on LLM. 2024. [arxiv]
71. Xu et al. Course-Correction: Safety Alignment Using Synthetic Preferences. 2024. [arxiv]
72. Sun et al. LAMBDA: A Large Model Based Data Agent. 2024. [arxiv]
73. Zhu et al. CollectiveSFT: Scaling Large Language Models for Chinese Medical Benchmark with Collective Instructions in Healthcare. 2024. [arxiv]
74. Yu et al. Correcting Negative Bias in Large Language Models through Negative Attention Score Alignment. 2024. [arxiv]
75. Xie et al. The Power of Personalized Datasets: Advancing Chinese Composition Writing for Elementary School through Targeted Model Fine-Tuning. IALP 2024. [paper]
76. Liu et al. Instruct-Code-Llama: Improving Capabilities of Language Model in Competition Level Code Generation by Online Judge Feedback. ICIC 2024. [paper]
77. Wang et al. Cybernetic Sentinels: Unveiling the Impact of Safety Data Selection on Model Security in Supervised Fine-Tuning. ICIC 2024. [paper]
78. Xia et al. Understanding the Performance and Estimating the Cost of LLM Fine-Tuning. 2024. [arxiv]
79. Zeng et al. Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructions. 2024. [arxiv]
80. Xia et al. Using Pre-trained Language Model for Accurate ESG Prediction. FinNLP 2024. [paper]
81. Liang et al. I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigm. 2024. [arxiv]
82. Bai et al. Aligning Large Language Model with Direct Multi-Preference Optimization for Recommendation. CIKM 2024. [paper]
83. Zhang et al. CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework for Chinese Psychological Counseling. ACL 2024. [paper]
84. StarWhisper: A large language model for Astronomy, based on ChatGLM2-6B and Qwen-14B.
85. DISC-LawLLM: A large language model specialized in Chinese legal domain, based on Baichuan-13B, is capable of retrieving and reasoning on legal knowledge.
86. Sunsimiao: A large language model specialized in Chinese medical domain, based on Baichuan-7B and ChatGLM-6B.
87. CareGPT: A series of large language models for Chinese medical domain, based on LLaMA2-7B and Baichuan-13B.
88. MachineMindset: A series of MBTI Personality large language models, capable of giving any LLM 16 different personality types based on different datasets and training methods.
89. Luminia-13B-v3: A large language model specialized in generate metadata for stable diffusion. [demo]
90. Chinese-LLaVA-Med: A multimodal large language model specialized in Chinese medical domain, based on LLaVA-1.5-7B.
91. AutoRE: A document-level relation extraction system based on large language models.
92. NVIDIA RTX AI Toolkit: SDKs for fine-tuning LLMs on Windows PC for NVIDIA RTX.
93. LazyLLM: An easy and lazy way for building multi-agent LLMs applications and supports model fine-tuning via LlamaFactory.
94. RAG-Retrieval: A full pipeline for RAG retrieval model fine-tuning, inference, and distillation. [blog]
95. 360-LLaMA-Factory: A modified library that supports long sequence SFT & DPO using ring attention.
96. Sky-T1: An o1-like model fine-tuned by NovaSky AI with very small cost.
97. WeClone: One-stop solution for creating your digital avatar from chat logs.
98. EmoLLM: A project about large language models (LLMs) and mental health.

License Copy Chapter

This repository is licensed under the Apache-2.0 License.

Please follow the model licenses to use the corresponding model weights: BLOOM / DeepSeek / Falcon / Gemma / GLM-4 / GPT-2 / Granite / InternLM / Llama / Llama 2 / Llama 3 / Llama 4 / MiniCPM / Mistral/Mixtral/Pixtral / Phi-3/Phi-4 / Qwen / StarCoder 2 / TeleChat2 / Yuan 2

Citation Copy Chapter

If this work is helpful, please kindly cite as:

@inproceedings{zheng2024llamafactory,
  title={LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models},
  author={Yaowei Zheng and Richong Zhang and Junhao Zhang and Yanhan Ye and Zheyan Luo and Zhangchi Feng and Yongqiang Ma},
  booktitle={Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  address={Bangkok, Thailand},
  publisher={Association for Computational Linguistics},
  year={2024},
  url={http://arxiv.org/abs/2403.13372}
}

Acknowledgement Copy Chapter

This repo benefits from PEFT, TRL, QLoRA and FastChat. Thanks for their wonderful works.

File: data/README.md Copy Chapter

The dataset_info.json contains all available datasets. If you are using a custom dataset, please make sure to add a dataset description in dataset_info.json and specify dataset: dataset_name before training to use it.

The dataset_info.json file should be put in the dataset_dir directory. You can change dataset_dir to use another directory. The default value is ./data.

Currently we support datasets in alpaca and sharegpt format. Allowed file types include json, jsonl, csv, parquet, arrow.

/* Detailed source-code truncated for AI context efficiency. */

Alpaca Format Copy Chapter

Supervised Fine-Tuning Dataset Copy Section

• Example dataset

In supervised fine-tuning, the instruction column will be concatenated with the input column and used as the user prompt, then the user prompt would be instruction\ninput. The output column represents the model response.

For reasoning models, if the dataset contains chain-of-thought (CoT), the CoT needs to be placed in the model responses, such as cotoutput.

The system column will be used as the system prompt if specified.

The history column is a list consisting of string tuples representing prompt-response pairs in the history messages. Note that the responses in the history will also be learned by the model in supervised fine-tuning.

[
  {
    "instruction": "user instruction (required)",
    "input": "user input (optional)",
    "output": "model response (required)",
    "system": "system prompt (optional)",
    "history": [
      ["user instruction in the first round (optional)", "model response in the first round (optional)"],
      ["user instruction in the second round (optional)", "model response in the second round (optional)"]
    ]
  }
]

Regarding the above dataset, the dataset description in dataset_info.json should be:

"dataset_name": {
  "file_name": "data.json",
  "columns": {
    "prompt": "instruction",
    "query": "input",
    "response": "output",
    "system": "system",
    "history": "history"
  }
}

[!TIP]
If the model has reasoning capabilities (e.g. Qwen3) but the dataset does not contain chain-of-thought (CoT), LLaMA-Factory will automatically add empty CoT to the data. When enable_thinking is True (slow thinking, by default), the empty CoT will be added to the model responses and loss computation will be considered; otherwise (fast thinking), it will be added to the user prompts and loss computation will be ignored. Please keep the enable_thinking parameter consistent during training and inference.

If you want to train data containing CoT with slow thinking and data without CoT with fast thinking, you can set enable_thinking to None. However, this feature is relatively complicated and should be used with caution.

Pre-training Dataset Copy Section

• Example dataset

In pre-training, only the text column will be used for model learning.

[
  {"text": "document"},
  {"text": "document"}
]

Regarding the above dataset, the dataset description in dataset_info.json should be:

"dataset_name": {
  "file_name": "data.json",
  "columns": {
    "prompt": "text"
  }
}

Preference Dataset Copy Section

Preference datasets are used for reward modeling, DPO training, ORPO and SimPO training.

It requires a better response in chosen column and a worse response in rejected column.

[
  {
    "instruction": "user instruction (required)",
    "input": "user input (optional)",
    "chosen": "chosen answer (required)",
    "rejected": "rejected answer (required)"
  }
]

Regarding the above dataset, the dataset description in dataset_info.json should be:

"dataset_name": {
  "file_name": "data.json",
  "ranking": true,
  "columns": {
    "prompt": "instruction",
    "query": "input",
    "chosen": "chosen",
    "rejected": "rejected"
  }
}

KTO Dataset Copy Section

An additional column kto_tag is required. Please refer to the sharegpt format for details.

Multimodal Image Dataset Copy Section

An additional column images is required. Please refer to the sharegpt format for details.

Multimodal Video Dataset Copy Section

An additional column videos is required. Please refer to the sharegpt format for details.

Multimodal Audio Dataset Copy Section

An additional column audios is required. Please refer to the sharegpt format for details.

Sharegpt Format Copy Chapter

Supervised Fine-Tuning Dataset Copy Section

• Example dataset

Compared to the alpaca format, the sharegpt format allows the datasets have more roles, such as human, gpt, observation and function. They are presented in a list of objects in the conversations column.

Note that the human and observation should appear in odd positions, while gpt and function should appear in even positions. The gpt and function will be learned by the model.

[
  {
    "conversations": [
      {
        "from": "human",
        "value": "user instruction"
      },
      {
        "from": "function_call",
        "value": "tool arguments"
      },
      {
        "from": "observation",
        "value": "tool result"
      },
      {
        "from": "gpt",
        "value": "model response"
      }
    ],
    "system": "system prompt (optional)",
    "tools": "tool description (optional)"
  }
]

Regarding the above dataset, the dataset description in dataset_info.json should be:

"dataset_name": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "conversations",
    "system": "system",
    "tools": "tools"
  }
}

Pre-training Dataset Copy Section

Not yet supported, please use the alpaca format.

Preference Dataset Copy Section

• Example dataset

Preference datasets in sharegpt format also require a better message in chosen column and a worse message in rejected column.

[
  {
    "conversations": [
      {
        "from": "human",
        "value": "user instruction"
      },
      {
        "from": "gpt",
        "value": "model response"
      },
      {
        "from": "human",
        "value": "user instruction"
      }
    ],
    "chosen": {
      "from": "gpt",
      "value": "chosen answer (required)"
    },
    "rejected": {
      "from": "gpt",
      "value": "rejected answer (required)"
    }
  }
]

Regarding the above dataset, the dataset description in dataset_info.json should be:

"dataset_name": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "ranking": true,
  "columns": {
    "messages": "conversations",
    "chosen": "chosen",
    "rejected": "rejected"
  }
}

KTO Dataset Copy Section

• Example dataset

KTO datasets require a extra kto_tag column containing the boolean human feedback.

[
  {
    "conversations": [
      {
        "from": "human",
        "value": "user instruction"
      },
      {
        "from": "gpt",
        "value": "model response"
      }
    ],
    "kto_tag": "human feedback [true/false] (required)"
  }
]

Regarding the above dataset, the dataset description in dataset_info.json should be:

"dataset_name": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "conversations",
    "kto_tag": "kto_tag"
  }
}

Multimodal Image Dataset Copy Section

• Example dataset

Multimodal image datasets require an images column containing the paths to the input images.

The number of images should be identical to the `` tokens in the conversations.

[
  {
    "conversations": [
      {
        "from": "human",
        "value": "user instruction"
      },
      {
        "from": "gpt",
        "value": "model response"
      }
    ],
    "images": [
      "image path (required)"
    ]
  }
]

Regarding the above dataset, the dataset description in dataset_info.json should be:

"dataset_name": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "conversations",
    "images": "images"
  }
}

Multimodal Video Dataset Copy Section

• Example dataset

Multimodal video datasets require a videos column containing the paths to the input videos.

The number of videos should be identical to the <video> tokens in the conversations.

[
  {
    "conversations": [
      {
        "from": "human",
        "value": "<video>user instruction"
      },
      {
        "from": "gpt",
        "value": "model response"
      }
    ],
    "videos": [
      "video path (required)"
    ]
  }
]

Regarding the above dataset, the dataset description in dataset_info.json should be:

"dataset_name": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "conversations",
    "videos": "videos"
  }
}

Multimodal Audio Dataset Copy Section

• Example dataset

Multimodal audio datasets require an audios column containing the paths to the input audios.

The number of audios should be identical to the <audio> tokens in the conversations.

[
  {
    "conversations": [
      {
        "from": "human",
        "value": "<audio>user instruction"
      },
      {
        "from": "gpt",
        "value": "model response"
      }
    ],
    "audios": [
      "audio path (required)"
    ]
  }
]

Regarding the above dataset, the dataset description in dataset_info.json should be:

"dataset_name": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "conversations",
    "audios": "audios"
  }
}

OpenAI Format Copy Section

The openai format is simply a special case of the sharegpt format, where the first message may be a system prompt.

[
  {
    "messages": [
      {
        "role": "system",
        "content": "system prompt (optional)"
      },
      {
        "role": "user",
        "content": "user instruction"
      },
      {
        "role": "assistant",
        "content": "model response"
      }
    ]
  }
]

Regarding the above dataset, the dataset description in dataset_info.json should be:

"dataset_name": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "messages"
  },
  "tags": {
    "role_tag": "role",
    "content_tag": "content",
    "user_tag": "user",
    "assistant_tag": "assistant",
    "system_tag": "system"
  }
}

File: data/README_zh.md Copy Chapter

dataset_info.json 包含了所有可用的数据集。如果您希望使用自定义数据集，请务必在 dataset_info.json 文件中添加数据集描述，并通过修改 dataset: 数据集名称 配置来使用数据集。

其中 dataset_info.json 文件应放置在 dataset_dir 目录下。您可以通过修改 dataset_dir 参数来使用其他目录。默认值为 ./data。

目前我们支持 alpaca 格式和 sharegpt 格式的数据集。允许的文件类型包括 json、jsonl、csv、parquet 和 arrow。

"数据集名称": {
  "hf_hub_url": "Hugging Face 的数据集仓库地址（若指定，则忽略 script_url 和 file_name）",
  "ms_hub_url": "ModelScope 的数据集仓库地址（若指定，则忽略 script_url 和 file_name）",
  "script_url": "包含数据加载脚本的本地文件夹名称（若指定，则忽略 file_name）",
  "file_name": "该目录下数据集文件夹或文件的名称（若上述参数未指定，则此项必需）",
  "formatting": "数据集格式（可选，默认：alpaca，可以为 alpaca 或 sharegpt）",
  "ranking": "是否为偏好数据集（可选，默认：False）",
  "subset": "数据集子集的名称（可选，默认：None）",
  "split": "所使用的数据集切分（可选，默认：train）",
  "folder": "Hugging Face 仓库的文件夹名称（可选，默认：None）",
  "num_samples": "该数据集所使用的样本数量。（可选，默认：None）",
  "columns（可选）": {
    "prompt": "数据集代表提示词的表头名称（默认：instruction）",
    "query": "数据集代表请求的表头名称（默认：input）",
    "response": "数据集代表回答的表头名称（默认：output）",
    "history": "数据集代表历史对话的表头名称（默认：None）",
    "messages": "数据集代表消息列表的表头名称（默认：conversations）",
    "system": "数据集代表系统提示的表头名称（默认：None）",
    "tools": "数据集代表工具描述的表头名称（默认：None）",
    "images": "数据集代表图像输入的表头名称（默认：None）",
    "videos": "数据集代表视频输入的表头名称（默认：None）",
    "audios": "数据集代表音频输入的表头名称（默认：None）",
    "chosen": "数据集代表更优回答的表头名称（默认：None）",
    "rejected": "数据集代表更差回答的表头名称（默认：None）",
    "kto_tag": "数据集代表 KTO 标签的表头名称（默认：None）"
  },
  "tags（可选，用于 sharegpt 格式）": {
    "role_tag": "消息中代表发送者身份的键名（默认：from）",
    "content_tag": "消息中代表文本内容的键名（默认：value）",
    "user_tag": "消息中代表用户的 role_tag（默认：human）",
    "assistant_tag": "消息中代表助手的 role_tag（默认：gpt）",
    "observation_tag": "消息中代表工具返回结果的 role_tag（默认：observation）",
    "function_tag": "消息中代表工具调用的 role_tag（默认：function_call）",
    "system_tag": "消息中代表系统提示的 role_tag（默认：system，会覆盖 system column）"
  }
}

Alpaca 格式 Copy Chapter

指令监督微调数据集 Copy Section

• 样例数据集

在指令监督微调时，instruction 列对应的内容会与 input 列对应的内容拼接后作为提示词，即提示词为 instruction\ninput。而 output 列对应的内容为模型回答。

对于推理类模型的微调，如果数据集包含思维链，则需要把思维链放在模型回答中，例如 cotoutput。

如果指定，system 列对应的内容将被作为系统提示词。

history 列是由多个字符串二元组构成的列表，分别代表历史消息中每轮对话的指令和回答。注意在指令监督微调时，历史消息中的回答内容也会被用于模型学习。

[
  {
    "instruction": "用户指令（必填）",
    "input": "用户输入（选填）",
    "output": "模型回答（必填）",
    "system": "系统提示词（选填）",
    "history": [
      ["第一轮指令（选填）", "第一轮回答（选填）"],
      ["第二轮指令（选填）", "第二轮回答（选填）"]
    ]
  }
]

对于上述格式的数据，dataset_info.json 中的数据集描述应为：

"数据集名称": {
  "file_name": "data.json",
  "columns": {
    "prompt": "instruction",
    "query": "input",
    "response": "output",
    "system": "system",
    "history": "history"
  }
}

[!TIP]
如果模型本身具备推理能力（如 Qwen3）而数据集不包含思维链，LLaMA-Factory 会自动为数据添加空思维链。当 enable_thinking 为 True 时（慢思考，默认），空思维链会添加到模型回答中并且计算损失，否则会添加到用户指令中并且不计算损失（快思考）。请在训练和推理时保持 enable_thinking 参数一致。

如果您希望训练包含思维链的数据时使用慢思考，训练不包含思维链的数据时使用快思考，可以设置 enable_thinking 为 None。但该功能较为复杂，请谨慎使用。

预训练数据集 Copy Section

• 样例数据集

在预训练时，只有 text 列中的内容会用于模型学习。

[
  {"text": "document"},
  {"text": "document"}
]

对于上述格式的数据，dataset_info.json 中的数据集描述应为：

"数据集名称": {
  "file_name": "data.json",
  "columns": {
    "prompt": "text"
  }
}

偏好数据集 Copy Section

偏好数据集用于奖励模型训练、DPO 训练、ORPO 训练和 SimPO 训练。

它需要在 chosen 列中提供更优的回答，并在 rejected 列中提供更差的回答。

[
  {
    "instruction": "用户指令（必填）",
    "input": "用户输入（选填）",
    "chosen": "优质回答（必填）",
    "rejected": "劣质回答（必填）"
  }
]

对于上述格式的数据，dataset_info.json 中的数据集描述应为：

"数据集名称": {
  "file_name": "data.json",
  "ranking": true,
  "columns": {
    "prompt": "instruction",
    "query": "input",
    "chosen": "chosen",
    "rejected": "rejected"
  }
}

KTO 数据集 Copy Section

KTO 数据集需要提供额外的 kto_tag 列。详情请参阅 sharegpt。

多模态图像数据集 Copy Section

多模态图像数据集需要提供额外的 images 列。详情请参阅 sharegpt。

多模态视频数据集 Copy Section

多模态视频数据集需要提供额外的 videos 列。详情请参阅 sharegpt。

多模态音频数据集 Copy Section

多模态音频数据集需要提供额外的 audios 列。详情请参阅 sharegpt。

Sharegpt 格式 Copy Chapter

指令监督微调数据集 Copy Section

• 样例数据集

相比 alpaca 格式的数据集，sharegpt 格式支持更多的角色种类，例如 human、gpt、observation、function 等等。它们构成一个对象列表呈现在 conversations 列中。

注意其中 human 和 observation 必须出现在奇数位置，gpt 和 function 必须出现在偶数位置。默认所有的 gpt 和 function 会被用于学习。

[
  {
    "conversations": [
      {
        "from": "human",
        "value": "用户指令"
      },
      {
        "from": "function_call",
        "value": "工具参数"
      },
      {
        "from": "observation",
        "value": "工具结果"
      },
      {
        "from": "gpt",
        "value": "模型回答"
      }
    ],
    "system": "系统提示词（选填）",
    "tools": "工具描述（选填）"
  }
]

对于上述格式的数据，dataset_info.json 中的数据集描述应为：

"数据集名称": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "conversations",
    "system": "system",
    "tools": "tools"
  }
}

预训练数据集 Copy Section

尚不支持，请使用 alpaca 格式。

偏好数据集 Copy Section

• 样例数据集

Sharegpt 格式的偏好数据集同样需要在 chosen 列中提供更优的消息，并在 rejected 列中提供更差的消息。

[
  {
    "conversations": [
      {
        "from": "human",
        "value": "用户指令"
      },
      {
        "from": "gpt",
        "value": "模型回答"
      },
      {
        "from": "human",
        "value": "用户指令"
      }
    ],
    "chosen": {
      "from": "gpt",
      "value": "优质回答"
    },
    "rejected": {
      "from": "gpt",
      "value": "劣质回答"
    }
  }
]

对于上述格式的数据，dataset_info.json 中的数据集描述应为：

"数据集名称": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "ranking": true,
  "columns": {
    "messages": "conversations",
    "chosen": "chosen",
    "rejected": "rejected"
  }
}

KTO 数据集 Copy Section

• 样例数据集

KTO 数据集需要额外添加一个 kto_tag 列，包含 bool 类型的人类反馈。

[
  {
    "conversations": [
      {
        "from": "human",
        "value": "用户指令"
      },
      {
        "from": "gpt",
        "value": "模型回答"
      }
    ],
    "kto_tag": "人类反馈 [true/false]（必填）"
  }
]

对于上述格式的数据，dataset_info.json 中的数据集描述应为：

"数据集名称": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "conversations",
    "kto_tag": "kto_tag"
  }
}

多模态图像数据集 Copy Section

• 样例数据集

多模态图像数据集需要额外添加一个 images 列，包含输入图像的路径。

注意图片的数量必须与文本中所有 `` 标记的数量严格一致。

[
  {
    "conversations": [
      {
        "from": "human",
        "value": "用户指令"
      },
      {
        "from": "gpt",
        "value": "模型回答"
      }
    ],
    "images": [
      "图像路径（必填）",
      "图像路径（必填）"
    ]
  }
]

对于上述格式的数据，dataset_info.json 中的数据集描述应为：

"数据集名称": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "conversations",
    "images": "images"
  }
}

多模态视频数据集 Copy Section

• 样例数据集

多模态视频数据集需要额外添加一个 videos 列，包含输入视频的路径。

注意视频的数量必须与文本中所有 <video> 标记的数量严格一致。

[
  {
    "conversations": [
      {
        "from": "human",
        "value": "<video><video>用户指令"
      },
      {
        "from": "gpt",
        "value": "模型回答"
      }
    ],
    "videos": [
      "视频路径（必填）",
      "视频路径（必填）"
    ]
  }
]

对于上述格式的数据，dataset_info.json 中的数据集描述应为：

"数据集名称": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "conversations",
    "videos": "videos"
  }
}

多模态音频数据集 Copy Section

• 样例数据集

多模态音频数据集需要额外添加一个 audios 列，包含输入音频的路径。

注意音频的数量必须与文本中所有 <audio> 标记的数量严格一致。

[
  {
    "conversations": [
      {
        "from": "human",
        "value": "<audio><audio>用户指令"
      },
      {
        "from": "gpt",
        "value": "模型回答"
      }
    ],
    "audios": [
      "音频路径（必填）",
      "音频路径（必填）"
    ]
  }
]

对于上述格式的数据，dataset_info.json 中的数据集描述应为：

"数据集名称": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "conversations",
    "audios": "audios"
  }
}

OpenAI 格式 Copy Section

OpenAI 格式仅仅是 sharegpt 格式的一种特殊情况，其中第一条消息可能是系统提示词。

[
  {
    "messages": [
      {
        "role": "system",
        "content": "系统提示词（选填）"
      },
      {
        "role": "user",
        "content": "用户指令"
      },
      {
        "role": "assistant",
        "content": "模型回答"
      }
    ]
  }
]

对于上述格式的数据，dataset_info.json 中的数据集描述应为：

"数据集名称": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "messages"
  },
  "tags": {
    "role_tag": "role",
    "content_tag": "content",
    "user_tag": "user",
    "assistant_tag": "assistant",
    "system_tag": "system"
  }
}

File: docker/docker-cuda/README.md Copy Chapter

Docker Setup for NVIDIA GPUs

This directory contains Docker configuration files for running LLaMA Factory with NVIDIA GPU support.

Prerequisites Copy Chapter

Linux-specific Requirements Copy Section

Before running the Docker container with GPU support, you need to install the following packages:

1. Docker: The container runtime

   bash Copy

   # Ubuntu/Debian
   sudo apt-get update
   sudo apt-get install docker.io
   
   # Or install Docker Engine from the official repository:
   # https://docs.docker.com/engine/install/

2. Docker Compose (if using the docker-compose method):

   bash Copy

   # Ubuntu/Debian
   sudo apt-get install docker-compose
   
   # Or install the latest version:
   # https://docs.docker.com/compose/install/

3. NVIDIA Container Toolkit (required for GPU support):

   bash Copy

   # Add the NVIDIA GPG key and repository
   distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
   curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
   curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
   
   # Install nvidia-container-toolkit
   sudo apt-get update
   sudo apt-get install -y nvidia-container-toolkit
   
   # Restart Docker to apply changes
   sudo systemctl restart docker

   Note: Without nvidia-container-toolkit, the Docker container will not be able to access your NVIDIA GPU.

Verify GPU Access Copy Section

After installation, verify that Docker can access your GPU:

sudo docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

If successful, you should see your GPU information displayed.

Usage Copy Chapter

Using Docker Compose (Recommended) Copy Section

cd docker/docker-cuda/
docker compose up -d
docker compose exec llamafactory bash

Using Docker Run Copy Section

# Build the image
docker build -f ./docker/docker-cuda/Dockerfile \
    --build-arg PIP_INDEX=https://pypi.org/simple \
    --build-arg EXTRAS=metrics \
    -t llamafactory:latest .

# Run the container
docker run -dit --ipc=host --gpus=all \
    -p 7860:7860 \
    -p 8000:8000 \
    --name llamafactory \
    llamafactory:latest

# Enter the container
docker exec -it llamafactory bash

Troubleshooting Copy Chapter

GPU Not Detected Copy Section

If your GPU is not detected inside the container:

1. Ensure nvidia-container-toolkit is installed
2. Check that the Docker daemon has been restarted after installation
3. Verify your NVIDIA drivers are properly installed: nvidia-smi
4. Check Docker GPU support: docker run --rm --gpus all ubuntu nvidia-smi

Permission Denied Copy Section

If you get permission errors, ensure your user is in the docker group:

sudo usermod -aG docker $USER
# Log out and back in for changes to take effect

Megatron Bridge Image Copy Chapter

Dockerfile.megatron builds a CUDA runtime for LLaMA-Factory + Megatron Bridge:

Build Copy Section

From repo root:

docker build -f docker/docker-cuda/Dockerfile.megatron \
  -t llamafactory-megatron-bridge:latest .

Run training Copy Section

docker run --rm -it --gpus all --ipc=host --shm-size=16g \
  -e DISABLE_VERSION_CHECK=1 \
  -e USE_MEGATRON_BRIDGE=1 \
  -v "$PWD":/app -w /app \
  llamafactory-megatron-bridge:latest

Additional Notes Copy Chapter

• The default image is built on Ubuntu 22.04 (x86_64), CUDA 12.4, Python 3.11, PyTorch 2.6.0, and Flash-attn 2.7.4
• For different CUDA versions, you may need to adjust the base image in the Dockerfile
• Make sure your NVIDIA driver version is compatible with the CUDA version used in the Docker image

File: docs/en/advanced/lora-and-quantization/lora.md Copy Chapter

LoRA

This page is not yet available in English. Use the language switcher to view Simplified Chinese.

File: docs/en/advanced/lora-and-quantization/quantization.md Copy Chapter

Quantization

This page is not yet available in English. Use the language switcher to view Simplified Chinese.

File: docs/en/advanced/distributed/deepspeed.md Copy Chapter

DeepSpeed

This page is not yet available in English. Use the language switcher to view Simplified Chinese.

File: docs/en/advanced/distributed/fsdp.md Copy Chapter

FSDP

This page is not yet available in English. Use the language switcher to view Simplified Chinese.

File: docs/en/advanced/distributed/parallel-dp-tp-ep-sp-cp.md Copy Chapter

Parallel (DP, TP, EP, SP, CP)

This page is not yet available in English. Use the language switcher to view Simplified Chinese.

File: docs/en/advanced/custom-kernels/custom-kernels.md Copy Chapter

Custom Kernels

This page is not yet available in English. Use the language switcher to view Simplified Chinese.