# Technical Documentation: axolotl-ai-cloud/axolotl > ℹ️ **Provenance:** Hybrid Fusion: `axolotl-ai-cloud/axolotl` (README + 4 In-Tree Chapters) Β· [CodeWiki Reference](https://codewiki.google/github.com/axolotl-ai-cloud/axolotl) Β· Recency: Active (< 180 days) ## 1. Project Overview & Quickstart (axolotl-ai-cloud/axolotl) ## πŸŽ‰ Latest Updates - 2026/08: - New model support has been added in Axolotl for [Muse Glimmer](https://docs.axolotl.ai/docs/models/muse-glimmer.html), [North Micro Vision Instruct](https://docs.axolotl.ai/docs/models/cohere-north-micro-vision-instruct.html) and [Shieldstral](https://docs.axolotl.ai/docs/models/shieldstral.html). - 2026/07: - [NVFP4 (4-bit) MoE LoRA training](https://docs.axolotl.ai/docs/nvfp4_lora.html) is now supported via ScatterMoE (W4A16) and SonicMoE (W4A4), including adapter merge back into a plain NVFP4 checkpoint. - 2026/06: - [Expert Parallelism (EP)](https://docs.axolotl.ai/docs/nd_parallelism.html) for distributed MoE training via DeepEP, remote training through [Tinker-compatible APIs](https://github.com/axolotl-ai-cloud/axolotl/pull/3614), [Context Parallelism for hybrid SSM models](https://github.com/axolotl-ai-cloud/axolotl/pull/3572) (Nemotron-H, Falcon-H1, Bamba), [BitNet 1.58-bit](https://github.com/axolotl-ai-cloud/axolotl/pull/3634) fine-tuning, and a [multimodal assistant-only loss-masking fix](https://github.com/axolotl-ai-cloud/axolotl/pull/3625). - 2026/04: - New model support has been added in Axolotl for [Mistral Medium 3.5](https://docs.axolotl.ai/docs/models/mistral-medium-3_5.html) and [Gemma 4](https://docs.axolotl.ai/docs/models/gemma4.html). - New RL and kernels: [Async GRPO](https://github.com/axolotl-ai-cloud/axolotl/pull/3486) (up to 58% faster steps), [Flash Attention 4](https://docs.axolotl.ai/docs/attention.html#flash-attention), [NeMo Gym](https://github.com/axolotl-ai-cloud/axolotl/pull/3516), and [EBFT](https://github.com/axolotl-ai-cloud/axolotl/pull/3527). - Axolotl is now [uv-first](https://github.com/axolotl-ai-cloud/axolotl/pull/3545) and has [SonicMoE fused LoRA](https://github.com/axolotl-ai-cloud/axolotl/pull/3519) support. - 2026/03: - New model support has been added in Axolotl for [Mistral Small 4](https://docs.axolotl.ai/docs/models/mistral4.html), [Qwen3.5, Qwen3.5 MoE](https://docs.axolotl.ai/docs/models/qwen3.5.html), [GLM-4.7-Flash](https://docs.axolotl.ai/docs/models/glm47-flash.html), [GLM-4.6V](https://docs.axolotl.ai/docs/models/glm46v.html), and [GLM-4.5-Air](https://docs.axolotl.ai/docs/models/glm45.html). - [MoE expert quantization](https://docs.axolotl.ai/docs/expert_quantization.html) support (via `quantize_moe_experts: true`) greatly reduces VRAM when training MoE models (FSDP2 compat). Expand older updates - 2026/02: - [ScatterMoE LoRA](https://github.com/axolotl-ai-cloud/axolotl/pull/3410) support. LoRA fine-tuning directly on MoE expert weights using custom Triton kernels. - Axolotl now has support for [SageAttention](https://github.com/axolotl-ai-cloud/axolotl/pull/2823) and [GDPO](https://github.com/axolotl-ai-cloud/axolotl/pull/3353) (Generalized DPO). - 2026/01: - New integration for [EAFT](https://github.com/axolotl-ai-cloud/axolotl/pull/3366) (Entropy-Aware Focal Training), weights loss by entropy of the top-k logit distribution, and [Scalable Softmax](https://github.com/axolotl-ai-cloud/axolotl/pull/3338), improves long context in attention. - 2025/12: - Axolotl now includes support for [Kimi-Linear](https://docs.axolotl.ai/docs/models/kimi-linear.html), [Plano-Orchestrator](https://docs.axolotl.ai/docs/models/plano.html), [MiMo](https://docs.axolotl.ai/docs/models/mimo.html), [InternVL 3.5](https://docs.axolotl.ai/docs/models/internvl3_5.html), [Olmo3](https://docs.axolotl.ai/docs/models/olmo3.html), [Trinity](https://docs.axolotl.ai/docs/models/trinity.html), and [Ministral3](https://docs.axolotl.ai/docs/models/ministral3.html). - [Distributed Muon Optimizer](https://github.com/axolotl-ai-cloud/axolotl/pull/3264) support has been added for FSDP2 pretraining. - 2025/10: New model support has been added in Axolotl for: [Qwen3 Next](https://docs.axolotl.ai/docs/models/qwen3-next.html), [Qwen2.5-vl, Qwen3-vl](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/qwen2_5-vl), [Qwen3, Qwen3MoE](https://docs.axolotl.ai/docs/models/qwen3.html), [Granite 4](https://docs.axolotl.ai/docs/models/granite4.html), [HunYuan](https://docs.axolotl.ai/docs/models/hunyuan.html), [Magistral 2509](https://docs.axolotl.ai/docs/models/magistral/vision.html), [Apertus](https://docs.axolotl.ai/docs/models/apertus.html), and [Seed-OSS](https://docs.axolotl.ai/docs/models/seed-oss.html). - 2025/09: Axolotl now has text diffusion training. Read more [here](https://github.com/axolotl-ai-cloud/axolotl/tree/main/src/axolotl/integrations/diffusion). - 2025/08: QAT has been updated to include NVFP4 support. See [PR](https://github.com/axolotl-ai-cloud/axolotl/pull/3107). - 2025/07: - ND Parallelism support has been added into Axolotl. Compose Context Parallelism (CP), Tensor Parallelism (TP), and Fully Sharded Data Parallelism (FSDP) within a single node and across multiple nodes. Check out the [blog post](https://huggingface.co/blog/accelerate-nd-parallel) for more info. - Axolotl adds more models: [GPT-OSS](https://docs.axolotl.ai/docs/models/gpt-oss.html), [Gemma 3n](https://docs.axolotl.ai/docs/models/gemma3n.html), [Liquid Foundation Model 2 (LFM2)](https://docs.axolotl.ai/docs/models/LiquidAI.html), and [Arcee Foundation Models (AFM)](https://docs.axolotl.ai/docs/models/arcee.html). - FP8 finetuning with fp8 gather op is now possible in Axolotl via `torchao`. Get started [here](https://docs.axolotl.ai/docs/mixed_precision.html#sec-fp8)! - [Voxtral](https://docs.axolotl.ai/docs/models/voxtral.html), [Magistral 1.1](https://docs.axolotl.ai/docs/models/magistral.html), and [Devstral](https://docs.axolotl.ai/docs/models/devstral.html) with mistral-common tokenizer support has been integrated in Axolotl! - TiledMLP support for single-GPU to multi-GPU training with DDP, DeepSpeed and FSDP support has been added to support Arctic Long Sequence Training. (ALST). See [examples](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/alst) for using ALST with Axolotl! - 2025/06: Magistral with mistral-common tokenizer support has been added to Axolotl. See [docs](https://docs.axolotl.ai/docs/models/magistral.html) to start training your own Magistral models with Axolotl! - 2025/05: Quantization Aware Training (QAT) support has been added to Axolotl. Explore the [docs](https://docs.axolotl.ai/docs/qat.html) to learn more! - 2025/04: Llama 4 support has been added in Axolotl. See [docs](https://docs.axolotl.ai/docs/models/llama-4.html) to start training your own Llama 4 models with Axolotl's linearized version! - 2025/03: Axolotl has implemented Sequence Parallelism (SP) support. Read the [blog](https://huggingface.co/blog/axolotl-ai-co/long-context-with-sequence-parallelism-in-axolotl) and [docs](https://docs.axolotl.ai/docs/sequence_parallelism.html) to learn how to scale your context length when fine-tuning. - 2025/03: (Beta) Fine-tuning Multimodal models is now supported in Axolotl. Check out the [docs](https://docs.axolotl.ai/docs/multimodal.html) to fine-tune your own! - 2025/02: Axolotl has added LoRA optimizations to reduce memory usage and improve training speed for LoRA and QLoRA in single GPU and multi-GPU training (DDP and DeepSpeed). Jump into the [docs](https://docs.axolotl.ai/docs/lora_optims.html) to give it a try. - 2025/02: Axolotl has added GRPO support. Dive into our [blog](https://huggingface.co/blog/axolotl-ai-co/training-llms-w-interpreter-feedback-wasm) and [GRPO example](https://github.com/axolotl-ai-cloud/grpo_code) and have some fun! - 2025/01: Axolotl has added Reward Modelling / Process Reward Modelling fine-tuning support. See [docs](https://docs.axolotl.ai/docs/reward_modelling.html). ## ✨ Overview Axolotl is a free and open-source tool designed to streamline post-training and fine-tuning for the latest large language models (LLMs). Features: - **Multiple Model Support**: Train various models like GPT-OSS, LLaMA, Mistral, Mixtral, Pythia, and many more models available on the Hugging Face Hub. - **Multimodal Training**: Fine-tune vision-language models (VLMs) including LLaMA-Vision, Qwen2-VL, Pixtral, LLaVA, SmolVLM2, GLM-4.6V, InternVL 3.5, Gemma 3n, PaddleOCR-VL, Muse Glimmer, and audio models like Voxtral with image, video, and audio support. - **Training Methods**: Full fine-tuning, LoRA, QLoRA, GPTQ, QAT (int8/int4/FP8/NVFP4/MXFP4), FP8 mixed-precision training, NVFP4/MXFP4 MoE LoRA, Preference Tuning (DPO, IPO, KTO, ORPO), RL (GRPO, GDPO), and Reward Modelling (RM) / Process Reward Modelling (PRM). - **Easy Configuration**: Re-use a single YAML configuration file across the full fine-tuning pipeline: dataset preprocessing, training, evaluation, quantization, and inference. - **Performance Optimizations**: [Multipacking](https://docs.axolotl.ai/docs/multipack.html), [Flash Attention 2/3/4](https://docs.axolotl.ai/docs/attention.html#flash-attention), [Xformers](https://docs.axolotl.ai/docs/attention.html#xformers), [Flex Attention](https://docs.axolotl.ai/docs/attention.html#flex-attention), [SageAttention](https://docs.axolotl.ai/docs/attention.html#sageattention), [Liger Kernel](https://docs.axolotl.ai/docs/custom_integrations.html#liger-kernels), [Cut Cross Entropy](https://docs.axolotl.ai/docs/custom_integrations.html#cut-cross-entropy), [ScatterMoE](https://docs.axolotl.ai/docs/custom_integrations.html#kernels-integration), [Sequence Parallelism (SP)](https://docs.axolotl.ai/docs/sequence_parallelism.html), [LoRA optimizations](https://docs.axolotl.ai/docs/lora_optims.html), [Multi-GPU training (FSDP1, FSDP2, DeepSpeed)](https://docs.axolotl.ai/docs/multi-gpu.html), [Multi-node training (Torchrun, Ray)](https://docs.axolotl.ai/docs/multi-node.html), and many more! - **Flexible Dataset Handling**: Load from local, HuggingFace, and cloud (S3, Azure, GCP, OCI) datasets. - **Cloud Ready**: We ship [Docker images](https://hub.docker.com/u/axolotlai) and also [PyPI packages](https://pypi.org/project/axolotl/) for use on cloud platforms and local hardware. ## πŸš€ Quick Start - LLM Fine-tuning in Minutes **Requirements**: - NVIDIA GPU (Ampere or newer for `bf16` and Flash Attention) or AMD GPU - Python >=3.11 (3.12 recommended) - PyTorch β‰₯2.11.0 ### Google Colab [](https://colab.research.google.com/github/axolotl-ai-cloud/axolotl/blob/main/examples/colab-notebooks/colab-axolotl-example.ipynb#scrollTo=msOCO4NRmRLa) ### Installation ```bash # install uv if you don't already have it installed (restart shell after) curl -LsSf https://astral.sh/uv/install.sh | sh # change depending on system export UV_TORCH_BACKEND=cu130 # create a new virtual environment uv venv --python 3.12 source .venv/bin/activate uv pip install torch==2.12.0 torchvision uv pip install --no-build-isolation axolotl[deepspeed] # Download example axolotl configs, deepspeed configs axolotl fetch examples axolotl fetch deepspeed_configs # OPTIONAL ``` #### Using Docker Installing with Docker can be less error prone than installing in your own environment. ```bash docker run --gpus '"all"' --ipc=host --rm -it axolotlai/axolotl:main-latest ``` Other installation approaches are described [here](https://docs.axolotl.ai/docs/installation.html). #### Cloud Providers - [RunPod](https://runpod.io/gsc?template=v2ickqhz9s&ref=6i7fkpdz) - [Vast.ai](https://cloud.vast.ai?ref_id=62897&template_id=bdd4a49fa8bce926defc99471864cace&utm_source=github&utm_medium=developer_community&utm_campaign=template_launch_axolotl&utm_content=readme) - [PRIME Intellect](https://app.primeintellect.ai/dashboard/create-cluster?image=axolotl&location=Cheapest&security=Cheapest&show_spot=true) - [Modal](https://www.modal.com?utm_source=github&utm_medium=github&utm_campaign=axolotl) - [Novita](https://novita.ai/gpus-console?templateId=311) - [JarvisLabs.ai](https://jarvislabs.ai/templates/axolotl) - [Latitude.sh](https://latitude.sh/blueprint/989e0e79-3bf6-41ea-a46b-1f246e309d5c) ### Your First Fine-tune ```bash # Fetch axolotl examples axolotl fetch examples # Or, specify a custom path axolotl fetch examples --dest path/to/folder # Train a model using LoRA axolotl train examples/llama-3/lora-1b.yml ``` That's it! Check out our [Getting Started Guide](https://docs.axolotl.ai/docs/getting-started.html) for a more detailed walkthrough. ## πŸ“š Documentation - [Installation Options](https://docs.axolotl.ai/docs/installation.html) - Detailed setup instructions for different environments - [Support Matrix](https://docs.axolotl.ai/docs/support-matrix.html) - Feature support, compatibility, and known gaps - [Configuration Guide](https://docs.axolotl.ai/docs/config-reference.html) - Full configuration options and examples - [Dataset Loading](https://docs.axolotl.ai/docs/dataset_loading.html) - Loading datasets from various sources - [Dataset Guide](https://docs.axolotl.ai/docs/dataset-formats/) - Supported formats and how to use them - [Multi-GPU Training](https://docs.axolotl.ai/docs/multi-gpu.html) - [Multi-Node Training](https://docs.axolotl.ai/docs/multi-node.html) - [Multipacking](https://docs.axolotl.ai/docs/multipack.html) - [API Reference](https://docs.axolotl.ai/docs/api/) - Auto-generated code documentation - [FAQ](https://docs.axolotl.ai/docs/faq.html) - Frequently asked questions ## AI Agent Support Axolotl ships with built-in documentation optimized for AI coding agents (Claude Code, Cursor, Copilot, etc.). These docs are bundled with the pip package, no repo clone needed. ```bash # Show overview and available training methods axolotl agent-docs # Topic-specific references axolotl agent-docs sft # supervised fine-tuning axolotl agent-docs grpo # GRPO online RL axolotl agent-docs preference_tuning # DPO, KTO, ORPO, SimPO axolotl agent-docs reward_modelling # outcome and process reward models axolotl agent-docs pretraining # continual pretraining axolotl agent-docs --list # list all topics # Dump config schema for programmatic use axolotl config-schema axolotl config-schema --field adapter ``` If you're working with the source repo, agent docs are also available at `docs/agents/` and the project overview is in `AGENTS.md`. ## 🀝 Getting Help - Join our [Discord community](https://discord.gg/HhrNrHJPRb) for support - Check out our [Examples](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/) directory - Read our [Debugging Guide](https://docs.axolotl.ai/docs/debugging.html) - Need dedicated support? Please contact [βœ‰οΈwing@axolotl.ai](mailto:wing@axolotl.ai) for options ## 🌟 Contributing Contributions are welcome! Please see our [Contributing Guide](https://github.com/axolotl-ai-cloud/axolotl/blob/main/.github/CONTRIBUTING.md) for details. ## πŸ“ˆ Telemetry Axolotl has opt-out telemetry that helps us understand how the project is being used and prioritize improvements. We collect basic system information, model types, and error rates, never personal data or file paths. Telemetry is enabled by default. To disable it, set AXOLOTL_DO_NOT_TRACK=1. For more details, see our [telemetry documentation](https://docs.axolotl.ai/docs/telemetry.html). ## ❀️ Sponsors Interested in sponsoring? Contact us at [wing@axolotl.ai](mailto:wing@axolotl.ai) ## πŸ“ Citing Axolotl If you use Axolotl in your research or projects, please cite it as follows: ```bibtex @software{axolotl, title = {Axolotl: Open Source LLM Post-Training}, author = {{Axolotl maintainers and contributors}}, url = {https://github.com/axolotl-ai-cloud/axolotl}, license = {Apache-2.0}, year = {2023} } ``` ## πŸ“œ License This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details. ## 2. In-Tree Documentation Chapters (axolotl-ai-cloud/axolotl) ## File: README.md ## πŸŽ‰ Latest Updates - 2026/08: - New model support has been added in Axolotl for [Muse Glimmer](https://docs.axolotl.ai/docs/models/muse-glimmer.html), [North Micro Vision Instruct](https://docs.axolotl.ai/docs/models/cohere-north-micro-vision-instruct.html) and [Shieldstral](https://docs.axolotl.ai/docs/models/shieldstral.html). - 2026/07: - [NVFP4 (4-bit) MoE LoRA training](https://docs.axolotl.ai/docs/nvfp4_lora.html) is now supported via ScatterMoE (W4A16) and SonicMoE (W4A4), including adapter merge back into a plain NVFP4 checkpoint. - 2026/06: - [Expert Parallelism (EP)](https://docs.axolotl.ai/docs/nd_parallelism.html) for distributed MoE training via DeepEP, remote training through [Tinker-compatible APIs](https://github.com/axolotl-ai-cloud/axolotl/pull/3614), [Context Parallelism for hybrid SSM models](https://github.com/axolotl-ai-cloud/axolotl/pull/3572) (Nemotron-H, Falcon-H1, Bamba), [BitNet 1.58-bit](https://github.com/axolotl-ai-cloud/axolotl/pull/3634) fine-tuning, and a [multimodal assistant-only loss-masking fix](https://github.com/axolotl-ai-cloud/axolotl/pull/3625). - 2026/04: - New model support has been added in Axolotl for [Mistral Medium 3.5](https://docs.axolotl.ai/docs/models/mistral-medium-3_5.html) and [Gemma 4](https://docs.axolotl.ai/docs/models/gemma4.html). - New RL and kernels: [Async GRPO](https://github.com/axolotl-ai-cloud/axolotl/pull/3486) (up to 58% faster steps), [Flash Attention 4](https://docs.axolotl.ai/docs/attention.html#flash-attention), [NeMo Gym](https://github.com/axolotl-ai-cloud/axolotl/pull/3516), and [EBFT](https://github.com/axolotl-ai-cloud/axolotl/pull/3527). - Axolotl is now [uv-first](https://github.com/axolotl-ai-cloud/axolotl/pull/3545) and has [SonicMoE fused LoRA](https://github.com/axolotl-ai-cloud/axolotl/pull/3519) support. - 2026/03: - New model support has been added in Axolotl for [Mistral Small 4](https://docs.axolotl.ai/docs/models/mistral4.html), [Qwen3.5, Qwen3.5 MoE](https://docs.axolotl.ai/docs/models/qwen3.5.html), [GLM-4.7-Flash](https://docs.axolotl.ai/docs/models/glm47-flash.html), [GLM-4.6V](https://docs.axolotl.ai/docs/models/glm46v.html), and [GLM-4.5-Air](https://docs.axolotl.ai/docs/models/glm45.html). - [MoE expert quantization](https://docs.axolotl.ai/docs/expert_quantization.html) support (via `quantize_moe_experts: true`) greatly reduces VRAM when training MoE models (FSDP2 compat). Expand older updates - 2026/02: - [ScatterMoE LoRA](https://github.com/axolotl-ai-cloud/axolotl/pull/3410) support. LoRA fine-tuning directly on MoE expert weights using custom Triton kernels. - Axolotl now has support for [SageAttention](https://github.com/axolotl-ai-cloud/axolotl/pull/2823) and [GDPO](https://github.com/axolotl-ai-cloud/axolotl/pull/3353) (Generalized DPO). - 2026/01: - New integration for [EAFT](https://github.com/axolotl-ai-cloud/axolotl/pull/3366) (Entropy-Aware Focal Training), weights loss by entropy of the top-k logit distribution, and [Scalable Softmax](https://github.com/axolotl-ai-cloud/axolotl/pull/3338), improves long context in attention. - 2025/12: - Axolotl now includes support for [Kimi-Linear](https://docs.axolotl.ai/docs/models/kimi-linear.html), [Plano-Orchestrator](https://docs.axolotl.ai/docs/models/plano.html), [MiMo](https://docs.axolotl.ai/docs/models/mimo.html), [InternVL 3.5](https://docs.axolotl.ai/docs/models/internvl3_5.html), [Olmo3](https://docs.axolotl.ai/docs/models/olmo3.html), [Trinity](https://docs.axolotl.ai/docs/models/trinity.html), and [Ministral3](https://docs.axolotl.ai/docs/models/ministral3.html). - [Distributed Muon Optimizer](https://github.com/axolotl-ai-cloud/axolotl/pull/3264) support has been added for FSDP2 pretraining. - 2025/10: New model support has been added in Axolotl for: [Qwen3 Next](https://docs.axolotl.ai/docs/models/qwen3-next.html), [Qwen2.5-vl, Qwen3-vl](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/qwen2_5-vl), [Qwen3, Qwen3MoE](https://docs.axolotl.ai/docs/models/qwen3.html), [Granite 4](https://docs.axolotl.ai/docs/models/granite4.html), [HunYuan](https://docs.axolotl.ai/docs/models/hunyuan.html), [Magistral 2509](https://docs.axolotl.ai/docs/models/magistral/vision.html), [Apertus](https://docs.axolotl.ai/docs/models/apertus.html), and [Seed-OSS](https://docs.axolotl.ai/docs/models/seed-oss.html). - 2025/09: Axolotl now has text diffusion training. Read more [here](https://github.com/axolotl-ai-cloud/axolotl/tree/main/src/axolotl/integrations/diffusion). - 2025/08: QAT has been updated to include NVFP4 support. See [PR](https://github.com/axolotl-ai-cloud/axolotl/pull/3107). - 2025/07: - ND Parallelism support has been added into Axolotl. Compose Context Parallelism (CP), Tensor Parallelism (TP), and Fully Sharded Data Parallelism (FSDP) within a single node and across multiple nodes. Check out the [blog post](https://huggingface.co/blog/accelerate-nd-parallel) for more info. - Axolotl adds more models: [GPT-OSS](https://docs.axolotl.ai/docs/models/gpt-oss.html), [Gemma 3n](https://docs.axolotl.ai/docs/models/gemma3n.html), [Liquid Foundation Model 2 (LFM2)](https://docs.axolotl.ai/docs/models/LiquidAI.html), and [Arcee Foundation Models (AFM)](https://docs.axolotl.ai/docs/models/arcee.html). - FP8 finetuning with fp8 gather op is now possible in Axolotl via `torchao`. Get started [here](https://docs.axolotl.ai/docs/mixed_precision.html#sec-fp8)! - [Voxtral](https://docs.axolotl.ai/docs/models/voxtral.html), [Magistral 1.1](https://docs.axolotl.ai/docs/models/magistral.html), and [Devstral](https://docs.axolotl.ai/docs/models/devstral.html) with mistral-common tokenizer support has been integrated in Axolotl! - TiledMLP support for single-GPU to multi-GPU training with DDP, DeepSpeed and FSDP support has been added to support Arctic Long Sequence Training. (ALST). See [examples](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/alst) for using ALST with Axolotl! - 2025/06: Magistral with mistral-common tokenizer support has been added to Axolotl. See [docs](https://docs.axolotl.ai/docs/models/magistral.html) to start training your own Magistral models with Axolotl! - 2025/05: Quantization Aware Training (QAT) support has been added to Axolotl. Explore the [docs](https://docs.axolotl.ai/docs/qat.html) to learn more! - 2025/04: Llama 4 support has been added in Axolotl. See [docs](https://docs.axolotl.ai/docs/models/llama-4.html) to start training your own Llama 4 models with Axolotl's linearized version! - 2025/03: Axolotl has implemented Sequence Parallelism (SP) support. Read the [blog](https://huggingface.co/blog/axolotl-ai-co/long-context-with-sequence-parallelism-in-axolotl) and [docs](https://docs.axolotl.ai/docs/sequence_parallelism.html) to learn how to scale your context length when fine-tuning. - 2025/03: (Beta) Fine-tuning Multimodal models is now supported in Axolotl. Check out the [docs](https://docs.axolotl.ai/docs/multimodal.html) to fine-tune your own! - 2025/02: Axolotl has added LoRA optimizations to reduce memory usage and improve training speed for LoRA and QLoRA in single GPU and multi-GPU training (DDP and DeepSpeed). Jump into the [docs](https://docs.axolotl.ai/docs/lora_optims.html) to give it a try. - 2025/02: Axolotl has added GRPO support. Dive into our [blog](https://huggingface.co/blog/axolotl-ai-co/training-llms-w-interpreter-feedback-wasm) and [GRPO example](https://github.com/axolotl-ai-cloud/grpo_code) and have some fun! - 2025/01: Axolotl has added Reward Modelling / Process Reward Modelling fine-tuning support. See [docs](https://docs.axolotl.ai/docs/reward_modelling.html). ## ✨ Overview Axolotl is a free and open-source tool designed to streamline post-training and fine-tuning for the latest large language models (LLMs). Features: - **Multiple Model Support**: Train various models like GPT-OSS, LLaMA, Mistral, Mixtral, Pythia, and many more models available on the Hugging Face Hub. - **Multimodal Training**: Fine-tune vision-language models (VLMs) including LLaMA-Vision, Qwen2-VL, Pixtral, LLaVA, SmolVLM2, GLM-4.6V, InternVL 3.5, Gemma 3n, PaddleOCR-VL, Muse Glimmer, and audio models like Voxtral with image, video, and audio support. - **Training Methods**: Full fine-tuning, LoRA, QLoRA, GPTQ, QAT (int8/int4/FP8/NVFP4/MXFP4), FP8 mixed-precision training, NVFP4/MXFP4 MoE LoRA, Preference Tuning (DPO, IPO, KTO, ORPO), RL (GRPO, GDPO), and Reward Modelling (RM) / Process Reward Modelling (PRM). - **Easy Configuration**: Re-use a single YAML configuration file across the full fine-tuning pipeline: dataset preprocessing, training, evaluation, quantization, and inference. - **Performance Optimizations**: [Multipacking](https://docs.axolotl.ai/docs/multipack.html), [Flash Attention 2/3/4](https://docs.axolotl.ai/docs/attention.html#flash-attention), [Xformers](https://docs.axolotl.ai/docs/attention.html#xformers), [Flex Attention](https://docs.axolotl.ai/docs/attention.html#flex-attention), [SageAttention](https://docs.axolotl.ai/docs/attention.html#sageattention), [Liger Kernel](https://docs.axolotl.ai/docs/custom_integrations.html#liger-kernels), [Cut Cross Entropy](https://docs.axolotl.ai/docs/custom_integrations.html#cut-cross-entropy), [ScatterMoE](https://docs.axolotl.ai/docs/custom_integrations.html#kernels-integration), [Sequence Parallelism (SP)](https://docs.axolotl.ai/docs/sequence_parallelism.html), [LoRA optimizations](https://docs.axolotl.ai/docs/lora_optims.html), [Multi-GPU training (FSDP1, FSDP2, DeepSpeed)](https://docs.axolotl.ai/docs/multi-gpu.html), [Multi-node training (Torchrun, Ray)](https://docs.axolotl.ai/docs/multi-node.html), and many more! - **Flexible Dataset Handling**: Load from local, HuggingFace, and cloud (S3, Azure, GCP, OCI) datasets. - **Cloud Ready**: We ship [Docker images](https://hub.docker.com/u/axolotlai) and also [PyPI packages](https://pypi.org/project/axolotl/) for use on cloud platforms and local hardware. ## πŸš€ Quick Start - LLM Fine-tuning in Minutes **Requirements**: - NVIDIA GPU (Ampere or newer for `bf16` and Flash Attention) or AMD GPU - Python >=3.11 (3.12 recommended) - PyTorch β‰₯2.11.0 ### Google Colab [](https://colab.research.google.com/github/axolotl-ai-cloud/axolotl/blob/main/examples/colab-notebooks/colab-axolotl-example.ipynb#scrollTo=msOCO4NRmRLa) ### Installation ```bash # install uv if you don't already have it installed (restart shell after) curl -LsSf https://astral.sh/uv/install.sh | sh # change depending on system export UV_TORCH_BACKEND=cu130 # create a new virtual environment uv venv --python 3.12 source .venv/bin/activate uv pip install torch==2.12.0 torchvision uv pip install --no-build-isolation axolotl[deepspeed] # Download example axolotl configs, deepspeed configs axolotl fetch examples axolotl fetch deepspeed_configs # OPTIONAL ``` #### Using Docker Installing with Docker can be less error prone than installing in your own environment. ```bash docker run --gpus '"all"' --ipc=host --rm -it axolotlai/axolotl:main-latest ``` Other installation approaches are described [here](https://docs.axolotl.ai/docs/installation.html). #### Cloud Providers - [RunPod](https://runpod.io/gsc?template=v2ickqhz9s&ref=6i7fkpdz) - [Vast.ai](https://cloud.vast.ai?ref_id=62897&template_id=bdd4a49fa8bce926defc99471864cace&utm_source=github&utm_medium=developer_community&utm_campaign=template_launch_axolotl&utm_content=readme) - [PRIME Intellect](https://app.primeintellect.ai/dashboard/create-cluster?image=axolotl&location=Cheapest&security=Cheapest&show_spot=true) - [Modal](https://www.modal.com?utm_source=github&utm_medium=github&utm_campaign=axolotl) - [Novita](https://novita.ai/gpus-console?templateId=311) - [JarvisLabs.ai](https://jarvislabs.ai/templates/axolotl) - [Latitude.sh](https://latitude.sh/blueprint/989e0e79-3bf6-41ea-a46b-1f246e309d5c) ### Your First Fine-tune ```bash # Fetch axolotl examples axolotl fetch examples # Or, specify a custom path axolotl fetch examples --dest path/to/folder # Train a model using LoRA axolotl train examples/llama-3/lora-1b.yml ``` That's it! Check out our [Getting Started Guide](https://docs.axolotl.ai/docs/getting-started.html) for a more detailed walkthrough. ## πŸ“š Documentation - [Installation Options](https://docs.axolotl.ai/docs/installation.html) - Detailed setup instructions for different environments - [Support Matrix](https://docs.axolotl.ai/docs/support-matrix.html) - Feature support, compatibility, and known gaps - [Configuration Guide](https://docs.axolotl.ai/docs/config-reference.html) - Full configuration options and examples - [Dataset Loading](https://docs.axolotl.ai/docs/dataset_loading.html) - Loading datasets from various sources - [Dataset Guide](https://docs.axolotl.ai/docs/dataset-formats/) - Supported formats and how to use them - [Multi-GPU Training](https://docs.axolotl.ai/docs/multi-gpu.html) - [Multi-Node Training](https://docs.axolotl.ai/docs/multi-node.html) - [Multipacking](https://docs.axolotl.ai/docs/multipack.html) - [API Reference](https://docs.axolotl.ai/docs/api/) - Auto-generated code documentation - [FAQ](https://docs.axolotl.ai/docs/faq.html) - Frequently asked questions ## AI Agent Support Axolotl ships with built-in documentation optimized for AI coding agents (Claude Code, Cursor, Copilot, etc.). These docs are bundled with the pip package, no repo clone needed. ```bash # Show overview and available training methods axolotl agent-docs # Topic-specific references axolotl agent-docs sft # supervised fine-tuning axolotl agent-docs grpo # GRPO online RL axolotl agent-docs preference_tuning # DPO, KTO, ORPO, SimPO axolotl agent-docs reward_modelling # outcome and process reward models axolotl agent-docs pretraining # continual pretraining axolotl agent-docs --list # list all topics # Dump config schema for programmatic use axolotl config-schema axolotl config-schema --field adapter ``` If you're working with the source repo, agent docs are also available at `docs/agents/` and the project overview is in `AGENTS.md`. ## 🀝 Getting Help - Join our [Discord community](https://discord.gg/HhrNrHJPRb) for support - Check out our [Examples](https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/) directory - Read our [Debugging Guide](https://docs.axolotl.ai/docs/debugging.html) - Need dedicated support? Please contact [βœ‰οΈwing@axolotl.ai](mailto:wing@axolotl.ai) for options ## 🌟 Contributing Contributions are welcome! Please see our [Contributing Guide](https://github.com/axolotl-ai-cloud/axolotl/blob/main/.github/CONTRIBUTING.md) for details. ## πŸ“ˆ Telemetry Axolotl has opt-out telemetry that helps us understand how the project is being used and prioritize improvements. We collect basic system information, model types, and error rates, never personal data or file paths. Telemetry is enabled by default. To disable it, set AXOLOTL_DO_NOT_TRACK=1. For more details, see our [telemetry documentation](https://docs.axolotl.ai/docs/telemetry.html). ## ❀️ Sponsors Interested in sponsoring? Contact us at [wing@axolotl.ai](mailto:wing@axolotl.ai) ## πŸ“ Citing Axolotl If you use Axolotl in your research or projects, please cite it as follows: ```bibtex @software{axolotl, title = {Axolotl: Open Source LLM Post-Training}, author = {{Axolotl maintainers and contributors}}, url = {https://github.com/axolotl-ai-cloud/axolotl}, license = {Apache-2.0}, year = {2023} } ``` ## πŸ“œ License This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details. --- ## File: .runpod/README.md

LLM Post Training- Full fine-tune, LoRA, QLoRa etc. Llama/Mistral/Gemma and more

# Configuration Options This document outlines all available configuration options for training models. The configuration can be provided as a JSON request. ## Usage You can use these configuration Options: 1. As a JSON request body: ```json { "input": { "user_id": "user", "model_id": "model-name", "run_id": "run-id", "credentials": { "wandb_api_key": "", # add your Weights & biases key. TODO: you will be able to set this in Enviornment variables. "hf_token": "", # add your HF_token. TODO: you will be able to set this in Enviornment variables. }, "args": { "base_model": "NousResearch/Llama-3.2-1B", // ... other options } } } ``` ## Configuration Options ### Model Configuration | Option | Description | Default | | ------------------- | --------------------------------------------------------------------------------------------- | -------------------- | | `base_model` | Path to the base model (local or HuggingFace) | Required | | `base_model_config` | Configuration path for the base model | Same as base_model | | `revision_of_model` | Specific model revision from HuggingFace hub | Latest | | `tokenizer_config` | Custom tokenizer configuration path | Optional | | `model_type` | Type of model to load | AutoModelForCausalLM | | `tokenizer_type` | Type of tokenizer to use | AutoTokenizer | | `hub_model_id` | Repository ID where the model will be pushed on Hugging Face Hub (format: username/repo-name) | Optional | ## Model Family Identification | Option | Default | Description | | -------------------------- | ------- | ------------------------------ | | `is_falcon_derived_model` | `false` | Whether model is Falcon-based | | `is_llama_derived_model` | `false` | Whether model is LLaMA-based | | `is_qwen_derived_model` | `false` | Whether model is Qwen-based | | `is_mistral_derived_model` | `false` | Whether model is Mistral-based | ## Model Configuration Overrides | Option | Default | Description | | ----------------------------------------------- | ---------- | ---------------------------------- | | `overrides_of_model_config.rope_scaling.type` | `"linear"` | RoPE scaling type (linear/dynamic) | | `overrides_of_model_config.rope_scaling.factor` | `1.0` | RoPE scaling factor | ### Model Loading Options | Option | Description | Default | | -------------- | ----------------------------- | ------- | | `load_in_8bit` | Load model in 8-bit precision | false | | `load_in_4bit` | Load model in 4-bit precision | false | | `bf16` | Use bfloat16 precision | false | | `fp16` | Use float16 precision | false | | `tf32` | Use tensor float 32 precision | false | ## Memory and Device Settings | Option | Default | Description | | ------------------ | --------- | ----------------------- | | `gpu_memory_limit` | `"20GiB"` | GPU memory limit | | `lora_on_cpu` | `false` | Load LoRA on CPU | | `device_map` | `"auto"` | Device mapping strategy | | `max_memory` | `null` | Max memory per device | ## Training Hyperparameters | Option | Default | Description | | ----------------------------- | --------- | --------------------------- | | `gradient_accumulation_steps` | `1` | Gradient accumulation steps | | `micro_batch_size` | `2` | Batch size per GPU | | `eval_batch_size` | `null` | Evaluation batch size | | `num_epochs` | `4` | Number of training epochs | | `warmup_steps` | `100` | Warmup steps | | `warmup_ratio` | `0.05` | Warmup ratio | | `learning_rate` | `0.00003` | Learning rate | | `lr_quadratic_warmup` | `false` | Quadratic warmup | | `logging_steps` | `null` | Logging frequency | | `eval_steps` | `null` | Evaluation frequency | | `evals_per_epoch` | `null` | Evaluations per epoch | | `save_strategy` | `"epoch"` | Checkpoint saving strategy | | `save_steps` | `null` | Saving frequency | | `saves_per_epoch` | `null` | Saves per epoch | | `save_total_limit` | `null` | Maximum checkpoints to keep | | `max_steps` | `null` | Maximum training steps | ### Dataset Configuration ```yaml datasets: - path: vicgalle/alpaca-gpt4 # HuggingFace dataset or TODO: You will be able to add the local path. type: alpaca # Format type (alpaca, gpteacher, oasst, etc.) ds_type: json # Dataset type data_files: path/to/data # Source data files train_on_split: train # Dataset split to use ``` ## Chat Template Settings | Option | Default | Description | | ------------------------ | -------------------------------- | ---------------------- | | `chat_template` | `"tokenizer_default"` | Chat template type | | `chat_template_jinja` | `null` | Custom Jinja template | | `default_system_message` | `"You are a helpful assistant."` | Default system message | ## Dataset Processing | Option | Default | Description | | --------------------------------- | -------------------------- | ----------------------------------- | | `dataset_prepared_path` | `"data/last_run_prepared"` | Path for prepared dataset | | `push_dataset_to_hub` | `""` | Push dataset to HF hub | | `dataset_num_proc` | `4` | Number of preprocessing processes | | `dataset_keep_in_memory` | `false` | Keep dataset in memory | | `shuffle_merged_datasets` | `true` | Shuffle merged datasets | | `shuffle_before_merging_datasets` | `false` | Shuffle each dataset before merging | | `dataset_exact_deduplication` | `true` | Deduplicate datasets | ## LoRA Configuration | Option | Default | Description | | -------------------------- | ---------------------- | ------------------------------ | | `adapter` | `"lora"` | Adapter type (lora/qlora) | | `lora_model_dir` | `""` | Directory with pretrained LoRA | | `lora_r` | `8` | LoRA attention dimension | | `lora_alpha` | `16` | LoRA alpha parameter | | `lora_dropout` | `0.05` | LoRA dropout | | `lora_target_modules` | `["q_proj", "v_proj"]` | Modules to apply LoRA | | `lora_target_linear` | `false` | Target all linear modules | | `peft_layers_to_transform` | `[]` | Layers to transform | | `lora_modules_to_save` | `[]` | Modules to save | | `lora_fan_in_fan_out` | `false` | Fan in/out structure | ## Optimization Settings | Option | Default | Description | | ------------------------- | ------- | -------------------------- | | `train_on_inputs` | `false` | Train on input prompts | | `group_by_length` | `false` | Group by sequence length | | `gradient_checkpointing` | `false` | Use gradient checkpointing | | `early_stopping_patience` | `3` | Early stopping patience | ## Learning Rate Scheduling | Option | Default | Description | | -------------------------- | ---------- | -------------------- | | `lr_scheduler` | `"cosine"` | Scheduler type | | `lr_scheduler_kwargs` | `{}` | Scheduler parameters | | `cosine_min_lr_ratio` | `null` | Minimum LR ratio | | `cosine_constant_lr_ratio` | `null` | Constant LR ratio | | `lr_div_factor` | `null` | LR division factor | ## Optimizer Settings | Option | Default | Description | | ---------------------- | ------------ | ------------------- | | `optimizer` | `"adamw_hf"` | Optimizer choice | | `optim_args` | `{}` | Optimizer arguments | | `optim_target_modules` | `[]` | Target modules | | `weight_decay` | `null` | Weight decay | | `adam_beta1` | `null` | Adam beta1 | | `adam_beta2` | `null` | Adam beta2 | | `adam_epsilon` | `null` | Adam epsilon | | `max_grad_norm` | `null` | Gradient clipping | ## Attention Implementations | Option | Default | Description | | -------------------------- | ------- | ----------------------------- | | `flash_optimum` | `false` | Use better transformers | | `xformers_attention` | `false` | Use xformers | | `flash_attention` | `false` | Use flash attention | | `flash_attn_cross_entropy` | `false` | Flash attention cross entropy | | `flash_attn_rms_norm` | `false` | Flash attention RMS norm | | `flash_attn_fuse_mlp` | `false` | Fuse MLP operations | | `sdp_attention` | `false` | Use scaled dot product | | `s2_attention` | `false` | Use shifted sparse attention | ## Tokenizer Modifications | Option | Default | Description | | ---------------- | ------- | ---------------------------- | | `special_tokens` | - | Special tokens to add/modify | | `tokens` | `[]` | Additional tokens | ## Distributed Training | Option | Default | Description | | ----------------------- | ------- | --------------------- | | `fsdp` | `null` | FSDP configuration | | `fsdp_config` | `null` | FSDP config options | | `deepspeed` | `null` | Deepspeed config path | | `ddp_timeout` | `null` | DDP timeout | | `ddp_bucket_cap_mb` | `null` | DDP bucket capacity | | `ddp_broadcast_buffers` | `null` | DDP broadcast buffers |

Example Configuration Request:

Here's a complete example for fine-tuning a LLaMA model using LoRA: ```json { "input": { "user_id": "user", "model_id": "llama-test", "run_id": "test-run", "credentials": { "wandb_api_key": "", "hf_token": "" }, "args": { "base_model": "NousResearch/Llama-3.2-1B", "load_in_8bit": false, "load_in_4bit": false, "strict": false, "datasets": [ { "path": "teknium/GPT4-LLM-Cleaned", "type": "alpaca" } ], "dataset_prepared_path": "last_run_prepared", "val_set_size": 0.1, "output_dir": "./outputs/lora-out", "adapter": "lora", "sequence_len": 2048, "sample_packing": true, "eval_sample_packing": true, "pad_to_sequence_len": true, "lora_r": 16, "lora_alpha": 32, "lora_dropout": 0.05, "lora_target_modules": [ "gate_proj", "down_proj", "up_proj", "q_proj", "v_proj", "k_proj", "o_proj" ], "gradient_accumulation_steps": 2, "micro_batch_size": 2, "num_epochs": 1, "optimizer": "adamw_8bit", "lr_scheduler": "cosine", "learning_rate": 0.0002, "train_on_inputs": false, "group_by_length": false, "bf16": "auto", "tf32": false, "gradient_checkpointing": true, "logging_steps": 1, "flash_attention": true, "loss_watchdog_threshold": 5, "loss_watchdog_patience": 3, "warmup_steps": 10, "evals_per_epoch": 4, "saves_per_epoch": 1, "weight_decay": 0, "hub_model_id": "runpod/llama-fr-lora", "wandb_name": "test-run-1", "wandb_project": "test-run-1", "wandb_entity": "axo-test", "special_tokens": { "pad_token": "<|end_of_text|>" } } } } ``` ### Advanced Features #### Wandb Integration - `wandb_project`: Project name for Weights & Biases - `wandb_entity`: Team name in W&B - `wandb_watch`: Monitor model with W&B - `wandb_name`: Name of the W&B run - `wandb_run_id`: ID for the W&B run #### Performance Optimization - `sample_packing`: Enable efficient sequence packing - `eval_sample_packing`: Use sequence packing during evaluation - `torch_compile`: Enable PyTorch 2.0 compilation - `flash_attention`: Use Flash Attention implementation - `xformers_attention`: Use xFormers attention implementation ### Available Optimizers The following optimizers are supported: - `adamw_hf`: HuggingFace's AdamW implementation - `adamw_torch`: PyTorch's AdamW - `adamw_torch_fused`: Fused AdamW implementation - `adamw_torch_xla`: XLA-optimized AdamW - `adamw_apex_fused`: NVIDIA Apex fused AdamW - `adafactor`: Adafactor optimizer - `adamw_anyprecision`: Anyprecision AdamW - `adamw_bnb_8bit`: 8-bit AdamW from bitsandbytes - `lion_8bit`: 8-bit Lion optimizer - `lion_32bit`: 32-bit Lion optimizer - `sgd`: Stochastic Gradient Descent - `adagrad`: Adagrad optimizer ## Notes - Set `load_in_8bit: true` or `load_in_4bit: true` for memory-efficient training - Enable `flash_attention: true` for faster training on modern GPUs - Use `gradient_checkpointing: true` to reduce memory usage - Adjust `micro_batch_size` and `gradient_accumulation_steps` based on your GPU memory For more detailed information, please refer to the [documentation](https://axolotl-ai-cloud.github.io/axolotl/docs/config-reference.html). ### Errors: - if you face any issues with the Flash Attention-2, Delete yoor worker and Re-start. --- ## File: .vscode/README.md See [docs/debugging.md](../docs/debugging.md) for guidance on how to modify these files to debug axolotl with VSCode. --- ## File: devtools/README.md This directory contains example config files that might be useful for debugging. Please see [docs/debugging.qmd](../docs/debugging.qmd) for more information. --- METRICS --- - Files Extracted: 5 - Estimated Token Budget: ~11693 tokens - Recency Window: Active (< 180 days) - Canonical Reference: https://codewiki.google/github.com/axolotl-ai-cloud/axolotl