# Technical Documentation: modelscope/DiffSynth-Studio > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/modelscope/DiffSynth-Studio) ### Overview (`README.md`) # DiffSynth-Studio modelscope%2FDiffSynth-Studio | Trendshift

[](https://pypi.org/project/DiffSynth/) [](https://github.com/modelscope/DiffSynth-Studio/blob/master/LICENSE) [](https://github.com/modelscope/DiffSynth-Studio/issues) [](https://GitHub.com/modelscope/DiffSynth-Studio/pull/) [](https://GitHub.com/modelscope/DiffSynth-Studio/commit/) [](https://discord.gg/Mm9suEeUDc) [切换到中文版](./README_zh.md) ## Introduction Welcome to the magical world of Diffusion models! DiffSynth-Studio is an open-source Diffusion model engine developed and maintained by the [ModelScope Community](https://www.modelscope.cn/) team. We hope to foster technological innovation through framework construction, aggregate the power of the open-source community, and explore the rich capabilities of generative model technology! Framework features: * [Model Support](#all-supported-models): The framework integrates mainstream open-source Diffusion models, covering image generation, video generation, audio generation, and image quality metrics models. * [VRAM Management](https://diffsynth-studio-doc.readthedocs.io/en/latest/Pipeline_Usage/VRAM_management.html): Dynamically schedules model parameters across disk, memory, and VRAM, allowing consumer-grade GPUs with low VRAM to run inference with large models. * [Parameter Quantization](https://diffsynth-studio-doc.readthedocs.io/en/latest/Pipeline_Usage/Quantization.html): Converts model parameters to quantized precisions such as NF4 and INT8, significantly reducing the VRAM requirements of model inference and LoRA training. * [Arbitrary Training](https://diffsynth-studio-doc.readthedocs.io/en/latest/Pipeline_Usage/Model_Training.html): Almost every model that supports inference also supports training, whether base models, LoRAs, or any Adapter models with additional inputs. * [Split Training](https://diffsynth-studio-doc.readthedocs.io/en/latest/Training/Split_Training.html): Uses a computational graph inference engine to track every variable in the Pipeline, splitting the training process into two stages for efficient training. References: * Developer documentation (for humans): [中文版](https://diffsynth-studio-doc.readthedocs.io/zh-cn/latest/README.html)、[English version](https://diffsynth-studio-doc.readthedocs.io/en/latest/README.html) * Agent Skills (for AI): [DiffSynth-Studio Model Integration Skills](https://www.modelscope.cn/collections/DiffSynth-Studio/DiffSynth-Studio-Model-Integration-Skills) See also: * [DiffSynth-ComfyUI](https://github.com/modelscope/DiffSynth-ComfyUI): A [ComfyUI](https://github.com/Comfy-Org/ComfyUI) node package built on DiffSynth-Studio, enabling you to build workflows in ComfyUI to run model inference. * [DiffSynth-WebUI](https://github.com/modelscope/DiffSynth-WebUI): A lightweight LoRA training tool built on DiffSynth-Studio, enabling LoRA training of models on consumer-grade GPUs. * [ModelScope AIGC Zone (for Chinese users)](https://modelscope.cn/aigc/home): Productized features powered by DiffSynth-Studio as the core inference and training engine; experience the rich potential of the open-source model ecosystem. * [ModelScope Civision (for global users)](https://modelscope.ai/civision/home): Unlock the vast potential of the open-source model ecosystem through productized capabilities powered by DiffSynth-Studio. ## Update History > DiffSynth-Studio has undergone major version updates, and some old features are no longer maintained. If you need to use old features, please switch to the [last historical version](https://github.com/modelscope/DiffSynth-Studio/tree/afd101f3452c9ecae0c87b79adfa2e22d65ffdc3) before the major version update. > Currently, the development personnel of this project are limited, with most of the work handled by [Artiprocher](https://github.com/Artiprocher) and [mi804](https://github.com/mi804). Therefore, the progress of new feature development will be relatively slow, and the speed of responding to and resolving issues is limited. We apologize for this and ask developers to understand. - **October 21, 2026** We now support Fibo 2.0, including unified text-to-image and image editing inference, masked and multi-reference editing, text rendering, low VRAM inference, and full/LoRA training for both text-to-image and image editing on [Fibo-2.0-dev](https://www.modelscope.cn/models/briaai/Fibo-2.0-dev). See the [documentation](/docs/en/Model_Details/Fibo.md) and [example code](/examples/fibo/). - **October 20, 2026** We now support MOSS-Image, including inference and training for [MOSS-Image-Base](https://modelscope.cn/models/openmoss/MOSS-Image-Base) and [MOSS-Image-SFT](https://modelscope.cn/models/openmoss/MOSS-Image-SFT). In addition, we provide an [EntroPack low-bit quantized model](https://www.modelscope.cn/models/DiffSynth-Studio/MOSS-Image-EntroPack), which can run the 3.5-bit quantized model with as little as 4 GB of VRAM; and an [accelerated model](https://modelscope.cn/models/DiffSynth-Studio/MOSS-Image-SFT-TurboLoRA) distillation-accelerated via [OPD training](https://diffsynth-studio-doc.readthedocs.io/en/latest/Training/Distillation.html), supporting 8-step ultra-fast inference. - **October 16, 2026** flux-3-image-klein open-sourced, welcome a new member to the image model family! Support includes text-to-image generation, multi-reference editing, layout control, low VRAM inference, and Base training capabilities. For details, please refer to the [documentation](/docs/en/Model_Details/FLUX.3.md) and [example code](/examples/flux3/). We quantized the model with [EntroPack](https://github.com/modelscope/entropack), providing pre-quantized models in 3-bit, 4-bit, and 6-bit versions at [DiffSynth-Studio/flux-3-image-klein-entropack](https://www.modelscope.cn/models/DiffSynth-Studio/flux-3-image-klein-entropack). - **October 12, 2026** We now support distillation accelerated training based on OPD (On-Policy Distillation), which enables distillation acceleration of models at an extremely low cost. For details, please refer to [Distillation Accelerated Training](/docs/en/Training/Distillation.md). - **September 29, 2026** We are excited to open-source EntroPack, a general-purpose tensor compression library for PyTorch that provides lossless compression with exact recovery and rate-controlled lossy compression, with GPU encoding and decoding. It is integrated into DiffSynth-Studio as a quantization backend, compressing model weights at arbitrary target bitrates in both inference and LoRA training. For more information, please refer to: * Project Page: https://github.com/modelscope/entropack * Technical Report: https://arxiv.org/abs/2609.34185 * Pre-quantized Models: https://www.modelscope.cn/collections/DiffSynth-Studio/EntroPack-Quantized-Models * Example Code: [Z-Image-Turbo](/examples/z_image/model_inference_low_vram/Z-Image-Turbo-EntroPack.py), [Qwen-Image-2.1](/examples/qwen_image_21/model_inference_low_vram/Qwen-Image-2.1-EntroPack.py), [MiniMax-H3](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-EntroPack.py), [LoRA Training Example](/examples/z_image/model_training/special/quant_training/Z-Image-Turbo-entropack_lossy_quant.sh) - **September 20, 2026** Qwen-Image-2.1 was open-sourced. The model supports text-to-image generation, image editing, and RGBA output with a transparency channel. DiffSynth-Studio now provides full support for it. For details, please refer to the [documentation](/docs/zh/Model_Details/Qwen-Image-2.1.md) and [example code](/examples/qwen_image_21/). - **September 14, 2026** We trained and open-sourced DiffSynth-Music, a controllable music generation model based on ACE-Step, which supports five control modes: Beats, Vocals, Accompany, Prosody, and Reference. For more information, please refer to: * Model: https://modelscope.cn/models/DiffSynth-Studio/DiffSynth-Music * Technical Report: https://arxiv.org/abs/2609.12774 * Code: [./examples/diffsynth_music/](./examples/diffsynth_music/) - **September 11, 2026** We have added support for YuE2, a new music generation model. For details, please refer to the [documentation](/docs/zh/Model_Details/YuE2.md) and [example code](/examples/yue2/). - **September 10, 2026** We have integrated [LTX-2.5](https://modelscope.cn/models/Lightricks/LTX-2.5), the latest audio-video generation model from Lightricks. The features include text-to-audio/video with automatic duration prediction, image-to-audio/video with keyframe interpolation, audio-to-video, audio-video retake, IC-LoRA pixel spatial upscaling, text-to-audio, INT8 quantized inference, low VRAM inference, and training. For details, please refer to the [documentation](/docs/en/Model_Details/LTX-2.md) and [code](/examples/ltx2/). - **September 10, 2026** We have open-sourced [DiffSynth-ComfyUI](https://github.com/modelscope/DiffSynth-ComfyUI). Now you can run model inference with DiffSynth-Studio in [ComfyUI](https://github.com/Comfy-Org/ComfyUI). - **September 8, 2026** We trained a [MiniMax-H3 Training Adapter](https://www.modelscope.cn/models/DiffSynth-Studio/MiniMax-H3-TrainingAdapter) for the CFG-distilled MiniMax-H3 base, a DeCFG LoRA training adapter offered in FL2VA and Ref2VA versions. The training data comes from the self-generated [MiniMax-H3-Self-Generated-Dataset](https://www.modelscope.cn/datasets/DiffSynth-Studio/MiniMax-H3-Self-Generated-Dataset). Based on this adapter, we trained two toy models, [MiniMax-H3-Songyu-LoRA](https://www.modelscope.cn/models/mibei0804/MiniMax-H3-Songyu-LoRA) (character identity, FL2VA) and [MiniMax-H3-Ref2VA-FirstFrame-Lineart](https://www.modelscope.cn/models/mibei0804/MiniMax-H3-Ref2VA-FirstFrame-Lineart) (lineart first-frame control, Ref2VA), as references for inference and fine-tuning. - **September 1, 2026** We have integrated [SenseNova-U1.5](https://www.modelscope.cn/models/SenseNova/SenseNova-U1.5-8B-MoT), SenseTime's unified multimodal model, for which we provide text-to-image generation, image editing, low VRAM inference, and training support. For details, please refer to the [documentation](/docs/en/Model_Details/SenseNova-U1.md) and [example code](/examples/sensenova_u1/). - **August 31, 2026** We have integrated [Qwen-Video-Edit](https://modelscope.cn/models/yunpeng1998/Qwen-Video-Edit), a video editing model developed by open-source community contributor [yunpeng1998](https://github.com/yunpeng1998) based on the image editing model Qwen-Image-Edit. This serves as an excellent example of exploring and expanding model capabilities. - **August 25, 2026** We have open-sourced [DiffSynth-WebUI](https://github.com/modelscope/DiffSynth-WebUI), enabling one-click private deployment of LoRA training services. Combined with the model quantization feature, you can train large models even with consumer-grade GPUs. - **August 19, 2026** We have released the model quantization feature. It provides a unified `QuantizeConfig` entry point supporting multiple quantization backends including bitsandbytes, torchao, and comfy-kitchen, with capabilities such as online quantization, loading pre-quantized weights, mixed quantization, saving quantized models, and quantization + LoRA training. For details, please refer to the [documentation](/docs/en/Pipeline_Usage/Quantization.md).
More - **August 17, 2026** MiniMax-Music3 open-sourced, welcome a new member to the audio model family! Support includes text-to-music generation and low VRAM inference. For details, please refer to the [documentation](/docs/en/Model_Details/MiniMax-Music3.md) and [example code](/examples/minimax_music3/). - **August 7, 2026** We add support for Wan-Animate-2 in the Wan series. Given a reference image and a driving video, it makes the reference character perform the motions in the driving video, generating high-quality character animation, with both standard and distilled variants. For details, please refer to the [documentation](/docs/en/Model_Details/Wan.md) and [example code](/examples/wanvideo/). - **August 3, 2026** MiniMax-H3 open-sourced, welcome a new member to the video model family! Support includes text-to-video-audio generation, keyframe-guided generation, reference-driven generation, low VRAM inference, and NF4-quantized inference. For details, please refer to the [documentation](/docs/en/Model_Details/MiniMax-H3.md) and [example code](/examples/minimax_h3/). - **July 28, 2026** LingBot-Video open-sourced, welcome a new member to the video model family! This release includes two variants, Dense-1.3B and MoE-30B-A3B (30B total parameters, ~3B active per token), both supporting text-to-video, image-to-video and text-to-image generation, low VRAM inference, and LoRA / full training capabilities. For details, please refer to the [documentation](/docs/en/Model_Details/LingBot-Video.md) and [example code](/examples/lingbot_video/). Huge thanks to [NancyFyong](https://github.com/NancyFyong) for contributing the integration of this model! - **July 21, 2026** We have open-sourced [DiffSynth-Studio Model Integration Skills](https://www.modelscope.cn/collections/DiffSynth-Studio/DiffSynth-Studio-Model-Integration-Skills). This is a composable collection of Agent Skills that automates the entire workflow of integrating external diffusion models into DiffSynth-Studio, significantly improving the standardization and efficiency of model integration. Get started with the [example](https://www.modelscope.cn/skills/DiffSynth-Studio/diffsynth-integrator/file/view/master/example.md?status=1)! - **June 29, 2026** Boogu-Image open-sourced. Support includes text-to-image generation, image editing, low VRAM inference, and training capabilities. For details, please refer to the [documentation](/docs/en/Model_Details/Boogu-Image.md) and [example code](/examples/boogu_image/). - **June 24, 2026** Krea-2 is now open-source, and we have provided full support. For more details, please refer to the [documentation](/docs/en/Model_Details/Krea-2.md) and [example code](/examples/krea2/). - **June 16, 2026**: We have added a new Template model for ACE-Step: [vocals2music](https://www.modelscope.cn/models/DiffSynth-Studio/acestep15xlsft-vocals2music). For more details, please refer to the [documentation](/docs/en/Model_Details/ACE-Step.md) and [example code](/examples/ace_step/). - **June 15, 2026** We have open-sourced Image-to-LoRA V2, compressing the hours-long training process for image style LoRAs into a single model inference step, thereby exploring a new paradigm for LoRA model training. The [technical report](https://arxiv.org/abs/2606.13809) has been released. This release includes three models: * [DiffSynth-Studio/ZImage-i2L-v2](https://modelscope.cn/models/DiffSynth-Studio/ZImage-i2L-v2): Adapted for the Z-Image model * [DiffSynth-Studio/KleinBase4B-i2L-v2](https://modelscope.cn/models/DiffSynth-Studio/KleinBase4B-i2L-v2): Adapted for the FLUX.2-klein-base-4B model * [DiffSynth-Studio/HidreamO1-i2L-v2](https://modelscope.cn/models/DiffSynth-Studio/HidreamO1-i2L-v2): Adapted for the Hidream-O1-Image model - **June 5, 2026** Ideogram 4 open-sourced. Support includes text-to-image inference. For details, please refer to the [documentation](/docs/en/Model_Details/Ideogram-4.md) and [example code](/examples/ideogram4/). - **May 21, 2026**: Added support for image quality metrics models, including FID, CLIP, Aesthetic, PickScore, ImageReward, HPSv2, and HPSv3. For details, refer to the [documentation](/docs/en/Model_Details/Image-Quality-Metrics.md) and [example code](/examples/image_quality_metric/). - **May 18, 2026** Added **CPU Offload Training** support. By moving model weights layer-by-layer between CPU and GPU, it significantly reduces GPU VRAM usage during training, enabling LoRA training of large models even on consumer-grade GPUs, compatible with all models. Simply add `--enable_model_cpu_offload` to your training command to enable (currently supports single-GPU training only). For details, see the [documentation](/docs/en/Training/Offload_Training.md). - **May 14, 2026** HiDream-O1-Image open-sourced, welcome a new member to the image model family! Support includes text-to-image generation, image editing, low VRAM inference, and training capabilities. For details, please refer to the [documentation](/docs/en/Model_Details/HiDream-O1-Image.md) and [example code](/examples/hidream_o1_image/). - **April 28, 2026** We released Diffusion Templates, a plugin framework designed for Diffusion models that significantly lowers the barrier to training controllable generative models. Let's explore this cutting-edge technology together! * Open-source code: [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) * Technical report: [arXiv](https://arxiv.org/abs/2604.24351) * Project homepage: [GitHub](https://modelscope.github.io/diffusion-templates-web/) * Documentation: [English Version](https://diffsynth-studio-doc.readthedocs.io/en/latest/Diffusion_Templates/Introducing_Diffusion_Templates.html) | [Chinese Version](https://diffsynth-studio-doc.readthedocs.io/zh-cn/latest/Diffusion_Templates/Introducing_Diffusion_Templates.html) * Online demo: [ModelScope](https://modelscope.cn/studios/DiffSynth-Studio/Diffusion-Templates) * Model collections: [ModelScope](https://modelscope.cn/collections/DiffSynth-Studio/KleinBase4B-Templates) | [ModelScope International](https://modelscope.ai/collections/DiffSynth-Studio/KleinBase4B-Templates) | [HuggingFace](https://huggingface.co/collections/DiffSynth-Studio/kleinbase4b-templates) * Datasets: [ModelScope](https://modelscope.cn/collections/DiffSynth-Studio/ImagePulseV2) | [ModelScope International](https://modelscope.ai/collections/DiffSynth-Studio/ImagePulseV2) | [HuggingFace](https://huggingface.co/collections/DiffSynth-Studio/imagepulsev2) - **April 27, 2026** We support ACE-Step-1.5! Support includes text-to-music generation, low VRAM inference, and LoRA training capabilities. For details, please refer to the [documentation](/docs/en/Model_Details/ACE-Step.md) and [example code](/examples/ace_step/). - **April 27, 2026**: We have reinstated support for the Stable Diffusion v1.5 and SDXL models, providing academic research support exclusively for these two model types. - **April 14, 2026** JoyAI-Image open-sourced, welcome a new member to the image editing model family! Support includes instruction-guided image editing, low VRAM inference, and training capabilities. For details, please refer to the [documentation](/docs/en/Model_Details/JoyAI-Image.md) and [example code](/examples/joyai_image/). - **March 19, 2026**: Added support for [openmoss/MOVA-720p](https://modelscope.cn/models/openmoss/MOVA-720p) and [openmoss/MOVA-360p](https://modelscope.cn/models/openmoss/MOVA-360p) models, including training and inference capabilities. [Documentation](/docs/en/Model_Details/Wan.md) and [example code](/examples/mova/) are now available. - **March 12, 2026**: We have added support for the [LTX-2.3](https://modelscope.cn/models/Lightricks/LTX-2.3) audio-video generation model. The features includes text-to-audio/video, image-to-audio/video, IC-LoRA control, audio-to-video, and audio-video inpainting. We have supported the complete inference and training functionalities. For details, please refer to the [documentation](/docs/en/Model_Details/LTX-2.md) and [code](/examples/ltx2/). - **March 3, 2026**: We released the [DiffSynth-Studio/Qwen-Image-Layered-Control-V2](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Layered-Control-V2) model, which is an updated version of Qwen-Image-Layered-Control. In addition to the originally supported text-guided functionality, it adds brush-controlled layer separation capabilities. - **March 2, 2026** Added support for [Anima](https://modelscope.cn/models/circlestone-labs/Anima). For details, please refer to the [documentation](docs/en/Model_Details/Anima.md). This is an interesting anime-style image generation model. We look forward to its future updates. - **February 26, 2026** Added full and lora training support for the LTX-2 audio-video generation model. See the [documentation](/docs/en/Model_Details/LTX-2.md) for details. - **February 10, 2026** Added inference support for the LTX-2 audio-video generation model. See the [documentation](/docs/en/Model_Details/LTX-2.md) for details. Support for model training will be implemented in the future. - **February 2, 2026** The first document of the Research Tutorial series is now available, guiding you through training a small 0.1B text-to-image model from scratch. For details, see the [documentation](/docs/en/Research_Tutorial/train_from_scratch.md) and [model](https://modelscope.cn/models/DiffSynth-Studio/AAAMyModel). We hope DiffSynth-Studio can evolve into a more powerful training framework for Diffusion models. - **January 27, 2026**: [Z-Image](https://modelscope.cn/models/Tongyi-MAI/Z-Image) is released, and our [Z-Image-i2L](https://www.modelscope.cn/models/DiffSynth-Studio/Z-Image-i2L) model is released concurrently. You can use it in [ModelScope Studios](https://modelscope.cn/studios/DiffSynth-Studio/Z-Image-i2L). For details, see the [documentation](/docs/zh/Model_Details/Z-Image.md). - **January 19, 2026**: Added support for [FLUX.2-klein-4B](https://modelscope.cn/models/black-forest-labs/FLUX.2-klein-4B) and [FLUX.2-klein-9B](https://modelscope.cn/models/black-forest-labs/FLUX.2-klein-9B) models, including training and inference capabilities. [Documentation](/docs/en/Model_Details/FLUX2.md) and [example code](/examples/flux2/) are now available. - **January 12, 2026**: We trained and open-sourced a text-guided image layer separation model ([Model Link](https://modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Layered-Control)). Given an input image and a textual description, the model isolates the image layer corresponding to the described content. For more details, please refer to our blog post ([Chinese version](https://modelscope.cn/learn/4938), [English version](https://huggingface.co/blog/kelseye/qwen-image-layered-control)). - **December 24, 2025**: Based on Qwen-Image-Edit-2511, we trained an In-Context Editing LoRA model ([Model Link](https://modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Edit-2511-ICEdit-LoRA)). This model takes three images as input (Image A, Image B, and Image C), and automatically analyzes the transformation from Image A to Image B, then applies the same transformation to Image C to generate Image D. For more details, please refer to our blog post ([Chinese version](https://mp.weixin.qq.com/s/41aEiN3lXKGCJs1-we4Q2g), [English version](https://huggingface.co/blog/kelseye/qwen-image-edit-2511-icedit-lora)). - **December 9, 2025** We release a wild model based on DiffSynth-Studio 2.0: [Qwen-Image-i2L](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-i2L) (Image-to-LoRA). This model takes an image as input and outputs a LoRA. Although this version still has significant room for improvement in terms of generalization, detail preservation, and other aspects, we are open-sourcing these models to inspire more innovative research. For more details, please refer to our [blog](https://huggingface.co/blog/kelseye/qwen-image-i2l). - **December 4, 2025** DiffSynth-Studio 2.0 released! Many new features online - [Documentation](/docs/en/README.md) online: Our documentation is still continuously being optimized and updated - [VRAM Management](/docs/en/Pipeline_Usage/VRAM_management.md) module upgraded, supporting layer-level disk offload, releasing both memory and VRAM simultaneously - New model support - Z-Image Turbo: [Model](https://www.modelscope.ai/models/Tongyi-MAI/Z-Image-Turbo), [Documentation](/docs/en/Model_Details/Z-Image.md), [Code](/examples/z_image/) - FLUX.2-dev: [Model](https://www.modelscope.cn/models/black-forest-labs/FLUX.2-dev), [Documentation](/docs/en/Model_Details/FLUX2.md), [Code](/examples/flux2/) - Training framework upgrade - [Split Training](/docs/en/Training/Split_Training.md): Supports automatically splitting the training process into two stages: data processing and training (even for training ControlNet or any other model). Computations that do not require gradient backpropagation, such as text encoding and VAE encoding, are performed during the data processing stage, while other computations are handled during the training stage. Faster speed, less VRAM requirement. - [Differential LoRA Training](/docs/en/Training/Differential_LoRA.md): This is a training technique we used in [ArtAug](https://www.modelscope.cn/models/DiffSynth-Studio/ArtAug-lora-FLUX.1dev-v1), now available for LoRA training of any model. - [FP8 Training](/docs/en/Training/FP8_Precision.md): FP8 can be applied to any non-training model during training, i.e., models with gradients turned off or gradients that only affect LoRA weights. - **November 4, 2025** Supported the [ByteDance/Video-As-Prompt-Wan2.1-14B](https://modelscope.cn/models/ByteDance/Video-As-Prompt-Wan2.1-14B) model, which is trained based on Wan 2.1 and supports generating corresponding actions based on reference videos. - **October 30, 2025** Supported the [meituan-longcat/LongCat-Video](https://www.modelscope.cn/models/meituan-longcat/LongCat-Video) model, which supports text-to-video, image-to-video, and video continuation. This model uses the Wan framework for inference and training in this project. - **October 27, 2025** Supported the [krea/krea-realtime-video](https://www.modelscope.cn/models/krea/krea-realtime-video) model, adding another member to the Wan model ecosystem. - **September 23, 2025** [DiffSynth-Studio/Qwen-Image-EliGen-Poster](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-EliGen-Poster) released! This model was jointly developed and open-sourced by us and Taobao Experience Design Team. Built upon Qwen-Image, the model is specifically designed for e-commerce poster scenarios, supporting precise partition layout control. Please refer to [our sample code](./examples/qwen_image/model_inference/Qwen-Image-EliGen-Poster.py). - **September 9, 2025** Our training framework supports various training modes. Currently adapted for Qwen-Image, in addition to the standard SFT training mode, Direct Distill is now supported. Please refer to [our sample code](./examples/qwen_image/model_training/lora/Qwen-Image-Distill-LoRA.sh). This feature is experimental, and we will continue to improve it to support more comprehensive model training functions. - **August 28, 2025** We support Wan2.2-S2V, an audio-driven cinematic video generation model. See [./examples/wanvideo/](./examples/wanvideo/). - **August 21, 2025** [DiffSynth-Studio/Qwen-Image-EliGen-V2](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-EliGen-V2) released! Compared to the V1 version, the training dataset has been changed to [Qwen-Image-Self-Generated-Dataset](https://www.modelscope.cn/datasets/DiffSynth-Studio/Qwen-Image-Self-Generated-Dataset), so the generated images better conform to Qwen-Image's own image distribution and style. Please refer to [our sample code](./examples/qwen_image/model_inference_low_vram/Qwen-Image-EliGen-V2.py). - **August 21, 2025** We open-sourced the [DiffSynth-Studio/Qwen-Image-In-Context-Control-Union](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-In-Context-Control-Union) structural control LoRA model, adopting the In Context technical route, supporting multiple categories of structural control conditions, including canny, depth, lineart, softedge, normal, and openpose. Please refer to [our sample code](./examples/qwen_image/model_inference/Qwen-Image-In-Context-Control-Union.py). - **August 20, 2025** We open-sourced the [DiffSynth-Studio/Qwen-Image-Edit-Lowres-Fix](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Edit-Lowres-Fix) model, improving the editing effect of Qwen-Image-Edit on low-resolution image inputs. Please refer to [our sample code](./examples/qwen_image/model_inference/Qwen-Image-Edit-Lowres-Fix.py) - **August 19, 2025** Qwen-Image-Edit open-sourced, welcome a new member to the image editing model family! - **August 18, 2025** We trained and open-sourced the Qwen-Image inpainting ControlNet model [DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Inpaint](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Inpaint). The model structure adopts a lightweight design. Please refer to [our sample code](./examples/qwen_image/model_inference/Qwen-Image-Blockwise-ControlNet-Inpaint.py). - **August 15, 2025** We open-sourced the [Qwen-Image-Self-Generated-Dataset](https://www.modelscope.cn/datasets/DiffSynth-Studio/Qwen-Image-Self-Generated-Dataset) dataset. This is an image dataset generated using the Qwen-Image model, containing 160,000 `1024 x 1024` images. It includes general, English text rendering, and Chinese text rendering subsets. We provide annotations for image descriptions, entities, and structural control images for each image. Developers can use this dataset to train Qwen-Image models' ControlNet and EliGen models. We aim to promote technological development through open-sourcing! - **August 13, 2025** We trained and open-sourced the Qwen-Image ControlNet model [DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Depth](https://modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Depth). The model structure adopts a lightweight design. Please refer to [our sample code](./examples/qwen_image/model_inference/Qwen-Image-Blockwise-ControlNet-Depth.py). - **August 12, 2025** We trained and open-sourced the Qwen-Image ControlNet model [DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Canny](https://modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Canny). The model structure adopts a lightweight design. Please refer to [our sample code](./examples/qwen_image/model_inference/Qwen-Image-Blockwise-ControlNet-Canny.py). - **August 11, 2025** We open-sourced the distilled acceleration model [DiffSynth-Studio/Qwen-Image-Distill-LoRA](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Distill-LoRA) for Qwen-Image, following the same training process as [DiffSynth-Studio/Qwen-Image-Distill-Full](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Distill-Full), but the model structure has been modified to LoRA, thus being better compatible with other open-source ecosystem models. - **August 7, 2025** We open-sourced the entity control LoRA model [DiffSynth-Studio/Qwen-Image-EliGen](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-EliGen) for Qwen-Image. Qwen-Image-EliGen can achieve entity-level controlled text-to-image generation. Technical details can be found in [the paper](https://arxiv.org/abs/2501.01097). Training dataset: [EliGenTrainSet](https://www.modelscope.cn/datasets/DiffSynth-Studio/EliGenTrainSet). - **August 5, 2025** We open-sourced the distilled acceleration model [DiffSynth-Studio/Qwen-Image-Distill-Full](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Distill-Full) for Qwen-Image, achieving approximately 5x acceleration. - **August 4, 2025** Qwen-Image open-sourced, welcome a new member to the image generation model family! - **August 1, 2025** [FLUX.1-Krea-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.1-Krea-dev) open-sourced, a text-to-image model focused on aesthetic photography. We provided comprehensive support in a timely manner, including low VRAM layer-by-layer offload, LoRA training, and full training. For more details, please refer to [./examples/flux/](./examples/flux/). - **July 28, 2025** Wan 2.2 open-sourced. We provided comprehensive support in a timely manner, including low VRAM layer-by-layer offload, FP8 quantization, sequence parallelism, LoRA training, and full training. For more details, please refer to [./examples/wanvideo/](./examples/wanvideo/). - **July 11, 2025** We propose Nexus-Gen, a unified framework that combines the language reasoning capabilities of Large Language Models (LLMs) with the image generation capabilities of diffusion models. This framework supports seamless image understanding, generation, and editing tasks. - Paper: [Nexus-Gen: Unified Image Understanding, Generation, and Editing via Prefilled Autoregression in Shared Embedding Space](https://arxiv.org/pdf/2504.21356) - GitHub Repository: https://github.com/modelscope/Nexus-Gen - Model: [ModelScope](https://www.modelscope.cn/models/DiffSynth-Studio/Nexus-GenV2), [HuggingFace](https://huggingface.co/modelscope/Nexus-GenV2) - Training Dataset: [ModelScope Dataset](https://www.modelscope.cn/datasets/DiffSynth-Studio/Nexus-Gen-Training-Dataset) - Online Experience: [ModelScope Nexus-Gen Studio](https://www.modelscope.cn/studios/DiffSynth-Studio/Nexus-Gen) - **June 15, 2025** ModelScope's official evaluation framework [EvalScope](https://github.com/modelscope/evalscope) now supports text-to-image generation evaluation. Please refer to the [best practices](https://evalscope.readthedocs.io/zh-cn/latest/best_practice/t2i_eval.html) guide to try it out. - **March 25, 2025** Our new open-source project [DiffSynth-Engine](https://github.com/modelscope/DiffSynth-Engine) is now open-sourced! Focused on stable model deployment, targeting industry, providing better engineering support, higher computational performance, and more stable features. - **March 31, 2025** We support InfiniteYou, a face feature preservation method for FLUX. More details can be found in [./examples/InfiniteYou/](./examples/InfiniteYou/). - **March 13, 2025** We support HunyuanVideo-I2V, the image-to-video generation version of Tencent's open-source HunyuanVideo. More details can be found in [./examples/HunyuanVideo/](./examples/HunyuanVideo/). - **February 25, 2025** We support Wan-Video, a series of state-of-the-art video synthesis models open-sourced by Alibaba. See [./examples/wanvideo/](./examples/wanvideo/). - **February 17, 2025** We support [StepVideo](https://modelscope.cn/models/stepfun-ai/stepvideo-t2v/summary)! Advanced video synthesis model! See [./examples/stepvideo](./examples/stepvideo/). - **December 31, 2024** We propose EliGen, a new framework for entity-level controlled text-to-image generation, supplemented with an inpainting fusion pipeline, extending its capabilities to image inpainting tasks. EliGen can seamlessly integrate existing community models such as IP-Adapter and In-Context LoRA, enhancing their versatility. For more details, see [./examples/EntityControl](./examples/EntityControl/). - Paper: [EliGen: Entity-Level Controlled Image Generation with Regional Attention](https://arxiv.org/abs/2501.01097) - Model: [ModelScope](https://www.modelscope.cn/models/DiffSynth-Studio/Eligen), [HuggingFace](https://huggingface.co/modelscope/EliGen) - Online Experience: [ModelScope EliGen Studio](https://www.modelscope.cn/studios/DiffSynth-Studio/EliGen) - Training Dataset: [EliGen Train Set](https://www.modelscope.cn/datasets/DiffSynth-Studio/EliGenTrainSet) - **December 19, 2024** We implemented advanced VRAM management for HunyuanVideo, enabling video generation with resolutions of 129x720x1280 on 24GB VRAM or 129x512x384 on just 6GB VRAM. More details can be found in [./examples/HunyuanVideo/](./examples/HunyuanVideo/). - **December 18, 2024** We propose ArtAug, a method to improve text-to-image models through synthesis-understanding interaction. We trained an ArtAug enhancement module for FLUX.1-dev in LoRA format. This model incorporates the aesthetic understanding of Qwen2-VL-72B into FLUX.1-dev, thereby improving the quality of generated images. - Paper: https://arxiv.org/abs/2412.12888 - Example: https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/ArtAug - Model: [ModelScope](https://www.modelscope.cn/models/DiffSynth-Studio/ArtAug-lora-FLUX.1dev-v1), [HuggingFace](https://huggingface.co/ECNU-CILab/ArtAug-lora-FLUX.1dev-v1) - Demo: [ModelScope](https://modelscope.cn/aigc/imageGeneration?tab=advanced&versionId=7228&modelType=LoRA&sdVersion=FLUX_1&modelUrl=modelscope%3A%2F%2FDiffSynth-Studio%2FArtAug-lora-FLUX.1dev-v1%3Frevision%3Dv1.0), HuggingFace (coming soon) - **October 25, 2024** We provide extensive FLUX ControlNet support. This project supports many different ControlNet models and can be freely combined, even if their structures are different. Additionally, ControlNet models are compatible with high-resolution optimization and partition control technologies, enabling very powerful controllable image generation. See [`./examples/ControlNet/`](./examples/ControlNet/). - **October 8, 2024** We released extended LoRAs based on CogVideoX-5B and ExVideo. You can download this model from [ModelScope](https://modelscope.cn/models/ECNU-CILab/ExVideo-CogVideoX-LoRA-129f-v1) or [HuggingFace](https://huggingface.co/ECNU-CILab/ExVideo-CogVideoX-LoRA-129f-v1). - **August 22, 2024** This project now supports CogVideoX-5B. See [here](/examples/video_synthesis/). We provide several interesting features for this text-to-video model, including: - Text-to-video - Video editing - Self super-resolution - Video interpolation - **August 22, 2024** We implemented an interesting brush feature that supports all text-to-image models. Now you can create stunning images with the assistance of AI using the brush! - Use it in our [WebUI](#usage-in-webui). - **August 21, 2024** DiffSynth-Studio now supports FLUX. - Enable CFG and high-resolution inpainting to improve visual quality. See [here](/examples/image_synthesis/README.md) - LoRA, ControlNet, and other addon models will be released soon. - **June 21, 2024** We propose ExVideo, a post-training fine-tuning technique aimed at enhancing the capabilities of video generation models. We extended Stable Video Diffusion to achieve long video generation of up to 128 frames. - [Project Page](https://ecnu-cilab.github.io/ExVideoProjectPage/) - Source code has been released in this repository. See [`examples/ExVideo`](./examples/ExVideo/). - Model has been released at [HuggingFace](https://huggingface.co/ECNU-CILab/ExVideo-SVD-128f-v1) and [ModelScope](https://modelscope.cn/models/ECNU-CILab/ExVideo-SVD-128f-v1). - Technical report has been released at [arXiv](https://arxiv.org/abs/2406.14130). - You can try ExVideo in this [demo](https://huggingface.co/spaces/modelscope/ExVideo-SVD-128f-v1)! - **June 13, 2024** DiffSynth Studio has migrated to ModelScope. The development team has also transitioned from "me" to "us". Of course, I will still participate in subsequent development and maintenance work. - **January 29, 2024** We propose Diffutoon, an excellent cartoon coloring solution. - [Project Page](https://ecnu-cilab.github.io/DiffutoonProjectPage/) - Source code has been released in this project. - Technical report (IJCAI 2024) has been released at [arXiv](https://arxiv.org/abs/2401.16224). - **December 8, 2023** We decided to initiate a new project aimed at unleashing the potential of diffusion models, especially in video synthesis. The development work of this project officially began. - **November 15, 2023** We propose FastBlend, a powerful video deflickering algorithm. - sd-webui extension has been released at [GitHub](https://github.com/Artiprocher/sd-webui-fastblend). - Demonstration videos have been showcased on Bilibili, including three tasks: - [Video Deflickering](https://www.bilibili.com/video/BV1d94y1W7PE) - [Video Interpolation](https://www.bilibili.com/video/BV1Lw411m71p) - [Image-Driven Video Rendering](https://www.bilibili.com/video/BV1RB4y1Z7LF) - Technical report has been released at [arXiv](https://arxiv.org/abs/2311.09265). - Unofficial ComfyUI extensions developed by other users have been released at [GitHub](https://github.com/AInseven/ComfyUI-fastblend). - **October 1, 2023** We released an early version of the project named FastSDXL. This was an initial attempt to build a diffusion engine. - Source code has been released at [GitHub](https://github.com/Artiprocher/FastSDXL). - FastSDXL includes a trainable OLSS scheduler to improve efficiency. - The original repository of OLSS is located [here](https://github.com/alibaba/EasyNLP/tree/master/diffusion/olss_scheduler). - Technical report (CIKM 2023) has been released at [arXiv](https://arxiv.org/abs/2305.14677). - Demonstration video has been released at [Bilibili](https://www.bilibili.com/video/BV1w8411y7uj). - Since OLSS requires additional training, we did not implement it in this project. - **August 29, 2023** We propose DiffSynth, a video synthesis framework. - [Project Page](https://ecnu-cilab.github.io/DiffSynth.github.io/). - Source code has been released at [EasyNLP](https://github.com/alibaba/EasyNLP/tree/master/diffusion/DiffSynth). - Technical report (ECML PKDD 2024) has been released at [arXiv](https://arxiv.org/abs/2308.03463).
## Installation Install from source (recommended): ``` git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` For more installation methods and instructions for non-NVIDIA GPUs, please refer to the [Installation Guide](/docs/en/Pipeline_Usage/Setup.md).
Download Source Configuration > Before model inference and training, you can configure the model download source and other options through [environment variables](/docs/en/Pipeline_Usage/Environment_Variables.md). > > This project downloads models from [ModelScope](https://modelscope.cn/) by default. For users outside China, you can download models from the [ModelScope International](https://modelscope.ai) site with the following configuration: > > ```shell > export MODELSCOPE_ENDPOINT=https://modelscope.ai > ``` > > To download models from [HuggingFace](https://huggingface.co/), please modify the [environment variables](/docs/en/Pipeline_Usage/Environment_Variables.md). Note that model IDs may differ across platforms: > > ```shell > export DIFFSYNTH_DOWNLOAD_SOURCE="huggingface" > ```
## Basic Framework DiffSynth-Studio redesigns the inference and training pipelines for mainstream Diffusion models (including FLUX, Wan, etc.), enabling efficient memory management and flexible model training. Quick start: experience popular and the latest models: | Architecture | Model ID | Inference | Low VRAM Inference | Full Training | Validation After Full Training | LoRA Training | Validation After LoRA Training | |-|-|-|-|-|-|-|-| | Fibo 2.0 | [briaai/Fibo-2.0-dev](https://www.modelscope.cn/models/briaai/Fibo-2.0-dev) | [code](/examples/fibo/model_inference/Fibo-2.0-dev.py) | [code](/examples/fibo/model_inference_low_vram/Fibo-2.0-dev.py) | [code](/examples/fibo/model_training/full/Fibo-2.0-dev-T2I.sh) | [code](/examples/fibo/model_training/validate_full/Fibo-2.0-dev-T2I.py) | [code](/examples/fibo/model_training/lora/Fibo-2.0-dev-T2I.sh) | [code](/examples/fibo/model_training/validate_lora/Fibo-2.0-dev-T2I.py) | | Fibo 2.0 | [briaai/Fibo-2.0-edit-turbo](https://www.modelscope.cn/models/briaai/Fibo-2.0-edit-turbo) | [code](/examples/fibo/model_inference/Fibo-2.0-edit-turbo.py) | [code](/examples/fibo/model_inference_low_vram/Fibo-2.0-edit-turbo.py) | - | - | - | - | | Qwen-Image-2.1 | [Qwen/Qwen-Image-2.1](https://www.modelscope.cn/models/Qwen/Qwen-Image-2.1) | [code](/examples/qwen_image_21/model_inference/Qwen-Image-2.1.py) | [code](/examples/qwen_image_21/model_inference_low_vram/Qwen-Image-2.1.py) | [code](/examples/qwen_image_21/model_training/full/Qwen-Image-2.1.sh) | [code](/examples/qwen_image_21/model_training/validate_full/Qwen-Image-2.1.py) | [code](/examples/qwen_image_21/model_training/lora/Qwen-Image-2.1.sh) | [code](/examples/qwen_image_21/model_training/validate_lora/Qwen-Image-2.1.py) | | Qwen-Image-2.1 | [Qwen/Qwen-Image-2.1-Turbo](https://www.modelscope.cn/models/Qwen/Qwen-Image-2.1-Turbo) | [code](/examples/qwen_image_21/model_inference/Qwen-Image-2.1-Turbo.py) | [code](/examples/qwen_image_21/model_inference_low_vram/Qwen-Image-2.1-Turbo.py) | [code](/examples/qwen_image_21/model_training/full/Qwen-Image-2.1-Turbo.sh) | [code](/examples/qwen_image_21/model_training/validate_full/Qwen-Image-2.1-Turbo.py) | [code](/examples/qwen_image_21/model_training/lora/Qwen-Image-2.1-Turbo.sh) | [code](/examples/qwen_image_21/model_training/validate_lora/Qwen-Image-2.1-Turbo.py) | | MiniMax-H3 | [MiniMax/MiniMax-H3: FL2VA](https://www.modelscope.cn/models/MiniMax/MiniMax-H3) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-FL2VA.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-FL2VA.py) | [code](/examples/minimax_h3/model_training/full/MiniMax-H3-FL2VA.sh) | [code](/examples/minimax_h3/model_training/validate_full/MiniMax-H3-FL2VA.py) | [code](/examples/minimax_h3/model_training/lora/MiniMax-H3-FL2VA.sh) | [code](/examples/minimax_h3/model_training/validate_lora/MiniMax-H3-FL2VA.py) | | MiniMax-H3 | [DiffSynth-Studio/MiniMax-H3-NF4: FL2VA pruned](https://www.modelscope.cn/models/DiffSynth-Studio/MiniMax-H3-NF4) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-NF4-Pruned-FL2VA.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-NF4-Pruned-FL2VA.py) | - | - | [code](/examples/minimax_h3/model_training/lora/MiniMax-H3-NF4-Pruned-FL2VA.sh) | [code](/examples/minimax_h3/model_training/validate_lora/MiniMax-H3-NF4-Pruned-FL2VA.py) | | ACE-Step | [ACE-Step/acestep-v15-xl-sft](https://www.modelscope.cn/models/ACE-Step/acestep-v15-xl-sft) | [code](/examples/ace_step/model_inference/acestep-v15-xl-sft.py) | [code](/examples/ace_step/model_inference_low_vram/acestep-v15-xl-sft.py) | [code](/examples/ace_step/model_training/full/acestep-v15-xl-sft.sh) | [code](/examples/ace_step/model_training/validate_full/acestep-v15-xl-sft.py) | [code](/examples/ace_step/model_training/lora/acestep-v15-xl-sft.sh) | [code](/examples/ace_step/model_training/validate_lora/acestep-v15-xl-sft.py) | | Z-Image | [Tongyi-MAI/Z-Image-Turbo](https://www.modelscope.cn/models/Tongyi-MAI/Z-Image-Turbo) | [code](/examples/z_image/model_inference/Z-Image-Turbo.py) | [code](/examples/z_image/model_inference_low_vram/Z-Image-Turbo.py) | [code](/examples/z_image/model_training/full/Z-Image-Turbo.sh) | [code](/examples/z_image/model_training/validate_full/Z-Image-Turbo.py) | [code](/examples/z_image/model_training/lora/Z-Image-Turbo.sh) | [code](/examples/z_image/model_training/validate_lora/Z-Image-Turbo.py) | | Krea-2 | [krea/Krea-2-Raw](https://www.modelscope.cn/models/krea/Krea-2-Raw) | [code](/examples/krea2/model_inference/Krea-2-Raw.py) | [code](/examples/krea2/model_inference_low_vram/Krea-2-Raw.py) | [code](/examples/krea2/model_training/full/Krea-2-Raw.sh) | [code](/examples/krea2/model_training/validate_full/Krea-2-Raw.py) | [code](/examples/krea2/model_training/lora/Krea-2-Raw.sh) | [code](/examples/krea2/model_training/validate_lora/Krea-2-Raw.py) | | Krea-2 | [krea/Krea-2-Turbo](https://www.modelscope.cn/models/krea/Krea-2-Turbo) | [code](/examples/krea2/model_inference/Krea-2-Turbo.py) | [code](/examples/krea2/model_inference_low_vram/Krea-2-Turbo.py) | [code](/examples/krea2/model_training/full/Krea-2-Turbo.sh) | [code](/examples/krea2/model_training/validate_full/Krea-2-Turbo.py) | [code](/examples/krea2/model_training/lora/Krea-2-Turbo.sh) | [code](/examples/krea2/model_training/validate_lora/Krea-2-Turbo.py) | Model overview: - Image generation - Fibo 2.0: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Fibo.html), [Example code](/examples/fibo/) - MOSS-Image: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/MOSS-Image.html), [Example code](/examples/moss_image/) - FLUX.3: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/FLUX.3.html), [Example code](/examples/flux3/) - Qwen-Image-2.1: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Qwen-Image-2.1.html), [Example code](/examples/qwen_image_21/) - SenseNova-U1: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/SenseNova-U1.html), [Example code](/examples/sensenova_u1/) - Boogu-Image: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Boogu-Image.html), [Example code](/examples/boogu_image/) - Krea-2: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Krea-2.html), [Example code](/examples/krea2/) - Ideogram 4: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Ideogram-4.html), [Example code](/examples/ideogram4/) - HiDream-O1-Image: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/HiDream-O1-Image.html), [Example code](/examples/hidream_o1_image/) - JoyAI-Image: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/JoyAI-Image.html), [Example code](/examples/joyai_image/) - ERNIE-Image: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/ERNIE-Image.html), [Example code](/examples/ernie_image/) - FLUX.2: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/FLUX2.html), [Example code](/examples/flux2/) - Z-Image: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Z-Image.html), [Example code](/examples/z_image/) - Anima: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Anima.html), [Example code](/examples/anima/) - Qwen-Image: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Qwen-Image.html), [Example code](/examples/qwen_image/) - FLUX.1: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/FLUX.html), [Example code](/examples/flux/) - Stable Diffusion XL: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Stable-Diffusion-XL.html), [Example code](/examples/stable_diffusion_xl/) - Stable Diffusion: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Stable-Diffusion.html), [Example code](/examples/stable_diffusion/) - Video generation - MiniMax-H3: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/MiniMax-H3.html), [Example code](/examples/minimax_h3/) - LingBot-Video: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/LingBot-Video.html), [Example code](/examples/lingbot_video/) - LTX-2: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/LTX-2.html), [Example code](/examples/ltx2/) - Wan: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Wan.html), [Example code](/examples/wanvideo/) - Audio generation - DiffSynth-Music: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/DiffSynth-Music.html), [Example code](/examples/diffsynth_music/) - MiniMax-Music3: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/MiniMax-Music3.html), [Example code](/examples/minimax_music3/) - YuE2: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/YuE2.html), [Example code](/examples/yue2/) - ACE-Step: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/ACE-Step.html), [Example code](/examples/ace_step/) - Evaluation models: [Documentation](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Image-Quality-Metrics.html), [Example code](/examples/image_quality_metric/) [View all supported models](#all-supported-models) ## Innovative Achievements We believe that a well-developed open-source code framework can lower the threshold for technical exploration. We have achieved many interesting technologies based on this codebase. Perhaps you also have many wild ideas, and with DiffSynth-Studio, you can quickly realize these ideas.
EntroPack: A General-Purpose Tensor Compression Library for PyTorch > Compress model weights at arbitrary target bitrates with GPU encoding and decoding, and run the compressed weights directly in inference and LoRA training. * Project Page: [GitHub](https://github.com/modelscope/entropack) * Technical Report: https://arxiv.org/abs/2609.34185 * Pre-quantized Models: [ModelScope](https://www.modelscope.cn/models/DiffSynth-Studio/EntroPackPreQuants) * Documentation: [English Version](/docs/en/Pipeline_Usage/Quantization.md) | [Chinese Version](/docs/zh/Pipeline_Usage/Quantization.md) * Example Code: [Z-Image-Turbo](/examples/z_image/model_inference_low_vram/Z-Image-Turbo-EntroPack.py), [Qwen-Image-2.1](/examples/qwen_image_21/model_inference_low_vram/Qwen-Image-2.1-EntroPack.py), [MiniMax-H3](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-EntroPack.py), [LoRA Training Example](/examples/z_image/model_training/special/quant_training/Z-Image-Turbo-entropack_lossy_quant.sh)
TreeAdapter: A Model System Built from Structured LoRAs > A model system composed of 10,000+ LoRAs challenges the precise generation of tens of thousands of rare species. * Paper: [TreeAdapter: Hierarchical Taxonomy-Guided Adapter Composition for Fine-Grained Species Image Generation](https://arxiv.org/abs/2607.24215) * Model: [ModelScope](https://modelscope.cn/models/DiffSynth-Studio/TreeAdapter-KleinBase4B)
Image-to-LoRA: Compressing Model Training into Model Inference > True Meta Learning: feed a dataset of images into one end of the model, and the trained LoRA model comes out the other end. * Paper: [Compressing Image Style Training into a Single Model Forward](https://arxiv.org/abs/2606.13809) * Model: * [DiffSynth-Studio/ZImage-i2L-v2](https://modelscope.cn/models/DiffSynth-Studio/ZImage-i2L-v2): Adapted for the Z-Image model * [DiffSynth-Studio/KleinBase4B-i2L-v2](https://modelscope.cn/models/DiffSynth-Studio/KleinBase4B-i2L-v2): Adapted for the FLUX.2-klein-base-4B model * [DiffSynth-Studio/HidreamO1-i2L-v2](https://modelscope.cn/models/DiffSynth-Studio/HidreamO1-i2L-v2): Adapted for the Hidream-O1-Image model |Input example 1|Output example 1|Input example 2|Output example 2| |-|-|-|-| |||||
Diffusion-Templates: A Plugin-Based Controllable Generation Framework > One framework that turns every controllable generation capability into a plugin, allowing multiple models to combine and emerge with rich generation capabilities. * Paper: [Diffusion Templates: A Unified Plugin Framework for Controllable Diffusion](https://arxiv.org/abs/2604.24351) * Project homepage: [GitHub](https://modelscope.github.io/diffusion-templates-web/) * Documentation: [English Version](https://diffsynth-studio-doc.readthedocs.io/en/latest/Diffusion_Templates/Introducing_Diffusion_Templates.html) | [Chinese Version](https://diffsynth-studio-doc.readthedocs.io/zh-cn/latest/Diffusion_Templates/Introducing_Diffusion_Templates.html) * Online demo: [ModelScope](https://modelscope.cn/studios/DiffSynth-Studio/Diffusion-Templates) * Model collections: [ModelScope](https://modelscope.cn/collections/DiffSynth-Studio/KleinBase4B-Templates) | [ModelScope International](https://modelscope.ai/collections/DiffSynth-Studio/KleinBase4B-Templates) | [HuggingFace](https://huggingface.co/collections/DiffSynth-Studio/kleinbase4b-templates) * Datasets: [ModelScope](https://modelscope.cn/collections/DiffSynth-Studio/ImagePulseV2) | [ModelScope International](https://modelscope.cn/collections/DiffSynth-Studio/ImagePulseV2) | [HuggingFace](https://huggingface.co/collections/DiffSynth-Studio/imagepulsev2) |Reference image|Local editing|Style transfer|Sharpness enhancement| |-|-|-|-| |||||
Spectral Evolution Search: Efficient Inference-Time Scaling for Reward-Aligned Image Generation > Trade inference time for higher quality of generated content. - Paper: [Spectral Evolution Search: Efficient Inference-Time Scaling for Reward-Aligned Image Generation ](https://arxiv.org/abs/2602.03208) - Sample Code: [/docs/en/Research_Tutorial/inference_time_scaling.md](/docs/en/Research_Tutorial/inference_time_scaling.md) |FLUX.1-dev|FLUX.1-dev + SES|Qwen-Image|Qwen-Image + SES| |-|-|-|-| |||||
VIRAL: Visual In-Context Reasoning via Analogy in Diffusion Transformers > Transform image 3 into image 4 based on the change from image 1 to image 2 — the emergent capability of image editing models. - Paper: [VIRAL: Visual In-Context Reasoning via Analogy in Diffusion Transformers ](https://arxiv.org/abs/2602.03210) - Sample code: [/examples/qwen_image/model_inference/Qwen-Image-Edit-2511-ICEdit.py](/examples/qwen_image/model_inference/Qwen-Image-Edit-2511-ICEdit.py) - Model: [ModelScope](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Edit-2511-ICEdit-LoRA) |Example 1|Example 2|Query|Output| |-|-|-|-| |||||
AttriCtrl: Attribute Intensity Control for Image Generation Models > Numerical attributes can also precisely control image generation models. - Paper: [AttriCtrl: Fine-Grained Control of Aesthetic Attribute Intensity in Diffusion Models](https://arxiv.org/abs/2508.02151) - Sample Code: [/examples/flux/model_inference/FLUX.1-dev-AttriCtrl.py](/examples/flux/model_inference/FLUX.1-dev-AttriCtrl.py) - Model: [ModelScope](https://www.modelscope.cn/models/DiffSynth-Studio/AttriCtrl-FLUX.1-Dev) |brightness scale = 0.1|brightness scale = 0.3|brightness scale = 0.5|brightness scale = 0.7|brightness scale = 0.9| |-|-|-|-|-| ||||||
AutoLoRA: Automated LoRA Retrieval and Fusion > LoRA is a product that unifies needs and solutions — how can we make better use of these LoRAs? - Paper: [AutoLoRA: Automatic LoRA Retrieval and Fine-Grained Gated Fusion for Text-to-Image Generation](https://arxiv.org/abs/2508.02107) - Sample Code: [/examples/flux/model_inference/FLUX.1-dev-LoRA-Fusion.py](/examples/flux/model_inference/FLUX.1-dev-LoRA-Fusion.py) - Model: [ModelScope](https://www.modelscope.cn/models/DiffSynth-Studio/LoRAFusion-preview-FLUX.1-dev) ||[LoRA 1](https://modelscope.cn/models/cancel13/cxsk)|[LoRA 2](https://modelscope.cn/models/wy413928499/xuancai2)|[LoRA 3](https://modelscope.cn/models/DiffSynth-Studio/ArtAug-lora-FLUX.1dev-v1)|[LoRA 4](https://modelscope.cn/models/hongyanbujian/JPL)| |-|-|-|-|-| |[LoRA 1](https://modelscope.cn/models/cancel13/cxsk) ||||| |[LoRA 2](https://modelscope.cn/models/wy413928499/xuancai2) ||||| |[LoRA 3](https://modelscope.cn/models/DiffSynth-Studio/ArtAug-lora-FLUX.1dev-v1) ||||| |[LoRA 4](https://modelscope.cn/models/hongyanbujian/JPL) |||||
Nexus-Gen: Unified Architecture for Image Understanding, Generation, and Editing > What happens when a single model combines image understanding, generation, and editing capabilities? - Detailed Page: https://github.com/modelscope/Nexus-Gen - Paper: [Nexus-Gen: Unified Image Understanding, Generation, and Editing via Prefilled Autoregression in Shared Embedding Space](https://arxiv.org/pdf/2504.21356) - Model: [ModelScope](https://www.modelscope.cn/models/DiffSynth-Studio/Nexus-GenV2), [HuggingFace](https://huggingface.co/modelscope/Nexus-GenV2) - Dataset: [ModelScope Dataset](https://www.modelscope.cn/datasets/DiffSynth-Studio/Nexus-Gen-Training-Dataset) - Online Experience: [ModelScope Nexus-Gen Studio](https://www.modelscope.cn/studios/DiffSynth-Studio/Nexus-Gen)
ArtAug: Aesthetic Enhancement for Image Generation Models > A single LoRA that significantly enhances detail and aesthetics. - Detailed Page: [./examples/ArtAug/](./examples/ArtAug/) - Paper: [ArtAug: Enhancing Text-to-Image Generation through Synthesis-Understanding Interaction](https://arxiv.org/abs/2412.12888) - Model: [ModelScope](https://www.modelscope.cn/models/DiffSynth-Studio/ArtAug-lora-FLUX.1dev-v1), [HuggingFace](https://huggingface.co/ECNU-CILab/ArtAug-lora-FLUX.1dev-v1) - Online Experience: [ModelScope AIGC Tab](https://www.modelscope.cn/aigc/imageGeneration?tab=advanced&versionId=7228&modelType=LoRA&sdVersion=FLUX_1&modelUrl=modelscope%3A%2F%2FDiffSynth-Studio%2FArtAug-lora-FLUX.1dev-v1%3Frevision%3Dv1.0) |FLUX.1-dev|FLUX.1-dev + ArtAug LoRA| |-|-| |||
EliGen: Precise Image Partition Control > How can region-based layers control the position of content in an image? - Paper: [EliGen: Entity-Level Controlled Image Generation with Regional Attention](https://arxiv.org/abs/2501.01097) - Sample Code: [/examples/flux/model_inference/FLUX.1-dev-EliGen.py](/examples/flux/model_inference/FLUX.1-dev-EliGen.py) - Model: [ModelScope](https://www.modelscope.cn/models/DiffSynth-Studio/Eligen), [HuggingFace](https://huggingface.co/modelscope/EliGen) - Online Experience: [ModelScope EliGen Studio](https://www.modelscope.cn/studios/DiffSynth-Studio/EliGen) - Dataset: [EliGen Train Set](https://www.modelscope.cn/datasets/DiffSynth-Studio/EliGenTrainSet) |Entity Control Region|Generated Image| |-|-| |||
ExVideo: Extended Training for Video Generation Models > If a video generation model can only generate 25 frames, how can we make it generate longer videos? - Project Page: [Project Page](https://ecnu-cilab.github.io/ExVideoProjectPage/) - Paper: [ExVideo: Extending Video Diffusion Models via Parameter-Efficient Post-Tuning](https://arxiv.org/abs/2406.14130) - Sample Code: Please refer to the [older version](https://github.com/modelscope/DiffSynth-Studio/tree/afd101f3452c9ecae0c87b79adfa2e22d65ffdc3/examples/ExVideo) - Model: [ModelScope](https://modelscope.cn/models/ECNU-CILab/ExVideo-SVD-128f-v1), [HuggingFace](https://huggingface.co/ECNU-CILab/ExVideo-SVD-128f-v1) https://github.com/modelscope/DiffSynth-Studio/assets/35051019/d97f6aa9-8064-4b5b-9d49-ed6001bb9acc
Diffutoon: High-Resolution Anime-Style Video Rendering > I don't care what you say, I just love anime! - Project Page: [Project Page](https://ecnu-cilab.github.io/DiffutoonProjectPage/) - Paper: [Diffutoon: High-Resolution Editable Toon Shading via Diffusion Models](https://arxiv.org/abs/2401.16224) - Sample Code: Please refer to the [older version](https://github.com/modelscope/DiffSynth-Studio/tree/afd101f3452c9ecae0c87b79adfa2e22d65ffdc3/examples/Diffutoon) The Diffutoon examples are from the pre-2.0 codebase and are not included in the current `main` branch. Use the older source tree above with the corresponding older version of DiffSynth-Studio. https://github.com/Artiprocher/DiffSynth-Studio/assets/35051019/b54c05c5-d747-4709-be5e-b39af82404dd
DiffSynth: The Original Version of This Project > In the era before video generation models, how could image generation models be used to process videos? - Project Page: [Project Page](https://ecnu-cilab.github.io/DiffSynth.github.io/) - Paper: [DiffSynth: Latent In-Iteration Deflickering for Realistic Video Synthesis](https://arxiv.org/abs/2308.03463) - Sample Code: Please refer to the [older version](https://github.com/modelscope/DiffSynth-Studio/tree/afd101f3452c9ecae0c87b79adfa2e22d65ffdc3/examples/diffsynth) https://github.com/Artiprocher/DiffSynth-Studio/assets/35051019/59fb2f7b-8de0-4481-b79f-0c3a7361a1ea
## Contact Us |Discord：https://discord.gg/Mm9suEeUDc| |-| |Image| ## All Supported Models | Architecture | Model ID | Inference | Low VRAM Inference | Full Training | Validation After Full Training | LoRA Training | Validation After LoRA Training | |-|-|-|-|-|-|-|-| | Fibo 2.0 | [briaai/Fibo-2.0-dev](https://www.modelscope.cn/models/briaai/Fibo-2.0-dev) | [code](/examples/fibo/model_inference/Fibo-2.0-dev.py) | [code](/examples/fibo/model_inference_low_vram/Fibo-2.0-dev.py) | [code](/examples/fibo/model_training/full/Fibo-2.0-dev-T2I.sh) | [code](/examples/fibo/model_training/validate_full/Fibo-2.0-dev-T2I.py) | [code](/examples/fibo/model_training/lora/Fibo-2.0-dev-T2I.sh) | [code](/examples/fibo/model_training/validate_lora/Fibo-2.0-dev-T2I.py) | | Fibo 2.0 | [briaai/Fibo-2.0-edit-turbo](https://www.modelscope.cn/models/briaai/Fibo-2.0-edit-turbo) | [code](/examples/fibo/model_inference/Fibo-2.0-edit-turbo.py) | [code](/examples/fibo/model_inference_low_vram/Fibo-2.0-edit-turbo.py) | - | - | - | - | | MOSS-Image | [openmoss/MOSS-Image-Base](https://www.modelscope.cn/models/openmoss/MOSS-Image-Base) | [code](/examples/moss_image/model_inference/MOSS-Image-Base.py) | [code](/examples/moss_image/model_inference_low_vram/MOSS-Image-Base.py) | [code](/examples/moss_image/model_training/full/MOSS-Image-Base.sh) | [code](/examples/moss_image/model_training/validate_full/MOSS-Image-Base.py) | [code](/examples/moss_image/model_training/lora/MOSS-Image-Base.sh) | [code](/examples/moss_image/model_training/validate_lora/MOSS-Image-Base.py) | | MOSS-Image | [openmoss/MOSS-Image-SFT](https://www.modelscope.cn/models/openmoss/MOSS-Image-SFT) | [code](/examples/moss_image/model_inference/MOSS-Image-SFT.py) | [code](/examples/moss_image/model_inference_low_vram/MOSS-Image-SFT.py) | [code](/examples/moss_image/model_training/full/MOSS-Image-SFT.sh) | [code](/examples/moss_image/model_training/validate_full/MOSS-Image-SFT.py) | [code](/examples/moss_image/model_training/lora/MOSS-Image-SFT.sh) | [code](/examples/moss_image/model_training/validate_lora/MOSS-Image-SFT.py) | | MOSS-Image | [DiffSynth-Studio/MOSS-Image-EntroPack](https://www.modelscope.cn/models/DiffSynth-Studio/MOSS-Image-EntroPack) | [code](/examples/moss_image/model_inference/MOSS-Image-EntroPack.py) | [code](/examples/moss_image/model_inference_low_vram/MOSS-Image-EntroPack.py) | - | - | [code](/examples/moss_image/model_training/lora/MOSS-Image-entropack.sh) | [code](/examples/moss_image/model_training/validate_lora/MOSS-Image-entropack.py) | | FLUX.3 | [black-forest-labs/flux-3-image-klein](https://www.modelscope.cn/models/black-forest-labs/flux-3-image-klein) | [code](/examples/flux3/model_inference/flux-3-image-klein.py) | [code](/examples/flux3/model_inference_low_vram/flux-3-image-klein.py) | [code](/examples/flux3/model_training/full/flux-3-image-klein.sh) | [code](/examples/flux3/model_training/validate_full/flux-3-image-klein.py) | [code](/examples/flux3/model_training/lora/flux-3-image-klein.sh) | [code](/examples/flux3/model_training/validate_lora/flux-3-image-klein.py) | | FLUX.3 | [black-forest-labs/flux-3-image-klein-base](https://www.modelscope.cn/models/black-forest-labs/flux-3-image-klein-base) | [code](/examples/flux3/model_inference/flux-3-image-klein-base.py) | [code](/examples/flux3/model_inference_low_vram/flux-3-image-klein-base.py) | [code](/examples/flux3/model_training/full/flux-3-image-klein-base.sh) | [code](/examples/flux3/model_training/validate_full/flux-3-image-klein-base.py) | [code](/examples/flux3/model_training/lora/flux-3-image-klein-base.sh) | [code](/examples/flux3/model_training/validate_lora/flux-3-image-klein-base.py) | | FLUX.3 | [DiffSynth-Studio/flux-3-image-klein-entropack](https://www.modelscope.cn/models/DiffSynth-Studio/flux-3-image-klein-entropack) | [code](/examples/flux3/model_inference/flux-3-image-klein-EntroPack.py) | [code](/examples/flux3/model_inference_low_vram/flux-3-image-klein-EntroPack.py) | - | - | [code](/examples/flux3/model_training/lora/flux-3-image-klein-entropack.sh) | [code](/examples/flux3/model_training/validate_lora/flux-3-image-klein-entropack.py) | | Qwen-Image-2.1 | [Qwen/Qwen-Image-2.1](https://www.modelscope.cn/models/Qwen/Qwen-Image-2.1) | [code](/examples/qwen_image_21/model_inference/Qwen-Image-2.1.py) | [code](/examples/qwen_image_21/model_inference_low_vram/Qwen-Image-2.1.py) | [code](/examples/qwen_image_21/model_training/full/Qwen-Image-2.1.sh) | [code](/examples/qwen_image_21/model_training/validate_full/Qwen-Image-2.1.py) | [code](/examples/qwen_image_21/model_training/lora/Qwen-Image-2.1.sh) | [code](/examples/qwen_image_21/model_training/validate_lora/Qwen-Image-2.1.py) | | Qwen-Image-2.1 | [Qwen/Qwen-Image-2.1-Turbo](https://www.modelscope.cn/models/Qwen/Qwen-Image-2.1-Turbo) | [code](/examples/qwen_image_21/model_inference/Qwen-Image-2.1-Turbo.py) | [code](/examples/qwen_image_21/model_inference_low_vram/Qwen-Image-2.1-Turbo.py) | [code](/examples/qwen_image_21/model_training/full/Qwen-Image-2.1-Turbo.sh) | [code](/examples/qwen_image_21/model_training/validate_full/Qwen-Image-2.1-Turbo.py) | [code](/examples/qwen_image_21/model_training/lora/Qwen-Image-2.1-Turbo.sh) | [code](/examples/qwen_image_21/model_training/validate_lora/Qwen-Image-2.1-Turbo.py) | | Qwen-Image-2.1 | [PAI/Qwen-Image-2.1-Fun-Controlnet-Union](https://www.modelscope.cn/models/PAI/Qwen-Image-2.1-Fun-Controlnet-Union) | [code](/examples/qwen_image_21/model_inference/Qwen-Image-2.1-Fun-Controlnet-Union.py) | [code](/examples/qwen_image_21/model_inference_low_vram/Qwen-Image-2.1-Fun-Controlnet-Union.py) | [code](/examples/qwen_image_21/model_training/full/Qwen-Image-2.1-Fun-Controlnet-Union.sh) | [code](/examples/qwen_image_21/model_training/validate_full/Qwen-Image-2.1-Fun-Controlnet-Union.py) | [code](/examples/qwen_image_21/model_training/lora/Qwen-Image-2.1-Fun-Controlnet-Union.sh) | [code](/examples/qwen_image_21/model_training/validate_lora/Qwen-Image-2.1-Fun-Controlnet-Union.py) | | SenseNova-U1 | [SenseNova/SenseNova-U1.5-8B-MoT: T2I](https://www.modelscope.cn/models/SenseNova/SenseNova-U1.5-8B-MoT) | [code](/examples/sensenova_u1/model_inference/SenseNova-U1.5-8B-MoT.py) | [code](/examples/sensenova_u1/model_inference_low_vram/SenseNova-U1.5-8B-MoT.py) | [code](/examples/sensenova_u1/model_training/full/SenseNova-U1.5-8B-MoT.sh) | [code](/examples/sensenova_u1/model_training/validate_full/SenseNova-U1.5-8B-MoT.py) | [code](/examples/sensenova_u1/model_training/lora/SenseNova-U1.5-8B-MoT.sh) | [code](/examples/sensenova_u1/model_training/validate_lora/SenseNova-U1.5-8B-MoT.py) | | SenseNova-U1 | [SenseNova/SenseNova-U1.5-8B-MoT: Edit](https://www.modelscope.cn/models/SenseNova/SenseNova-U1.5-8B-MoT) | [code](/examples/sensenova_u1/model_inference/SenseNova-U1.5-8B-MoT-Edit.py) | [code](/examples/sensenova_u1/model_inference_low_vram/SenseNova-U1.5-8B-MoT-Edit.py) | [code](/examples/sensenova_u1/model_training/full/SenseNova-U1.5-8B-MoT-Edit.sh) | [code](/examples/sensenova_u1/model_training/validate_full/SenseNova-U1.5-8B-MoT-Edit.py) | [code](/examples/sensenova_u1/model_training/lora/SenseNova-U1.5-8B-MoT-Edit.sh) | [code](/examples/sensenova_u1/model_training/validate_lora/SenseNova-U1.5-8B-MoT-Edit.py) | | SenseNova-U1 | [SenseNova/SenseNova-U1.5-8B-MoT-LoRAs: 8-step](https://www.modelscope.cn/models/SenseNova/SenseNova-U1.5-8B-MoT-LoRAs) | [code](/examples/sensenova_u1/model_inference/SenseNova-U1.5-8B-MoT-LoRA-8step.py) | [code](/examples/sensenova_u1/model_inference_low_vram/SenseNova-U1.5-8B-MoT-LoRA-8step.py) | - | - | - | - | | SenseNova-U1 | [SenseNova/SenseNova-U1.5-8B-MoT-SFT: T2I](https://www.modelscope.cn/models/SenseNova/SenseNova-U1.5-8B-MoT-SFT) | [code](/examples/sensenova_u1/model_inference/SenseNova-U1.5-8B-MoT-SFT.py) | [code](/examples/sensenova_u1/model_inference_low_vram/SenseNova-U1.5-8B-MoT-SFT.py) | [code](/examples/sensenova_u1/model_training/full/SenseNova-U1.5-8B-MoT-SFT.sh) | [code](/examples/sensenova_u1/model_training/validate_full/SenseNova-U1.5-8B-MoT-SFT.py) | [code](/examples/sensenova_u1/model_training/lora/SenseNova-U1.5-8B-MoT-SFT.sh) | [code](/examples/sensenova_u1/model_training/validate_lora/SenseNova-U1.5-8B-MoT-SFT.py) | | SenseNova-U1 | [SenseNova/SenseNova-U1.5-8B-MoT-SFT: Edit](https://www.modelscope.cn/models/SenseNova/SenseNova-U1.5-8B-MoT-SFT) | [code](/examples/sensenova_u1/model_inference/SenseNova-U1.5-8B-MoT-SFT-Edit.py) | [code](/examples/sensenova_u1/model_inference_low_vram/SenseNova-U1.5-8B-MoT-SFT-Edit.py) | [code](/examples/sensenova_u1/model_training/full/SenseNova-U1.5-8B-MoT-SFT-Edit.sh) | [code](/examples/sensenova_u1/model_training/validate_full/SenseNova-U1.5-8B-MoT-SFT-Edit.py) | [code](/examples/sensenova_u1/model_training/lora/SenseNova-U1.5-8B-MoT-SFT-Edit.sh) | [code](/examples/sensenova_u1/model_training/validate_lora/SenseNova-U1.5-8B-MoT-SFT-Edit.py) | | MiniMax-Music3 | [MiniMax/MiniMax-Music3](https://www.modelscope.cn/models/MiniMax/MiniMax-Music3) | [code](/examples/minimax_music3/model_inference/MiniMax-Music3.py) | [code](/examples/minimax_music3/model_inference_low_vram/MiniMax-Music3.py) | — | — | — | — | | YuE2 | [m-a-p/YuE2-3B](https://www.modelscope.cn/models/m-a-p/YuE2-3B) | [code](/examples/yue2/model_inference/YuE2.py) | [code](/examples/yue2/model_inference_low_vram/YuE2.py) | — | — | — | — | | DiffSynth-Music | [DiffSynth-Studio/DiffSynth-Music](https://www.modelscope.cn/models/DiffSynth-Studio/DiffSynth-Music) | [code](/examples/diffsynth_music/model_inference/DiffSynth-Music.py) | [code](/examples/diffsynth_music/model_inference_low_vram/DiffSynth-Music.py) | [code](/examples/diffsynth_music/model_training/full/DiffSynth-Music.sh) | [code](/examples/diffsynth_music/model_training/validate_full/DiffSynth-Music.py) | - | - | | MiniMax-H3 | [MiniMax/MiniMax-H3: FL2VA](https://www.modelscope.cn/models/MiniMax/MiniMax-H3) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-FL2VA.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-FL2VA.py) | [code](/examples/minimax_h3/model_training/full/MiniMax-H3-FL2VA.sh) | [code](/examples/minimax_h3/model_training/validate_full/MiniMax-H3-FL2VA.py) | [code](/examples/minimax_h3/model_training/lora/MiniMax-H3-FL2VA.sh) | [code](/examples/minimax_h3/model_training/validate_lora/MiniMax-H3-FL2VA.py) | | MiniMax-H3 | [MiniMax/MiniMax-H3: Ref2VA](https://www.modelscope.cn/models/MiniMax/MiniMax-H3) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-Ref2VA.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-Ref2VA.py) | [code](/examples/minimax_h3/model_training/full/MiniMax-H3-Ref2VA.sh) | [code](/examples/minimax_h3/model_training/validate_full/MiniMax-H3-Ref2VA.py) | [code](/examples/minimax_h3/model_training/lora/MiniMax-H3-Ref2VA.sh) | [code](/examples/minimax_h3/model_training/validate_lora/MiniMax-H3-Ref2VA.py) | | MiniMax-H3 | [MiniMax/MiniMax-H3: Retake](https://www.modelscope.cn/models/MiniMax/MiniMax-H3) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-Retake.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-Retake.py) | - | - | - | - | | MiniMax-H3 | [DiffSynth-Studio/MiniMax-H3-NF4: FL2VA](https://www.modelscope.cn/models/DiffSynth-Studio/MiniMax-H3-NF4) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-NF4-FL2VA.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-NF4-FL2VA.py) | - | - | [code](/examples/minimax_h3/model_training/lora/MiniMax-H3-NF4-FL2VA.sh) | [code](/examples/minimax_h3/model_training/validate_lora/MiniMax-H3-NF4-FL2VA.py) | | MiniMax-H3 | [DiffSynth-Studio/MiniMax-H3-NF4: Ref2VA](https://www.modelscope.cn/models/DiffSynth-Studio/MiniMax-H3-NF4) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-NF4-Ref2VA.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-NF4-Ref2VA.py) | - | - | [code](/examples/minimax_h3/model_training/lora/MiniMax-H3-NF4-Ref2VA.sh) | [code](/examples/minimax_h3/model_training/validate_lora/MiniMax-H3-NF4-Ref2VA.py) | | MiniMax-H3 | [Comfy-Org/MiniMax-H3: FL2VA pruned](https://www.modelscope.cn/models/Comfy-Org/MiniMax-H3) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-Pruned-FL2VA.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-Pruned-FL2VA.py) | [code](/examples/minimax_h3/model_training/full/MiniMax-H3-Pruned-FL2VA.sh) | [code](/examples/minimax_h3/model_training/validate_full/MiniMax-H3-Pruned-FL2VA.py) | [code](/examples/minimax_h3/model_training/lora/MiniMax-H3-Pruned-FL2VA.sh) | [code](/examples/minimax_h3/model_training/validate_lora/MiniMax-H3-Pruned-FL2VA.py) | | MiniMax-H3 | [Comfy-Org/MiniMax-H3: Ref2VA pruned](https://www.modelscope.cn/models/Comfy-Org/MiniMax-H3) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-Pruned-Ref2VA.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-Pruned-Ref2VA.py) | [code](/examples/minimax_h3/model_training/full/MiniMax-H3-Pruned-Ref2VA.sh) | [code](/examples/minimax_h3/model_training/validate_full/MiniMax-H3-Pruned-Ref2VA.py) | [code](/examples/minimax_h3/model_training/lora/MiniMax-H3-Pruned-Ref2VA.sh) | [code](/examples/minimax_h3/model_training/validate_lora/MiniMax-H3-Pruned-Ref2VA.py) | | MiniMax-H3 | [DiffSynth-Studio/MiniMax-H3-NF4: FL2VA pruned](https://www.modelscope.cn/models/DiffSynth-Studio/MiniMax-H3-NF4) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-NF4-Pruned-FL2VA.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-NF4-Pruned-FL2VA.py) | - | - | [code](/examples/minimax_h3/model_training/lora/MiniMax-H3-NF4-Pruned-FL2VA.sh) | [code](/examples/minimax_h3/model_training/validate_lora/MiniMax-H3-NF4-Pruned-FL2VA.py) | | MiniMax-H3 | [DiffSynth-Studio/MiniMax-H3-NF4: Ref2VA pruned](https://www.modelscope.cn/models/DiffSynth-Studio/MiniMax-H3-NF4) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-NF4-Pruned-Ref2VA.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-NF4-Pruned-Ref2VA.py) | - | - | [code](/examples/minimax_h3/model_training/lora/MiniMax-H3-NF4-Pruned-Ref2VA.sh) | [code](/examples/minimax_h3/model_training/validate_lora/MiniMax-H3-NF4-Pruned-Ref2VA.py) | | MiniMax-H3 | [Comfy-Org/MiniMax-H3: FL2VA int8_convrot](https://www.modelscope.cn/models/Comfy-Org/MiniMax-H3) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-Int8-ConvRot-FL2VA.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-Int8-ConvRot-FL2VA.py) | - | - | [code](/examples/minimax_h3/model_training/lora/MiniMax-H3-Int8-ConvRot-FL2VA.sh) | [code](/examples/minimax_h3/model_training/validate_lora/MiniMax-H3-Int8-ConvRot-FL2VA.py) | | MiniMax-H3 | [Comfy-Org/MiniMax-H3: Ref2VA int8_convrot](https://www.modelscope.cn/models/Comfy-Org/MiniMax-H3) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-Int8-ConvRot-Ref2VA.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-Int8-ConvRot-Ref2VA.py) | - | - | [code](/examples/minimax_h3/model_training/lora/MiniMax-H3-Int8-ConvRot-Ref2VA.sh) | [code](/examples/minimax_h3/model_training/validate_lora/MiniMax-H3-Int8-ConvRot-Ref2VA.py) | | MiniMax-H3 | [Comfy-Org/MiniMax-H3: FL2VA pruned int8_convrot](https://www.modelscope.cn/models/Comfy-Org/MiniMax-H3) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-Int8-ConvRot-Pruned-FL2VA.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-Int8-ConvRot-Pruned-FL2VA.py) | - | - | [code](/examples/minimax_h3/model_training/lora/MiniMax-H3-Int8-ConvRot-Pruned-FL2VA.sh) | [code](/examples/minimax_h3/model_training/validate_lora/MiniMax-H3-Int8-ConvRot-Pruned-FL2VA.py) | | MiniMax-H3 | [Comfy-Org/MiniMax-H3: Ref2VA pruned int8_convrot](https://www.modelscope.cn/models/Comfy-Org/MiniMax-H3) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-Int8-ConvRot-Pruned-Ref2VA.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-Int8-ConvRot-Pruned-Ref2VA.py) | - | - | [code](/examples/minimax_h3/model_training/lora/MiniMax-H3-Int8-ConvRot-Pruned-Ref2VA.sh) | [code](/examples/minimax_h3/model_training/validate_lora/MiniMax-H3-Int8-ConvRot-Pruned-Ref2VA.py) | | MiniMax-H3 | [Comfy-Org/MiniMax-H3: FL2VA pruned fp8](https://www.modelscope.cn/models/Comfy-Org/MiniMax-H3) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-FP8-Pruned-FL2VA.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-FP8-Pruned-FL2VA.py) | - | - | [code](/examples/minimax_h3/model_training/lora/MiniMax-H3-FP8-Pruned-FL2VA.sh) | [code](/examples/minimax_h3/model_training/validate_lora/MiniMax-H3-FP8-Pruned-FL2VA.py) | | MiniMax-H3 | [Comfy-Org/MiniMax-H3: Ref2VA pruned fp8](https://www.modelscope.cn/models/Comfy-Org/MiniMax-H3) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-FP8-Pruned-Ref2VA.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-FP8-Pruned-Ref2VA.py) | - | - | [code](/examples/minimax_h3/model_training/lora/MiniMax-H3-FP8-Pruned-Ref2VA.sh) | [code](/examples/minimax_h3/model_training/validate_lora/MiniMax-H3-FP8-Pruned-Ref2VA.py) | | MiniMax-H3 | [lightx2v/Minimax-h3-Turbo: FL2VA 4steps](https://www.modelscope.cn/models/lightx2v/Minimax-h3-Turbo) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/minimax_h3/model_inference/MiniMax-H3-FL2VA-Turbo.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-FL2VA-Turbo.py) | - | - | - | - | | MiniMax-H3 | [DiffSynth-Studio/MiniMax-H3-Text-Embeddings](https://www.modelscope.cn/models/DiffSynth-Studio/MiniMax-H3-Text-Embeddings) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-Text-Embeddings.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-Text-Embeddings.py) | [code](/examples/minimax_h3/model_training/full/MiniMax-H3-Text-Embeddings.sh) | [code](/examples/minimax_h3/model_training/validate_full/MiniMax-H3-Text-Embeddings.py) | - | - | | MiniMax-H3 | [PAI/MiniMax-H3-Fun-Controlnet-Union](https://www.modelscope.cn/models/PAI/MiniMax-H3-Fun-Controlnet-Union) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-Fun-Controlnet-Union.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-Fun-Controlnet-Union.py) | [code](/examples/minimax_h3/model_training/full/MiniMax-H3-Fun-Controlnet-Union.sh) | [code](/examples/minimax_h3/model_training/validate_full/MiniMax-H3-Fun-Controlnet-Union.py) | [code](/examples/minimax_h3/model_training/lora/MiniMax-H3-Fun-Controlnet-Union.sh) | [code](/examples/minimax_h3/model_training/validate_lora/MiniMax-H3-Fun-Controlnet-Union.py) | | MiniMax-H3 | [OpenVDN/vdn-minimax-h3: VDN-H3 50 steps](https://www.modelscope.cn/models/OpenVDN/vdn-minimax-h3) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-VDN-FL2VA.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-VDN-FL2VA.py) | - | - | - | - | | MiniMax-H3 | [OpenVDN/vdn-minimax-h3: VDN-H3 8 steps](https://www.modelscope.cn/models/OpenVDN/vdn-minimax-h3) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-VDN-Turbo-FL2VA.py) | [code](/examples/minimax_h3/model_inference_low_vram/MiniMax-H3-VDN-Turbo-FL2VA.py) | - | - | - | - | | MiniMax-H3 | [OpenVDN/vdn-minimax-h3: VDN-H3 FP8](https://www.modelscope.cn/models/OpenVDN/vdn-minimax-h3) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-VDN-FP8-FL2VA.py) | - | - | - | - | - | | MiniMax-H3 | [OpenVDN/vdn-minimax-h3: VDN-H3 8 steps FP8](https://www.modelscope.cn/models/OpenVDN/vdn-minimax-h3) | [code](/examples/minimax_h3/model_inference/MiniMax-H3-VDN-Turbo-FP8-FL2VA.py) | - | - | - | - | - | | LingBot-Video | [Robbyant/lingbot-video-dense-1.3b: T2V](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b) | [code](/examples/lingbot_video/model_inference/lingbot-video-dense-1.3b_t2v.py) | [code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-dense-1.3b_t2v.py) | [code](/examples/lingbot_video/model_training/full/lingbot-video-dense-1.3b_t2v.sh) | [code](/examples/lingbot_video/model_training/validate_full/lingbot-video-dense-1.3b_t2v.py) | [code](/examples/lingbot_video/model_training/lora/lingbot-video-dense-1.3b_t2v.sh) | [code](/examples/lingbot_video/model_training/validate_lora/lingbot-video-dense-1.3b_t2v.py) | | LingBot-Video | [Robbyant/lingbot-video-dense-1.3b: TI2V](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b) | [code](/examples/lingbot_video/model_inference/lingbot-video-dense-1.3b_ti2v.py) | [code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-dense-1.3b_ti2v.py) | [code](/examples/lingbot_video/model_training/full/lingbot-video-dense-1.3b_ti2v.sh) | [code](/examples/lingbot_video/model_training/validate_full/lingbot-video-dense-1.3b_ti2v.py) | [code](/examples/lingbot_video/model_training/lora/lingbot-video-dense-1.3b_ti2v.sh) | [code](/examples/lingbot_video/model_training/validate_lora/lingbot-video-dense-1.3b_ti2v.py) | | LingBot-Video | [Robbyant/lingbot-video-dense-1.3b: T2I](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b) | [code](/examples/lingbot_video/model_inference/lingbot-video-dense-1.3b_t2i.py) | [code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-dense-1.3b_t2i.py) | - | - | - | - | | LingBot-Video | [Robbyant/lingbot-video-moe-30b-a3b: T2V](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b) | [code](/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2v.py) | [code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2v.py) | [code](/examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_t2v.sh) | [code](/examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_t2v.py) | [code](/examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_t2v.sh) | [code](/examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_t2v.py) | | LingBot-Video | [Robbyant/lingbot-video-moe-30b-a3b: TI2V](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b) | [code](/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_ti2v.py) | [code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_ti2v.py) | [code](/examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_ti2v.sh) | [code](/examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_ti2v.py) | [code](/examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_ti2v.sh) | [code](/examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_ti2v.py) | | LingBot-Video | [Robbyant/lingbot-video-moe-30b-a3b: T2I](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b) | [code](/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2i.py) | [code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2i.py) | - | - | - | - | | LingBot-Video | [Robbyant/lingbot-video-moe-30b-a3b: T2V + Refinement](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b) | [code](/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2v_refiner.py) | [code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2v_refiner.py) | - | - | - | - | | LingBot-Video | [Robbyant/lingbot-video-moe-30b-a3b: TI2V + Refinement](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b) | [code](/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_ti2v_refiner.py) | [code](/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_ti2v_refiner.py) | - | - | - | - | | ACE-Step | [ACE-Step/Ace-Step1.5](https://www.modelscope.cn/models/ACE-Step/Ace-Step1.5) | [code](/examples/ace_step/model_inference/Ace-Step1.5.py) | [code](/examples/ace_step/model_inference_low_vram/Ace-Step1.5.py) | [code](/examples/ace_step/model_training/full/Ace-Step1.5.sh) | [code](/examples/ace_step/model_training/validate_full/Ace-Step1.5.py) | [code](/examples/ace_step/model_training/lora/Ace-Step1.5.sh) | [code](/examples/ace_step/model_training/validate_lora/Ace-Step1.5.py) | | ACE-Step | [ACE-Step/acestep-v15-turbo-shift1](https://www.modelscope.cn/models/ACE-Step/acestep-v15-turbo-shift1) | [code](/examples/ace_step/model_inference/acestep-v15-turbo-shift1.py) | [code](/examples/ace_step/model_inference_low_vram/acestep-v15-turbo-shift1.py) | [code](/examples/ace_step/model_training/full/acestep-v15-turbo-shift1.sh) | [code](/examples/ace_step/model_training/validate_full/acestep-v15-turbo-shift1.py) | [code](/examples/ace_step/model_training/lora/acestep-v15-turbo-shift1.sh) | [code](/examples/ace_step/model_training/validate_lora/acestep-v15-turbo-shift1.py) | | ACE-Step | [ACE-Step/acestep-v15-turbo-shift3](https://www.modelscope.cn/models/ACE-Step/acestep-v15-turbo-shift3) | [code](/examples/ace_step/model_inference/acestep-v15-turbo-shift3.py) | [code](/examples/ace_step/model_inference_low_vram/acestep-v15-turbo-shift3.py) | [code](/examples/ace_step/model_training/full/acestep-v15-turbo-shift3.sh) | [code](/examples/ace_step/model_training/validate_full/acestep-v15-turbo-shift3.py) | [code](/examples/ace_step/model_training/lora/acestep-v15-turbo-shift3.sh) | [code](/examples/ace_step/model_training/validate_lora/acestep-v15-turbo-shift3.py) | | ACE-Step | [ACE-Step/acestep-v15-turbo-continuous](https://www.modelscope.cn/models/ACE-Step/acestep-v15-turbo-continuous) | [code](/examples/ace_step/model_inference/acestep-v15-turbo-continuous.py) | [code](/examples/ace_step/model_inference_low_vram/acestep-v15-turbo-continuous.py) | [code](/examples/ace_step/model_training/full/acestep-v15-turbo-continuous.sh) | [code](/examples/ace_step/model_training/validate_full/acestep-v15-turbo-continuous.py) | [code](/examples/ace_step/model_training/lora/acestep-v15-turbo-continuous.sh) | [code](/examples/ace_step/model_training/validate_lora/acestep-v15-turbo-continuous.py) | | ACE-Step | [ACE-Step/acestep-v15-base](https://www.modelscope.cn/models/ACE-Step/acestep-v15-base) | [code](/examples/ace_step/model_inference/acestep-v15-base.py) | [code](/examples/ace_step/model_inference_low_vram/acestep-v15-base.py) | [code](/examples/ace_step/model_training/full/acestep-v15-base.sh) | [code](/examples/ace_step/model_training/validate_full/acestep-v15-base.py) | [code](/examples/ace_step/model_training/lora/acestep-v15-base.sh) | [code](/examples/ace_step/model_training/validate_lora/acestep-v15-base.py) | | ACE-Step | [ACE-Step/acestep-v15-base: CoverTask](https://www.modelscope.cn/models/ACE-Step/acestep-v15-base) | [code](/examples/ace_step/model_inference/acestep-v15-base-CoverTask.py) | [code](/examples/ace_step/model_inference_low_vram/acestep-v15-base-CoverTask.py) | — | — | — | — | | ACE-Step | [ACE-Step/acestep-v15-base: RepaintTask](https://www.modelscope.cn/models/ACE-Step/acestep-v15-base) | [code](/examples/ace_step/model_inference/acestep-v15-base-RepaintTask.py) | [code](/examples/ace_step/model_inference_low_vram/acestep-v15-base-RepaintTask.py) | — | — | — | — | | ACE-Step | [ACE-Step/acestep-v15-sft](https://www.modelscope.cn/models/ACE-Step/acestep-v15-sft) | [code](/examples/ace_step/model_inference/acestep-v15-sft.py) | [code](/examples/ace_step/model_inference_low_vram/acestep-v15-sft.py) | [code](/examples/ace_step/model_training/full/acestep-v15-sft.sh) | [code](/examples/ace_step/model_training/validate_full/acestep-v15-sft.py) | [code](/examples/ace_step/model_training/lora/acestep-v15-sft.sh) | [code](/examples/ace_step/model_training/validate_lora/acestep-v15-sft.py) | | ACE-Step | [ACE-Step/acestep-v15-xl-base](https://www.modelscope.cn/models/ACE-Step/acestep-v15-xl-base) | [code](/examples/ace_step/model_inference/acestep-v15-xl-base.py) | [code](/examples/ace_step/model_inference_low_vram/acestep-v15-xl-base.py) | [code](/examples/ace_step/model_training/full/acestep-v15-xl-base.sh) | [code](/examples/ace_step/model_training/validate_full/acestep-v15-xl-base.py) | [code](/examples/ace_step/model_training/lora/acestep-v15-xl-base.sh) | [code](/examples/ace_step/model_training/validate_lora/acestep-v15-xl-base.py) | | ACE-Step | [ACE-Step/acestep-v15-xl-sft](https://www.modelscope.cn/models/ACE-Step/acestep-v15-xl-sft) | [code](/examples/ace_step/model_inference/acestep-v15-xl-sft.py) | [code](/examples/ace_step/model_inference_low_vram/acestep-v15-xl-sft.py) | [code](/examples/ace_step/model_training/full/acestep-v15-xl-sft.sh) | [code](/examples/ace_step/model_training/validate_full/acestep-v15-xl-sft.py) | [code](/examples/ace_step/model_training/lora/acestep-v15-xl-sft.sh) | [code](/examples/ace_step/model_training/validate_lora/acestep-v15-xl-sft.py) | | ACE-Step | [ACE-Step/acestep-v15-xl-turbo](https://www.modelscope.cn/models/ACE-Step/acestep-v15-xl-turbo) | [code](/examples/ace_step/model_inference/acestep-v15-xl-turbo.py) | [code](/examples/ace_step/model_inference_low_vram/acestep-v15-xl-turbo.py) | [code](/examples/ace_step/model_training/full/acestep-v15-xl-turbo.sh) | [code](/examples/ace_step/model_training/validate_full/acestep-v15-xl-turbo.py) | [code](/examples/ace_step/model_training/lora/acestep-v15-xl-turbo.sh) | [code](/examples/ace_step/model_training/validate_lora/acestep-v15-xl-turbo.py) | | ACE-Step | [DiffSynth-Studio/acestep15xlsft-lora-music](https://www.modelscope.cn/models/DiffSynth-Studio/acestep15xlsft-lora-music) | [code](/examples/ace_step/model_inference/acestep15xlsft-vocals2music.py) | [code](/examples/ace_step/model_inference_low_vram/acestep15xlsft-vocals2music.py) | [code](/examples/ace_step/model_training/full/acestep15xlsft-vocals2music.sh) | [code](/examples/ace_step/model_training/validate_full/acestep15xlsft-vocals2music.py) | - | - | | Boogu-Image | [Boogu/Boogu-Image-0.1-Base](https://modelscope.cn/models/Boogu/Boogu-Image-0.1-Base) | [code](/examples/boogu_image/model_inference/Boogu-Image-0.1-Base.py) | [code](/examples/boogu_image/model_inference_low_vram/Boogu-Image-0.1-Base.py) | [code](/examples/boogu_image/model_training/full/Boogu-Image-0.1-Base.sh) | [code](/examples/boogu_image/model_training/validate_full/Boogu-Image-0.1-Base.py) | [code](/examples/boogu_image/model_training/lora/Boogu-Image-0.1-Base.sh) | [code](/examples/boogu_image/model_training/validate_lora/Boogu-Image-0.1-Base.py) | | Boogu-Image | [Boogu/Boogu-Image-0.1-Turbo](https://modelscope.cn/models/Boogu/Boogu-Image-0.1-Turbo) | [code](/examples/boogu_image/model_inference/Boogu-Image-0.1-Turbo.py) | [code](/examples/boogu_image/model_inference_low_vram/Boogu-Image-0.1-Turbo.py) | [code](/examples/boogu_image/model_training/full/Boogu-Image-0.1-Turbo.sh) | [code](/examples/boogu_image/model_training/validate_full/Boogu-Image-0.1-Turbo.py) | [code](/examples/boogu_image/model_training/lora/Boogu-Image-0.1-Turbo.sh) | [code](/examples/boogu_image/model_training/validate_lora/Boogu-Image-0.1-Turbo.py) | | Boogu-Image | [Boogu/Boogu-Image-0.1-Edit](https://modelscope.cn/models/Boogu/Boogu-Image-0.1-Edit) | [code](/examples/boogu_image/model_inference/Boogu-Image-0.1-Edit.py) | [code](/examples/boogu_image/model_inference_low_vram/Boogu-Image-0.1-Edit.py) | [code](/examples/boogu_image/model_training/full/Boogu-Image-0.1-Edit.sh) | [code](/examples/boogu_image/model_training/validate_full/Boogu-Image-0.1-Edit.py) | [code](/examples/boogu_image/model_training/lora/Boogu-Image-0.1-Edit.sh) | [code](/examples/boogu_image/model_training/validate_lora/Boogu-Image-0.1-Edit.py) | | Krea-2 | [krea/Krea-2-Raw](https://www.modelscope.cn/models/krea/Krea-2-Raw) | [code](/examples/krea2/model_inference/Krea-2-Raw.py) | [code](/examples/krea2/model_inference_low_vram/Krea-2-Raw.py) | [code](/examples/krea2/model_training/full/Krea-2-Raw.sh) | [code](/examples/krea2/model_training/validate_full/Krea-2-Raw.py) | [code](/examples/krea2/model_training/lora/Krea-2-Raw.sh) | [code](/examples/krea2/model_training/validate_lora/Krea-2-Raw.py) | | Krea-2 | [krea/Krea-2-Turbo](https://www.modelscope.cn/models/krea/Krea-2-Turbo) | [code](/examples/krea2/model_inference/Krea-2-Turbo.py) | [code](/examples/krea2/model_inference_low_vram/Krea-2-Turbo.py) | [code](/examples/krea2/model_training/full/Krea-2-Turbo.sh) | [code](/examples/krea2/model_training/validate_full/Krea-2-Turbo.py) | [code](/examples/krea2/model_training/lora/Krea-2-Turbo.sh) | [code](/examples/krea2/model_training/validate_lora/Krea-2-Turbo.py) | | Ideogram 4 | [ideogram-ai/ideogram-4-fp8](https://www.modelscope.cn/models/ideogram-ai/ideogram-4-fp8) | [code](/examples/ideogram4/model_inference/ideogram-4-fp8.py) | - | - | - | - | - | | Ideogram 4 | [DiffSynth-Studio/ideogram-4-bf16-repackage](https://www.modelscope.cn/models/DiffSynth-Studio/ideogram-4-bf16-repackage) | [code](/examples/ideogram4/model_inference/ideogram-4-bf16-repackage.py) | [code](/examples/ideogram4/model_inference_low_vram/ideogram-4-bf16-repackage.py) | [code](/examples/ideogram4/model_training/full/Ideogram-4-bf16-repackage.sh) | - | [code](/examples/ideogram4/model_training/lora/Ideogram-4-bf16-repackage.sh) | [code](/examples/ideogram4/model_training/validate_lora/Ideogram-4-bf16-repackage.py) | | HiDream-O1-Image | [HiDream-ai/HiDream-O1-Image](https://modelscope.cn/models/HiDream-ai/HiDream-O1-Image) | [code](/examples/hidream_o1_image/model_inference/HiDream-O1-Image.py) | [code](/examples/hidream_o1_image/model_inference_low_vram/HiDream-O1-Image.py) | [code](/examples/hidream_o1_image/model_training/full/HiDream-O1-Image.sh) | [code](/examples/hidream_o1_image/model_training/validate_full/HiDream-O1-Image.py) | [code](/examples/hidream_o1_image/model_training/lora/HiDream-O1-Image.sh) | [code](/examples/hidream_o1_image/model_training/validate_lora/HiDream-O1-Image.py) | | HiDream-O1-Image | [HiDream-ai/HiDream-O1-Image-Dev](https://modelscope.cn/models/HiDream-ai/HiDream-O1-Image-Dev) | [code](/examples/hidream_o1_image/model_inference/HiDream-O1-Image-Dev.py) | [code](/examples/hidream_o1_image/model_inference_low_vram/HiDream-O1-Image-Dev.py) | [code](/examples/hidream_o1_image/model_training/full/HiDream-O1-Image-Dev.sh) | [code](/examples/hidream_o1_image/model_training/validate_full/HiDream-O1-Image-Dev.py) | [code](/examples/hidream_o1_image/model_training/lora/HiDream-O1-Image-Dev.sh) | [code](/examples/hidream_o1_image/model_training/validate_lora/HiDream-O1-Image-Dev.py) | | HiDream-O1-Image | [DiffSynth-Studio/HidreamO1-i2L-v2](https://www.modelscope.cn/models/DiffSynth-Studio/HidreamO1-i2L-v2) | [code](/examples/hidream_o1_image/model_inference/HidreamO1-i2L-v2.py) | [code](/examples/hidream_o1_image/model_inference_low_vram/HidreamO1-i2L-v2.py) | [code](/examples/hidream_o1_image/model_training/full/HidreamO1-i2L-v2.sh) | [code](/examples/hidream_o1_image/model_training/validate_full/HidreamO1-i2L-v2.py) | - | - | | JoyAI-Image | [jd-opensource/JoyAI-Image-Edit](https://modelscope.cn/models/jd-opensource/JoyAI-Image-Edit) | [code](/examples/joyai_image/model_inference/JoyAI-Image-Edit.py) | [code](/examples/joyai_image/model_inference_low_vram/JoyAI-Image-Edit.py) | [code](/examples/joyai_image/model_training/full/JoyAI-Image-Edit.sh) | [code](/examples/joyai_image/model_training/validate_full/JoyAI-Image-Edit.py) | [code](/examples/joyai_image/model_training/lora/JoyAI-Image-Edit.sh) | [code](/examples/joyai_image/model_training/validate_lora/JoyAI-Image-Edit.py) | | ERNIE-Image | [PaddlePaddle/ERNIE-Image](https://www.modelscope.cn/models/PaddlePaddle/ERNIE-Image) | [code](/examples/ernie_image/model_inference/ERNIE-Image.py) | [code](/examples/ernie_image/model_inference_low_vram/ERNIE-Image.py) | [code](/examples/ernie_image/model_training/full/ERNIE-Image.sh) | [code](/examples/ernie_image/model_training/validate_full/ERNIE-Image.py) | [code](/examples/ernie_image/model_training/lora/ERNIE-Image.sh) | [code](/examples/ernie_image/model_training/validate_lora/ERNIE-Image.py) | | ERNIE-Image | [PaddlePaddle/ERNIE-Image-Turbo](https://www.modelscope.cn/models/PaddlePaddle/ERNIE-Image-Turbo) | [code](/examples/ernie_image/model_inference/ERNIE-Image-Turbo.py) | [code](/examples/ernie_image/model_inference_low_vram/ERNIE-Image-Turbo.py) | — | — | — | — | | LTX-2.5 | [Lightricks/LTX-2.5: OneStagePipeline-T2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.5) | [code](/examples/ltx2/model_inference/LTX-2.5-T2AV-OneStage.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.5-T2AV-OneStage.py) | [code](/examples/ltx2/model_training/full/LTX-2.5-T2AV-splited.sh) | [code](/examples/ltx2/model_training/validate_full/LTX-2.5-T2AV.py) | [code](/examples/ltx2/model_training/lora/LTX-2.5-T2AV-splited.sh) | [code](/examples/ltx2/model_training/validate_lora/LTX-2.5-T2AV.py) | | LTX-2.5 | [Lightricks/LTX-2.5: TwoStagePipeline-T2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.5) | [code](/examples/ltx2/model_inference/LTX-2.5-T2AV-TwoStage.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.5-T2AV-TwoStage.py) | - | - | - | - | | LTX-2.5 | [Lightricks/LTX-2.5: OneStagePipeline-I2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.5) | [code](/examples/ltx2/model_inference/LTX-2.5-I2AV-OneStage.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.5-I2AV-OneStage.py) | - | - | - | - | | LTX-2.5 | [Lightricks/LTX-2.5: TwoStagePipeline-I2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.5) | [code](/examples/ltx2/model_inference/LTX-2.5-I2AV-TwoStage.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.5-I2AV-TwoStage.py) | - | - | - | - | | LTX-2.5 | [Lightricks/LTX-2.5: TwoStagePipeline-A2V](https://www.modelscope.cn/models/Lightricks/LTX-2.5) | [code](/examples/ltx2/model_inference/LTX-2.5-A2V-TwoStage.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.5-A2V-TwoStage.py) | - | - | - | - | | LTX-2.5 | [Lightricks/LTX-2.5: TwoStagePipeline-Retake](https://www.modelscope.cn/models/Lightricks/LTX-2.5) | [code](/examples/ltx2/model_inference/LTX-2.5-T2AV-TwoStage-Retake.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.5-T2AV-TwoStage-Retake.py) | - | - | - | - | | LTX-2.5 | [Lightricks/LTX-2.5: T2A](https://www.modelscope.cn/models/Lightricks/LTX-2.5) | [code](/examples/ltx2/model_inference/LTX-2.5-T2A.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.5-T2A.py) | - | - | - | - | | LTX-2.5 | [Lightricks/LTX-2.5: DistilledPipeline-T2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.5) | [code](/examples/ltx2/model_inference/LTX-2.5-T2AV-DistilledPipeline.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.5-T2AV-DistilledPipeline.py) | - | - | - | - | | LTX-2.5 | [Lightricks/LTX-2.5-22b-IC-LoRA-Pixel-Spatial-Upscaler](https://www.modelscope.cn/models/Lightricks/LTX-2.5-22b-IC-LoRA-Pixel-Spatial-Upscaler) | [code](/examples/ltx2/model_inference/LTX-2.5-IC-LoRA-Pixel-Spatial-Upscaler.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.5-IC-LoRA-Pixel-Spatial-Upscaler.py) | - | - | - | - | | LTX-2.5 | [Lightricks/LTX-2.5: INT8-ConvRot](https://www.modelscope.cn/models/Lightricks/LTX-2.5) | [code](/examples/ltx2/model_inference/LTX-2.5-T2AV-INT8-ConvRot.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.5-T2AV-INT8-ConvRot.py) | - | - | [code](/examples/ltx2/model_training/lora/LTX-2.5-Int8-ConvRot-splited.sh) | [code](/examples/ltx2/model_training/validate_lora/LTX-2.5-Int8-ConvRot.py) | | LTX-2 | [jd-opensource/JoyAI-Echo](https://modelscope.cn/models/jd-opensource/JoyAI-Echo) | [code](/examples/ltx2/model_inference/JoyAI-Echo-T2AV.py) | [code](/examples/ltx2/model_inference_low_vram/JoyAI-Echo-T2AV.py) | [code](/examples/ltx2/model_training/full/JoyAI-Echo-T2AV-splited.sh) | [code](/examples/ltx2/model_training/validate_full/JoyAI-Echo-T2AV.py) | [code](/examples/ltx2/model_training/lora/JoyAI-Echo-T2AV-splited.sh) | [code](/examples/ltx2/model_training/validate_lora/JoyAI-Echo-T2AV.py) | | LTX-2 | [Lightricks/LTX-2.3: OneStagePipeline-I2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.3) | [code](/examples/ltx2/model_inference/LTX-2.3-I2AV-OneStage.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.3-I2AV-OneStage.py) | [code](/examples/ltx2/model_training/full/LTX-2.3-I2AV-splited.sh) | [code](/examples/ltx2/model_training/validate_full/LTX-2.3-I2AV.py) | [code](/examples/ltx2/model_training/lora/LTX-2.3-I2AV-splited.sh) | [code](/examples/ltx2/model_training/validate_lora/LTX-2.3-I2AV.py) | | LTX-2 | [Lightricks/LTX-2.3: TwoStagePipeline-I2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.3) | [code](/examples/ltx2/model_inference/LTX-2.3-I2AV-TwoStage.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.3-I2AV-TwoStage.py) | - | - | - | - | | LTX-2 | [Lightricks/LTX-2.3: DistilledPipeline-I2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.3) | [code](/examples/ltx2/model_inference/LTX-2.3-I2AV-DistilledPipeline.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.3-I2AV-DistilledPipeline.py) | - | - | - | - | | LTX-2 | [Lightricks/LTX-2.3: OneStagePipeline-T2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.3) | [code](/examples/ltx2/model_inference/LTX-2.3-T2AV-OneStage.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.3-T2AV-OneStage.py) | [code](/examples/ltx2/model_training/full/LTX-2.3-T2AV-splited.sh) | [code](/examples/ltx2/model_training/validate_full/LTX-2.3-T2AV.py) | [code](/examples/ltx2/model_training/lora/LTX-2.3-T2AV-splited.sh) | [code](/examples/ltx2/model_training/validate_lora/LTX-2.3-T2AV.py) | | LTX-2 | [Lightricks/LTX-2.3: TwoStagePipeline-T2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.3) | [code](/examples/ltx2/model_inference/LTX-2.3-T2AV-TwoStage.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.3-T2AV-TwoStage.py) | - | - | - | - | | LTX-2 | [Lightricks/LTX-2.3: DistilledPipeline-T2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.3) | [code](/examples/ltx2/model_inference/LTX-2.3-T2AV-DistilledPipeline.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.3-T2AV-DistilledPipeline.py) | - | - | - | - | | LTX-2 | [Lightricks/LTX-2.3: A2V](https://www.modelscope.cn/models/Lightricks/LTX-2.3) | [code](/examples/ltx2/model_inference/LTX-2.3-A2V-TwoStage.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.3-A2V-TwoStage.py) | - | - | - | - | | LTX-2 | [Lightricks/LTX-2.3: Retake](https://www.modelscope.cn/models/Lightricks/LTX-2.3) | [code](/examples/ltx2/model_inference/LTX-2.3-T2AV-TwoStage-Retake.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.3-T2AV-TwoStage-Retake.py) | - | - | - | - | | LTX-2 | [Lightricks/LTX-2.3-22b-IC-LoRA-Union-Control](https://www.modelscope.cn/models/Lightricks/LTX-2.3-22b-IC-LoRA-Union-Control) | [code](/examples/ltx2/model_inference/LTX-2.3-T2AV-IC-LoRA-Union-Control.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.3-T2AV-IC-LoRA-Union-Control.py) | - | - | [code](/examples/ltx2/model_training/lora/LTX-2.3-T2AV-IC-LoRA-splited.sh) | [code](/examples/ltx2/model_training/validate_lora/LTX-2.3-T2AV-IC-LoRA.py) | | LTX-2 | [Lightricks/LTX-2.3-22b-IC-LoRA-Motion-Track-Control](https://www.modelscope.cn/models/Lightricks/LTX-2.3-22b-IC-LoRA-Motion-Track-Control) | [code](/examples/ltx2/model_inference/LTX-2.3-T2AV-IC-LoRA-Motion-Track-Control.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2.3-T2AV-IC-LoRA-Motion-Track-Control.py) | - | - | [code](/examples/ltx2/model_training/lora/LTX-2.3-T2AV-IC-LoRA-splited.sh) | [code](/examples/ltx2/model_training/validate_lora/LTX-2.3-T2AV-IC-LoRA.py) | | LTX-2 | [Lightricks/LTX-2: OneStagePipeline-T2AV](https://www.modelscope.cn/models/Lightricks/LTX-2) | [code](/examples/ltx2/model_inference/LTX-2-T2AV-OneStage.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-OneStage.py) | [code](/examples/ltx2/model_training/full/LTX-2-T2AV-splited.sh) | [code](/examples/ltx2/model_training/validate_full/LTX-2-T2AV.py) | [code](/examples/ltx2/model_training/lora/LTX-2-T2AV-splited.sh) | [code](/examples/ltx2/model_training/validate_lora/LTX-2-T2AV.py) | | LTX-2 | [Lightricks/LTX-2-19b-IC-LoRA-Union-Control](https://www.modelscope.cn/models/Lightricks/LTX-2-19b-IC-LoRA-Union-Control) | [code](/examples/ltx2/model_inference/LTX-2-T2AV-IC-LoRA-Union-Control.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-IC-LoRA-Union-Control.py) | - | - | [code](/examples/ltx2/model_training/lora/LTX-2-T2AV-IC-LoRA-splited.sh) | [code](/examples/ltx2/model_training/validate_lora/LTX-2-T2AV-IC-LoRA.py) | | LTX-2 | [Lightricks/LTX-2-19b-IC-LoRA-Detailer](https://www.modelscope.cn/models/Lightricks/LTX-2-19b-IC-LoRA-Detailer) | [code](/examples/ltx2/model_inference/LTX-2-T2AV-IC-LoRA-Detailer.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-IC-LoRA-Detailer.py) | - | - | [code](/examples/ltx2/model_training/lora/LTX-2-T2AV-IC-LoRA-splited.sh) | [code](/examples/ltx2/model_training/validate_lora/LTX-2-T2AV-IC-LoRA.py) | | LTX-2 | [Lightricks/LTX-2: TwoStagePipeline-T2AV](https://www.modelscope.cn/models/Lightricks/LTX-2) | [code](/examples/ltx2/model_inference/LTX-2-T2AV-TwoStage.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-TwoStage.py) | - | - | - | - | | LTX-2 | [Lightricks/LTX-2: DistilledPipeline-T2AV](https://www.modelscope.cn/models/Lightricks/LTX-2) | [code](/examples/ltx2/model_inference/LTX-2-T2AV-DistilledPipeline.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-DistilledPipeline.py) | - | - | - | - | | LTX-2 | [Lightricks/LTX-2: OneStagePipeline-I2AV](https://www.modelscope.cn/models/Lightricks/LTX-2) | [code](/examples/ltx2/model_inference/LTX-2-I2AV-OneStage.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2-I2AV-OneStage.py) | - | - | - | - | | LTX-2 | [Lightricks/LTX-2: TwoStagePipeline-I2AV](https://www.modelscope.cn/models/Lightricks/LTX-2) | [code](/examples/ltx2/model_inference/LTX-2-I2AV-TwoStage.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2-I2AV-TwoStage.py) | - | - | - | - | | LTX-2 | [Lightricks/LTX-2: DistilledPipeline-I2AV](https://www.modelscope.cn/models/Lightricks/LTX-2) | [code](/examples/ltx2/model_inference/LTX-2-I2AV-DistilledPipeline.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2-I2AV-DistilledPipeline.py) | - | - | - | - | | LTX-2 | [Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-In](https://www.modelscope.cn/models/Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-In) | [code](/examples/ltx2/model_inference/LTX-2-T2AV-Camera-Control-Dolly-In.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-Camera-Control-Dolly-In.py) | - | - | - | - | | LTX-2 | [Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-Out](https://www.modelscope.cn/models/Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-Out) | [code](/examples/ltx2/model_inference/LTX-2-T2AV-Camera-Control-Dolly-Out.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-Camera-Control-Dolly-Out.py) | - | - | - | - | | LTX-2 | [Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-Left](https://www.modelscope.cn/models/Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-Left) | [code](/examples/ltx2/model_inference/LTX-2-T2AV-Camera-Control-Dolly-Left.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-Camera-Control-Dolly-Left.py) | - | - | - | - | | LTX-2 | [Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-Right](https://www.modelscope.cn/models/Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-Right) | [code](/examples/ltx2/model_inference/LTX-2-T2AV-Camera-Control-Dolly-Right.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-Camera-Control-Dolly-Right.py) | - | - | - | - | | LTX-2 | [Lightricks/LTX-2-19b-LoRA-Camera-Control-Jib-Up](https://www.modelscope.cn/models/Lightricks/LTX-2-19b-LoRA-Camera-Control-Jib-Up) | [code](/examples/ltx2/model_inference/LTX-2-T2AV-Camera-Control-Jib-Up.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-Camera-Control-Jib-Up.py) | - | - | - | - | | LTX-2 | [Lightricks/LTX-2-19b-LoRA-Camera-Control-Jib-Down](https://www.modelscope.cn/models/Lightricks/LTX-2-19b-LoRA-Camera-Control-Jib-Down) | [code](/examples/ltx2/model_inference/LTX-2-T2AV-Camera-Control-Jib-Down.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-Camera-Control-Jib-Down.py) | - | - | - | - | | LTX-2 | [Lightricks/LTX-2-19b-LoRA-Camera-Control-Static](https://www.modelscope.cn/models/Lightricks/LTX-2-19b-LoRA-Camera-Control-Static) | [code](/examples/ltx2/model_inference/LTX-2-T2AV-Camera-Control-Static.py) | [code](/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-Camera-Control-Static.py) | - | - | - | - | | FLUX.2 | [black-forest-labs/FLUX.2-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.2-dev) | [code](/examples/flux2/model_inference/FLUX.2-dev.py) | [code](/examples/flux2/model_inference_low_vram/FLUX.2-dev.py) | - | - | [code](/examples/flux2/model_training/lora/FLUX.2-dev.sh) | [code](/examples/flux2/model_training/validate_lora/FLUX.2-dev.py) | | FLUX.2 | [black-forest-labs/FLUX.2-klein-4B](https://www.modelscope.cn/models/black-forest-labs/FLUX.2-klein-4B) | [code](/examples/flux2/model_inference/FLUX.2-klein-4B.py) | [code](/examples/flux2/model_inference_low_vram/FLUX.2-klein-4B.py) | [code](/examples/flux2/model_training/full/FLUX.2-klein-4B.sh) | [code](/examples/flux2/model_training/validate_full/FLUX.2-klein-4B.py) | [code](/examples/flux2/model_training/lora/FLUX.2-klein-4B.sh) | [code](/examples/flux2/model_training/validate_lora/FLUX.2-klein-4B.py) | | FLUX.2 | [black-forest-labs/FLUX.2-klein-9B](https://www.modelscope.cn/models/black-forest-labs/FLUX.2-klein-9B) | [code](/examples/flux2/model_inference/FLUX.2-klein-9B.py) | [code](/examples/flux2/model_inference_low_vram/FLUX.2-klein-9B.py) | [code](/examples/flux2/model_training/full/FLUX.2-klein-9B.sh) | [code](/examples/flux2/model_training/validate_full/FLUX.2-klein-9B.py) | [code](/examples/flux2/model_training/lora/FLUX.2-klein-9B.sh) | [code](/examples/flux2/model_training/validate_lora/FLUX.2-klein-9B.py) | | FLUX.2 | [black-forest-labs/FLUX.2-klein-base-4B](https://www.modelscope.cn/models/black-forest-labs/FLUX.2-klein-base-4B) | [code](/examples/flux2/model_inference/FLUX.2-klein-base-4B.py) | [code](/examples/flux2/model_inference_low_vram/FLUX.2-klein-base-4B.py) | [code](/examples/flux2/model_training/full/FLUX.2-klein-base-4B.sh) | [code](/examples/flux2/model_training/validate_full/FLUX.2-klein-base-4B.py) | [code](/examples/flux2/model_training/lora/FLUX.2-klein-base-4B.sh) | [code](/examples/flux2/model_training/validate_lora/FLUX.2-klein-base-4B.py) | | FLUX.2 | [black-forest-labs/FLUX.2-klein-base-9B](https://www.modelscope.cn/models/black-forest-labs/FLUX.2-klein-base-9B) | [code](/examples/flux2/model_inference/FLUX.2-klein-base-9B.py) | [code](/examples/flux2/model_inference_low_vram/FLUX.2-klein-base-9B.py) | [code](/examples/flux2/model_training/full/FLUX.2-klein-base-9B.sh) | [code](/examples/flux2/model_training/validate_full/FLUX.2-klein-base-9B.py) | [code](/examples/flux2/model_training/lora/FLUX.2-klein-base-9B.sh) | [code](/examples/flux2/model_training/validate_lora/FLUX.2-klein-base-9B.py) | | FLUX.2 | [DiffSynth-Studio/Template-KleinBase4B-Aesthetic](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Aesthetic) | [code](/examples/flux2/model_inference/Template-KleinBase4B-Aesthetic.py) | [code](/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Aesthetic.py) | [code](/examples/flux2/model_training/full/Template-KleinBase4B-Aesthetic.sh) | [code](/examples/flux2/model_training/validate_full/Template-KleinBase4B-Aesthetic.py) | - | - | | FLUX.2 | [DiffSynth-Studio/Template-KleinBase4B-Brightness](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Brightness) | [code](/examples/flux2/model_inference/Template-KleinBase4B-Brightness.py) | [code](/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Brightness.py) | [code](/examples/flux2/model_training/full/Template-KleinBase4B-Brightness.sh) | [code](/examples/flux2/model_training/validate_full/Template-KleinBase4B-Brightness.py) | - | - | | FLUX.2 | [DiffSynth-Studio/Template-KleinBase4B-Age](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Age) | [code](/examples/flux2/model_inference/Template-KleinBase4B-Age.py) | [code](/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Age.py) | [code](/examples/flux2/model_training/full/Template-KleinBase4B-Age.sh) | [code](/examples/flux2/model_training/validate_full/Template-KleinBase4B-Age.py) | - | - | | FLUX.2 | [DiffSynth-Studio/Template-KleinBase4B-ControlNet](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-ControlNet) | [code](/examples/flux2/model_inference/Template-KleinBase4B-ControlNet.py) | [code](/examples/flux2/model_inference_low_vram/Template-KleinBase4B-ControlNet.py) | [code](/examples/flux2/model_training/full/Template-KleinBase4B-ControlNet.sh) | [code](/examples/flux2/model_training/validate_full/Template-KleinBase4B-ControlNet.py) | - | - | | FLUX.2 | [DiffSynth-Studio/Template-KleinBase4B-Edit](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Edit) | [code](/examples/flux2/model_inference/Template-KleinBase4B-Edit.py) | [code](/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Edit.py) | [code](/examples/flux2/model_training/full/Template-KleinBase4B-Edit.sh) | [code](/examples/flux2/model_training/validate_full/Template-KleinBase4B-Edit.py) | - | - | | FLUX.2 | [DiffSynth-Studio/Template-KleinBase4B-Inpaint](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Inpaint) | [code](/examples/flux2/model_inference/Template-KleinBase4B-Inpaint.py) | [code](/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Inpaint.py) | [code](/examples/flux2/model_training/full/Template-KleinBase4B-Inpaint.sh) | [code](/examples/flux2/model_training/validate_full/Template-KleinBase4B-Inpaint.py) | - | - | | FLUX.2 | [DiffSynth-Studio/Template-KleinBase4B-PandaMeme](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-PandaMeme) | [code](/examples/flux2/model_inference/Template-KleinBase4B-PandaMeme.py) | [code](/examples/flux2/model_inference_low_vram/Template-KleinBase4B-PandaMeme.py) | [code](/examples/flux2/model_training/full/Template-KleinBase4B-PandaMeme.sh) | [code](/examples/flux2/model_training/validate_full/Template-KleinBase4B-PandaMeme.py) | - | - | | FLUX.2 | [DiffSynth-Studio/Template-KleinBase4B-Sharpness](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Sharpness) | [code](/examples/flux2/model_inference/Template-KleinBase4B-Sharpness.py) | [code](/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Sharpness.py) | [code](/examples/flux2/model_training/full/Template-KleinBase4B-Sharpness.sh) | [code](/examples/flux2/model_training/validate_full/Template-KleinBase4B-Sharpness.py) | - | - | | FLUX.2 | [DiffSynth-Studio/Template-KleinBase4B-SoftRGB](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-SoftRGB) | [code](/examples/flux2/model_inference/Template-KleinBase4B-SoftRGB.py) | [code](/examples/flux2/model_inference_low_vram/Template-KleinBase4B-SoftRGB.py) | [code](/examples/flux2/model_training/full/Template-KleinBase4B-SoftRGB.sh) | [code](/examples/flux2/model_training/validate_full/Template-KleinBase4B-SoftRGB.py) | - | - | | FLUX.2 | [DiffSynth-Studio/Template-KleinBase4B-Upscaler](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Upscaler) | [code](/examples/flux2/model_inference/Template-KleinBase4B-Upscaler.py) | [code](/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Upscaler.py) | [code](/examples/flux2/model_training/full/Template-KleinBase4B-Upscaler.sh) | [code](/examples/flux2/model_training/validate_full/Template-KleinBase4B-Upscaler.py) | - | - | | FLUX.2 | [DiffSynth-Studio/Template-KleinBase4B-ContentRef](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-ContentRef) | [code](/examples/flux2/model_inference/Template-KleinBase4B-ContentRef.py) | [code](/examples/flux2/model_inference_low_vram/Template-KleinBase4B-ContentRef.py) | [code](/examples/flux2/model_training/full/Template-KleinBase4B-ContentRef.sh) | [code](/examples/flux2/model_training/validate_full/Template-KleinBase4B-ContentRef.py) | - | - | | FLUX.2 | [DiffSynth-Studio/KleinBase4B-i2L-v2](https://www.modelscope.cn/models/DiffSynth-Studio/KleinBase4B-i2L-v2) | [code](/examples/flux2/model_inference/KleinBase4B-i2L-v2.py) | [code](/examples/flux2/model_inference_low_vram/KleinBase4B-i2L-v2.py) | [code](/examples/flux2/model_training/full/KleinBase4B-i2L-v2.sh) | [code](/examples/flux2/model_training/validate_full/KleinBase4B-i2L-v2.py) | - | - | | Z-Image | [Tongyi-MAI/Z-Image](https://www.modelscope.cn/models/Tongyi-MAI/Z-Image) | [code](/examples/z_image/model_inference/Z-Image.py) | [code](/examples/z_image/model_inference_low_vram/Z-Image.py) | [code](/examples/z_image/model_training/full/Z-Image.sh) | [code](/examples/z_image/model_training/validate_full/Z-Image.py) | [code](/examples/z_image/model_training/lora/Z-Image.sh) | [code](/examples/z_image/model_training/validate_lora/Z-Image.py) | | Z-Image | [DiffSynth-Studio/Z-Image-i2L](https://www.modelscope.cn/models/DiffSynth-Studio/Z-Image-i2L) | [code](/examples/z_image/model_inference/Z-Image-i2L.py) | [code](/examples/z_image/model_inference_low_vram/Z-Image-i2L.py) | - | - | - | - | | Z-Image | [Tongyi-MAI/Z-Image-Turbo](https://www.modelscope.cn/models/Tongyi-MAI/Z-Image-Turbo) | [code](/examples/z_image/model_inference/Z-Image-Turbo.py) | [code](/examples/z_image/model_inference_low_vram/Z-Image-Turbo.py) | [code](/examples/z_image/model_training/full/Z-Image-Turbo.sh) | [code](/examples/z_image/model_training/validate_full/Z-Image-Turbo.py) | [code](/examples/z_image/model_training/lora/Z-Image-Turbo.sh) | [code](/examples/z_image/model_training/validate_lora/Z-Image-Turbo.py) | | Z-Image | [PAI/Z-Image-Turbo-Fun-Controlnet-Union-2.1](https://www.modelscope.cn/models/PAI/Z-Image-Turbo-Fun-Controlnet-Union-2.1) | [code](/examples/z_image/model_inference/Z-Image-Turbo-Fun-Controlnet-Union-2.1.py) | [code](/examples/z_image/model_inference_low_vram/Z-Image-Turbo-Fun-Controlnet-Union-2.1.py) | [code](/examples/z_image/model_training/full/Z-Image-Turbo-Fun-Controlnet-Union-2.1.sh) | [code](/examples/z_image/model_training/validate_full/Z-Image-Turbo-Fun-Controlnet-Union-2.1.py) | [code](/examples/z_image/model_training/lora/Z-Image-Turbo-Fun-Controlnet-Union-2.1.sh) | [code](/examples/z_image/model_training/validate_lora/Z-Image-Turbo-Fun-Controlnet-Union-2.1.py) | | Z-Image | [PAI/Z-Image-Turbo-Fun-Controlnet-Union-2.1-8steps](https://www.modelscope.cn/models/PAI/Z-Image-Turbo-Fun-Controlnet-Union-2.1) | [code](/examples/z_image/model_inference/Z-Image-Turbo-Fun-Controlnet-Union-2.1-8steps.py) | [code](/examples/z_image/model_inference_low_vram/Z-Image-Turbo-Fun-Controlnet-Union-2.1-8steps.py) | [code](/examples/z_image/model_training/full/Z-Image-Turbo-Fun-Controlnet-Union-2.1-8steps.sh) | [code](/examples/z_image/model_training/validate_full/Z-Image-Turbo-Fun-Controlnet-Union-2.1-8steps.py) | [code](/examples/z_image/model_training/lora/Z-Image-Turbo-Fun-Controlnet-Union-2.1-8steps.sh) | [code](/examples/z_image/model_training/validate_lora/Z-Image-Turbo-Fun-Controlnet-Union-2.1-8steps.py) | | Z-Image | [PAI/Z-Image-Turbo-Fun-Controlnet-Tile-2.1-8steps](https://www.modelscope.cn/models/PAI/Z-Image-Turbo-Fun-Controlnet-Union-2.1) | [code](/examples/z_image/model_inference/Z-Image-Turbo-Fun-Controlnet-Tile-2.1-8steps.py) | [code](/examples/z_image/model_inference_low_vram/Z-Image-Turbo-Fun-Controlnet-Tile-2.1-8steps.py) | [code](/examples/z_image/model_training/full/Z-Image-Turbo-Fun-Controlnet-Tile-2.1-8steps.sh) | [code](/examples/z_image/model_training/validate_full/Z-Image-Turbo-Fun-Controlnet-Tile-2.1-8steps.py) | [code](/examples/z_image/model_training/lora/Z-Image-Turbo-Fun-Controlnet-Tile-2.1-8steps.sh) | [code](/examples/z_image/model_training/validate_lora/Z-Image-Turbo-Fun-Controlnet-Tile-2.1-8steps.py) | | Z-Image | [DiffSynth-Studio/ZImage-i2L-v2](https://www.modelscope.cn/models/DiffSynth-Studio/ZImage-i2L-v2) | [code](/examples/z_image/model_inference/ZImage-i2L-v2.py) | [code](/examples/z_image/model_inference_low_vram/ZImage-i2L-v2.py) | [code](/examples/z_image/model_training/full/ZImage-i2L-v2.sh) | [code](/examples/z_image/model_training/validate_full/ZImage-i2L-v2.py) | - | - | | Anima | [circlestone-labs/Anima](https://www.modelscope.cn/models/circlestone-labs/Anima) | [code](/examples/anima/model_inference/anima-preview.py) | [code](/examples/anima/model_inference_low_vram/anima-preview.py) | [code](/examples/anima/model_training/full/anima-preview.sh) | [code](/examples/anima/model_training/validate_full/anima-preview.py) | [code](/examples/anima/model_training/lora/anima-preview.sh) | [code](/examples/anima/model_training/validate_lora/anima-preview.py) | | Qwen-Image | [Qwen/Qwen-Image](https://www.modelscope.cn/models/Qwen/Qwen-Image) | [code](/examples/qwen_image/model_inference/Qwen-Image.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image.py) | [code](/examples/qwen_image/model_training/full/Qwen-Image.sh) | [code](/examples/qwen_image/model_training/validate_full/Qwen-Image.py) | [code](/examples/qwen_image/model_training/lora/Qwen-Image.sh) | [code](/examples/qwen_image/model_training/validate_lora/Qwen-Image.py) | | Qwen-Image | [Qwen/Qwen-Image-2512](https://www.modelscope.cn/models/Qwen/Qwen-Image-2512) | [code](/examples/qwen_image/model_inference/Qwen-Image-2512.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-2512.py) | [code](/examples/qwen_image/model_training/full/Qwen-Image-2512.sh) | [code](/examples/qwen_image/model_training/validate_full/Qwen-Image-2512.py) | [code](/examples/qwen_image/model_training/lora/Qwen-Image-2512.sh) | [code](/examples/qwen_image/model_training/validate_lora/Qwen-Image-2512.py) | | Qwen-Image | [Qwen/Qwen-Image-Edit](https://www.modelscope.cn/models/Qwen/Qwen-Image-Edit) | [code](/examples/qwen_image/model_inference/Qwen-Image-Edit.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-Edit.py) | [code](/examples/qwen_image/model_training/full/Qwen-Image-Edit.sh) | [code](/examples/qwen_image/model_training/validate_full/Qwen-Image-Edit.py) | [code](/examples/qwen_image/model_training/lora/Qwen-Image-Edit.sh) | [code](/examples/qwen_image/model_training/validate_lora/Qwen-Image-Edit.py) | | Qwen-Image | [Qwen/Qwen-Image-Edit-2509](https://www.modelscope.cn/models/Qwen/Qwen-Image-Edit-2509) | [code](/examples/qwen_image/model_inference/Qwen-Image-Edit-2509.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-Edit-2509.py) | [code](/examples/qwen_image/model_training/full/Qwen-Image-Edit-2509.sh) | [code](/examples/qwen_image/model_training/validate_full/Qwen-Image-Edit-2509.py) | [code](/examples/qwen_image/model_training/lora/Qwen-Image-Edit-2509.sh) | [code](/examples/qwen_image/model_training/validate_lora/Qwen-Image-Edit-2509.py) | | Qwen-Image | [Qwen/Qwen-Image-Edit-2511](https://www.modelscope.cn/models/Qwen/Qwen-Image-Edit-2511) | [code](/examples/qwen_image/model_inference/Qwen-Image-Edit-2511.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-Edit-2511.py) | [code](/examples/qwen_image/model_training/full/Qwen-Image-Edit-2511.sh) | [code](/examples/qwen_image/model_training/validate_full/Qwen-Image-Edit-2511.py) | [code](/examples/qwen_image/model_training/lora/Qwen-Image-Edit-2511.sh) | [code](/examples/qwen_image/model_training/validate_lora/Qwen-Image-Edit-2511.py) | | Qwen-Image | [FireRedTeam/FireRed-Image-Edit-1.0](https://www.modelscope.cn/models/FireRedTeam/FireRed-Image-Edit-1.0) | [code](/examples/qwen_image/model_inference/FireRed-Image-Edit-1.0.py) | [code](/examples/qwen_image/model_inference_low_vram/FireRed-Image-Edit-1.0.py) | [code](/examples/qwen_image/model_training/full/FireRed-Image-Edit-1.0.sh) | [code](/examples/qwen_image/model_training/validate_full/FireRed-Image-Edit-1.0.py) | [code](/examples/qwen_image/model_training/lora/FireRed-Image-Edit-1.0.sh) | [code](/examples/qwen_image/model_training/validate_lora/FireRed-Image-Edit-1.0.py) | | Qwen-Image | [FireRedTeam/FireRed-Image-Edit-1.1](https://www.modelscope.cn/models/FireRedTeam/FireRed-Image-Edit-1.1) | [code](/examples/qwen_image/model_inference/FireRed-Image-Edit-1.1.py) | [code](/examples/qwen_image/model_inference_low_vram/FireRed-Image-Edit-1.1.py) | [code](/examples/qwen_image/model_training/full/FireRed-Image-Edit-1.1.sh) | [code](/examples/qwen_image/model_training/validate_full/FireRed-Image-Edit-1.1.py) | [code](/examples/qwen_image/model_training/lora/FireRed-Image-Edit-1.1.sh) | [code](/examples/qwen_image/model_training/validate_lora/FireRed-Image-Edit-1.1.py) | | Qwen-Image | [lightx2v/Qwen-Image-Edit-2511-Lightning](https://modelscope.cn/models/lightx2v/Qwen-Image-Edit-2511-Lightning) | [code](/examples/qwen_image/model_inference/Qwen-Image-Edit-2511-Lightning.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-Edit-2511-Lightning.py) | - | - | - | - | | Qwen-Image | [Qwen/Qwen-Image-Layered](https://www.modelscope.cn/models/Qwen/Qwen-Image-Layered) | [code](/examples/qwen_image/model_inference/Qwen-Image-Layered.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-Layered.py) | [code](/examples/qwen_image/model_training/full/Qwen-Image-Layered.sh) | [code](/examples/qwen_image/model_training/validate_full/Qwen-Image-Layered.py) | [code](/examples/qwen_image/model_training/lora/Qwen-Image-Layered.sh) | [code](/examples/qwen_image/model_training/validate_lora/Qwen-Image-Layered.py) | | Qwen-Image | [DiffSynth-Studio/Qwen-Image-Layered-Control](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Layered-Control) | [code](/examples/qwen_image/model_inference/Qwen-Image-Layered-Control.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-Layered-Control.py) | [code](/examples/qwen_image/model_training/full/Qwen-Image-Layered-Control.sh) | [code](/examples/qwen_image/model_training/validate_full/Qwen-Image-Layered-Control.py) | [code](/examples/qwen_image/model_training/lora/Qwen-Image-Layered-Control.sh) | [code](/examples/qwen_image/model_training/validate_lora/Qwen-Image-Layered-Control.py) | | Qwen-Image | [DiffSynth-Studio/Qwen-Image-Layered-Control-V2](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Layered-Control-V2) | [code](/examples/qwen_image/model_inference/Qwen-Image-Layered-Control-V2.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-Layered-Control-V2.py) | - | - | [code](/examples/qwen_image/model_training/lora/Qwen-Image-Layered-Control-V2.sh) | [code](/examples/qwen_image/model_training/validate_lora/Qwen-Image-Layered-Control-V2.py) | | Qwen-Image | [DiffSynth-Studio/Qwen-Image-EliGen](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-EliGen) | [code](/examples/qwen_image/model_inference/Qwen-Image-EliGen.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-EliGen.py) | - | - | [code](/examples/qwen_image/model_training/lora/Qwen-Image-EliGen.sh) | [code](/examples/qwen_image/model_training/validate_lora/Qwen-Image-EliGen.py) | | Qwen-Image | [DiffSynth-Studio/Qwen-Image-EliGen-V2](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-EliGen-V2) | [code](/examples/qwen_image/model_inference/Qwen-Image-EliGen-V2.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-EliGen-V2.py) | - | - | [code](/examples/qwen_image/model_training/lora/Qwen-Image-EliGen.sh) | [code](/examples/qwen_image/model_training/validate_lora/Qwen-Image-EliGen.py) | | Qwen-Image | [DiffSynth-Studio/Qwen-Image-EliGen-Poster](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-EliGen-Poster) | [code](/examples/qwen_image/model_inference/Qwen-Image-EliGen-Poster.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-EliGen-Poster.py) | - | - | [code](/examples/qwen_image/model_training/lora/Qwen-Image-EliGen-Poster.sh) | [code](/examples/qwen_image/model_training/validate_lora/Qwen-Image-EliGen-Poster.py) | | Qwen-Image | [DiffSynth-Studio/Qwen-Image-Distill-Full](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Distill-Full) | [code](/examples/qwen_image/model_inference/Qwen-Image-Distill-Full.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-Distill-Full.py) | [code](/examples/qwen_image/model_training/full/Qwen-Image-Distill-Full.sh) | [code](/examples/qwen_image/model_training/validate_full/Qwen-Image-Distill-Full.py) | [code](/examples/qwen_image/model_training/lora/Qwen-Image-Distill-Full.sh) | [code](/examples/qwen_image/model_training/validate_lora/Qwen-Image-Distill-Full.py) | | Qwen-Image | [DiffSynth-Studio/Qwen-Image-Distill-LoRA](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Distill-LoRA) | [code](/examples/qwen_image/model_inference/Qwen-Image-Distill-LoRA.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-Distill-LoRA.py) | - | - | [code](/examples/qwen_image/model_training/lora/Qwen-Image-Distill-LoRA.sh) | [code](/examples/qwen_image/model_training/validate_lora/Qwen-Image-Distill-LoRA.py) | | Qwen-Image | [DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Canny](https://modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Canny) | [code](/examples/qwen_image/model_inference/Qwen-Image-Blockwise-ControlNet-Canny.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-Blockwise-ControlNet-Canny.py) | [code](/examples/qwen_image/model_training/full/Qwen-Image-Blockwise-ControlNet-Canny.sh) | [code](/examples/qwen_image/model_training/validate_full/Qwen-Image-Blockwise-ControlNet-Canny.py) | [code](/examples/qwen_image/model_training/lora/Qwen-Image-Blockwise-ControlNet-Canny.sh) | [code](/examples/qwen_image/model_training/validate_lora/Qwen-Image-Blockwise-ControlNet-Canny.py) | | Qwen-Image | [DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Depth](https://modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Depth) | [code](/examples/qwen_image/model_inference/Qwen-Image-Blockwise-ControlNet-Depth.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-Blockwise-ControlNet-Depth.py) | [code](/examples/qwen_image/model_training/full/Qwen-Image-Blockwise-ControlNet-Depth.sh) | [code](/examples/qwen_image/model_training/validate_full/Qwen-Image-Blockwise-ControlNet-Depth.py) | [code](/examples/qwen_image/model_training/lora/Qwen-Image-Blockwise-ControlNet-Depth.sh) | [code](/examples/qwen_image/model_training/validate_lora/Qwen-Image-Blockwise-ControlNet-Depth.py) | | Qwen-Image | [DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Inpaint](https://modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Inpaint) | [code](/examples/qwen_image/model_inference/Qwen-Image-Blockwise-ControlNet-Inpaint.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-Blockwise-ControlNet-Inpaint.py) | [code](/examples/qwen_image/model_training/full/Qwen-Image-Blockwise-ControlNet-Inpaint.sh) | [code](/examples/qwen_image/model_training/validate_full/Qwen-Image-Blockwise-ControlNet-Inpaint.py) | [code](/examples/qwen_image/model_training/lora/Qwen-Image-Blockwise-ControlNet-Inpaint.sh) | [code](/examples/qwen_image/model_training/validate_lora/Qwen-Image-Blockwise-ControlNet-Inpaint.py) | | Qwen-Image | [DiffSynth-Studio/Qwen-Image-In-Context-Control-Union](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-In-Context-Control-Union) | [code](/examples/qwen_image/model_inference/Qwen-Image-In-Context-Control-Union.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-In-Context-Control-Union.py) | - | - | [code](/examples/qwen_image/model_training/lora/Qwen-Image-In-Context-Control-Union.sh) | [code](/examples/qwen_image/model_training/validate_lora/Qwen-Image-In-Context-Control-Union.py) | | Qwen-Image | [DiffSynth-Studio/Qwen-Image-Edit-Lowres-Fix](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Edit-Lowres-Fix) | [code](/examples/qwen_image/model_inference/Qwen-Image-Edit-Lowres-Fix.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-Edit-Lowres-Fix.py) | - | - | - | - | | Qwen-Image | [DiffSynth-Studio/Qwen-Image-i2L](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-i2L) | [code](/examples/qwen_image/model_inference/Qwen-Image-i2L.py) | [code](/examples/qwen_image/model_inference_low_vram/Qwen-Image-i2L.py) | - | - | - | - | | Qwen-Video-Edit | [yunpeng1998/Qwen-Video-Edit](https://www.modelscope.cn/models/yunpeng1998/Qwen-Video-Edit) | [code](/examples/qwen_video_edit/model_inference/Qwen-Video-Edit.py) | [code](/examples/qwen_video_edit/model_inference_low_vram/Qwen-Video-Edit.py) | [code](/examples/qwen_video_edit/model_training/full/Qwen-Video-Edit.sh) | [code](/examples/qwen_video_edit/model_training/validate_full/Qwen-Video-Edit.py) | [code](/examples/qwen_video_edit/model_training/lora/Qwen-Video-Edit.sh) | [code](/examples/qwen_video_edit/model_training/validate_lora/Qwen-Video-Edit.py) | | Wan | [Wan-AI/Wan2.1-T2V-1.3B](https://modelscope.cn/models/Wan-AI/Wan2.1-T2V-1.3B) | [code](/examples/wanvideo/model_inference/Wan2.1-T2V-1.3B.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-T2V-1.3B.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-T2V-1.3B.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-T2V-1.3B.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-T2V-1.3B.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-T2V-1.3B.py) | | Wan | [Wan-AI/Wan2.1-T2V-14B](https://modelscope.cn/models/Wan-AI/Wan2.1-T2V-14B) | [code](/examples/wanvideo/model_inference/Wan2.1-T2V-14B.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-T2V-14B.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-T2V-14B.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-T2V-14B.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-T2V-14B.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-T2V-14B.py) | | Wan | [Wan-AI/Wan2.1-I2V-14B-480P](https://modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-480P) | [code](/examples/wanvideo/model_inference/Wan2.1-I2V-14B-480P.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-I2V-14B-480P.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-I2V-14B-480P.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-I2V-14B-480P.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-I2V-14B-480P.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-I2V-14B-480P.py) | | Wan | [Wan-AI/Wan2.1-I2V-14B-720P](https://modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-720P) | [code](/examples/wanvideo/model_inference/Wan2.1-I2V-14B-720P.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-I2V-14B-720P.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-I2V-14B-720P.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-I2V-14B-720P.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-I2V-14B-720P.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-I2V-14B-720P.py) | | Wan | [Wan-AI/Wan2.1-FLF2V-14B-720P](https://modelscope.cn/models/Wan-AI/Wan2.1-FLF2V-14B-720P) | [code](/examples/wanvideo/model_inference/Wan2.1-FLF2V-14B-720P.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-FLF2V-14B-720P.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-FLF2V-14B-720P.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-FLF2V-14B-720P.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-FLF2V-14B-720P.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-FLF2V-14B-720P.py) | | Wan | [iic/VACE-Wan2.1-1.3B-Preview](https://modelscope.cn/models/iic/VACE-Wan2.1-1.3B-Preview) | [code](/examples/wanvideo/model_inference/Wan2.1-VACE-1.3B-Preview.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-VACE-1.3B-Preview.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-VACE-1.3B-Preview.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-VACE-1.3B-Preview.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-VACE-1.3B-Preview.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-VACE-1.3B-Preview.py) | | Wan | [Wan-AI/Wan2.1-VACE-1.3B](https://modelscope.cn/models/Wan-AI/Wan2.1-VACE-1.3B) | [code](/examples/wanvideo/model_inference/Wan2.1-VACE-1.3B.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-VACE-1.3B.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-VACE-1.3B.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-VACE-1.3B.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-VACE-1.3B.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-VACE-1.3B.py) | | Wan | [Wan-AI/Wan2.1-VACE-14B](https://modelscope.cn/models/Wan-AI/Wan2.1-VACE-14B) | [code](/examples/wanvideo/model_inference/Wan2.1-VACE-14B.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-VACE-14B.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-VACE-14B.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-VACE-14B.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-VACE-14B.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-VACE-14B.py) | | Wan | [PAI/Wan2.1-Fun-1.3B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-1.3B-InP) | [code](/examples/wanvideo/model_inference/Wan2.1-Fun-1.3B-InP.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-1.3B-InP.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-Fun-1.3B-InP.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-1.3B-InP.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-1.3B-InP.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-1.3B-InP.py) | | Wan | [PAI/Wan2.1-Fun-1.3B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-1.3B-Control) | [code](/examples/wanvideo/model_inference/Wan2.1-Fun-1.3B-Control.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-1.3B-Control.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-Fun-1.3B-Control.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-1.3B-Control.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-1.3B-Control.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-1.3B-Control.py) | | Wan | [PAI/Wan2.1-Fun-14B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-14B-InP) | [code](/examples/wanvideo/model_inference/Wan2.1-Fun-14B-InP.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-14B-InP.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-Fun-14B-InP.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-14B-InP.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-14B-InP.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-14B-InP.py) | | Wan | [PAI/Wan2.1-Fun-14B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-14B-Control) | [code](/examples/wanvideo/model_inference/Wan2.1-Fun-14B-Control.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-14B-Control.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-Fun-14B-Control.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-14B-Control.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-14B-Control.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-14B-Control.py) | | Wan | [PAI/Wan2.1-Fun-V1.1-1.3B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-Control) | [code](/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-1.3B-Control.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-1.3B-Control.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-1.3B-Control.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-1.3B-Control.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-1.3B-Control.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-1.3B-Control.py) | | Wan | [PAI/Wan2.1-Fun-V1.1-14B-Control](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-Control) | [code](/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-14B-Control.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-14B-Control.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-14B-Control.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-14B-Control.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-14B-Control.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-14B-Control.py) | | Wan | [PAI/Wan2.1-Fun-V1.1-1.3B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-InP) | [code](/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-1.3B-InP.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-1.3B-InP.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-1.3B-InP.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-1.3B-InP.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-1.3B-InP.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-1.3B-InP.py) | | Wan | [PAI/Wan2.1-Fun-V1.1-14B-InP](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-InP) | [code](/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-14B-InP.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-14B-InP.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-14B-InP.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-14B-InP.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-14B-InP.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-14B-InP.py) | | Wan | [PAI/Wan2.1-Fun-V1.1-1.3B-Control-Camera](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-Control-Camera) | [code](/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-1.3B-Control-Camera.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-1.3B-Control-Camera.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-1.3B-Control-Camera.py) | | Wan | [PAI/Wan2.1-Fun-V1.1-14B-Control-Camera](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-Control-Camera) | [code](/examples/wanvideo/model_inference/Wan2.1-Fun-V1.1-14B-Control-Camera.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-Fun-V1.1-14B-Control-Camera.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-Fun-V1.1-14B-Control-Camera.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-Fun-V1.1-14B-Control-Camera.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-Fun-V1.1-14B-Control-Camera.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-Fun-V1.1-14B-Control-Camera.py) | | Wan | [DiffSynth-Studio/Wan2.1-1.3b-speedcontrol-v1](https://modelscope.cn/models/DiffSynth-Studio/Wan2.1-1.3b-speedcontrol-v1) | [code](/examples/wanvideo/model_inference/Wan2.1-1.3b-speedcontrol-v1.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.1-1.3b-speedcontrol-v1.py) | [code](/examples/wanvideo/model_training/full/Wan2.1-1.3b-speedcontrol-v1.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.1-1.3b-speedcontrol-v1.py) | [code](/examples/wanvideo/model_training/lora/Wan2.1-1.3b-speedcontrol-v1.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.1-1.3b-speedcontrol-v1.py) | | Wan | [krea/krea-realtime-video](https://www.modelscope.cn/models/krea/krea-realtime-video) | [code](/examples/wanvideo/model_inference/krea-realtime-video.py) | [code](/examples/wanvideo/model_inference_low_vram/krea-realtime-video.py) | [code](/examples/wanvideo/model_training/full/krea-realtime-video.sh) | [code](/examples/wanvideo/model_training/validate_full/krea-realtime-video.py) | [code](/examples/wanvideo/model_training/lora/krea-realtime-video.sh) | [code](/examples/wanvideo/model_training/validate_lora/krea-realtime-video.py) | | Wan | [meituan-longcat/LongCat-Video](https://www.modelscope.cn/models/meituan-longcat/LongCat-Video) | [code](/examples/wanvideo/model_inference/LongCat-Video.py) | [code](/examples/wanvideo/model_inference_low_vram/LongCat-Video.py) | [code](/examples/wanvideo/model_training/full/LongCat-Video.sh) | [code](/examples/wanvideo/model_training/validate_full/LongCat-Video.py) | [code](/examples/wanvideo/model_training/lora/LongCat-Video.sh) | [code](/examples/wanvideo/model_training/validate_lora/LongCat-Video.py) | | Wan | [ByteDance/Video-As-Prompt-Wan2.1-14B](https://modelscope.cn/models/ByteDance/Video-As-Prompt-Wan2.1-14B) | [code](/examples/wanvideo/model_inference/Video-As-Prompt-Wan2.1-14B.py) | [code](/examples/wanvideo/model_inference_low_vram/Video-As-Prompt-Wan2.1-14B.py) | [code](/examples/wanvideo/model_training/full/Video-As-Prompt-Wan2.1-14B.sh) | [code](/examples/wanvideo/model_training/validate_full/Video-As-Prompt-Wan2.1-14B.py) | [code](/examples/wanvideo/model_training/lora/Video-As-Prompt-Wan2.1-14B.sh) | [code](/examples/wanvideo/model_training/validate_lora/Video-As-Prompt-Wan2.1-14B.py) | | Wan | [Wan-AI/Wan2.2-T2V-A14B](https://modelscope.cn/models/Wan-AI/Wan2.2-T2V-A14B) | [code](/examples/wanvideo/model_inference/Wan2.2-T2V-A14B.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.2-T2V-A14B.py) | [code](/examples/wanvideo/model_training/full/Wan2.2-T2V-A14B.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.2-T2V-A14B.py) | [code](/examples/wanvideo/model_training/lora/Wan2.2-T2V-A14B.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.2-T2V-A14B.py) | | Wan | [Wan-AI/Wan2.2-I2V-A14B](https://modelscope.cn/models/Wan-AI/Wan2.2-I2V-A14B) | [code](/examples/wanvideo/model_inference/Wan2.2-I2V-A14B.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.2-I2V-A14B.py) | [code](/examples/wanvideo/model_training/full/Wan2.2-I2V-A14B.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.2-I2V-A14B.py) | [code](/examples/wanvideo/model_training/lora/Wan2.2-I2V-A14B.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.2-I2V-A14B.py) | | Wan | [Wan-AI/Wan2.2-TI2V-5B](https://modelscope.cn/models/Wan-AI/Wan2.2-TI2V-5B) | [code](/examples/wanvideo/model_inference/Wan2.2-TI2V-5B.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.2-TI2V-5B.py) | [code](/examples/wanvideo/model_training/full/Wan2.2-TI2V-5B.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.2-TI2V-5B.py) | [code](/examples/wanvideo/model_training/lora/Wan2.2-TI2V-5B.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.2-TI2V-5B.py) | | Wan | [Wan-AI/Wan2.2-Animate-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-Animate-14B) | [code](/examples/wanvideo/model_inference/Wan2.2-Animate-14B.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-14B.py) | [code](/examples/wanvideo/model_training/full/Wan2.2-Animate-14B.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-14B.py) | [code](/examples/wanvideo/model_training/lora/Wan2.2-Animate-14B.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-14B.py) | | Wan | [Wan-AI/Wan2.2-Animate-2-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-Animate-2-14B) | [code](/examples/wanvideo/model_inference/Wan2.2-Animate-2-14B.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-2-14B.py) | [code](/examples/wanvideo/model_training/full/Wan2.2-Animate-2-14B.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-2-14B.py) | [code](/examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-2-14B.py) | | Wan | [Wan-AI/Wan2.2-Animate-2-14B: Distilled](https://www.modelscope.cn/models/Wan-AI/Wan2.2-Animate-2-14B) | [code](/examples/wanvideo/model_inference/Wan2.2-Animate-2-14B-Distilled.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.2-Animate-2-14B-Distilled.py) | [code](/examples/wanvideo/model_training/full/Wan2.2-Animate-2-14B-Distilled.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.2-Animate-2-14B-Distilled.py) | [code](/examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B-Distilled.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.2-Animate-2-14B-Distilled.py) | | Wan | [Wan-AI/Wan2.2-S2V-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.2-S2V-14B) | [code](/examples/wanvideo/model_inference/Wan2.2-S2V-14B_multi_clips.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.2-S2V-14B_multi_clips.py) | [code](/examples/wanvideo/model_training/full/Wan2.2-S2V-14B.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.2-S2V-14B.py) | [code](/examples/wanvideo/model_training/lora/Wan2.2-S2V-14B.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.2-S2V-14B.py) | | Wan | [PAI/Wan2.2-VACE-Fun-A14B](https://www.modelscope.cn/models/PAI/Wan2.2-VACE-Fun-A14B) | [code](/examples/wanvideo/model_inference/Wan2.2-VACE-Fun-A14B.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.2-VACE-Fun-A14B.py) | [code](/examples/wanvideo/model_training/full/Wan2.2-VACE-Fun-A14B.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.2-VACE-Fun-A14B.py) | [code](/examples/wanvideo/model_training/lora/Wan2.2-VACE-Fun-A14B.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.2-VACE-Fun-A14B.py) | | Wan | [PAI/Wan2.2-Fun-A14B-InP](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-InP) | [code](/examples/wanvideo/model_inference/Wan2.2-Fun-A14B-InP.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.2-Fun-A14B-InP.py) | [code](/examples/wanvideo/model_training/full/Wan2.2-Fun-A14B-InP.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.2-Fun-A14B-InP.py) | [code](/examples/wanvideo/model_training/lora/Wan2.2-Fun-A14B-InP.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.2-Fun-A14B-InP.py) | | Wan | [PAI/Wan2.2-Fun-A14B-Control](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-Control) | [code](/examples/wanvideo/model_inference/Wan2.2-Fun-A14B-Control.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.2-Fun-A14B-Control.py) | [code](/examples/wanvideo/model_training/full/Wan2.2-Fun-A14B-Control.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.2-Fun-A14B-Control.py) | [code](/examples/wanvideo/model_training/lora/Wan2.2-Fun-A14B-Control.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.2-Fun-A14B-Control.py) | | Wan | [PAI/Wan2.2-Fun-A14B-Control-Camera](https://modelscope.cn/models/PAI/Wan2.2-Fun-A14B-Control-Camera) | [code](/examples/wanvideo/model_inference/Wan2.2-Fun-A14B-Control-Camera.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan2.2-Fun-A14B-Control-Camera.py) | [code](/examples/wanvideo/model_training/full/Wan2.2-Fun-A14B-Control-Camera.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan2.2-Fun-A14B-Control-Camera.py) | [code](/examples/wanvideo/model_training/lora/Wan2.2-Fun-A14B-Control-Camera.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan2.2-Fun-A14B-Control-Camera.py) | | Wan | [openmoss/MOVA-360p](https://modelscope.cn/models/openmoss/MOVA-360p) | [code](/examples/mova/model_inference/MOVA-360p-I2AV.py) | [code](/examples/mova/model_inference_low_vram/MOVA-360p-I2AV.py) | [code](/examples/mova/model_training/full/MOVA-360P-I2AV.sh) | [code](/examples/mova/model_training/validate_full/MOVA-360p-I2AV.py) | [code](/examples/mova/model_training/lora/MOVA-360P-I2AV.sh) | [code](/examples/mova/model_training/validate_lora/MOVA-360p-I2AV.py) | | Wan | [openmoss/MOVA-720p](https://modelscope.cn/models/openmoss/MOVA-720p) | [code](/examples/mova/model_inference/MOVA-720p-I2AV.py) | [code](/examples/mova/model_inference_low_vram/MOVA-720p-I2AV.py) | [code](/examples/mova/model_training/full/MOVA-720P-I2AV.sh) | [code](/examples/mova/model_training/validate_full/MOVA-720p-I2AV.py) | [code](/examples/mova/model_training/lora/MOVA-720P-I2AV.sh) | [code](/examples/mova/model_training/validate_lora/MOVA-720p-I2AV.py) | | Wan | [Wan-AI/Wan-Dancer-14B (global model)](https://modelscope.cn/models/Wan-AI/Wan-Dancer-14B) | [code](/examples/wanvideo/model_inference/Wan-Dancer-14B-global.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan-Dancer-14B-global.py) | [code](/examples/wanvideo/model_training/full/Wan-Dancer-14B-global.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan-Dancer-14B-global.py) | [code](/examples/wanvideo/model_training/lora/Wan-Dancer-14B-global.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan-Dancer-14B-global.py) | | Wan | [Wan-AI/Wan-Dancer-14B (local model)](https://modelscope.cn/models/Wan-AI/Wan-Dancer-14B) | [code](/examples/wanvideo/model_inference/Wan-Dancer-14B-local.py) | [code](/examples/wanvideo/model_inference_low_vram/Wan-Dancer-14B-local.py) | [code](/examples/wanvideo/model_training/full/Wan-Dancer-14B-local.sh) | [code](/examples/wanvideo/model_training/validate_full/Wan-Dancer-14B-local.py) | [code](/examples/wanvideo/model_training/lora/Wan-Dancer-14B-local.sh) | [code](/examples/wanvideo/model_training/validate_lora/Wan-Dancer-14B-local.py) | | FLUX.1 | [black-forest-labs/FLUX.1-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.1-dev) | [code](/examples/flux/model_inference/FLUX.1-dev.py) | [code](/examples/flux/model_inference_low_vram/FLUX.1-dev.py) | [code](/examples/flux/model_training/full/FLUX.1-dev.sh) | [code](/examples/flux/model_training/validate_full/FLUX.1-dev.py) | [code](/examples/flux/model_training/lora/FLUX.1-dev.sh) | [code](/examples/flux/model_training/validate_lora/FLUX.1-dev.py) | | FLUX.1 | [black-forest-labs/FLUX.1-Krea-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.1-Krea-dev) | [code](/examples/flux/model_inference/FLUX.1-Krea-dev.py) | [code](/examples/flux/model_inference_low_vram/FLUX.1-Krea-dev.py) | [code](/examples/flux/model_training/full/FLUX.1-Krea-dev.sh) | [code](/examples/flux/model_training/validate_full/FLUX.1-Krea-dev.py) | [code](/examples/flux/model_training/lora/FLUX.1-Krea-dev.sh) | [code](/examples/flux/model_training/validate_lora/FLUX.1-Krea-dev.py) | | FLUX.1 | [black-forest-labs/FLUX.1-Kontext-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.1-Kontext-dev) | [code](/examples/flux/model_inference/FLUX.1-Kontext-dev.py) | [code](/examples/flux/model_inference_low_vram/FLUX.1-Kontext-dev.py) | [code](/examples/flux/model_training/full/FLUX.1-Kontext-dev.sh) | [code](/examples/flux/model_training/validate_full/FLUX.1-Kontext-dev.py) | [code](/examples/flux/model_training/lora/FLUX.1-Kontext-dev.sh) | [code](/examples/flux/model_training/validate_lora/FLUX.1-Kontext-dev.py) | | FLUX.1 | [black-forest-labs/FLUX.1-Fill-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.1-Fill-dev) | [code](/examples/flux/model_inference/FLUX.1-Fill-dev.py) | [code](/examples/flux/model_inference_low_vram/FLUX.1-Fill-dev.py) | [code](/examples/flux/model_training/full/FLUX.1-Fill-dev.sh) | [code](/examples/flux/model_training/validate_full/FLUX.1-Fill-dev.py) | [code](/examples/flux/model_training/lora/FLUX.1-Fill-dev.sh) | [code](/examples/flux/model_training/validate_lora/FLUX.1-Fill-dev.py) | | FLUX.1 | [black-forest-labs/FLUX.1-Redux-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.1-Redux-dev) | [code](/examples/flux/model_inference/FLUX.1-Redux-dev.py) | [code](/examples/flux/model_inference_low_vram/FLUX.1-Redux-dev.py) | [code](/examples/flux/model_training/full/FLUX.1-Redux-dev.sh) | [code](/examples/flux/model_training/validate_full/FLUX.1-Redux-dev.py) | [code](/examples/flux/model_training/lora/FLUX.1-Redux-dev.sh) | [code](/examples/flux/model_training/validate_lora/FLUX.1-Redux-dev.py) | | FLUX.1 | [HuanJue/Insert-Anything](https://www.modelscope.cn/models/HuanJue/Insert-Anything) | [code](/examples/flux/model_inference/Insert-Anything.py) | [code](/examples/flux/model_inference_low_vram/Insert-Anything.py) | - | - | [code](/examples/flux/model_training/lora/Insert-Anything.sh) | [code](/examples/flux/model_training/validate_lora/Insert-Anything.py) | | FLUX.1 | [alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Beta](https://www.modelscope.cn/models/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Beta) | [code](/examples/flux/model_inference/FLUX.1-dev-Controlnet-Inpainting-Beta.py) | [code](/examples/flux/model_inference_low_vram/FLUX.1-dev-Controlnet-Inpainting-Beta.py) | [code](/examples/flux/model_training/full/FLUX.1-dev-Controlnet-Inpainting-Beta.sh) | [code](/examples/flux/model_training/validate_full/FLUX.1-dev-Controlnet-Inpainting-Beta.py) | [code](/examples/flux/model_training/lora/FLUX.1-dev-Controlnet-Inpainting-Beta.sh) | [code](/examples/flux/model_training/validate_lora/FLUX.1-dev-Controlnet-Inpainting-Beta.py) | | FLUX.1 | [InstantX/FLUX.1-dev-Controlnet-Union-alpha](https://www.modelscope.cn/models/InstantX/FLUX.1-dev-Controlnet-Union-alpha) | [code](/examples/flux/model_inference/FLUX.1-dev-Controlnet-Union-alpha.py) | [code](/examples/flux/model_inference_low_vram/FLUX.1-dev-Controlnet-Union-alpha.py) | [code](/examples/flux/model_training/full/FLUX.1-dev-Controlnet-Union-alpha.sh) | [code](/examples/flux/model_training/validate_full/FLUX.1-dev-Controlnet-Union-alpha.py) | [code](/examples/flux/model_training/lora/FLUX.1-dev-Controlnet-Union-alpha.sh) | [code](/examples/flux/model_training/validate_lora/FLUX.1-dev-Controlnet-Union-alpha.py) | | FLUX.1 | [jasperai/Flux.1-dev-Controlnet-Upscaler](https://www.modelscope.cn/models/jasperai/Flux.1-dev-Controlnet-Upscaler) | [code](/examples/flux/model_inference/FLUX.1-dev-Controlnet-Upscaler.py) | [code](/examples/flux/model_inference_low_vram/FLUX.1-dev-Controlnet-Upscaler.py) | [code](/examples/flux/model_training/full/FLUX.1-dev-Controlnet-Upscaler.sh) | [code](/examples/flux/model_training/validate_full/FLUX.1-dev-Controlnet-Upscaler.py) | [code](/examples/flux/model_training/lora/FLUX.1-dev-Controlnet-Upscaler.sh) | [code](/examples/flux/model_training/validate_lora/FLUX.1-dev-Controlnet-Upscaler.py) | | FLUX.1 | [InstantX/FLUX.1-dev-IP-Adapter](https://www.modelscope.cn/models/InstantX/FLUX.1-dev-IP-Adapter) | [code](/examples/flux/model_inference/FLUX.1-dev-IP-Adapter.py) | [code](/examples/flux/model_inference_low_vram/FLUX.1-dev-IP-Adapter.py) | [code](/examples/flux/model_training/full/FLUX.1-dev-IP-Adapter.sh) | [code](/examples/flux/model_training/validate_full/FLUX.1-dev-IP-Adapter.py) | [code](/examples/flux/model_training/lora/FLUX.1-dev-IP-Adapter.sh) | [code](/examples/flux/model_training/validate_lora/FLUX.1-dev-IP-Adapter.py) | | FLUX.1 | [ByteDance/InfiniteYou](https://www.modelscope.cn/models/ByteDance/InfiniteYou) | [code](/examples/flux/model_inference/FLUX.1-dev-InfiniteYou.py) | [code](/examples/flux/model_inference_low_vram/FLUX.1-dev-InfiniteYou.py) | [code](/examples/flux/model_training/full/FLUX.1-dev-InfiniteYou.sh) | [code](/examples/flux/model_training/validate_full/FLUX.1-dev-InfiniteYou.py) | [code](/examples/flux/model_training/lora/FLUX.1-dev-InfiniteYou.sh) | [code](/examples/flux/model_training/validate_lora/FLUX.1-dev-InfiniteYou.py) | | FLUX.1 | [DiffSynth-Studio/Eligen](https://www.modelscope.cn/models/DiffSynth-Studio/Eligen) | [code](/examples/flux/model_inference/FLUX.1-dev-EliGen.py) | [code](/examples/flux/model_inference_low_vram/FLUX.1-dev-EliGen.py) | - | - | [code](/examples/flux/model_training/lora/FLUX.1-dev-EliGen.sh) | [code](/examples/flux/model_training/validate_lora/FLUX.1-dev-EliGen.py) | | FLUX.1 | [DiffSynth-Studio/LoRA-Encoder-FLUX.1-Dev](https://www.modelscope.cn/models/DiffSynth-Studio/LoRA-Encoder-FLUX.1-Dev) | [code](/examples/flux/model_inference/FLUX.1-dev-LoRA-Encoder.py) | [code](/examples/flux/model_inference_low_vram/FLUX.1-dev-LoRA-Encoder.py) | [code](/examples/flux/model_training/full/FLUX.1-dev-LoRA-Encoder.sh) | [code](/examples/flux/model_training/validate_full/FLUX.1-dev-LoRA-Encoder.py) | - | - | | FLUX.1 | [DiffSynth-Studio/LoRAFusion-preview-FLUX.1-dev](https://modelscope.cn/models/DiffSynth-Studio/LoRAFusion-preview-FLUX.1-dev) | [code](/examples/flux/model_inference/FLUX.1-dev-LoRA-Fusion.py) | - | - | - | - | - | | FLUX.1 | [stepfun-ai/Step1X-Edit](https://www.modelscope.cn/models/stepfun-ai/Step1X-Edit) | [code](/examples/flux/model_inference/Step1X-Edit.py) | [code](/examples/flux/model_inference_low_vram/Step1X-Edit.py) | [code](/examples/flux/model_training/full/Step1X-Edit.sh) | [code](/examples/flux/model_training/validate_full/Step1X-Edit.py) | [code](/examples/flux/model_training/lora/Step1X-Edit.sh) | [code](/examples/flux/model_training/validate_lora/Step1X-Edit.py) | | FLUX.1 | [ostris/Flex.2-preview](https://www.modelscope.cn/models/ostris/Flex.2-preview) | [code](/examples/flux/model_inference/FLEX.2-preview.py) | [code](/examples/flux/model_inference_low_vram/FLEX.2-preview.py) | [code](/examples/flux/model_training/full/FLEX.2-preview.sh) | [code](/examples/flux/model_training/validate_full/FLEX.2-preview.py) | [code](/examples/flux/model_training/lora/FLEX.2-preview.sh) | [code](/examples/flux/model_training/validate_lora/FLEX.2-preview.py) | | FLUX.1 | [DiffSynth-Studio/Nexus-GenV2](https://www.modelscope.cn/models/DiffSynth-Studio/Nexus-GenV2) | [code](/examples/flux/model_inference/Nexus-Gen-Editing.py) | [code](/examples/flux/model_inference_low_vram/Nexus-Gen-Editing.py) | [code](/examples/flux/model_training/full/Nexus-Gen.sh) | [code](/examples/flux/model_training/validate_full/Nexus-Gen.py) | [code](/examples/flux/model_training/lora/Nexus-Gen.sh) | [code](/examples/flux/model_training/validate_lora/Nexus-Gen.py) | | Stable Diffusion XL | [stabilityai/stable-diffusion-xl-base-1.0](https://www.modelscope.cn/models/stabilityai/stable-diffusion-xl-base-1.0) | [code](/examples/stable_diffusion_xl/model_inference/stable-diffusion-xl-base-1.0.py) | [code](/examples/stable_diffusion_xl/model_inference_low_vram/stable-diffusion-xl-base-1.0.py) | [code](/examples/stable_diffusion_xl/model_training/full/stable-diffusion-xl-base-1.0.sh) | [code](/examples/stable_diffusion_xl/model_training/validate_full/stable-diffusion-xl-base-1.0.py) | [code](/examples/stable_diffusion_xl/model_training/lora/stable-diffusion-xl-base-1.0.sh) | [code](/examples/stable_diffusion_xl/model_training/validate_lora/stable-diffusion-xl-base-1.0.py) | | Stable Diffusion | [AI-ModelScope/stable-diffusion-v1-5](https://www.modelscope.cn/models/AI-ModelScope/stable-diffusion-v1-5) | [code](/examples/stable_diffusion/model_inference/stable-diffusion-v1-5.py) | [code](/examples/stable_diffusion/model_inference_low_vram/stable-diffusion-v1-5.py) | [code](/examples/stable_diffusion/model_training/full/stable-diffusion-v1-5.sh) | [code](/examples/stable_diffusion/model_training/validate_full/stable-diffusion-v1-5.py) | [code](/examples/stable_diffusion/model_training/lora/stable-diffusion-v1-5.sh) | [code](/examples/stable_diffusion/model_training/validate_lora/stable-diffusion-v1-5.py) | --- ### Overview (`docs/en/README.md`) # DiffSynth-Studio Documentation Welcome to the magical world of Diffusion models! `DiffSynth-Studio` is an open-source Diffusion model engine developed and maintained by the [ModelScope Community](https://www.modelscope.cn/). We aim to build a universal Diffusion model framework that fosters technological innovation through framework construction, aggregates the power of the open-source community, and explores the boundaries of generative model technology!
Documentation Reading Guide ```mermaid graph LR; use["I want to use models for inference and training"]-->sec1["Section 1: Getting Started"]; use["I want to use models for inference and training"]-->sec2["Section 2: Model Details"]; use["I want to use models for inference and training"]-->sec3["Section 3: Training Framework"]; develop["I want to develop based on this framework"]-->sec3["Section 3: Training Framework"]; develop["I want to develop based on this framework"]-->sec4["Section 4: Model Integration"]; develop["I want to develop based on this framework"]-->sec5["Section 5: API Reference"]; explore["I want to explore new technologies based on this project"]-->sec4["Section 4: Model Integration"]; explore["I want to explore new technologies based on this project"]-->sec5["Section 5: API Reference"]; explore["I want to explore new technologies based on this project"]-->sec6["Section 6: Diffusion Templates"]; explore["I want to explore new technologies based on this project"]-->sec7["Section 7: Research Guide"]; problem["I encountered a problem"]-->sec8["Section 8: Frequently Asked Questions"]; ```
## Section 1: Getting Started This section introduces the basic usage of `DiffSynth-Studio`, including how to enable VRAM management for inference on GPUs with extremely low VRAM, and how to train various base models, LoRAs, ControlNets, and other models. * [Installation Dependencies](./Pipeline_Usage/Setup.md) * [Model Inference](./Pipeline_Usage/Model_Inference.md) * [Accelerated Inference](./Pipeline_Usage/Accelerated_Inference.md) * [VRAM Management](./Pipeline_Usage/VRAM_management.md) * [Model Quantization](./Pipeline_Usage/Quantization.md) * [Model Training](./Pipeline_Usage/Model_Training.md) * [Environment Variables](./Pipeline_Usage/Environment_Variables.md) * [GPU/NPU Support](./Pipeline_Usage/GPU_support.md) * [Inference WebUI](./Pipeline_Usage/Inference_WebUI.md) ## Section 2: Model Details This section introduces the Diffusion models supported by `DiffSynth-Studio`. Some model pipelines feature special functionalities such as controllable generation and parallel acceleration. * [FLUX.1](./Model_Details/FLUX.md) * [Wan](./Model_Details/Wan.md) * [Qwen-Image](./Model_Details/Qwen-Image.md) * [Qwen-Video-Edit](./Model_Details/Qwen-Video-Edit.md) * [FLUX.2](./Model_Details/FLUX2.md) * [Z-Image](./Model_Details/Z-Image.md) * [Anima](./Model_Details/Anima.md) * [LTX-2](./Model_Details/LTX-2.md) * [ERNIE-Image](./Model_Details/ERNIE-Image.md) * [JoyAI-Image](./Model_Details/JoyAI-Image.md) * [ACE-Step](./Model_Details/ACE-Step.md) * [HiDream-O1-Image](./Model_Details/HiDream-O1-Image.md) * [Stable Diffusion](./Model_Details/Stable-Diffusion.md) * [Stable Diffusion XL](./Model_Details/Stable-Diffusion-XL.md) * [Image Quality Metrics](./Model_Details/Image-Quality-Metrics.md) * [Ideogram 4](./Model_Details/Ideogram-4.md) * [Krea-2](./Model_Details/Krea-2.md) * [Fibo 2.0](./Model_Details/Fibo.md) * [Boogu-Image](./Model_Details/Boogu-Image.md) * [LingBot-Video](./Model_Details/LingBot-Video.md) * [MiniMax-H3](./Model_Details/MiniMax-H3.md) * [MiniMax-Music3](./Model_Details/MiniMax-Music3.md) * [SenseNova-U1](./Model_Details/SenseNova-U1.md) * [YuE2](./Model_Details/YuE2.md) * [Qwen-Image-2.1](./Model_Details/Qwen-Image-2.1.md) * [FLUX.3](./Model_Details/FLUX.3.md) ## Section 3: Training Framework This section introduces the design philosophy of the training framework in `DiffSynth-Studio`, helping developers understand the principles of Diffusion model training algorithms. * [Basic Principles of Diffusion Models](./Training/Understanding_Diffusion_models.md) * [Standard Supervised Training](./Training/Supervised_Fine_Tuning.md) * [Enabling FP8 Precision in Training](./Training/FP8_Precision.md) * [End-to-End Distillation Accelerated Training](./Training/Direct_Distill.md) * [Two-Stage Split Training](./Training/Split_Training.md) * [Differential LoRA Training](./Training/Differential_LoRA.md) * [Enabling DeepSpeed](./Training/DeepSpeed.md) * [Offload Training](./Training/Offload_Training.md) ## Section 4: Model Integration This section introduces how to integrate models into `DiffSynth-Studio` to utilize the framework's basic functions, helping developers provide support for new models in this project or perform inference and training of private models. * [Integrating Model Architecture](./Developer_Guide/Integrating_Your_Model.md) * [Building a Pipeline](./Developer_Guide/Building_a_Pipeline.md) * [Enabling Fine-Grained VRAM Management](./Developer_Guide/Enabling_VRAM_management.md) * [Model Training Integration](./Developer_Guide/Training_Diffusion_Models.md) * [Integrating a Quantization Backend](./Developer_Guide/Integrating_Quantization_Backend.md) > We have open-sourced [**DiffSynth-Studio Model Integration Skills**](https://www.modelscope.cn/collections/DiffSynth-Studio/DiffSynth-Studio-Model-Integration-Skills). This is a composable collection of Agent Skills that automates the entire workflow of integrating external diffusion models (image / video / audio) into DiffSynth-Studio. It distills and defines the model integration standards of DiffSynth-Studio, consolidating best practices such as codebase analysis, model code integration, Pipeline design, low-VRAM management, and training support into reusable standard procedures. Following these standards can significantly lower the integration barrier, reduce repetitive debugging, and greatly improve the efficiency of integrating new models. We recommend starting from the [`diffsynth-integrator`](https://www.modelscope.cn/skills/DiffSynth-Studio/diffsynth-integrator) [example](https://www.modelscope.cn/skills/DiffSynth-Studio/diffsynth-integrator/file/view/master/example.md?status=1) to get started quickly and accelerate model integration. ## Section 5: API Reference This section introduces the independent core module `diffsynth.core` in `DiffSynth-Studio`, explaining how internal functions are designed and operate. Developers can use these functional modules in other codebase developments if needed. * [`diffsynth.core.attention`](./API_Reference/core/attention.md): Attention mechanism implementation * [`diffsynth.core.data`](./API_Reference/core/data.md): Data processing operators and general datasets * [`diffsynth.core.gradient`](./API_Reference/core/gradient.md): Gradient checkpointing * [`diffsynth.core.loader`](./API_Reference/core/loader.md): Model download and loading * [`diffsynth.core.quant`](./API_Reference/core/quant.md): Model quantization * [`diffsynth.core.vram`](./API_Reference/core/vram.md): VRAM management ## Section 6: Diffusion Templates This section introduces the controllable generation plugin framework for Diffusion models, explaining the framework's operation mechanism and how to use Template models for inference and training. * [Introducing Diffusion Templates](./Diffusion_Templates/Introducing_Diffusion_Templates.md) * [Diffusion Templates Architecture Details](./Diffusion_Templates/Understanding_Diffusion_Templates.md) * [Template Model Inference](./Diffusion_Templates/Template_Model_Inference.md) * [Template Model Training](./Diffusion_Templates/Template_Model_Training.md) ## Section 7: Research Guide This section introduces how to use `DiffSynth-Studio` to train new models, helping researchers explore new model technologies. * [Training models from scratch](./Research_Tutorial/train_from_scratch.md) * [Inference improvement techniques](./Research_Tutorial/inference_time_scaling.md) * [Designing controllable generation models](./Research_Tutorial/controllable_models.md) * Creating new training paradigms 【coming soon】 ## Section 8: Frequently Asked Questions This section summarizes common developer questions. If you encounter issues during usage or development, please refer to this section. If you still cannot resolve the problem, please submit an issue on GitHub. * [Frequently Asked Questions](./QA.md) --- ### Overview (`docs/zh/README.md`) # DiffSynth-Studio 文档 欢迎来到 Diffusion 模型的魔法世界！`DiffSynth-Studio` 是由[魔搭社区](https://www.modelscope.cn/)团队开发和维护的开源 Diffusion 模型引擎。我们期望构建一个通用的 Diffusion 模型框架，以框架建设孵化技术创新，凝聚开源社区的力量，探索生成式模型技术的边界！
文档阅读导引 ```mermaid graph LR; 我想要使用模型进行推理和训练-->sec1[Section 1: 上手使用]; 我想要使用模型进行推理和训练-->sec2[Section 2: 模型详解]; 我想要使用模型进行推理和训练-->sec3[Section 3: 训练框架]; 我想要基于此框架进行二次开发-->sec3[Section 3: 训练框架]; 我想要基于此框架进行二次开发-->sec4[Section 4: 模型接入]; 我想要基于此框架进行二次开发-->sec5[Section 5: API 参考]; 我想要基于本项目探索新的技术-->sec4[Section 4: 模型接入]; 我想要基于本项目探索新的技术-->sec5[Section 5: API 参考]; 我想要基于本项目探索新的技术-->sec6[Section 6: Diffusion Templates] 我想要基于本项目探索新的技术-->sec7[Section 7: 学术导引]; 我遇到了问题-->sec8[Section 8: 常见问题]; ```
## Section 1: 上手使用 本节介绍 `DiffSynth-Studio` 的基本使用方式，包括如何启用显存管理从而在极低显存的 GPU 上进行推理，以及如何训练任意基础模型、LoRA、ControlNet 等模型。 * [安装依赖](./Pipeline_Usage/Setup.md) * [模型推理](./Pipeline_Usage/Model_Inference.md) * [加速推理](./Pipeline_Usage/Accelerated_Inference.md) * [显存管理](./Pipeline_Usage/VRAM_management.md) * [模型量化](./Pipeline_Usage/Quantization.md) * [模型训练](./Pipeline_Usage/Model_Training.md) * [环境变量](./Pipeline_Usage/Environment_Variables.md) * [GPU/NPU 支持](./Pipeline_Usage/GPU_support.md) * [推理 WebUI](./Pipeline_Usage/Inference_WebUI.md) ## Section 2: 模型详解 本节介绍 `DiffSynth-Studio` 所支持的 Diffusion 模型，部分模型 Pipeline 具备可控生成、并行加速等特色功能。 * [FLUX.1](./Model_Details/FLUX.md) * [Wan](./Model_Details/Wan.md) * [Qwen-Image](./Model_Details/Qwen-Image.md) * [Qwen-Video-Edit](./Model_Details/Qwen-Video-Edit.md) * [FLUX.2](./Model_Details/FLUX2.md) * [Z-Image](./Model_Details/Z-Image.md) * [Anima](./Model_Details/Anima.md) * [LTX-2](./Model_Details/LTX-2.md) * [ERNIE-Image](./Model_Details/ERNIE-Image.md) * [JoyAI-Image](./Model_Details/JoyAI-Image.md) * [ACE-Step](./Model_Details/ACE-Step.md) * [HiDream-O1-Image](./Model_Details/HiDream-O1-Image.md) * [Stable Diffusion](./Model_Details/Stable-Diffusion.md) * [Stable Diffusion XL](./Model_Details/Stable-Diffusion-XL.md) * [图像质量评估指标](./Model_Details/Image-Quality-Metrics.md) * [Ideogram 4](./Model_Details/Ideogram-4.md) * [Krea-2](./Model_Details/Krea-2.md) * [Fibo 2.0](./Model_Details/Fibo.md) * [Boogu-Image](./Model_Details/Boogu-Image.md) * [LingBot-Video](./Model_Details/LingBot-Video.md) * [MiniMax-H3](./Model_Details/MiniMax-H3.md) * [MiniMax-Music3](./Model_Details/MiniMax-Music3.md) * [SenseNova-U1](./Model_Details/SenseNova-U1.md) * [YuE2](./Model_Details/YuE2.md) * [Qwen-Image-2.1](./Model_Details/Qwen-Image-2.1.md) * [FLUX.3](./Model_Details/FLUX.3.md) ## Section 3: 训练框架 本节介绍 `DiffSynth-Studio` 中训练框架的设计思路，帮助开发者理解 Diffusion 模型训练算法的原理。 * [Diffusion 模型基本原理](./Training/Understanding_Diffusion_models.md) * [标准监督训练](./Training/Supervised_Fine_Tuning.md) * [在训练中启用 FP8 精度](./Training/FP8_Precision.md) * [端到端的蒸馏加速训练](./Training/Direct_Distill.md) * [两阶段拆分训练](./Training/Split_Training.md) * [差分 LoRA 训练](./Training/Differential_LoRA.md) * [启用 DeepSpeed](./Training/DeepSpeed.md) * [Offload Training](./Training/Offload_Training.md) ## Section 4: 模型接入 本节介绍如何将模型接入 `DiffSynth-Studio` 从而使用框架基础功能，帮助开发者为本项目提供新模型的支持，或进行私有化模型的推理和训练。 * [接入模型结构](./Developer_Guide/Integrating_Your_Model.md) * [接入 Pipeline](./Developer_Guide/Building_a_Pipeline.md) * [接入细粒度显存管理](./Developer_Guide/Enabling_VRAM_management.md) * [接入模型训练](./Developer_Guide/Training_Diffusion_Models.md) * [接入量化后端](./Developer_Guide/Integrating_Quantization_Backend.md) > 我们开源了 [**DiffSynth-Studio Model Integration Skills**](https://www.modelscope.cn/collections/DiffSynth-Studio/DiffSynth-Studio-Model-Integration-Skills)。这是一套可组合的 Agent Skill 合集，将外部扩散模型（图像 / 视频 / 音频）接入 DiffSynth-Studio 的全流程自动化。它沉淀并定义了 DiffSynth-Studio 的模型接入规范，将代码库分析、模型代码接入、Pipeline 设计、低显存管理与训练支持等最佳实践固化为可复用的标准流程。遵循这套规范，能显著降低接入门槛、减少反复调试，大幅提升新模型的接入效率。建议从 [diffsynth-integrator](https://www.modelscope.cn/skills/DiffSynth-Studio/diffsynth-integrator) 的[使用示例](https://www.modelscope.cn/skills/DiffSynth-Studio/diffsynth-integrator/file/view/master/example.md?status=1)开始，快速上手、加速模型接入。 ## Section 5: API 参考 本节介绍 `DiffSynth-Studio` 中的独立核心模块 `diffsynth.core`，介绍内部的功能是如何设计和运作的，开发者如有需要，可将其中的功能模块用于其他代码库的开发中。 * [`diffsynth.core.attention`](./API_Reference/core/attention.md): 注意力机制实现 * [`diffsynth.core.data`](./API_Reference/core/data.md): 数据处理算子与通用数据集 * [`diffsynth.core.gradient`](./API_Reference/core/gradient.md): 梯度检查点 * [`diffsynth.core.loader`](./API_Reference/core/loader.md): 模型下载与加载 * [`diffsynth.core.quant`](./API_Reference/core/quant.md): 模型量化 * [`diffsynth.core.vram`](./API_Reference/core/vram.md): 显存管理 ## Section 6: Diffusion Templates 本节介绍 Diffusion 模型可控生成插件框架 Diffusion Templates，讲解 Diffusion Templates 框架的运行机制，展示如何使用 Template 模型进行推理和训练。 * [Diffusion Templates 简介](./Diffusion_Templates/Introducing_Diffusion_Templates.md) * [Diffusion Templates 架构详解](./Diffusion_Templates/Understanding_Diffusion_Templates.md) * [Template 模型推理](./Diffusion_Templates/Template_Model_Inference.md) * [Template 模型训练](./Diffusion_Templates/Template_Model_Training.md) ## Section 7: 学术导引 本节介绍如何利用 `DiffSynth-Studio` 训练新的模型，帮助科研工作者探索新的模型技术。 * [从零开始训练模型](./Research_Tutorial/train_from_scratch.md) * [推理改进优化技术](./Research_Tutorial/inference_time_scaling.md) * [设计可控生成模型](./Research_Tutorial/controllable_models.md) * 创建新的训练范式【coming soon】 ## Section 8: 常见问题 本节总结了开发者常见的问题，如果你在使用和开发中遇到了问题，请参考本节内容，如果仍无法解决，请到 GitHub 上给我们提 issue。 * [常见问题](./QA.md) --- ### Overview (`examples/anima/README.md`) English Document: https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Anima.html 中文文档：https://diffsynth-studio-doc.readthedocs.io/zh-cn/latest/Model_Details/Anima.html --- ### Overview (`examples/fibo/README.md`) English Document: https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Fibo.html 中文文档：https://diffsynth-studio-doc.readthedocs.io/zh-cn/latest/Model_Details/Fibo.html ## Inference - Text-to-image, editing, masked editing, text rendering: [`model_inference/Fibo-2.0-dev.py`](model_inference/Fibo-2.0-dev.py) - Editing and masked editing: [`model_inference/Fibo-2.0-edit-turbo.py`](model_inference/Fibo-2.0-edit-turbo.py) - Low VRAM versions: [`model_inference_low_vram/`](model_inference_low_vram/) ## Training Only Fibo-2.0-dev is trainable. | Mode | Full Training | LoRA Training | Full Validation | LoRA Validation | |-|-|-|-|-| | Text-to-image | [`full/Fibo-2.0-dev-T2I.sh`](model_training/full/Fibo-2.0-dev-T2I.sh) | [`lora/Fibo-2.0-dev-T2I.sh`](model_training/lora/Fibo-2.0-dev-T2I.sh) | [`validate_full/Fibo-2.0-dev-T2I.py`](model_training/validate_full/Fibo-2.0-dev-T2I.py) | [`validate_lora/Fibo-2.0-dev-T2I.py`](model_training/validate_lora/Fibo-2.0-dev-T2I.py) | | Image editing | [`full/Fibo-2.0-dev-Edit.sh`](model_training/full/Fibo-2.0-dev-Edit.sh) | [`lora/Fibo-2.0-dev-Edit.sh`](model_training/lora/Fibo-2.0-dev-Edit.sh) | [`validate_full/Fibo-2.0-dev-Edit.py`](model_training/validate_full/Fibo-2.0-dev-Edit.py) | [`validate_lora/Fibo-2.0-dev-Edit.py`](model_training/validate_lora/Fibo-2.0-dev-Edit.py) | --- ### Overview (`examples/flux/README.md`) English Document: https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/FLUX.html 中文文档：https://diffsynth-studio-doc.readthedocs.io/zh-cn/latest/Model_Details/FLUX.html --- ### Overview (`examples/flux2/README.md`) English Document: https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/FLUX2.html 中文文档：https://diffsynth-studio-doc.readthedocs.io/zh-cn/latest/Model_Details/FLUX2.html --- ### Overview (`examples/lingbot_video/README.md`) English Document: https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/LingBot-Video.html 中文文档：https://diffsynth-studio-doc.readthedocs.io/zh-cn/latest/Model_Details/LingBot-Video.html --- ### Overview (`examples/ltx2/README.md`) English Document: https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/LTX-2.html 中文文档：https://diffsynth-studio-doc.readthedocs.io/zh-cn/latest/Model_Details/LTX-2.html --- ### Overview (`examples/mova/README.md`) English Document: https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Wan.html 中文文档：https://diffsynth-studio-doc.readthedocs.io/zh-cn/latest/Model_Details/Wan.html --- ### Overview (`examples/qwen_image/README.md`) English Document: https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Qwen-Image.html 中文文档：https://diffsynth-studio-doc.readthedocs.io/zh-cn/latest/Model_Details/Qwen-Image.html --- ### Overview (`examples/wanvideo/README.md`) English Document: https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Wan.html 中文文档：https://diffsynth-studio-doc.readthedocs.io/zh-cn/latest/Model_Details/Wan.html --- ### Overview (`examples/yue2/README.md`) # YuE2 YuE2 is a music generation model with editable symbolic planning. - Inference: [YuE2.py](./model_inference/YuE2.py) - Low VRAM inference: [YuE2.py](./model_inference_low_vram/YuE2.py) - Plan and render an edited score: [YuE2-Edit.py](./model_inference/YuE2-Edit.py) - Low VRAM plan and render an edited score: [YuE2-Edit.py](./model_inference_low_vram/YuE2-Edit.py) - Documentation: [English](/docs/en/Model_Details/YuE2.md), [Chinese](/docs/zh/Model_Details/YuE2.md) --- ### Overview (`examples/z_image/README.md`) English Document: https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Z-Image.html 中文文档：https://diffsynth-studio-doc.readthedocs.io/zh-cn/latest/Model_Details/Z-Image.html --- ### Overview (`diffsynth/utils/ses/README.md`) Please see `docs/en/Research_Tutorial/inference_time_scaling.md` or `docs/zh/Research_Tutorial/inference_time_scaling.md` for more details. --- ### Requirements (`docs/requirements.txt`) docutils>=0.16.0 myst_parser recommonmark sphinx>=5.3.0 sphinx-book-theme sphinx-copybutton sphinx-autobuild sphinx-rtd-theme sphinx_markdown_tables sphinxcontrib-mermaid pymdown-extensions --- ### En .Readthedocs.Yaml (`docs/en/.readthedocs.yaml`) # .readthedocs.yaml # Read the Docs configuration file # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details # Required version: 2 # Set the OS, Python version and other tools you might need build: os: ubuntu-22.04 tools: python: "3.10" # Build documentation in the "docs/" directory with Sphinx sphinx: configuration: docs/en/conf.py # Optionally build your docs in additional formats such as PDF and ePub # formats: # - pdf # - epub # Optional but recommended, declare the Python requirements required # to build your documentation # See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html python: install: - requirements: docs/requirements.txt --- ### En QA (`docs/en/QA.md`) # Frequently Asked Questions ## Why doesn't the training framework support batch size > 1? * **Larger batch sizes no longer achieve significant acceleration**: Due to acceleration technologies such as flash attention that have fully improved GPU utilization, larger batch sizes will only bring greater VRAM usage without significant acceleration. The experience with small models like Stable Diffusion 1.5 is no longer applicable to the latest large models. * **Larger batch sizes can be achieved through other solutions**: Multi-GPU training and Gradient Accumulation can both mathematically equivalently achieve larger batch sizes. * **Larger batch sizes contradict the framework's general design**: We hope to build a general training framework. Many models cannot accommodate larger batch sizes, such as text encodings of different lengths and images of different resolutions, which cannot be merged into larger batches. ## Why aren't redundant parameters removed from certain models? In some models, redundant parameters exist. For example, in Qwen-Image's DiT model, the text portion of the last layer does not participate in any calculations. This is a minor bug left by the model developers. Setting it as trainable directly will also cause errors in multi-GPU training. To maintain compatibility with other models in the open-source community, we have decided to retain these parameters. These redundant parameters can avoid errors in multi-GPU training through the `--find_unused_parameters` parameter. ## Why does FP8 quantization show no acceleration effect? Native FP8 computation relies on Hopper architecture GPUs and has significant precision errors. It is currently immature technology, so this project does not support native FP8 computation. FP8 computation in VRAM management refers to storing model parameters in memory or VRAM with FP8 precision and temporarily converting them to other precisions when needed for computation. Therefore, it can only reduce VRAM usage without acceleration effects. ## Why doesn't the training framework support native FP8 precision training? Even with suitable hardware conditions, we currently have no plans to support native FP8 precision training. * The main challenge of native FP8 precision training is precision overflow caused by gradient explosion. To ensure training stability, the model structure needs to be redesigned accordingly. However, no model developers are willing to do so at present. * Additionally, models trained with native FP8 precision can only be computed with BF16 precision during inference without Hopper architecture GPUs, theoretically resulting in generation quality inferior to FP8. Therefore, native FP8 precision training technology is extremely immature. We will observe the technological developments in the open-source community. ## How to dynamically load LoRA models during inference? We support two loading methods for LoRA models. See [LoRA Loading](./Pipeline_Usage/Model_Inference.md#loading-lora) for details: * Cold Loading: When [VRAM Management](./Pipeline_Usage/VRAM_management.md) is not enabled for the base model, LoRA will be fused into the base model weights. In this case, inference speed remains unchanged, and LoRA cannot be unloaded after loading. * Hot Loading: When [VRAM Management](./Pipeline_Usage/VRAM_management.md) is enabled for the base model, LoRA will not be fused into the base model weights. In this case, inference speed will slow down, and LoRA can be unloaded after loading via `pipe.clear_lora()`. ## How to reduce VRAM usage during training? The framework supports multiple methods to reduce VRAM usage during training, including Gradient Checkpointing, DeepSpeed, FP8, Two-Stage Split Training, and CPU Offload. Please refer to [Low VRAM Training](./Pipeline_Usage/Model_Training.md#low-vram-training). --- ### En Index (`docs/en/index.rst`) Welcome to DiffSynth-Studio's Documentation ========================================== .. toctree:: :maxdepth: 2 :caption: Documentation Introduction README .. toctree:: :maxdepth: 2 :caption: Getting Started Pipeline_Usage/Setup Pipeline_Usage/Model_Inference Pipeline_Usage/Accelerated_Inference Pipeline_Usage/VRAM_management Pipeline_Usage/Quantization Pipeline_Usage/Model_Training Pipeline_Usage/Environment_Variables Pipeline_Usage/GPU_support Pipeline_Usage/Inference_WebUI .. toctree:: :maxdepth: 2 :caption: Model Details Model_Details/FLUX Model_Details/Wan Model_Details/Qwen-Image Model_Details/Qwen-Video-Edit Model_Details/FLUX2 Model_Details/Z-Image Model_Details/Anima Model_Details/LTX-2 Model_Details/LTX-2.5 Model_Details/ERNIE-Image Model_Details/JoyAI-Image Model_Details/ACE-Step Model_Details/HiDream-O1-Image Model_Details/Stable-Diffusion Model_Details/Stable-Diffusion-XL Model_Details/Image-Quality-Metrics Model_Details/Ideogram-4 Model_Details/Krea-2 Model_Details/Boogu-Image Model_Details/LingBot-Video Model_Details/MiniMax-H3 Model_Details/DiffSynth-Music Model_Details/MiniMax-Music3 Model_Details/YuE2 Model_Details/SenseNova-U1 Model_Details/Qwen-Image-2.1 Model_Details/FLUX.3 Model_Details/MOSS-Image Model_Details/Fibo .. toctree:: :maxdepth: 2 :caption: Training Framework Training/Understanding_Diffusion_models Training/Supervised_Fine_Tuning Training/FP8_Precision Training/Distillation Training/Direct_Distill Training/Split_Training Training/Differential_LoRA Training/DeepSpeed Training/Offload_Training .. toctree:: :maxdepth: 2 :caption: Model Integration Developer_Guide/Integrating_Your_Model Developer_Guide/Building_a_Pipeline Developer_Guide/Enabling_VRAM_management Developer_Guide/Training_Diffusion_Models Developer_Guide/Integrating_Quantization_Backend .. toctree:: :maxdepth: 2 :caption: API Reference API_Reference/core/attention API_Reference/core/data API_Reference/core/gradient API_Reference/core/loader API_Reference/core/quant API_Reference/core/vram .. toctree:: :maxdepth: 2 :caption: Diffusion Templates Diffusion_Templates/Introducing_Diffusion_Templates.md Diffusion_Templates/Understanding_Diffusion_Templates.md Diffusion_Templates/Template_Model_Inference.md Diffusion_Templates/Template_Model_Training.md .. toctree:: :maxdepth: 2 :caption: Research Guide Research_Tutorial/train_from_scratch Research_Tutorial/inference_time_scaling Research_Tutorial/controllable_models .. toctree:: :maxdepth: 2 :caption: FAQ QA Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search` --- ### Zh .Readthedocs.Yaml (`docs/zh/.readthedocs.yaml`) # .readthedocs.yaml # Read the Docs configuration file # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details # Required version: 2 # Set the OS, Python version and other tools you might need build: os: ubuntu-22.04 tools: python: "3.10" # Build documentation in the "docs/" directory with Sphinx sphinx: configuration: docs/zh/conf.py # Optionally build your docs in additional formats such as PDF and ePub # formats: # - pdf # - epub # Optional but recommended, declare the Python requirements required # to build your documentation # See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html python: install: - requirements: docs/requirements.txt --- ### Zh QA (`docs/zh/QA.md`) # 常见问题 ## 为什么训练框架不支持 batch size > 1？ * **更大的 batch size 已无法实现显著加速**：由于 flash attention 等加速技术已经充分提高了 GPU 的利用率，因此更大的 batch size 只会带来更大的显存占用，无法带来显著加速。在 Stable Diffusion 1.5 这类小模型上的经验已不再适用于最新的大模型。 * **更大的 batch size 可以用其他方案实现**：多 GPU 训练和 Gradient Accumulation 都可以在数学意义上等价地实现更大的 batch size。 * **更大的 batch size 与框架的通用性设计相悖**：我们希望构建通用的训练框架，大量模型无法适配更大的 batch size，例如不同长度的文本编码、不同分辨率的图像等，都是无法合并为更大的 batch 的。 ## 为什么不删除某些模型中的冗余参数？ 在部分模型中，模型存在冗余参数，例如 Qwen-Image 的 DiT 模型最后一层的文本部分，这部分参数不会参与任何计算，这是模型开发者留下的小 bug。直接将其设置为可训练时还会在多 GPU 训练中出现报错。 为了与开源社区中其他模型保持兼容性，我们决定保留这些参数。这些冗余参数在多 GPU 训练中可以通过 `--find_unused_parameters` 参数避免报错。 ## 为什么 FP8 量化没有任何加速效果？ 原生 FP8 计算需要依赖 Hopper 架构的 GPU，同时在计算精度上有较大误差，目前仍然是不成熟的技术，因此本项目不支持原生 FP8 计算。 显存管理中的 FP8 计算是指将模型参数以 FP8 精度存储在内存或显存中，在需要计算时临时转换为其他精度，因此仅能减少显存占用，没有加速效果。 ## 为什么训练框架不支持原生 FP8 精度训练？ 即使硬件条件允许，我们目前也没有任何支持原生 FP8 精度训练的规划。 * 目前原生 FP8 精度训练的主要挑战是梯度爆炸导致的精度溢出，为了保证训练的稳定性，需针对性地重新设计模型结构，然而目前还没有任何模型开发者愿意这么做。 * 此外，使用原生 FP8 精度训练的模型，在推理时若没有 Hopper 架构 GPU，则只能以 BF16 精度进行计算，理论上其生成效果反而不如 FP8。 因此，原生 FP8 精度训练技术是极不成熟的，我们静观开源社区的技术发展。 ## 如何在推理时动态加载 LoRA 模型？ 我们支持 LoRA 模型的两种加载方式，详见[LoRA 加载](./Pipeline_Usage/Model_Inference.md#加载-lora)： * 冷加载：当基础模型未开启[显存管理](./Pipeline_Usage/VRAM_management.md)时，LoRA 会融合进基础模型权重，此时推理速度没有变化，LoRA 加载后无法卸载。 * 热加载：当基础模型开启[显存管理](./Pipeline_Usage/VRAM_management.md)时，LoRA 不会融合进基础模型权重，此时推理速度会变慢，LoRA 加载后可通过 `pipe.clear_lora()` 卸载。 ## 如何减少训练所需的显存？ 框架支持多种方式减少训练所需的显存，包括 Gradient Checkpointing、DeepSpeed、FP8、两阶段拆分训练、CPU Offload，请参考[低显存训练](./Pipeline_Usage/Model_Training.md#低显存训练)。 --- ### Zh Index (`docs/zh/index.rst`) 欢迎来到 DiffSynth-Studio 的文档 ===================== .. toctree:: :maxdepth: 2 :caption: 文档介绍 README .. toctree:: :maxdepth: 2 :caption: 上手使用 Pipeline_Usage/Setup Pipeline_Usage/Model_Inference Pipeline_Usage/Accelerated_Inference Pipeline_Usage/VRAM_management Pipeline_Usage/Quantization Pipeline_Usage/Model_Training Pipeline_Usage/Environment_Variables Pipeline_Usage/GPU_support Pipeline_Usage/Inference_WebUI .. toctree:: :maxdepth: 2 :caption: 模型详解 Model_Details/FLUX Model_Details/Wan Model_Details/Qwen-Image Model_Details/Qwen-Video-Edit Model_Details/FLUX2 Model_Details/Z-Image Model_Details/Anima Model_Details/LTX-2 Model_Details/LTX-2.5 Model_Details/ERNIE-Image Model_Details/JoyAI-Image Model_Details/ACE-Step Model_Details/HiDream-O1-Image Model_Details/Stable-Diffusion Model_Details/Stable-Diffusion-XL Model_Details/Image-Quality-Metrics Model_Details/Ideogram-4 Model_Details/Krea-2 Model_Details/Boogu-Image Model_Details/LingBot-Video Model_Details/MiniMax-H3 Model_Details/DiffSynth-Music Model_Details/MiniMax-Music3 Model_Details/YuE2 Model_Details/SenseNova-U1 Model_Details/Qwen-Image-2.1 Model_Details/FLUX.3 Model_Details/MOSS-Image Model_Details/Fibo .. toctree:: :maxdepth: 2 :caption: 训练框架 Training/Understanding_Diffusion_models Training/Supervised_Fine_Tuning Training/FP8_Precision Training/Distillation Training/Direct_Distill Training/Split_Training Training/Differential_LoRA Training/DeepSpeed Training/Offload_Training .. toctree:: :maxdepth: 2 :caption: 模型接入 Developer_Guide/Integrating_Your_Model Developer_Guide/Building_a_Pipeline Developer_Guide/Enabling_VRAM_management Developer_Guide/Training_Diffusion_Models Developer_Guide/Integrating_Quantization_Backend .. toctree:: :maxdepth: 2 :caption: API 参考 API_Reference/core/attention API_Reference/core/data API_Reference/core/gradient API_Reference/core/loader API_Reference/core/quant API_Reference/core/vram .. toctree:: :maxdepth: 2 :caption: Diffusion Templates Diffusion_Templates/Introducing_Diffusion_Templates.md Diffusion_Templates/Understanding_Diffusion_Templates.md Diffusion_Templates/Template_Model_Inference.md Diffusion_Templates/Template_Model_Training.md .. toctree:: :maxdepth: 2 :caption: 学术导引 Research_Tutorial/train_from_scratch Research_Tutorial/inference_time_scaling Research_Tutorial/controllable_models .. toctree:: :maxdepth: 2 :caption: 常见问题 QA Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search` --- ### En Developer Guide Building A Pipeline (`docs/en/Developer_Guide/Building_a_Pipeline.md`) # Building a Pipeline After [integrating the required models for the Pipeline](../Developer_Guide/Integrating_Your_Model.md), you also need to build a `Pipeline` for model inference. This document provides a standardized process for building a `Pipeline`. Developers can also refer to existing `Pipeline` implementations for construction. The `Pipeline` implementation is located in `diffsynth/pipelines`. Each `Pipeline` contains the following essential key components: * `__init__` * `from_pretrained` * `__call__` * `units` * `model_fn` ## `__init__` In `__init__`, the `Pipeline` is initialized. Here is a simple implementation: ```python import torch from PIL import Image from typing import Union from tqdm import tqdm from ..diffusion import FlowMatchScheduler from ..core import ModelConfig from ..diffusion.base_pipeline import BasePipeline, PipelineUnit from ..models.new_models import XXX_Model, YYY_Model, ZZZ_Model class NewDiffSynthPipeline(BasePipeline): def __init__(self, device="cuda", torch_dtype=torch.bfloat16): super().__init__(device=device, torch_dtype=torch_dtype) self.scheduler = FlowMatchScheduler() self.text_encoder: XXX_Model = None self.dit: YYY_Model = None self.vae: ZZZ_Model = None self.in_iteration_models = ("dit",) self.units = [ NewDiffSynthPipelineUnit_xxx(), ... ] self.model_fn = model_fn_new ``` This includes the following parts: * `scheduler`: Scheduler, used to control the coefficients in the iterative formula during inference, controlling the noise content at each step. * `text_encoder`, `dit`, `vae`: Models. Since [Latent Diffusion](https://arxiv.org/abs/2112.10752) was proposed, this three-stage model architecture has become the mainstream Diffusion model architecture. However, this is not immutable, and any number of models can be added to the `Pipeline`. * `in_iteration_models`: Iteration models. This tuple marks which models will be called during iteration. * `units`: Pre-processing units for model iteration. See [`units`](#units) for details. * `model_fn`: The `forward` function of the denoising model during iteration. See [`model_fn`](#model_fn) for details. > Q: Model loading does not occur in `__init__`, why initialize each model as `None` here? > > A: By annotating the type of each model here, the code editor can provide code completion prompts based on each model, facilitating subsequent development. ## `from_pretrained` `from_pretrained` is responsible for loading the required models to make the `Pipeline` callable. Here is a simple implementation: ```python @staticmethod def from_pretrained( torch_dtype: torch.dtype = torch.bfloat16, device: Union[str, torch.device] = "cuda", model_configs: list[ModelConfig] = [], vram_limit: float = None, ): # Initialize pipeline pipe = NewDiffSynthPipeline(device=device, torch_dtype=torch_dtype) model_pool = pipe.download_and_load_models(model_configs, vram_limit) # Fetch models pipe.text_encoder = model_pool.fetch_model("xxx_text_encoder") pipe.dit = model_pool.fetch_model("yyy_dit") pipe.vae = model_pool.fetch_model("zzz_vae") # If necessary, load tokenizers here. # VRAM Management pipe.vram_management_enabled = pipe.check_vram_management_state() return pipe ``` Developers need to implement the logic for fetching models. The corresponding model names are the `"model_name"` in the [model Config filled in during model integration](../Developer_Guide/Integrating_Your_Model.md#step-3-writing-model-config). Some models also need to load `tokenizer`. Extra `tokenizer_config` parameters can be added to `from_pretrained` as needed, and this part can be implemented after fetching the models. ## `__call__` `__call__` implements the entire generation process of the Pipeline. Below is a common generation process template. Developers can modify it based on their needs. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## `units` `units` contains all the preprocessing processes, such as: width/height checking, prompt encoding, initial noise generation, etc. In the entire model preprocessing process, data is abstracted into three mutually exclusive parts, stored in corresponding dictionaries: * `inputs_shared`: Shared inputs, parameters unrelated to [Classifier-Free Guidance](https://arxiv.org/abs/2207.12598) (CFG for short). * `inputs_posi`: Positive side inputs for Classifier-Free Guidance, containing content related to positive prompts. * `inputs_nega`: Negative side inputs for Classifier-Free Guidance, containing content related to negative prompts. Pipeline Unit implementations include three types: direct mode, CFG separation mode, and takeover mode. If some calculations are unrelated to CFG, direct mode can be used, for example, Qwen-Image's random noise initialization: ```python class QwenImageUnit_NoiseInitializer(PipelineUnit): def __init__(self): super().__init__( input_params=("height", "width", "seed", "rand_device"), output_params=("noise",), ) def process(self, pipe: QwenImagePipeline, height, width, seed, rand_device): noise = pipe.generate_noise((1, 16, height//8, width//8), seed=seed, rand_device=rand_device, rand_torch_dtype=pipe.torch_dtype) return {"noise": noise} ``` If some calculations are related to CFG and need to separately process positive and negative prompts, but the input parameters on both sides are the same, CFG separation mode can be used, for example, Qwen-image's prompt encoding: ```python class QwenImageUnit_PromptEmbedder(PipelineUnit): def __init__(self): super().__init__( seperate_cfg=True, input_params_posi={"prompt": "prompt"}, input_params_nega={"prompt": "negative_prompt"}, input_params=("edit_image",), output_params=("prompt_emb", "prompt_emb_mask"), onload_model_names=("text_encoder",) ) def process(self, pipe: QwenImagePipeline, prompt, edit_image=None) -> dict: pipe.load_models_to_device(self.onload_model_names) # Do something return {"prompt_emb": prompt_embeds, "prompt_emb_mask": encoder_attention_mask} ``` If some calculations need global information, takeover mode is required, for example, Qwen-Image's entity partition control: ```python class QwenImageUnit_EntityControl(PipelineUnit): def __init__(self): super().__init__( take_over=True, input_params=("eligen_entity_prompts", "width", "height", "eligen_enable_on_negative", "cfg_scale"), output_params=("entity_prompt_emb", "entity_masks", "entity_prompt_emb_mask"), onload_model_names=("text_encoder",) ) def process(self, pipe: QwenImagePipeline, inputs_shared, inputs_posi, inputs_nega): # Do something return inputs_shared, inputs_posi, inputs_nega ``` The following are the parameter configurations required for Pipeline Unit: * `seperate_cfg`: Whether to enable CFG separation mode * `take_over`: Whether to enable takeover mode * `input_params`: Shared input parameters * `output_params`: Output parameters * `input_params_posi`: Positive side input parameters * `input_params_nega`: Negative side input parameters * `onload_model_names`: Names of model components to be called When designing `unit`, please try to follow these principles: * Default fallback: For optional function `unit` input parameters, the default is `None` rather than `False` or other values. Please provide fallback processing for this default value. * Parameter triggering: Some Adapter models may not be loaded, such as ControlNet. The corresponding `unit` should control triggering based on whether the parameter input is `None` rather than whether the model is loaded. For example, when the user inputs `controlnet_image` but does not load the ControlNet model, the code should give an error rather than ignore these input parameters and continue execution. * Simplicity first: Use direct mode as much as possible, only use takeover mode when the function cannot be implemented. * VRAM efficiency: When calling models in `unit`, please use `pipe.load_models_to_device(self.onload_model_names)` to activate the corresponding models. Do not call other models outside `onload_model_names`. After `unit` calculation is completed, do not manually release VRAM with `pipe.load_models_to_device([])`. > Q: Some parameters are not called during the inference process, such as `output_params`. Is it still necessary to configure them? > > A: These parameters will not affect the inference process, but they will affect some experimental features. Therefore, we recommend configuring them properly. For example, "split training" - we can complete the preprocessing offline during training, but some model calculations that require gradient backpropagation cannot be split. These parameters are used to build computational graphs to infer which calculations can be split. ## `model_fn` `model_fn` is the unified `forward` interface during iteration. For models where the open-source ecosystem is not yet formed, you can directly use the denoising model's `forward`, for example: ```python def model_fn_new(dit=None, latents=None, timestep=None, prompt_emb=None, **kwargs): return dit(latents, prompt_emb, timestep) ``` For models with rich open-source ecosystems, `model_fn` usually contains complex and chaotic cross-model inference. Taking `diffsynth/pipelines/qwen_image.py` as an example, the additional calculations implemented in this function include: entity partition control, three types of ControlNet, Gradient Checkpointing, etc. Developers need to be extra careful when implementing this part to avoid conflicts between module functions. ## Compilation Acceleration To enable compilation acceleration, please refer to [Inference Acceleration](../Pipeline_Usage/Accelerated_Inference.md). --- ### En Developer Guide Enabling VRAM Management (`docs/en/Developer_Guide/Enabling_VRAM_management.md`) # Fine-Grained VRAM Management Scheme This document introduces how to write reasonable fine-grained VRAM management schemes for models, and how to use the VRAM management functions in `DiffSynth-Studio` for other external code libraries. Before reading this document, please read the document [VRAM Management](../Pipeline_Usage/VRAM_management.md). ## How Much VRAM Does a 20B Model Need? Taking Qwen-Image's DiT model as an example, this model has reached 20B parameters. The following code will load this model and perform inference, requiring about 40G VRAM. This model obviously cannot run on consumer-grade GPUs with smaller VRAM. ```python from diffsynth.core import load_model from diffsynth.models.qwen_image_dit import QwenImageDiT from modelscope import snapshot_download import torch snapshot_download( model_id="Qwen/Qwen-Image", local_dir="models/Qwen/Qwen-Image", allow_file_pattern="transformer/*" ) prefix = "models/Qwen/Qwen-Image/transformer/diffusion_pytorch_model" model_path = [prefix + f"-0000{i}-of-00009.safetensors" for i in range(1, 10)] inputs = { "latents": torch.randn((1, 16, 128, 128), dtype=torch.bfloat16, device="cuda"), "timestep": torch.zeros((1,), dtype=torch.bfloat16, device="cuda"), "prompt_emb": torch.randn((1, 5, 3584), dtype=torch.bfloat16, device="cuda"), "prompt_emb_mask": torch.ones((1, 5), dtype=torch.int64, device="cuda"), "height": 1024, "width": 1024, } model = load_model(QwenImageDiT, model_path, torch_dtype=torch.bfloat16, device="cuda") with torch.no_grad(): output = model(**inputs) ``` ## Writing Fine-Grained VRAM Management Scheme To write a fine-grained VRAM management scheme, we need to use `print(model)` to observe and analyze the model structure: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` In VRAM management, we only care about layers containing parameters. In this model structure, `QwenEmbedRope`, `TemporalTimesteps`, `SiLU` and other Layers do not contain parameters. `LayerNorm` also does not contain parameters because `elementwise_affine=False` is set. Layers containing parameters are only `Linear` and `RMSNorm`. `diffsynth.core.vram` provides two replacement modules for VRAM management: * `AutoWrappedLinear`: Used to replace `Linear` layers * `AutoWrappedModule`: Used to replace any other layer Write a `module_map` to map `Linear` and `RMSNorm` in the model to the corresponding modules: ```python module_map={ torch.nn.Linear: AutoWrappedLinear, RMSNorm: AutoWrappedModule, } ``` In addition, `vram_config` and `vram_limit` are also required, which have been introduced in [VRAM Management](../Pipeline_Usage/VRAM_management.md#more-usage-methods). Call `enable_vram_management` to enable VRAM management. Note that the `device` when loading the model is `cpu`, consistent with `offload_device`: ```python from diffsynth.core import load_model, enable_vram_management, AutoWrappedLinear, AutoWrappedModule from diffsynth.models.qwen_image_dit import QwenImageDiT, RMSNorm import torch prefix = "models/Qwen/Qwen-Image/transformer/diffusion_pytorch_model" model_path = [prefix + f"-0000{i}-of-00009.safetensors" for i in range(1, 10)] inputs = { "latents": torch.randn((1, 16, 128, 128), dtype=torch.bfloat16, device="cuda"), "timestep": torch.zeros((1,), dtype=torch.bfloat16, device="cuda"), "prompt_emb": torch.randn((1, 5, 3584), dtype=torch.bfloat16, device="cuda"), "prompt_emb_mask": torch.ones((1, 5), dtype=torch.int64, device="cuda"), "height": 1024, "width": 1024, } model = load_model(QwenImageDiT, model_path, torch_dtype=torch.bfloat16, device="cpu") enable_vram_management( model, module_map={ torch.nn.Linear: AutoWrappedLinear, RMSNorm: AutoWrappedModule, }, vram_config = { "offload_dtype": torch.bfloat16, "offload_device": "cpu", "onload_dtype": torch.bfloat16, "onload_device": "cpu", "preparing_dtype": torch.bfloat16, "preparing_device": "cuda", "computation_dtype": torch.bfloat16, "computation_device": "cuda", }, vram_limit=0, ) with torch.no_grad(): output = model(**inputs) ``` The above code only requires 2G VRAM to run the `forward` of a 20B model. ## Disk Offload [Disk Offload](../Pipeline_Usage/VRAM_management.md#disk-offload) is a special VRAM management scheme that needs to be enabled during the model loading process, not after the model is loaded. Usually, when the above code can run smoothly, Disk Offload can be directly enabled: ```python from diffsynth.core import load_model, enable_vram_management, AutoWrappedLinear, AutoWrappedModule from diffsynth.models.qwen_image_dit import QwenImageDiT, RMSNorm import torch prefix = "models/Qwen/Qwen-Image/transformer/diffusion_pytorch_model" model_path = [prefix + f"-0000{i}-of-00009.safetensors" for i in range(1, 10)] inputs = { "latents": torch.randn((1, 16, 128, 128), dtype=torch.bfloat16, device="cuda"), "timestep": torch.zeros((1,), dtype=torch.bfloat16, device="cuda"), "prompt_emb": torch.randn((1, 5, 3584), dtype=torch.bfloat16, device="cuda"), "prompt_emb_mask": torch.ones((1, 5), dtype=torch.int64, device="cuda"), "height": 1024, "width": 1024, } model = load_model( QwenImageDiT, model_path, module_map={ torch.nn.Linear: AutoWrappedLinear, RMSNorm: AutoWrappedModule, }, vram_config={ "offload_dtype": "disk", "offload_device": "disk", "onload_dtype": "disk", "onload_device": "disk", "preparing_dtype": torch.bfloat16, "preparing_device": "cuda", "computation_dtype": torch.bfloat16, "computation_device": "cuda", }, vram_limit=0, ) with torch.no_grad(): output = model(**inputs) ``` Disk Offload is an extremely special VRAM management scheme. It only supports `.safetensors` format files, not binary files such as `.bin`, `.pth`, `.ckpt`, and does not support [state dict converter](../Developer_Guide/Integrating_Your_Model.md#step-2-model-file-format-conversion) with Tensor reshape. If there are situations where Disk Offload cannot run normally but non-Disk Offload can run normally, please submit an issue to us on GitHub. ## Writing Default Configuration To make it easier for users to use the VRAM management function, we write the fine-grained VRAM management configuration in `diffsynth/configs/vram_management_module_maps.py`. The configuration information for the above model is: ```python "diffsynth.models.qwen_image_dit.QwenImageDiT": { "diffsynth.models.qwen_image_dit.RMSNorm": "diffsynth.core.vram.layers.AutoWrappedModule", "torch.nn.Linear": "diffsynth.core.vram.layers.AutoWrappedLinear", } ```# Fine-Grained VRAM Management Scheme This document introduces how to write reasonable fine-grained VRAM management schemes for models, and how to use the VRAM management functions in `DiffSynth-Studio` for other external code libraries. Before reading this document, please read the document [VRAM Management](../Pipeline_Usage/VRAM_management.md). ## How Much VRAM Does a 20B Model Need? Taking Qwen-Image's DiT model as an example, this model has reached 20B parameters. The following code will load this model and perform inference, requiring about 40G VRAM. This model obviously cannot run on consumer-grade GPUs with smaller VRAM. ```python from diffsynth.core import load_model from diffsynth.models.qwen_image_dit import QwenImageDiT from modelscope import snapshot_download import torch snapshot_download( model_id="Qwen/Qwen-Image", local_dir="models/Qwen/Qwen-Image", allow_file_pattern="transformer/*" ) prefix = "models/Qwen/Qwen-Image/transformer/diffusion_pytorch_model" model_path = [prefix + f"-0000{i}-of-00009.safetensors" for i in range(1, 10)] inputs = { "latents": torch.randn((1, 16, 128, 128), dtype=torch.bfloat16, device="cuda"), "timestep": torch.zeros((1,), dtype=torch.bfloat16, device="cuda"), "prompt_emb": torch.randn((1, 5, 3584), dtype=torch.bfloat16, device="cuda"), "prompt_emb_mask": torch.ones((1, 5), dtype=torch.int64, device="cuda"), "height": 1024, "width": 1024, } model = load_model(QwenImageDiT, model_path, torch_dtype=torch.bfloat16, device="cuda") with torch.no_grad(): output = model(**inputs) ``` ## Writing Fine-Grained VRAM Management Scheme To write a fine-grained VRAM management scheme, we need to use `print(model)` to observe and analyze the model structure: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` In VRAM management, we only care about layers containing parameters. In this model structure, `QwenEmbedRope`, `TemporalTimesteps`, `SiLU` and other Layers do not contain parameters. `LayerNorm` also does not contain parameters because `elementwise_affine=False` is set. Layers containing parameters are only `Linear` and `RMSNorm`. `diffsynth.core.vram` provides two replacement modules for VRAM management: * `AutoWrappedLinear`: Used to replace `Linear` layers * `AutoWrappedModule`: Used to replace any other layer Write a `module_map` to map `Linear` and `RMSNorm` in the model to the corresponding modules: ```python module_map={ torch.nn.Linear: AutoWrappedLinear, RMSNorm: AutoWrappedModule, } ``` In addition, `vram_config` and `vram_limit` are also required, which have been introduced in [VRAM Management](../Pipeline_Usage/VRAM_management.md#more-usage-methods). Call `enable_vram_management` to enable VRAM management. Note that the `device` when loading the model is `cpu`, consistent with `offload_device`: ```python from diffsynth.core import load_model, enable_vram_management, AutoWrappedLinear, AutoWrappedModule from diffsynth.models.qwen_image_dit import QwenImageDiT, RMSNorm import torch prefix = "models/Qwen/Qwen-Image/transformer/diffusion_pytorch_model" model_path = [prefix + f"-0000{i}-of-00009.safetensors" for i in range(1, 10)] inputs = { "latents": torch.randn((1, 16, 128, 128), dtype=torch.bfloat16, device="cuda"), "timestep": torch.zeros((1,), dtype=torch.bfloat16, device="cuda"), "prompt_emb": torch.randn((1, 5, 3584), dtype=torch.bfloat16, device="cuda"), "prompt_emb_mask": torch.ones((1, 5), dtype=torch.int64, device="cuda"), "height": 1024, "width": 1024, } model = load_model(QwenImageDiT, model_path, torch_dtype=torch.bfloat16, device="cpu") enable_vram_management( model, module_map={ torch.nn.Linear: AutoWrappedLinear, RMSNorm: AutoWrappedModule, }, vram_config = { "offload_dtype": torch.bfloat16, "offload_device": "cpu", "onload_dtype": torch.bfloat16, "onload_device": "cpu", "preparing_dtype": torch.bfloat16, "preparing_device": "cuda", "computation_dtype": torch.bfloat16, "computation_device": "cuda", }, vram_limit=0, ) with torch.no_grad(): output = model(**inputs) ``` The above code only requires 2G VRAM to run the `forward` of a 20B model. ## Disk Offload [Disk Offload](../Pipeline_Usage/VRAM_management.md#disk-offload) is a special VRAM management scheme that needs to be enabled during the model loading process, not after the model is loaded. Usually, when the above code can run smoothly, Disk Offload can be directly enabled: ```python from diffsynth.core import load_model, enable_vram_management, AutoWrappedLinear, AutoWrappedModule from diffsynth.models.qwen_image_dit import QwenImageDiT, RMSNorm import torch prefix = "models/Qwen/Qwen-Image/transformer/diffusion_pytorch_model" model_path = [prefix + f"-0000{i}-of-00009.safetensors" for i in range(1, 10)] inputs = { "latents": torch.randn((1, 16, 128, 128), dtype=torch.bfloat16, device="cuda"), "timestep": torch.zeros((1,), dtype=torch.bfloat16, device="cuda"), "prompt_emb": torch.randn((1, 5, 3584), dtype=torch.bfloat16, device="cuda"), "prompt_emb_mask": torch.ones((1, 5), dtype=torch.int64, device="cuda"), "height": 1024, "width": 1024, } model = load_model( QwenImageDiT, model_path, module_map={ torch.nn.Linear: AutoWrappedLinear, RMSNorm: AutoWrappedModule, }, vram_config={ "offload_dtype": "disk", "offload_device": "disk", "onload_dtype": "disk", "onload_device": "disk", "preparing_dtype": torch.bfloat16, "preparing_device": "cuda", "computation_dtype": torch.bfloat16, "computation_device": "cuda", }, vram_limit=0, ) with torch.no_grad(): output = model(**inputs) ``` Disk Offload is an extremely special VRAM management scheme. It only supports `.safetensors` format files, not binary files such as `.bin`, `.pth`, `.ckpt`, and does not support [state dict converter](../Developer_Guide/Integrating_Your_Model.md#step-2-model-file-format-conversion) with Tensor reshape. If there are situations where Disk Offload cannot run normally but non-Disk Offload can run normally, please submit an issue to us on GitHub. ## Writing Default Configuration To make it easier for users to use the VRAM management function, we write the fine-grained VRAM management configuration in `diffsynth/configs/vram_management_module_maps.py`. The configuration information for the above model is: ```python "diffsynth.models.qwen_image_dit.QwenImageDiT": { "diffsynth.models.qwen_image_dit.RMSNorm": "diffsynth.core.vram.layers.AutoWrappedModule", "torch.nn.Linear": "diffsynth.core.vram.layers.AutoWrappedLinear", } ``` --- ### En Developer Guide Integrating Quantization Backend (`docs/en/Developer_Guide/Integrating_Quantization_Backend.md`) # Integrating a Quantization Backend The quantization framework of `DiffSynth-Studio` lives in `diffsynth.core.quant` and ships with bitsandbytes, torchao, and comfy-kitchen backends (see [Model Quantization](../Pipeline_Usage/Quantization.md)). If you have your own quantization algorithm, or want to plug in another quantization library, you only need to implement a `QuantBackend` — online quantization, saving/loading pre-quantized checkpoints, mixed quantization, VRAM management, and quantization + LoRA training are all reused as-is. This guide walks through the whole process with a toy backend: **INT9** — 9-bit symmetric weight-only quantization, genuinely stored at 9 bits per weight, with one fp32 scale per output channel. INT9 does not exist on any hardware; it is used here because it keeps the example short while still covering every interface you have to implement. For the full interface signatures and contracts, see the [`diffsynth.core.quant` API documentation](../API_Reference/core/quant.md#extension-interface-custom-backends). ## Framework Structure The framework has three layers: - **`QuantizeConfig`**: the user-facing config and entry point, responsible for traversing the model, matching layers, and replacing `nn.Linear`. You never need to touch it. - **`QuantBackend`**: the adapter layer, which only ever deals with a **single** `nn.Linear`: how to quantize it, how to build an empty shell, how to dequantize it, how to read and write its state dict. This is the part you implement. - **The quantized Linear**: the module that actually holds the quantized weight and performs dequantization + matmul in `forward`. The quantized Linear must satisfy four contract clauses: - **(a)** It is a drop-in replacement for `nn.Linear`, with `forward(x)` doing dequantization + matmul internally. It must subclass `torch.nn.Linear`, otherwise LoRA injection and VRAM management cannot see it. - **(b)** `.to(...)` only moves devices, never re-types the packed weight or quant state. VRAM management performs dtype conversions on the model; if a packed weight were cast to bf16, the quant state would be corrupted. - **(c)** `state_dict()` and `load_state_dict(assign=True)` round-trip, via `flatten_state_dict` / `unflatten_state_dict` when necessary. - **(d)** (Training only) `forward` is differentiable w.r.t. its input, so gradients can pass through frozen quantized layers to reach LoRA branches. ## Step 1: Write the Quantized Linear INT9's storage layout needs a little thought: there is no native 9-bit dtype, and simply putting the codes into an int16 tensor would still spend 16 bits per weight — exactly as much as bf16, so the quantization would save nothing. Each weight is therefore split in two: the low 8 bits go into a uint8 `weight` buffer, and the 9th (most significant) bit forms a separate bit plane where 8 weights are packed into one byte in `weight_msb`, plus one fp32 `weight_scale` per output channel. That is 9 bits per weight, 56% of bf16. Two more details matter: - Delete `nn.Linear`'s original `weight` parameter and register a buffer under the same name, so checkpoint keys stay `layer_name.weight`. Disk offload and mixed-quantization key ownership rely on this. - Guard the dtype of the packed tensors by overriding `_apply`, i.e. contract clause (b). Every conversion (`.to()`, `.half()`, `.float()`, ...) funnels through `_apply`, so a conversion that would change the dtype is downgraded to a device-only move. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Dequantization in `forward` uses ordinary tensor ops, so gradients flow back to the input `x` through `F.linear` and contract clause (d) holds automatically. Unpacking here is written bit by bit with PyTorch ops purely for clarity; a real backend fuses unpacking into the matmul kernel instead of materializing an fp weight on every forward. There is an easy trap in `dequantize_weight`: the integer codes must be reconstructed in fp32. bf16 only carries 8 bits of significand, so integers above 256 are not representable exactly; casting the codes to bf16 before applying the scale rounds the 9th bit away and throws the accuracy gain out (measured: the advantage over int8 collapses from 2.25x to 1.15x). This applies to any format whose code width exceeds the significand of the compute dtype. ## Step 2: Write the Backend Every backend method operates on a single `nn.Linear`: - `capabilities()`: declares what the backend supports; all four flags default to `False`. Saving quantized weights requires `is_serializable=True`, and quantization + LoRA training requires `is_differentiable=True`. - `quantized_linear_classes()`: declares the Linear classes this backend produces; `is_quantized_linear` defaults to an `isinstance` check against them. - `create_quantized_linear()`: online quantization, turning an fp `nn.Linear` into a quantized one. `compute_device` is where quantization runs and `model_device` is where the result is stored, so the two together stream the work layer by layer with only one layer on the accelerator at a time. - `create_quantized_linear_shell()`: builds an empty shell, used for loading pre-quantized checkpoints and for disk offload. It is rebuilt on every offload cycle, so build it on the `meta` device and keep it cheap. - `dequantize_to_linear()`: restores a plain `nn.Linear`, used by `mode="dequant_once"`. - `flatten_state_dict` / `unflatten_state_dict`: conversion between the state dict and flat tensors. INT9's state dict already holds plain tensors, so the base class implementation is enough; only backends with composite tensors (tensor subclasses, nested quant state) such as bitsandbytes and torchao need to override them. Unimplemented methods raise a descriptive exception from the base class, so a backend that only supports some capabilities can implement just what it needs. `self.config` is the backend config instance injected by the framework — the `Int9WeightOnlyConfig` written in the next step. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Step 3: Write the Backend Config The backend config subclasses `BackendConfig`: user-tunable parameters are ordinary dataclass fields, while values pinned by the method are declared with `field(init=False, default=...)`. `describe_quant_method` reports the two groups separately, and `from_kwargs` raises when a user passes unknown `backend_config_kwargs`. ```python @dataclass class Int9WeightOnlyConfig(BackendConfig): per_channel: bool = True # user-tunable: per-channel or per-tensor bits: int = field(init=False, default=9) # pinned by the method, not overridable ``` ## Step 4: Register the Quantization Method One backend can register several methods, distinguished by the fields pinned in its config (the bitsandbytes backend, for example, distinguishes nf4 from fp4 via `quant_type`). Method names should follow the `__wa` convention: ```python register_quant_method("toy_int9_w9a16", "toy_int9", Int9WeightOnlyConfig.from_kwargs, label="9bit, int9, weight-only (toy)") ``` There are two ways to register a backend and its methods: **Option 1: keep it in your own code (recommended, plug-and-play).** Put the code above in any module; as long as that module is imported before you construct `QuantizeConfig`, the method is already in `QUANT_METHODS` and can be used just like a built-in one, with no framework changes: ```python import my_project.toy_int9 # triggers register_quant_backend / register_quant_method from diffsynth.core.quant import QuantizeConfig quantize = QuantizeConfig(method="toy_int9_w9a16", backend_config_kwargs={"per_channel": True}) ``` **Option 2: ship it as a built-in backend (permanent).** Put the backend file under `diffsynth/core/quant/backends/` and register it in `_LAZY_BACKENDS` in `diffsynth/core/quant/backends/__init__.py`; the framework then imports it on demand and users do not need to import anything: ```python _LAZY_BACKENDS = { "bitsandbytes": ".bitsandbytes", "torchao": ".torchao", "comfy_kitchen": ".comfy_kitchen", "toy_int9": ".toy_int9", } ``` If your quantization algorithm or library is generally useful, you are welcome to submit it as a PR following Option 2, so that more users can benefit from it. A backend that depends on a third-party library should check its dependencies in `validate_environment()` with an installation hint, and point `project_url` at the upstream project. ## Step 5: Self-Check The framework provides two verification tools; run them right after integrating. `check_backend_contract` verifies that the backend declares its Linear classes, that both factory methods return instances of those classes, that all declared classes subclass `nn.Linear`, and that every checkpoint key the backend actually writes lives under the layer name (a missing scale would make disk offload silently load corrupted layers). Unsupported factory methods are skipped rather than counted as failures. ```python from diffsynth.core.quant import QUANT_BACKENDS, QUANT_METHODS, check_backend_contract, check_differentiable, describe_quant_method describe_quant_method("toy_int9_w9a16") spec = QUANT_METHODS["toy_int9_w9a16"] check_backend_contract(QUANT_BACKENDS[spec.backend](spec.config_factory({})), compute_device="cpu") ``` The output is as follows; `describe_quant_method` also confirms that the split between user-tunable and pinned parameters is what you intended: ``` method: toy_int9_w9a16 backend: toy_int9 detail: 9bit, int9, weight-only (toy) backend config: my_project.toy_int9.Int9WeightOnlyConfig backend_config_kwargs (user-tunable): per_channel = True pinned by method (not overridable): bits = 9 check_backend_contract (toy_int9): [PASS] quantized_linear_classes() is non-empty: ['Int9Linear'] [PASS] Int9Linear subclasses torch.nn.Linear [PASS] a plain nn.Linear is not reported as quantized [PASS] create_quantized_linear_shell() returns a declared class, got Int9Linear [PASS] the shell is recognized before load_state_dict (disk offload routing) [PASS] create_quantized_linear() returns a declared class, got Int9Linear [PASS] every stored key lives under the layer name; uncovered: [] => OK ``` Next, check the numerical error, the real memory saving, the dtype guard of clause (b), and the differentiability of clause (d) on a small model: ```python import torch from diffsynth.core.quant import QuantizeConfig, check_differentiable class ToyModel(torch.nn.Module): def __init__(self): super().__init__() self.fc1 = torch.nn.Linear(256, 512) self.fc2 = torch.nn.Linear(512, 256, bias=False) def forward(self, x): return self.fc2(torch.nn.functional.silu(self.fc1(x))) def footprint(model): return sum(t.numel() * t.element_size() for t in list(model.parameters()) + list(model.buffers())) torch.manual_seed(0) model = ToyModel().to(torch.bfloat16) x = torch.randn(4, 256, dtype=torch.bfloat16) reference = model(x) fp_bytes = footprint(model) QuantizeConfig(method="toy_int9_w9a16").quantize_model(model, compute_device="cpu") print("relative error:", ((model(x) - reference).norm() / reference.norm()).item()) print(f"footprint: {fp_bytes} -> {footprint(model)} bytes ({footprint(model) / fp_bytes:.3f} of bf16)") model.to(torch.float32) # clause (b): packed dtypes must not change print(model.fc1.weight.dtype, model.fc1.weight_msb.dtype, model.fc1.weight_scale.dtype, model.fc1.bias.dtype) check_differentiable(model.fc1) # clause (d) ``` ``` 2 nn.Linear layers quantized (method: toy_int9_w9a16). relative error: 0.004150390625 footprint: 525312 -> 299008 bytes (0.569 of bf16) torch.uint8 torch.uint8 torch.float32 torch.float32 check_differentiable (Int9Linear): OK -- gradients pass through the module to its input ``` The measured footprint is 0.569 of bf16, slightly above 9/16 = 0.5625 because of the fp32 scales and the unquantized bias. If this ratio comes out close to 1, the packing format is not actually compressing the weights and you should revisit the storage layout in Step 1. ### Inference on a Real Model: Z-Image Once the small-model checks pass, the backend is ready for real models — a custom backend is used exactly like a built-in method. Import the module that registers it, then pass the method to `ModelConfig(quantize=...)`: ```python import torch import my_project.toy_int9 # registers the toy_int9 backend and the toy_int9_w9a16 method from diffsynth.core.quant import QuantizeConfig from diffsynth.pipelines.z_image import ModelConfig, ZImagePipeline pipe = ZImagePipeline.from_pretrained( torch_dtype=torch.bfloat16, device="cuda", model_configs=[ ModelConfig( model_id="Tongyi-MAI/Z-Image-Turbo", origin_file_pattern="transformer/*.safetensors", quantize=QuantizeConfig(method="toy_int9_w9a16"), ), ModelConfig(model_id="Tongyi-MAI/Z-Image-Turbo", origin_file_pattern="text_encoder/*.safetensors"), ModelConfig(model_id="Tongyi-MAI/Z-Image-Turbo", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"), ], tokenizer_config=ModelConfig(model_id="Tongyi-MAI/Z-Image-Turbo", origin_file_pattern="tokenizer/"), ) dit_bytes = sum(t.numel() * t.element_size() for t in list(pipe.dit.parameters()) + list(pipe.dit.buffers())) print(f"dit weights: {dit_bytes / 1024 ** 3:.3f} GiB") prompt = "A delicate portrait of an underwater girl, blue dress flowing, hair gently drifting, light and shadow clear, surrounded by bubbles, serene expression, exquisite details, dreamlike and beautiful." image = pipe(prompt=prompt, seed=42, rand_device="cuda") image.save("z_image_toy_int9.jpg") ``` Measured DiT weight footprint on Z-Image Turbo (the 8-step Turbo generation works normally, with no visible quality difference from bf16): | | DiT weights | | --- | --- | | bf16 | 11.464 GiB | | `toy_int9_w9a16` | 6.456 GiB (0.563x) | Note that peak memory and weight footprint are not the same thing: this toy materializes a temporary fp weight on every forward, so the peak saving is smaller than the storage saving. Measured on a synthetic model with 48 Linears, all weights resident on the GPU: | | Weights | Forward peak | | --- | --- | --- | | bf16 | 1.500 GiB | 1.527 GiB | | `toy_int9_w9a16` | 0.845 GiB (0.563x) | 1.036 GiB (0.678x) | That temporary weight depends only on the **largest single layer** and does not grow with depth, so the deeper the model, the closer the peak saving gets to the weight ratio; a real backend fusing unpacking into the matmul kernel does not need it at all. To push the peak down further, stack [VRAM management](../Pipeline_Usage/VRAM_management.md) on top and move weights layer by layer (pass a `vram_config` to each `ModelConfig` above — measured peak drops to 2.1 GiB). ### Accuracy: int9 vs int8 Does the extra bit actually buy accuracy? Quantize the same weight with the **identical** per-channel symmetric scheme at 8 and 9 bits, then compare the dequantized weight error and the layer output error. This is the general recipe for an accuracy regression on a new backend: hold everything else fixed and change only the bit width. ```python import torch from my_project.toy_int9 import Int9QuantBackend, Int9WeightOnlyConfig def quantize_int8(linear): """The same per-channel symmetric scheme with one bit less: codes in [-128, 127].""" weight = linear.weight.data scale = (weight.abs().amax(dim=1).float() / 127).clamp(min=1e-8) codes = (weight.float() / scale.unsqueeze(1)).round().clamp(-128, 127) return (codes * scale.unsqueeze(1)).to(weight.dtype) def relative_error(reference, value): return ((value.float() - reference.float()).norm() / reference.float().norm()).item() torch.manual_seed(0) backend = Int9QuantBackend(Int9WeightOnlyConfig()) linear = torch.nn.Linear(2048, 2048, bias=False).to(torch.bfloat16) fp_weight = linear.weight.data.clone() int9_weight = backend.create_quantized_linear(linear).dequantize_weight(torch.bfloat16) int8_weight = quantize_int8(linear) error8, error9 = relative_error(fp_weight, int8_weight), relative_error(fp_weight, int9_weight) print(f"weight error: int8 {error8:.6f} | int9 {error9:.6f} ({error8 / error9:.2f}x lower)") x = torch.randn(64, 2048, dtype=torch.bfloat16) reference = torch.nn.functional.linear(x, fp_weight) out8 = relative_error(reference, torch.nn.functional.linear(x, int8_weight)) out9 = relative_error(reference, torch.nn.functional.linear(x, int9_weight)) print(f"output error: int8 {out8:.6f} | int9 {out9:.6f} ({out8 / out9:.2f}x lower)") ``` ``` weight error: int8 0.004353 | int9 0.001937 (2.25x lower) output error: int8 0.004947 | int9 0.002816 (1.76x lower) ``` This matches the theory: going from 255 to 511 levels halves the quantization step, and for uniform quantization the error is proportional to the step, so the weight error drops to roughly half (2.25x measured). The end-to-end layer output gain is smaller (1.76x) because the activations themselves are bf16 and the matmul's own rounding noise eats part of the benefit — a reminder to evaluate bit-width gains at the actual compute precision, not only on the weights. Finally, verify clause (c): save the quantized weights, load them back into shells, and confirm both produce identical outputs. ```python from safetensors.torch import load_file, save_file save_config = QuantizeConfig(method="toy_int9_w9a16") tensors, metadata = save_config.flatten_state_dict(model.state_dict()) save_file(tensors, "toy_int9.safetensors", metadata=metadata) loaded = ToyModel().to(torch.bfloat16) load_config = QuantizeConfig(method="toy_int9_w9a16", load_prequantized=True) load_config.prepare_for_prequantized_load(loaded, compute_dtype=torch.bfloat16) loaded.load_state_dict(load_config.unflatten_state_dict(load_file("toy_int9.safetensors"), metadata), assign=True) print("reload match:", torch.equal(loaded(x.float()), model(x.float()))) ``` ``` reload match: True ``` ### Combining with Disk Offload Disk offload, part of [VRAM management](../Pipeline_Usage/VRAM_management.md), places the strictest demands on a quantization backend: the resident model keeps only `meta` shells, and each layer's tensors are streamed back from disk at forward time and dropped right after. It relies on two things: - It only supports **pre-quantized checkpoints**, so `load_prequantized=True` is required and `prepare_for_prequantized_load` must first swap the target layers for shells. - Which tensors a layer needs is resolved by a prefix scan over the checkpoint keys using the layer's dotted name, and the result is loaded with a strict `load_state_dict(assign=True)`. So the only requirement on a backend is that every tensor lives under `{layer_name}.` — flat siblings like `layer.weight_scale` and nested quant state like bnb's both work. A missing or extra key raises instead of silently loading a corrupted layer. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Measured on the same `ToyModel` (`torch_dtype=None` on `DiskMap` is essential — it guarantees the packed tensors are not re-typed while being read): ``` 2 nn.Linear layers replaced for loading the pre-quantized checkpoint (method: toy_int9_w9a16). fc1: ['fc1.bias', 'fc1.weight', 'fc1.weight_msb', 'fc1.weight_scale'] fc2: ['fc2.weight', 'fc2.weight_msb', 'fc2.weight_scale'] resident 299008 bytes -> in memory after disk offload 0 bytes output matches: True | repeatable: True ``` Each layer's `weight` / `weight_msb` / `weight_scale` / `bias` is correctly attributed to that layer, the resident footprint drops to 0 bytes (everything is a `meta` shell), the output is bit-identical to the resident quantized model, and repeated forwards stay stable — so rebuilding shells and streaming from disk has no side effects. On a real model, use the standard workflows from [Model Quantization](../Pipeline_Usage/Quantization.md) for end-to-end validation: pass `QuantizeConfig(method="toy_int9_w9a16")` to `ModelConfig(quantize=...)` for online-quantized inference, save the quantized weights with `save_quantized_model` and load them back after registering the hash, and inject LoRA into the quantized model for training. ## Integration Checklist - The packing format really shrinks the weights: the measured footprint ratio should be close to the theoretical bit-width ratio, not close to 1. - The accuracy gain is verified: compared against the same scheme with one bit less, the error really goes down; otherwise precision is being lost somewhere in the dequantization path. - The quantized Linear subclasses `torch.nn.Linear`, and all `state_dict` keys live under the layer name. - `_apply` guards the dtype of every packed tensor and quant state. - `capabilities()` matches reality: declaring `is_serializable` requires a round-tripping state dict, and declaring `is_differentiable` requires passing `check_differentiable`. - `create_quantized_linear` honors `compute_device` / `model_device`, so layer-by-layer streaming quantization works. - Disk offload works: the shell is built on `meta` and cheap to rebuild, every stored tensor lives under the layer's dotted name, and `unflatten_state_dict` tolerates being called with a single layer's subdict plus whole-file metadata. - When depending on a third-party library, `validate_environment()` gives a clear installation hint and `project_url` points at the upstream project. - `check_backend_contract` passes completely. --- ### En Developer Guide Integrating Your Model (`docs/en/Developer_Guide/Integrating_Your_Model.md`) # Integrating Model Architecture This document introduces how to integrate models into the `DiffSynth-Studio` framework for use by modules such as `Pipeline`. ## Step 1: Integrate Model Architecture Code All model architecture implementations in `DiffSynth-Studio` are unified in `diffsynth/models`. Each `.py` code file implements a model architecture, and all models are loaded through `ModelPool` in `diffsynth/models/model_loader.py`. When integrating new model architectures, please create a new `.py` file under this path. ```shell diffsynth/models/ ├── general_modules.py ├── model_loader.py ├── qwen_image_controlnet.py ├── qwen_image_dit.py ├── qwen_image_text_encoder.py ├── qwen_image_vae.py └── ... ``` In most cases, we recommend integrating models in native `PyTorch` code form, with the model architecture class directly inheriting from `torch.nn.Module`, for example: ```python import torch class NewDiffSynthModel(torch.nn.Module): def __init__(self, dim=1024): super().__init__() self.linear = torch.nn.Linear(dim, dim) self.activation = torch.nn.Sigmoid() def forward(self, x): x = self.linear(x) x = self.activation(x) return x ``` If the model architecture implementation contains additional dependencies, we strongly recommend removing them, otherwise this will cause heavy package dependency issues. In our existing models, Qwen-Image's Blockwise ControlNet is integrated in this way. The code is lightweight, please refer to `diffsynth/models/qwen_image_controlnet.py`. If the model has been integrated by Huggingface Library ([`transformers`](https://huggingface.co/docs/transformers/main/index), [`diffusers`](https://huggingface.co/docs/diffusers/main/index), etc.), we can integrate the model in a simpler way:
Integrating Huggingface Library Style Model Architecture Code The loading method for these models in Huggingface Library is: ```python from transformers import XXX_Model model = XXX_Model.from_pretrained("path_to_your_model") ``` `DiffSynth-Studio` does not support loading models through `from_pretrained` because this conflicts with VRAM management and other functions. Please rewrite the model architecture in the following format: ```python import torch class DiffSynth_XXX_Model(torch.nn.Module): def __init__(self): super().__init__() from transformers import XXX_Config, XXX_Model config = XXX_Config(**{ "architectures": ["XXX_Model"], "other_configs": "Please copy and paste the other configs here.", }) self.model = XXX_Model(config) def forward(self, x): outputs = self.model(x) return outputs ``` Where `XXX_Config` is the Config class corresponding to the model. For example, the Config class for `Qwen2_5_VLModel` is `Qwen2_5_VLConfig`, which can be found by consulting its source code. The content inside Config can usually be found in the `config.json` file in the model library. `DiffSynth-Studio` will not read the `config.json` file, so the content needs to be copied and pasted into the code. In rare cases, version updates of `transformers` and `diffusers` may cause some models to be unable to import. Therefore, if possible, we still recommend using the model integration method in Step 1.1. In our existing models, Qwen-Image's Text Encoder is integrated in this way. The code is lightweight, please refer to `diffsynth/models/qwen_image_text_encoder.py`.
## Step 2: Model File Format Conversion Due to the variety of model file formats provided by developers in the open-source community, we sometimes need to convert model file formats to form correctly formatted [state dict](https://docs.pytorch.org/tutorials/recipes/recipes/what_is_state_dict.html). This is common in the following situations: * Model files built by different code libraries, for example [Wan-AI/Wan2.1-T2V-1.3B](https://www.modelscope.cn/models/Wan-AI/Wan2.1-T2V-1.3B) and [Wan-AI/Wan2.1-T2V-1.3B-Diffusers](https://www.modelscope.cn/models/Wan-AI/Wan2.1-T2V-1.3B-Diffusers). * Models modified during integration, for example, the Text Encoder of [Qwen/Qwen-Image](https://www.modelscope.cn/models/Qwen/Qwen-Image) adds a `model.` prefix in `diffsynth/models/qwen_image_text_encoder.py`. * Model files containing multiple models, for example, the VACE Adapter and base DiT model of [Wan-AI/Wan2.1-VACE-14B](https://www.modelscope.cn/models/Wan-AI/Wan2.1-VACE-14B) are mixed and stored in the same set of model files. In our development philosophy, we hope to respect the wishes of model authors as much as possible. If we repackage the model files, for example [Comfy-Org/Qwen-Image_ComfyUI](https://www.modelscope.cn/models/Comfy-Org/Qwen-Image_ComfyUI), although we can call the model more conveniently, traffic (model page views and downloads, etc.) will be directed elsewhere, and the original author of the model will also lose the power to delete the model. Therefore, we have added the `diffsynth/utils/state_dict_converters` module to the framework for file format conversion during model loading. This part of logic is very simple. Taking Qwen-Image's Text Encoder as an example, only 10 lines of code are needed: ```python def QwenImageTextEncoderStateDictConverter(state_dict): state_dict_ = {} for k in state_dict: v = state_dict[k] if k.startswith("visual."): k = "model." + k elif k.startswith("model."): k = k.replace("model.", "model.language_model.") state_dict_[k] = v return state_dict_ ``` ## Step 3: Writing Model Config Model Config is located in `diffsynth/configs/model_configs.py`, used to identify model types and load them. The following fields need to be filled in: * `model_hash`: Model file hash value, which can be obtained through the `hash_model_file` function. This hash value is only related to the keys and tensor shapes in the model file's state dict, and is unrelated to other information in the file. * `model_name`: Model name, used for `Pipeline` to identify the required model. If different structured models play the same role in `Pipeline`, the same `model_name` can be used. When integrating new models, just ensure that `model_name` is different from other existing functional models. The corresponding model is fetched through `model_name` in the `Pipeline`'s `from_pretrained`. * `model_class`: Model architecture import path, pointing to the model architecture class implemented in Step 1, for example `diffsynth.models.qwen_image_text_encoder.QwenImageTextEncoder`. * `state_dict_converter`: Optional parameter. If model file format conversion is needed, the import path of the model conversion logic needs to be filled in, for example `diffsynth.utils.state_dict_converters.qwen_image_text_encoder.QwenImageTextEncoderStateDictConverter`. * `extra_kwargs`: Optional parameter. If additional parameters need to be passed when initializing the model, these parameters need to be filled in. For example, models [DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Canny](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Canny) and [DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Inpaint](https://www.modelscope.cn/models/DiffSynth-Studio/Qwen-Image-Blockwise-ControlNet-Inpaint) both adopt the `QwenImageBlockWiseControlNet` structure in `diffsynth/models/qwen_image_controlnet.py`, but the latter also needs additional configuration `additional_in_dim=4`. Therefore, this configuration information needs to be filled in the `extra_kwargs` field. We provide a piece of code to quickly understand how models are loaded through this configuration information: ```python from diffsynth.core import hash_model_file, load_state_dict, skip_model_initialization from diffsynth.models.qwen_image_text_encoder import QwenImageTextEncoder from diffsynth.utils.state_dict_converters.qwen_image_text_encoder import QwenImageTextEncoderStateDictConverter import torch model_hash = "8004730443f55db63092006dd9f7110e" model_name = "qwen_image_text_encoder" model_class = QwenImageTextEncoder state_dict_converter = QwenImageTextEncoderStateDictConverter extra_kwargs = {} model_path = [ "models/Qwen/Qwen-Image/text_encoder/model-00001-of-00004.safetensors", "models/Qwen/Qwen-Image/text_encoder/model-00002-of-00004.safetensors", "models/Qwen/Qwen-Image/text_encoder/model-00003-of-00004.safetensors", "models/Qwen/Qwen-Image/text_encoder/model-00004-of-00004.safetensors", ] if hash_model_file(model_path) == model_hash: with skip_model_initialization(): model = model_class(**extra_kwargs) state_dict = load_state_dict(model_path, torch_dtype=torch.bfloat16, device="cuda") state_dict = state_dict_converter(state_dict) model.load_state_dict(state_dict, assign=True) print("Done!") ``` > Q: The logic of the above code looks very simple, why is this part of code in `DiffSynth-Studio` extremely complex? > > A: Because we provide aggressive VRAM management functions that are coupled with the model loading logic, this leads to the complexity of the framework structure. We have tried our best to simplify the interface exposed to developers. The `model_hash` in `diffsynth/configs/model_configs.py` is not uniquely existing. Multiple models may exist in the same model file. For this situation, please use multiple model Configs to load each model separately, and write the corresponding `state_dict_converter` to separate the parameters required by each model. ## Step 4: Verifying Whether the Model Can Be Recognized and Loaded After model integration, the following code can be used to verify whether the model can be correctly recognized and loaded. The following code will attempt to load the model into memory: ```python from diffsynth.models.model_loader import ModelPool model_pool = ModelPool() model_pool.auto_load_model( [ "models/Qwen/Qwen-Image/text_encoder/model-00001-of-00004.safetensors", "models/Qwen/Qwen-Image/text_encoder/model-00002-of-00004.safetensors", "models/Qwen/Qwen-Image/text_encoder/model-00003-of-00004.safetensors", "models/Qwen/Qwen-Image/text_encoder/model-00004-of-00004.safetensors", ], ) ``` If the model can be recognized and loaded, you will see the following output: ``` Loading models from: [ "models/Qwen/Qwen-Image/text_encoder/model-00001-of-00004.safetensors", "models/Qwen/Qwen-Image/text_encoder/model-00002-of-00004.safetensors", "models/Qwen/Qwen-Image/text_encoder/model-00003-of-00004.safetensors", "models/Qwen/Qwen-Image/text_encoder/model-00004-of-00004.safetensors" ] Loaded model: { "model_name": "qwen_image_text_encoder", "model_class": "diffsynth.models.qwen_image_text_encoder.QwenImageTextEncoder", "extra_kwargs": null } ``` ## Step 5: Writing Model VRAM Management Scheme `DiffSynth-Studio` supports complex VRAM management. See [Enabling VRAM Management](../Developer_Guide/Enabling_VRAM_management.md) for details. --- ### En Developer Guide Training Diffusion Models (`docs/en/Developer_Guide/Training_Diffusion_Models.md`) # Integrating Model Training After [integrating models](../Developer_Guide/Integrating_Your_Model.md) and [implementing Pipeline](../Developer_Guide/Building_a_Pipeline.md), the next step is to integrate model training functionality. ## Training-Inference Consistent Pipeline Modification To ensure strict consistency between training and inference processes, we will use most of the inference code during training, but still need to make minor modifications. First, add extra logic during inference to switch the image-to-image/video-to-video logic based on the `scheduler` state. Taking Qwen-Image as an example: ```python class QwenImageUnit_InputImageEmbedder(PipelineUnit): def __init__(self): super().__init__( input_params=("input_image", "noise", "tiled", "tile_size", "tile_stride"), output_params=("latents", "input_latents"), onload_model_names=("vae",) ) def process(self, pipe: QwenImagePipeline, input_image, noise, tiled, tile_size, tile_stride): if input_image is None: return {"latents": noise, "input_latents": None} pipe.load_models_to_device(['vae']) image = pipe.preprocess_image(input_image).to(device=pipe.device, dtype=pipe.torch_dtype) input_latents = pipe.vae.encode(image, tiled=tiled, tile_size=tile_size, tile_stride=tile_stride) if pipe.scheduler.training: return {"latents": noise, "input_latents": input_latents} else: latents = pipe.scheduler.add_noise(input_latents, noise, timestep=pipe.scheduler.timesteps[0]) return {"latents": latents, "input_latents": input_latents} ``` Then, enable Gradient Checkpointing in `model_fn`, which will significantly reduce the VRAM required for training at the cost of computational speed. This is not mandatory, but we strongly recommend doing so. Taking Qwen-Image as an example, before modification: ```python text, image = block( image=image, text=text, temb=conditioning, image_rotary_emb=image_rotary_emb, attention_mask=attention_mask, ) ``` After modification: ```python from ..core import gradient_checkpoint_forward text, image = gradient_checkpoint_forward( block, use_gradient_checkpointing, use_gradient_checkpointing_offload, image=image, text=text, temb=conditioning, image_rotary_emb=image_rotary_emb, attention_mask=attention_mask, ) ``` ## Writing Training Scripts `DiffSynth-Studio` does not strictly encapsulate the training framework, but exposes the script content to developers. This approach makes it more convenient to modify training scripts to implement additional functions. Developers can refer to existing training scripts, such as `examples/qwen_image/model_training/train.py`, for modification to adapt to new model training. --- ### En Diffusion Templates Introducing Diffusion Templates (`docs/en/Diffusion_Templates/Introducing_Diffusion_Templates.md`) # Diffusion Templates Diffusion Templates is a controllable generation plugin framework for Diffusion models in DiffSynth-Studio, providing additional controllable generation capabilities for base models. * Open-source code: [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) * Technical report: [arXiv](https://arxiv.org/abs/2604.24351) * Project page: [GitHub](https://modelscope.github.io/diffusion-templates-web/) * Documentation reference * Introduction to Diffusion Templates: [English Version](https://diffsynth-studio-doc.readthedocs.io/en/latest/Diffusion_Templates/Introducing_Diffusion_Templates.html)、[中文版](https://diffsynth-studio-doc.readthedocs.io/zh-cn/latest/Diffusion_Templates/Introducing_Diffusion_Templates.html) * Detailed Architecture of Diffusion Templates: [English Version](https://diffsynth-studio-doc.readthedocs.io/en/latest/Diffusion_Templates/Understanding_Diffusion_Templates.html)、[中文版](https://diffsynth-studio-doc.readthedocs.io/zh-cn/latest/Diffusion_Templates/Understanding_Diffusion_Templates.html) * Template Model Inference: [English Version](https://diffsynth-studio-doc.readthedocs.io/en/latest/Diffusion_Templates/Template_Model_Inference.html)、[中文版](https://diffsynth-studio-doc.readthedocs.io/zh-cn/latest/Diffusion_Templates/Template_Model_Inference.html) * Template Model Training: [English Version](https://diffsynth-studio-doc.readthedocs.io/en/latest/Diffusion_Templates/Template_Model_Training.html)、[中文版](https://diffsynth-studio-doc.readthedocs.io/zh-cn/latest/Diffusion_Templates/Template_Model_Training.html) * Online demo: [ModelScope](https://modelscope.cn/studios/DiffSynth-Studio/Diffusion-Templates) * Model collection: [ModelScope](https://modelscope.cn/collections/DiffSynth-Studio/KleinBase4B-Templates)、[ModelScope International](https://modelscope.ai/collections/DiffSynth-Studio/KleinBase4B-Templates)、[HuggingFace](https://huggingface.co/collections/DiffSynth-Studio/kleinbase4b-templates) |Model Name|ModelScope|ModelScope International|HuggingFace|Inference Code|Low VRAM Inference Code|Training Code|Training Validation Code| |-|-|-|-|-|-|-|-| |Structure Control|[link](https://modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-ControlNet)|[link](https://modelscope.ai/models/DiffSynth-Studio/Template-KleinBase4B-ControlNet)|[link](https://huggingface.co/DiffSynth-Studio/Template-KleinBase4B-ControlNet)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-ControlNet.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-ControlNet.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-ControlNet.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-ControlNet.py)| |Brightness Adjustment|[link](https://modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Brightness)|[link](https://modelscope.ai/models/DiffSynth-Studio/Template-KleinBase4B-Brightness)|[link](https://huggingface.co/DiffSynth-Studio/Template-KleinBase4B-Brightness)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-Brightness.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Brightness.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-Brightness.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-Brightness.py)| |Color Adjustment|[link](https://modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-SoftRGB)|[link](https://modelscope.ai/models/DiffSynth-Studio/Template-KleinBase4B-SoftRGB)|[link](https://huggingface.co/DiffSynth-Studio/Template-KleinBase4B-SoftRGB)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-SoftRGB.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-SoftRGB.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-SoftRGB.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-SoftRGB.py)| |Image Editing|[link](https://modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Edit)|[link](https://modelscope.ai/models/DiffSynth-Studio/Template-KleinBase4B-Edit)|[link](https://huggingface.co/DiffSynth-Studio/Template-KleinBase4B-Edit)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-Edit.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Edit.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-Edit.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-Edit.py)| |Super-Resolution|[link](https://modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Upscaler)|[link](https://modelscope.ai/models/DiffSynth-Studio/Template-KleinBase4B-Upscaler)|[link](https://huggingface.co/DiffSynth-Studio/Template-KleinBase4B-Upscaler)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-Upscaler.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Upscaler.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-Upscaler.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-Upscaler.py)| |Sharpness Enhancement|[link](https://modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Sharpness)|[link](https://modelscope.ai/models/DiffSynth-Studio/Template-KleinBase4B-Sharpness)|[link](https://huggingface.co/DiffSynth-Studio/Template-KleinBase4B-Sharpness)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-Sharpness.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Sharpness.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-Sharpness.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-Sharpness.py)| |Aesthetic Alignment|[link](https://modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Aesthetic)|[link](https://modelscope.ai/models/DiffSynth-Studio/Template-KleinBase4B-Aesthetic)|[link](https://huggingface.co/DiffSynth-Studio/Template-KleinBase4B-Aesthetic)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-Aesthetic.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Aesthetic.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-Aesthetic.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-Aesthetic.py)| |Local Redrawing|[link](https://modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Inpaint)|[link](https://modelscope.ai/models/DiffSynth-Studio/Template-KleinBase4B-Inpaint)|[link](https://huggingface.co/DiffSynth-Studio/Template-KleinBase4B-Inpaint)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-Inpaint.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Inpaint.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-Inpaint.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-Inpaint.py)| |Content Reference|[link](https://modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-ContentRef)|[link](https://modelscope.ai/models/DiffSynth-Studio/Template-KleinBase4B-ContentRef)|[link](https://huggingface.co/DiffSynth-Studio/Template-KleinBase4B-ContentRef)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-ContentRef.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-ContentRef.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-ContentRef.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-ContentRef.py)| |Age Control|[link](https://modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Age)|[link](https://modelscope.ai/models/DiffSynth-Studio/Template-KleinBase4B-Age)|[link](https://huggingface.co/DiffSynth-Studio/Template-KleinBase4B-Age)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-Age.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Age.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-Age.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-Age.py)| |Panda Meme (Easter Egg Model)|[link](https://modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-PandaMeme)|[link](https://modelscope.ai/models/DiffSynth-Studio/Template-KleinBase4B-PandaMeme)|[link](https://huggingface.co/DiffSynth-Studio/Template-KleinBase4B-PandaMeme)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-PandaMeme.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-PandaMeme.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-PandaMeme.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-PandaMeme.py)| * Dataset: [ModelScope](https://modelscope.cn/collections/DiffSynth-Studio/ImagePulseV2)、[ModelScope International](https://modelscope.cn/collections/DiffSynth-Studio/ImagePulseV2)、[HuggingFace](https://huggingface.co/collections/DiffSynth-Studio/imagepulsev2) |Dataset Name|ModelScope|ModelScope International|HuggingFace| |-|-|-|-| |Text-to-Image|[link](https://modelscope.cn/datasets/DiffSynth-Studio/ImagePulseV2-TextImage)|[link](https://modelscope.ai/datasets/DiffSynth-Studio/ImagePulseV2-TextImage)|[link](https://huggingface.co/datasets/DiffSynth-Studio/ImagePulseV2-TextImage)| |Local Redrawing|[link](https://modelscope.cn/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Inpaint)|[link](https://modelscope.ai/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Inpaint)|[link](https://huggingface.co/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Inpaint)| |Background Replacement|[link](https://modelscope.cn/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Background)|[link](https://modelscope.ai/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Background)|[link](https://huggingface.co/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Background)| |Clothing Replacement|[link](https://modelscope.cn/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Clothes)|[link](https://modelscope.ai/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Clothes)|[link](https://huggingface.co/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Clothes)| |Pose Adjustment|[link](https://modelscope.cn/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Pose)|[link](https://modelscope.ai/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Pose)|[link](https://huggingface.co/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Pose)| |Foreground Modification|[link](https://modelscope.cn/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Change)|[link](https://modelscope.ai/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Change)|[link](https://huggingface.co/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Change)| |Local Addition/Removal|[link](https://modelscope.cn/datasets/DiffSynth-Studio/ImagePulseV2-Edit-AddRemove)|[link](https://modelscope.ai/datasets/DiffSynth-Studio/ImagePulseV2-Edit-AddRemove)|[link](https://huggingface.co/datasets/DiffSynth-Studio/ImagePulseV2-Edit-AddRemove)| |Super-Resolution|[link](https://modelscope.cn/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Upscale)|[link](https://modelscope.ai/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Upscale)|[link](https://huggingface.co/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Upscale)| |Portrait Generation|[link](https://modelscope.cn/datasets/DiffSynth-Studio/ImagePulseV2-TextImage-Human)|[link](https://modelscope.ai/datasets/DiffSynth-Studio/ImagePulseV2-TextImage-Human)|[link](https://huggingface.co/datasets/DiffSynth-Studio/ImagePulseV2-TextImage-Human)| |Random Cropping|[link](https://modelscope.cn/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Crop)|[link](https://modelscope.ai/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Crop)|[link](https://huggingface.co/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Crop)| |Lighting Adjustment|[link](https://modelscope.cn/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Light)|[link](https://modelscope.ai/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Light)|[link](https://huggingface.co/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Light)| |Scene Structure|[link](https://modelscope.cn/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Structure)|[link](https://modelscope.ai/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Structure)|[link](https://huggingface.co/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Structure)| |Facial Expression Editing|[link](https://modelscope.cn/datasets/DiffSynth-Studio/ImagePulseV2-Edit-HumanFace)|[link](https://modelscope.ai/datasets/DiffSynth-Studio/ImagePulseV2-Edit-HumanFace)|[link](https://huggingface.co/datasets/DiffSynth-Studio/ImagePulseV2-Edit-HumanFace)| |View Angle Adjustment|[link](https://modelscope.cn/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Angle)|[link](https://modelscope.ai/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Angle)|[link](https://huggingface.co/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Angle)| |Style Transfer|[link](https://modelscope.cn/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Style)|[link](https://modelscope.ai/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Style)|[link](https://huggingface.co/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Style)| |Multi-Resolution|[link](https://modelscope.cn/datasets/DiffSynth-Studio/ImagePulseV2-TextImage-MultiResolution)|[link](https://modelscope.ai/datasets/DiffSynth-Studio/ImagePulseV2-TextImage-MultiResolution)|[link](https://huggingface.co/datasets/DiffSynth-Studio/ImagePulseV2-TextImage-MultiResolution)| |Multi-Image Merge|[link](https://modelscope.cn/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Merge)|[link](https://modelscope.ai/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Merge)|[link](https://huggingface.co/datasets/DiffSynth-Studio/ImagePulseV2-Edit-Merge)| ## Model Performance Overview * Super-Resolution + Sharpness Enhancement: Generate ultra-high-resolution images |Low Resolution Input|High Resolution Output| |-|-| ||| * Structure Control + Aesthetic Alignment + Sharpness Enhancement: Fully-equipped ControlNet |Structure Control Image|Output Image| |-|-| ||| * Structure Control + Image Editing + Color Adjustment: Artistic Style Creation at Will |Structure Control Image|Editing Input Image|Output Image| |-|-|-| |||| * Brightness Control + Image Editing + Local Redrawing: Cross-dimensional Elements in Images |Reference Image|Redrawing Area|Output Image| |-|-|-| |||| --- ### En Diffusion Templates Template Model Inference (`docs/en/Diffusion_Templates/Template_Model_Inference.md`) # Template Model Inference ## Enabling Template Models on Base Model Pipelines Using the base model [black-forest-labs/FLUX.2-klein-base-4B](https://modelscope.cn/models/black-forest-labs/FLUX.2-klein-base-4B) as an example, when generating images using only the base model: ```python from diffsynth.diffusion.template import TemplatePipeline from diffsynth.pipelines.flux2_image import Flux2ImagePipeline, ModelConfig import torch # Load base model pipe = Flux2ImagePipeline.from_pretrained( torch_dtype=torch.bfloat16, device="cuda", model_configs=[ ModelConfig(model_id="black-forest-labs/FLUX.2-klein-4B", origin_file_pattern="text_encoder/*.safetensors"), ModelConfig(model_id="black-forest-labs/FLUX.2-klein-base-4B", origin_file_pattern="transformer/*.safetensors"), ModelConfig(model_id="black-forest-labs/FLUX.2-klein-4B", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"), ], tokenizer_config=ModelConfig(model_id="black-forest-labs/FLUX.2-klein-4B", origin_file_pattern="tokenizer/"), ) # Generate an image image = pipe( prompt="a cat", seed=0, cfg_scale=4, height=1024, width=1024, ) image.save("image.png") ``` The Template model [DiffSynth-Studio/Template-KleinBase4B-Brightness](https://modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Brightness) can control image brightness during generation. Through the `TemplatePipeline` model, it can be loaded from ModelScope (via `ModelConfig(model_id="xxx/xxx")`) or from a local path (via `ModelConfig(path="xxx")`). Inputting `scale=0.8` increases image brightness. Note that in the code, input parameters for `pipe` must be transferred to `template_pipeline`, and `template_inputs` should be added. ```python # Load Template model template_pipeline = TemplatePipeline.from_pretrained( torch_dtype=torch.bfloat16, device="cuda", model_configs=[ ModelConfig(model_id="DiffSynth-Studio/Template-KleinBase4B-Brightness") ], ) # Generate an image image = template_pipeline( pipe, prompt="a cat", seed=0, cfg_scale=4, height=1024, width=1024, template_inputs=[{"scale": 0.8}], ) image.save("image_0.8.png") ``` ## CFG Enhancement for Template Models Template models can enable CFG (Classifier-Free Guidance) to make control effects more pronounced. For example, with the model [DiffSynth-Studio/Template-KleinBase4B-Brightness](https://modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Brightness), adding `negative_template_inputs` to the TemplatePipeline input parameters and setting its scale to 0.5 will generate images with more noticeable brightness variations by contrasting both sides. ```python # Generate an image with CFG image = template_pipeline( pipe, prompt="a cat", seed=0, cfg_scale=4, height=1024, width=1024, template_inputs=[{"scale": 0.8}], negative_template_inputs=[{"scale": 0.5}], ) image.save("image_0.8_cfg.png") ``` ## Low VRAM Support Template models currently do not support the main framework's VRAM management, but lazy loading can be used - loading Template models only when needed for inference. This significantly reduces VRAM requirements when enabling multiple Template models, with peak VRAM usage being that of a single Template model. Add parameter `lazy_loading=True` to enable. ```python template_pipeline = TemplatePipeline.from_pretrained( torch_dtype=torch.bfloat16, device="cuda", model_configs=[ ModelConfig(model_id="DiffSynth-Studio/Template-KleinBase4B-Brightness") ], lazy_loading=True, ) ``` The base model's Pipeline and Template Pipeline are completely independent and can enable VRAM management on demand. When Template model outputs contain LoRA in Template Cache, you need to enable VRAM management for the base model's Pipeline or enable LoRA hot loading (using the code below), otherwise LoRA weights will be fused repeatedly. ```python pipe.dit = pipe.enable_lora_hot_loading(pipe.dit) ``` ## Enabling Multiple Template Models `TemplatePipeline` can load multiple Template models. During inference, use `model_id` in `template_inputs` to distinguish inputs for each Template model. After enabling VRAM management for the base model's Pipeline and lazy loading for Template Pipeline, you can load any number of Template models. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Super-Resolution + Sharpness Enhancement Combining [DiffSynth-Studio/Template-KleinBase4B-Upscaler](https://modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Upscaler) and [DiffSynth-Studio/Template-KleinBase4B-Sharpness](https://modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Sharpness) can upscale blurry images while improving detail clarity. ```python image = template( pipe, prompt="A cat is sitting on a stone.", seed=0, cfg_scale=4, num_inference_steps=50, template_inputs = [ { "model_id": 3, "image": Image.open("data/examples/templates/image_lowres_100.jpg"), "prompt": "A cat is sitting on a stone.", }, { "model_id": 5, "scale": 1, }, ], negative_template_inputs = [ { "model_id": 3, "image": Image.open("data/examples/templates/image_lowres_100.jpg"), "prompt": "", }, { "model_id": 5, "scale": 0, }, ], ) image.save("image_Upscaler_Sharpness.png") ``` | Low Resolution Input | High Resolution Output | |----------------------|------------------------| | | | ### Structure Control + Aesthetic Alignment + Sharpness Enhancement [DiffSynth-Studio/Template-KleinBase4B-ControlNet](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-ControlNet) controls composition, [DiffSynth-Studio/Template-KleinBase4B-Aesthetic](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Aesthetic) fills in details, and [DiffSynth-Studio/Template-KleinBase4B-Sharpness](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Sharpness) ensures clarity. Combining these three Template models produces exquisite images. ```python image = template( pipe, prompt="A cat is sitting on a stone, bathed in bright sunshine.", seed=0, cfg_scale=4, num_inference_steps=50, template_inputs = [ { "model_id": 1, "image": Image.open("data/examples/templates/image_depth.jpg"), "prompt": "A cat is sitting on a stone, bathed in bright sunshine.", }, { "model_id": 7, "lora_ids": list(range(1, 180, 2)), "lora_scales": 2.0, "merge_type": "mean", }, { "model_id": 5, "scale": 0.8, }, ], negative_template_inputs = [ { "model_id": 1, "image": Image.open("data/examples/templates/image_depth.jpg"), "prompt": "", }, { "model_id": 7, "lora_ids": list(range(1, 180, 2)), "lora_scales": 2.0, "merge_type": "mean", }, { "model_id": 5, "scale": 0, }, ], ) image.save("image_Controlnet_Aesthetic_Sharpness.png") ``` | Structure Control Image | Output Image | |-------------------------|--------------| | | | ### Structure Control + Image Editing + Color Adjustment [DiffSynth-Studio/Template-KleinBase4B-ControlNet](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-ControlNet) controls composition, [DiffSynth-Studio/Template-KleinBase4B-Edit](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Edit) preserves original image details like fur texture, and [DiffSynth-Studio/Template-KleinBase4B-SoftRGB](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-SoftRGB) controls color tones, creating an artistic masterpiece. ```python image = template( pipe, prompt="A cat is sitting on a stone. Colored ink painting.", seed=0, cfg_scale=4, num_inference_steps=50, template_inputs = [ { "model_id": 1, "image": Image.open("data/examples/templates/image_depth.jpg"), "prompt": "A cat is sitting on a stone. Colored ink painting.", }, { "model_id": 2, "image": Image.open("data/examples/templates/image_reference.jpg"), "prompt": "Convert the image style to colored ink painting.", }, { "model_id": 4, "R": 0.9, "G": 0.5, "B": 0.3, }, ], negative_template_inputs = [ { "model_id": 1, "image": Image.open("data/examples/templates/image_depth.jpg"), "prompt": "", }, { "model_id": 2, "image": Image.open("data/examples/templates/image_reference.jpg"), "prompt": "", }, ], ) image.save("image_Controlnet_Edit_SoftRGB.png") ``` | Structure Control Image | Editing Input Image | Output Image | |-------------------------|---------------------|--------------| | | | | ### Brightness Control + Image Editing + Local Redrawing [DiffSynth-Studio/Template-KleinBase4B-Brightness](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Brightness) generates bright scenes, [DiffSynth-Studio/Template-KleinBase4B-Edit](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Edit) references original image layout, and [DiffSynth-Studio/Template-KleinBase4B-Inpaint](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Inpaint) keeps background unchanged, generating cross-dimensional content. ```python image = template( pipe, prompt="A cat is sitting on a stone. Flat anime style.", seed=0, cfg_scale=4, num_inference_steps=50, template_inputs = [ { "model_id": 0, "scale": 0.6, }, { "model_id": 2, "image": Image.open("data/examples/templates/image_reference.jpg"), "prompt": "Convert the image style to flat anime style.", }, { "model_id": 6, "image": Image.open("data/examples/templates/image_reference.jpg"), "mask": Image.open("data/examples/templates/image_mask_1.jpg"), "force_inpaint": True, }, ], negative_template_inputs = [ { "model_id": 0, "scale": 0.5, }, { "model_id": 2, "image": Image.open("data/examples/templates/image_reference.jpg"), "prompt": "", }, { "model_id": 6, "image": Image.open("data/examples/templates/image_reference.jpg"), "mask": Image.open("data/examples/templates/image_mask_1.jpg"), }, ], ) image.save("image_Brightness_Edit_Inpaint.png") ``` | Reference Image | Redrawing Area | Output Image | |------------------|----------------|--------------| | | | | --- ### En Diffusion Templates Template Model Training (`docs/en/Diffusion_Templates/Template_Model_Training.md`) # Template Model Training DiffSynth-Studio currently provides comprehensive Template training support for [black-forest-labs/FLUX.2-klein-base-4B](https://www.modelscope.cn/models/black-forest-labs/FLUX.2-klein-base-4B), with more model adaptations coming soon. ## Continuing Training from Pretrained Models To continue training from our pretrained models, refer to the table in [FLUX.2](../Model_Details/FLUX2.md#model-overview) to find the corresponding training script. ## Building New Template Models ### Template Model Component Format A Template model binds to a model repository (or local folder) containing a code file `model.py` as the entry point. Here's the template for `model.py`: ```python import torch class CustomizedTemplateModel(torch.nn.Module): def __init__(self): super().__init__() @torch.no_grad() def process_inputs(self, xxx, **kwargs): yyy = xxx return {"yyy": yyy} def forward(self, yyy, **kwargs): zzz = yyy return {"zzz": zzz} class DataProcessor: def __call__(self, www, **kwargs): xxx = www return {"xxx": xxx} TEMPLATE_MODEL = CustomizedTemplateModel TEMPLATE_MODEL_PATH = "model.safetensors" TEMPLATE_DATA_PROCESSOR = DataProcessor ``` During Template model inference, Template Input passes through `TEMPLATE_MODEL`'s `process_inputs` and `forward` to generate Template Cache. ```mermaid flowchart LR; i@{shape: text, label: "Template Input"}-->p[process_inputs]; subgraph TEMPLATE_MODEL p[process_inputs]-->f[forward] end f[forward]-->c@{shape: text, label: "Template Cache"}; ``` During Template model training, Template Input comes from the dataset through `TEMPLATE_DATA_PROCESSOR`. ```mermaid flowchart LR; d@{shape: text, label: "Dataset"}-->dp[TEMPLATE_DATA_PROCESSOR]-->p[process_inputs]; subgraph TEMPLATE_MODEL p[process_inputs]-->f[forward] end f[forward]-->c@{shape: text, label: "Template Cache"}; ``` #### `TEMPLATE_MODEL` `TEMPLATE_MODEL` implements the Template model logic, inheriting from `torch.nn.Module` with required `process_inputs` and `forward` methods. These two methods form the complete Template model inference process, split into two stages to better support [two-stage split training](https://diffsynth-studio-doc.readthedocs.io/en/latest/Training/Split_Training.html). * `process_inputs` must use `@torch.no_grad()` for gradient-free computation * `forward` must contain all gradient computations required for training Both methods should accept `**kwargs` for compatibility. Reserved parameters include: * To interact with the base model Pipeline (e.g., call text encoder), add `pipe` parameter to method inputs * To enable Gradient Checkpointing, add `use_gradient_checkpointing` and `use_gradient_checkpointing_offload` to `forward` inputs * Multiple Template models use `model_id` to distinguish Template Inputs - do not use this field in method parameters #### `TEMPLATE_MODEL_PATH` (Optional) `TEMPLATE_MODEL_PATH` specifies the relative path to pretrained weights. For example: ```python TEMPLATE_MODEL_PATH = "model.safetensors" ``` For multi-file models: ```python TEMPLATE_MODEL_PATH = [ "model-00001-of-00003.safetensors", "model-00002-of-00003.safetensors", "model-00003-of-00003.safetensors", ] ``` Set to `None` for random initialization: ```python TEMPLATE_MODEL_PATH = None ``` #### `TEMPLATE_DATA_PROCESSOR` (Optional) To train Template models with DiffSynth-Studio, datasets should contain `template_inputs` fields in `metadata.json`. These fields pass through `TEMPLATE_DATA_PROCESSOR` to generate inputs for Template model methods. For example, the brightness control model [DiffSynth-Studio/Template-KleinBase4B-Brightness](https://modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Brightness) takes `scale` as input: ```json [ { "image": "images/image_1.jpg", "prompt": "a cat", "template_inputs": {"scale": 0.2} }, { "image": "images/image_2.jpg", "prompt": "a dog", "template_inputs": {"scale": 0.6} } ] ``` ```python class DataProcessor: def __call__(self, scale, **kwargs): return {"scale": scale} TEMPLATE_DATA_PROCESSOR = DataProcessor ``` Or calculate scale from image paths: ```json [ { "image": "images/image_1.jpg", "prompt": "a cat", "template_inputs": {"image": "/path/to/your/dataset/images/image_1.jpg"} } ] ``` ```python class DataProcessor: def __call__(self, image, **kwargs): image = Image.open(image) image = np.array(image) return {"scale": image.astype(np.float32).mean() / 255} TEMPLATE_DATA_PROCESSOR = DataProcessor ``` ### Training Template Models A Template model is "trainable" if its Template Cache variables are fully decoupled from the base model Pipeline - these variables should reach `model_fn` without participating in any Pipeline Unit calculations. For training with [black-forest-labs/FLUX.2-klein-base-4B](https://www.modelscope.cn/models/black-forest-labs/FLUX.2-klein-base-4B), use these training script parameters: * `--extra_inputs`: Additional inputs. Use `template_inputs` for text-to-image models, `edit_image,template_inputs` for image editing models * `--template_model_id_or_path`: Template model ID or local path (use `:` suffix for ModelScope IDs, e.g., `"DiffSynth-Studio/Template-KleinBase4B-Brightness:"`) * `--remove_prefix_in_ckpt`: State dict prefix to remove when saving models (use `"pipe.template_model."`) * `--trainable_models`: Trainable components (use `"template_model"` for full model, or `"template_model.xxx,template_model.yyy"` for specific components) Example training script: ```shell accelerate launch examples/flux2/model_training/train.py \ --dataset_base_path data/diffsynth_example_dataset/flux2/Template-KleinBase4B-Brightness \ --dataset_metadata_path data/diffsynth_example_dataset/flux2/Template-KleinBase4B-Brightness/metadata.jsonl \ --extra_inputs "template_inputs" \ --max_pixels 1048576 \ --dataset_repeat 50 \ --model_id_with_origin_paths "black-forest-labs/FLUX.2-klein-4B:text_encoder/*.safetensors,black-forest-labs/FLUX.2-klein-base-4B:transformer/*.safetensors,black-forest-labs/FLUX.2-klein-4B:vae/diffusion_pytorch_model.safetensors" \ --template_model_id_or_path "examples/flux2/model_training/scripts/brightness" \ --tokenizer_path "black-forest-labs/FLUX.2-klein-4B:tokenizer/" \ --learning_rate 1e-4 \ --num_epochs 2 \ --remove_prefix_in_ckpt "pipe.template_model." \ --output_path "./models/train/Template-KleinBase4B-Brightness_example" \ --trainable_models "template_model" \ --use_gradient_checkpointing \ --find_unused_parameters ``` ### Interacting with Base Model Pipeline Components Template models can interact with base model Pipelines. For example, using the text encoder: ```python class CustomizedTemplateModel(torch.nn.Module): def __init__(self): super().__init__() self.xxx = xxx() @torch.no_grad() def process_inputs(self, text, pipe, **kwargs): input_ids = pipe.tokenizer(text) text_emb = pipe.text_encoder(input_ids) return {"text_emb": text_emb} def forward(self, text_emb, pipe, **kwargs): kv_cache = self.xxx(text_emb) return {"kv_cache": kv_cache} TEMPLATE_MODEL = CustomizedTemplateModel ``` ### Using Non-Trainable Components For models with pretrained components: ```python class CustomizedTemplateModel(torch.nn.Module): def __init__(self): super().__init__() self.image_encoder = XXXEncoder.from_pretrained(xxx) self.mlp = MLP() @torch.no_grad() def process_inputs(self, image, **kwargs): emb = self.image_encoder(image) return {"emb": emb} def forward(self, emb, **kwargs): kv_cache = self.mlp(emb) return {"kv_cache": kv_cache} TEMPLATE_MODEL = CustomizedTemplateModel ``` Set `--trainable_models template_model.mlp` to train only the MLP component. ### Training on Low VRAM Devices The framework supports splitting Template model training into two stages: the first stage performs gradient-free computation, and the second stage performs gradient updates. For more information, refer to the documentation: [Two-stage Split Training](https://diffsynth-studio-doc.readthedocs.io/en/latest/Training/Split_Training.html). Here's a sample script: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Two-stage split training can reduce VRAM requirements and improve training speed. The training process is lossless in precision, but requires significant disk space for storing cache files. To further reduce VRAM requirements, you can enable fp8 precision by adding the parameters `--fp8_models "black-forest-labs/FLUX.2-klein-4B:text_encoder/*.safetensors,black-forest-labs/FLUX.2-klein-4B:vae/diffusion_pytorch_model.safetensors"` and `--fp8_models "black-forest-labs/FLUX.2-klein-base-4B:transformer/*.safetensors"` to the two-stage training. Note that fp8 precision can only be enabled on non-trainable model components and introduces minor errors. ### Uploading Template Models After training, follow these steps to upload Template models to ModelScope for wider distribution. 1. Set model path in `model.py`: ```python TEMPLATE_MODEL_PATH = "model.safetensors" ``` 2. Upload using ModelScope CLI: ```shell modelscope upload user_name/your_model_id /path/to/your/model.py model.py --token ms-xxx ``` 3. Package model files: ```python from diffsynth.diffusion.template import load_template_model, load_state_dict from safetensors.torch import save_file import torch model = load_template_model("path/to/your/template/model", torch_dtype=torch.bfloat16, device="cpu") state_dict = load_state_dict("path/to/your/ckpt/epoch-1.safetensors", torch_dtype=torch.bfloat16, device="cpu") state_dict.update(model.state_dict()) save_file(state_dict, "model.safetensors") ``` 4. Upload model file: ```shell modelscope upload user_name/your_model_id /path/to/your/model/epoch-1.safetensors model.safetensors --token ms-xxx ``` 5. Verify inference: ```python from diffsynth.diffusion.template import TemplatePipeline from diffsynth.pipelines.flux2_image import Flux2ImagePipeline, ModelConfig import torch # Load base model pipe = Flux2ImagePipeline.from_pretrained( torch_dtype=torch.bfloat16, device="cuda", model_configs=[ ModelConfig(model_id="black-forest-labs/FLUX.2-klein-4B", origin_file_pattern="text_encoder/*.safetensors"), ModelConfig(model_id="black-forest-labs/FLUX.2-klein-base-4B", origin_file_pattern="transformer/*.safetensors"), ModelConfig(model_id="black-forest-labs/FLUX.2-klein-4B", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"), ], tokenizer_config=ModelConfig(model_id="black-forest-labs/FLUX.2-klein-4B", origin_file_pattern="tokenizer/"), ) # Load Template model template_pipeline = TemplatePipeline.from_pretrained( torch_dtype=torch.bfloat16, device="cuda", model_configs=[ ModelConfig(model_id="user_name/your_model_id") ], ) # Generate image image = template_pipeline( pipe, prompt="a cat", seed=0, cfg_scale=4, height=1024, width=1024, template_inputs=[{xxx}], ) image.save("image.png") --- ### En Diffusion Templates Understanding Diffusion Templates (`docs/en/Diffusion_Templates/Understanding_Diffusion_Templates.md`) # Diffusion Templates Architecture Details The Diffusion Templates framework is a controllable generation plugin framework in DiffSynth-Studio that provides additional controllable generation capabilities for Diffusion models. ## Framework Structure The Diffusion Templates framework structure is shown below: ```mermaid flowchart TD; subgraph Template Pipeline si@{shape: text, label: "Template Input"}-->i1@{shape: text, label: "Template Input 1"}; si@{shape: text, label: "Template Input"}-->i2@{shape: text, label: "Template Input 2"}; si@{shape: text, label: "Template Input"}-->i3@{shape: text, label: "Template Input 3"}; i1@{shape: text, label: "Template Input 1"}-->m1[Template Model 1]-->c1@{shape: text, label: "Template Cache 1"}; i2@{shape: text, label: "Template Input 2"}-->m2[Template Model 2]-->c2@{shape: text, label: "Template Cache 2"}; i3@{shape: text, label: "Template Input 3"}-->m3[Template Model 3]-->c3@{shape: text, label: "Template Cache 3"}; c1-->c@{shape: text, label: "Template Cache"}; c2-->c; c3-->c; end i@{shape: text, label: "Model Input"}-->m[Diffusion Pipeline]-->o@{shape: text, label: "Model Output"}; c-->m; ``` The framework contains these module designs: * **Template Input**: Template model input. Format: Python dictionary with fields determined by each Template model (e.g., `{"scale": 0.8}`) * **Template Model**: Template model, loadable from ModelScope (`ModelConfig(model_id="xxx/xxx")`) or local path (`ModelConfig(path="xxx")`) * **Template Cache**: Template model output. Format: Python dictionary with fields matching base model Pipeline input parameters * **Template Pipeline**: Module for managing multiple Template models. Handles model loading and cache integration When the Diffusion Templates framework is disabled, base model components (Text Encoder, DiT, VAE) are loaded into the Diffusion Pipeline. Model Input (prompt, height, width) produces Model Output (e.g., images). When enabled, Template models are loaded into the Template Pipeline. The Template Pipeline outputs Template Cache (a subset of Diffusion Pipeline input parameters) for subsequent processing in the Diffusion Pipeline. This enables controllable generation by intercepting part of the Diffusion Pipeline's input parameters. ## Model Capability Medium Template Cache is defined as a subset of Diffusion Pipeline input parameters, ensuring framework generality. We restrict Template model inputs to only be Diffusion Pipeline parameters. The KV-Cache is particularly suitable as a Diffusion medium: * Proven effective in LLM Skills (prompts are converted to KV-Cache) * Has "high permission" in Diffusion models - can directly control image generation * Supports sequence-level concatenation for multiple Template models * Requires minimal development (add pipeline parameter and integrate to model) Other potential Template mediums: * **Residual**: Used in ControlNet for point-to-point control, but has resolution limitations and potential conflicts when merging * **LoRA**: Treated as input parameters rather than model components **Currently, we only support KV-Cache and LoRA as Template Cache mediums in FLUX.2 Pipeline, with plans to support more models and mediums in the future.** ## Template Model Format A Template model has this structure: ``` Template_Model ├── model.py └── model.safetensors ``` Where `model.py` is the entry point and `model.safetensors` contains model weights. For implementation details, see [Template Model Training](Template_Model_Training.md) or [existing Template models](https://modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Brightness). --- ### En Model Details ACE Step (`docs/en/Model_Details/ACE-Step.md`) # ACE-Step ACE-Step 1.5 is an open-source music generation model based on DiT architecture, supporting text-to-music, audio cover, repainting and other functionalities, running efficiently on consumer-grade hardware. ## Installation Before performing model inference and training, please install DiffSynth-Studio first. ```shell git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` For more information on installation, please refer to [Setup Dependencies](../Pipeline_Usage/Setup.md). ## Quick Start Running the following code will load the [ACE-Step/Ace-Step1.5](https://www.modelscope.cn/models/ACE-Step/Ace-Step1.5) model for inference. VRAM management is enabled, the framework automatically controls parameter loading based on available VRAM, requiring a minimum of 3GB VRAM. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Model Overview |Model ID|Inference|Low VRAM Inference|Full Training|Full Training Validation|LoRA Training|LoRA Training Validation| |-|-|-|-|-|-|-| |[ACE-Step/Ace-Step1.5](https://www.modelscope.cn/models/ACE-Step/Ace-Step1.5)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference/Ace-Step1.5.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference_low_vram/Ace-Step1.5.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/full/Ace-Step1.5.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_full/Ace-Step1.5.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/lora/Ace-Step1.5.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_lora/Ace-Step1.5.py)| |[ACE-Step/acestep-v15-turbo-shift1](https://www.modelscope.cn/models/ACE-Step/acestep-v15-turbo-shift1)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference/acestep-v15-turbo-shift1.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference_low_vram/acestep-v15-turbo-shift1.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/full/acestep-v15-turbo-shift1.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_full/acestep-v15-turbo-shift1.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/lora/acestep-v15-turbo-shift1.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_lora/acestep-v15-turbo-shift1.py)| |[ACE-Step/acestep-v15-turbo-shift3](https://www.modelscope.cn/models/ACE-Step/acestep-v15-turbo-shift3)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference/acestep-v15-turbo-shift3.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference_low_vram/acestep-v15-turbo-shift3.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/full/acestep-v15-turbo-shift3.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_full/acestep-v15-turbo-shift3.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/lora/acestep-v15-turbo-shift3.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_lora/acestep-v15-turbo-shift3.py)| |[ACE-Step/acestep-v15-turbo-continuous](https://www.modelscope.cn/models/ACE-Step/acestep-v15-turbo-continuous)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference/acestep-v15-turbo-continuous.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference_low_vram/acestep-v15-turbo-continuous.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/full/acestep-v15-turbo-continuous.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_full/acestep-v15-turbo-continuous.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/lora/acestep-v15-turbo-continuous.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_lora/acestep-v15-turbo-continuous.py)| |[ACE-Step/acestep-v15-base](https://www.modelscope.cn/models/ACE-Step/acestep-v15-base)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference/acestep-v15-base.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference_low_vram/acestep-v15-base.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/full/acestep-v15-base.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_full/acestep-v15-base.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/lora/acestep-v15-base.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_lora/acestep-v15-base.py)| |[ACE-Step/acestep-v15-base: CoverTask](https://www.modelscope.cn/models/ACE-Step/acestep-v15-base)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference/acestep-v15-base-CoverTask.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference_low_vram/acestep-v15-base-CoverTask.py)|—|—|—|—| |[ACE-Step/acestep-v15-base: RepaintTask](https://www.modelscope.cn/models/ACE-Step/acestep-v15-base)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference/acestep-v15-base-RepaintTask.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference_low_vram/acestep-v15-base-RepaintTask.py)|—|—|—|—| |[ACE-Step/acestep-v15-sft](https://www.modelscope.cn/models/ACE-Step/acestep-v15-sft)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference/acestep-v15-sft.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference_low_vram/acestep-v15-sft.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/full/acestep-v15-sft.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_full/acestep-v15-sft.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/lora/acestep-v15-sft.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_lora/acestep-v15-sft.py)| |[ACE-Step/acestep-v15-xl-base](https://www.modelscope.cn/models/ACE-Step/acestep-v15-xl-base)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference/acestep-v15-xl-base.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference_low_vram/acestep-v15-xl-base.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/full/acestep-v15-xl-base.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_full/acestep-v15-xl-base.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/lora/acestep-v15-xl-base.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_lora/acestep-v15-xl-base.py)| |[ACE-Step/acestep-v15-xl-sft](https://www.modelscope.cn/models/ACE-Step/acestep-v15-xl-sft)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference/acestep-v15-xl-sft.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference_low_vram/acestep-v15-xl-sft.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/full/acestep-v15-xl-sft.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_full/acestep-v15-xl-sft.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/lora/acestep-v15-xl-sft.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_lora/acestep-v15-xl-sft.py)| |[ACE-Step/acestep-v15-xl-turbo](https://www.modelscope.cn/models/ACE-Step/acestep-v15-xl-turbo)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference/acestep-v15-xl-turbo.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference_low_vram/acestep-v15-xl-turbo.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/full/acestep-v15-xl-turbo.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_full/acestep-v15-xl-turbo.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/lora/acestep-v15-xl-turbo.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_lora/acestep-v15-xl-turbo.py)| |[DiffSynth-Studio/acestep15xlsft-lora-music](https://www.modelscope.cn/models/DiffSynth-Studio/acestep15xlsft-lora-music)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference/acestep15xlsft-vocals2music.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_inference_low_vram/acestep15xlsft-vocals2music.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/full/acestep15xlsft-vocals2music.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ace_step/model_training/validate_full/acestep15xlsft-vocals2music.py)|-|-| ## Model Inference The model is loaded via `AceStepPipeline.from_pretrained`, see [Loading Models](../Pipeline_Usage/Model_Inference.md#loading-models) for details. The input parameters for `AceStepPipeline` inference include: * `prompt`: Text description of the music. * `cfg_scale`: Classifier-free guidance scale, defaults to 1.0. * `lyrics`: Lyrics text. * `task_type`: Task type,可选 values include `"text2music"` (text-to-music), `"cover"` (audio cover), `"repaint"` (repainting), defaults to `"text2music"`. * `reference_audios`: List of reference audio tensors for timbre reference. * `src_audio`: Source audio tensor for cover or repaint tasks. * `denoising_strength`: Denoising strength, controlling how much the output is influenced by source audio, defaults to 1.0. * `audio_cover_strength`: Audio cover step ratio, controlling how many steps use cover condition in cover tasks, defaults to 1.0. * `audio_code_string`: Input audio code string for cover tasks with discrete audio codes. * `repainting_ranges`: List of repainting time ranges (tuples of floats, in seconds) for repaint tasks. * `repainting_strength`: Repainting intensity, controlling the degree of change in repainted areas, defaults to 1.0. * `duration`: Audio duration in seconds, defaults to 60. * `bpm`: Beats per minute, defaults to 100. * `keyscale`: Musical key scale, defaults to "B minor". * `timesignature`: Time signature, defaults to "4". * `vocal_language`: Vocal language, defaults to "unknown". * `seed`: Random seed. * `rand_device`: Device for noise generation, defaults to "cpu". * `num_inference_steps`: Number of inference steps, defaults to 8. * `shift`: Timestep shift parameter for the scheduler, defaults to 3.0. ## Model Training Models in the ace_step series are trained uniformly via `examples/ace_step/model_training/train.py`. The script parameters include: * General Training Parameters * Dataset Configuration * `--dataset_base_path`: Root directory of the dataset. * `--dataset_metadata_path`: Path to the dataset metadata file. * `--dataset_repeat`: Number of dataset repeats per epoch. * `--dataset_num_workers`: Number of processes per DataLoader. * `--data_file_keys`: Field names to load from metadata, typically paths to image or video files, separated by `,`. * Model Loading Configuration * `--model_paths`: Paths to load models from, in JSON format. * `--model_id_with_origin_paths`: Model IDs with original paths, separated by commas. * `--extra_inputs`: Additional input parameters required by the model Pipeline, separated by `,`. * `--fp8_models`: Models to load in FP8 format, currently only supported for models whose parameters are not updated by gradients. * `--quant_options`: Dynamically quantize loaded models. Semicolon-separated entries, each `:[/]`, where `` matches an entry in `--model_paths`/`--model_id_with_origin_paths`, `method` is a registered method (e.g. `bitsandbytes_nf4`), and `exclude_modules` optionally lists layers kept in full precision. * Basic Training Configuration * `--learning_rate`: Learning rate. * `--num_epochs`: Number of epochs. * `--trainable_models`: Trainable models, e.g., `dit`, `vae`, `text_encoder`. * `--find_unused_parameters`: Whether unused parameters exist in DDP training. * `--weight_decay`: Weight decay magnitude. * `--task`: Training task, defaults to `sft`. * Output Configuration * `--output_path`: Path to save the model. * `--remove_prefix_in_ckpt`: Remove prefix in the model's state dict. * `--save_steps`: Interval in training steps to save the model. * LoRA Configuration * `--lora_base_model`: Which model to add LoRA to. * `--lora_target_modules`: Which layers to add LoRA to. * `--lora_rank`: Rank of LoRA. * `--lora_checkpoint`: Path to LoRA checkpoint. * `--preset_lora_path`: Path to preset LoRA checkpoint for LoRA differential training. * `--preset_lora_model`: Which model to integrate preset LoRA into, e.g., `dit`. * Gradient Configuration * `--use_gradient_checkpointing`: Whether to enable gradient checkpointing. * `--use_gradient_checkpointing_offload`: Whether to offload gradient checkpointing to CPU memory. * `--gradient_accumulation_steps`: Number of gradient accumulation steps. * Resolution Configuration * `--height`: Height of the image/video. Leave empty to enable dynamic resolution. * `--width`: Width of the image/video. Leave empty to enable dynamic resolution. * `--max_pixels`: Maximum pixel area, images larger than this will be scaled down during dynamic resolution. * `--num_frames`: Number of frames for video (video generation models only). * ACE-Step Specific Parameters * `--tokenizer_path`: Tokenizer path, in format model_id:origin_pattern. * `--silence_latent_path`: Silence latent path, in format model_id:origin_pattern. * `--initialize_model_on_cpu`: Whether to initialize models on CPU. ### Example Dataset ```shell modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --local_dir ./data/diffsynth_example_dataset ``` We provide recommended training scripts for each model, please refer to the table in "Model Overview" above. For guidance on writing model training scripts, see [Model Training](../Pipeline_Usage/Model_Training.md); for more advanced training algorithms, see [Training Framework Overview](https://github.com/modelscope/DiffSynth-Studio/tree/main/docs/en/Training/). --- ### En Model Details Anima (`docs/en/Model_Details/Anima.md`) # Anima Anima is an image generation model trained and open-sourced by CircleStone Labs and Comfy Org. ## Installation Before using this project for model inference and training, please install DiffSynth-Studio first. ```shell git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` For more installation information, please refer to [Install Dependencies](../Pipeline_Usage/Setup.md). ## Quick Start The following code demonstrates how to quickly load the [circlestone-labs/Anima](https://www.modelscope.cn/models/circlestone-labs/Anima) model for inference. VRAM management is enabled by default, allowing the framework to automatically control model parameter loading based on available VRAM. Minimum 8GB VRAM required. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Model Overview |Model ID|Inference|Low VRAM Inference|Full Training|Validation after Full Training|LoRA Training|Validation after LoRA Training| |-|-|-|-|-|-|-| |[circlestone-labs/Anima](https://www.modelscope.cn/models/circlestone-labs/Anima)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/anima/model_inference/anima-preview.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/anima/model_inference_low_vram/anima-preview.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/anima/model_training/full/anima-preview.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/anima/model_training/validate_full/anima-preview.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/anima/model_training/lora/anima-preview.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/anima/model_training/validate_lora/anima-preview.py)| Special training scripts: * Differential LoRA Training: [doc](../Training/Differential_LoRA.md) * FP8 Precision Training: [doc](../Training/FP8_Precision.md) * Two-Stage Split Training: [doc](../Training/Split_Training.md) * End-to-End Direct Distillation: [doc](../Training/Direct_Distill.md) ## Model Inference Models are loaded through `AnimaImagePipeline.from_pretrained`, see [Model Inference](../Pipeline_Usage/Model_Inference.md#loading-models) for details. Input parameters for `AnimaImagePipeline` inference include: * `prompt`: Text description of the desired image content. * `negative_prompt`: Content to exclude from the generated image (default: `""`). * `cfg_scale`: Classifier-free guidance parameter (default: 4.0). * `input_image`: Input image for image-to-image generation (default: `None`). * `denoising_strength`: Controls similarity to input image (default: 1.0). * `height`: Image height (must be multiple of 16, default: 1024). * `width`: Image width (must be multiple of 16, default: 1024). * `seed`: Random seed (default: `None`). * `rand_device`: Device for random noise generation (default: `"cpu"`). * `num_inference_steps`: Inference steps (default: 30). * `sigma_shift`: Scheduler sigma offset (default: `None`). * `progress_bar_cmd`: Progress bar implementation (default: `tqdm.tqdm`). For VRAM constraints, enable [VRAM Management](../Pipeline_Usage/VRAM_management.md). Recommended low-VRAM configurations are provided in the "Model Overview" table above. ## Model Training Anima models are trained through [`examples/anima/model_training/train.py`](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/anima/model_training/train.py) with parameters including: * General Training Parameters * Dataset Configuration * `--dataset_base_path`: Dataset root directory. * `--dataset_metadata_path`: Metadata file path. * `--dataset_repeat`: Dataset repetition per epoch. * `--dataset_num_workers`: Dataloader worker count. * `--data_file_keys`: Metadata fields to load (comma-separated). * Model Loading * `--model_paths`: Model paths (JSON format). * `--model_id_with_origin_paths`: Model IDs with origin paths (e.g., `"anima-team/anima-1B:text_encoder/*.safetensors"`). * `--extra_inputs`: Additional pipeline inputs (e.g., `controlnet_inputs` for ControlNet). * `--fp8_models`: FP8-formatted models (same format as `--model_paths`). * `--quant_options`: Dynamically quantize loaded models. Semicolon-separated entries, each `:[/]`, where `` matches an entry in `--model_paths`/`--model_id_with_origin_paths`, `method` is a registered method (e.g. `bitsandbytes_nf4`), and `exclude_modules` optionally lists layers kept in full precision. * Training Configuration * `--learning_rate`: Learning rate. * `--num_epochs`: Training epochs. * `--trainable_models`: Trainable components (e.g., `dit`, `vae`, `text_encoder`). * `--find_unused_parameters`: Handle unused parameters in DDP training. * `--weight_decay`: Weight decay value. * `--task`: Training task (default: `sft`). * Output Configuration * `--output_path`: Model output directory. * `--remove_prefix_in_ckpt`: Remove state dict prefixes. * `--save_steps`: Model saving interval. * LoRA Configuration * `--lora_base_model`: Target model for LoRA. * `--lora_target_modules`: Target modules for LoRA. * `--lora_rank`: LoRA rank. * `--lora_checkpoint`: LoRA checkpoint path. * `--preset_lora_path`: Preloaded LoRA checkpoint path. * `--preset_lora_model`: Model to merge LoRA with (e.g., `dit`). * Gradient Configuration * `--use_gradient_checkpointing`: Enable gradient checkpointing. * `--use_gradient_checkpointing_offload`: Offload checkpointing to CPU. * `--gradient_accumulation_steps`: Gradient accumulation steps. * Image Resolution * `--height`: Image height (empty for dynamic resolution). * `--width`: Image width (empty for dynamic resolution). * `--max_pixels`: Maximum pixel area for dynamic resolution. * Anima-Specific Parameters * `--tokenizer_path`: Tokenizer path for text-to-image models. * `--tokenizer_t5xxl_path`: T5-XXL tokenizer path. We provide a sample image dataset for testing: ```shell modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --local_dir ./data/diffsynth_example_dataset ``` For training script details, refer to [Model Training](../Pipeline_Usage/Model_Training.md). For advanced training techniques, see [Training Framework Documentation](https://github.com/modelscope/DiffSynth-Studio/tree/main/docs/zh/Training/). --- ### En Model Details Boogu Image (`docs/en/Model_Details/Boogu-Image.md`) # Boogu-Image Boogu-Image supports text-to-image, image-to-image, and instruction-guided image editing. ## Installation Before performing model inference and training, please install DiffSynth-Studio first. ```shell git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` For more information on installation, please refer to [Setup Dependencies](../Pipeline_Usage/Setup.md). ## Quick Start Running the following code will load the [Boogu/Boogu-Image-0.1-Base](https://modelscope.cn/models/Boogu/Boogu-Image-0.1-Base) model for inference. VRAM management is enabled, the framework automatically controls parameter loading based on available VRAM, requiring a minimum of 8GB VRAM. ```python from diffsynth.pipelines.boogu_image import BooguImagePipeline, ModelConfig import torch vram_config = { "offload_dtype": torch.float8_e4m3fn, "offload_device": "cpu", "onload_dtype": torch.float8_e4m3fn, "onload_device": "cpu", "preparing_dtype": torch.float8_e4m3fn, "preparing_device": "cuda", "computation_dtype": torch.bfloat16, "computation_device": "cuda", } pipe = BooguImagePipeline.from_pretrained( torch_dtype=torch.bfloat16, device="cuda", model_configs=[ ModelConfig(model_id="Boogu/Boogu-Image-0.1-Base", origin_file_pattern="transformer/*.safetensors", **vram_config), ModelConfig(model_id="Boogu/Boogu-Image-0.1-Base", origin_file_pattern="mllm/*.safetensors", **vram_config), ModelConfig(model_id="Boogu/Boogu-Image-0.1-Base", origin_file_pattern="vae/*.safetensors", **vram_config), ], processor_config=ModelConfig(model_id="Boogu/Boogu-Image-0.1-Base", origin_file_pattern="mllm/"), vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 0.5, ) output = pipe( prompt="a cat", negative_prompt="", height=1024, width=1024, seed=42, num_inference_steps=50, cfg_scale=4.0, ) output.save("image_Boogu-Image-0.1-Base.jpg") ``` ## Model Overview |Model ID|Inference|Low VRAM Inference|Full Training|Full Training Validation|LoRA Training|LoRA Training Validation| |-|-|-|-|-|-|-| |[Boogu/Boogu-Image-0.1-Base](https://modelscope.cn/models/Boogu/Boogu-Image-0.1-Base)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/boogu_image/model_inference/Boogu-Image-0.1-Base.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/boogu_image/model_inference_low_vram/Boogu-Image-0.1-Base.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/boogu_image/model_training/full/Boogu-Image-0.1-Base.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/boogu_image/model_training/validate_full/Boogu-Image-0.1-Base.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/boogu_image/model_training/lora/Boogu-Image-0.1-Base.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/boogu_image/model_training/validate_lora/Boogu-Image-0.1-Base.py)| |[Boogu/Boogu-Image-0.1-Turbo](https://modelscope.cn/models/Boogu/Boogu-Image-0.1-Turbo)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/boogu_image/model_inference/Boogu-Image-0.1-Turbo.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/boogu_image/model_inference_low_vram/Boogu-Image-0.1-Turbo.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/boogu_image/model_training/full/Boogu-Image-0.1-Turbo.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/boogu_image/model_training/validate_full/Boogu-Image-0.1-Turbo.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/boogu_image/model_training/lora/Boogu-Image-0.1-Turbo.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/boogu_image/model_training/validate_lora/Boogu-Image-0.1-Turbo.py)| |[Boogu/Boogu-Image-0.1-Edit](https://modelscope.cn/models/Boogu/Boogu-Image-0.1-Edit)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/boogu_image/model_inference/Boogu-Image-0.1-Edit.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/boogu_image/model_inference_low_vram/Boogu-Image-0.1-Edit.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/boogu_image/model_training/full/Boogu-Image-0.1-Edit.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/boogu_image/model_training/validate_full/Boogu-Image-0.1-Edit.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/boogu_image/model_training/lora/Boogu-Image-0.1-Edit.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/boogu_image/model_training/validate_lora/Boogu-Image-0.1-Edit.py)| ## Model Inference The model is loaded via `BooguImagePipeline.from_pretrained`, see [Loading Models](../Pipeline_Usage/Model_Inference.md#loading-models) for details. The input parameters for `BooguImagePipeline` inference include: * `prompt`: Text prompt describing the desired content or editing instruction. * `negative_prompt`: Negative prompt specifying what should not appear in the result, defaults to empty string. * `cfg_scale`: Classifier-free guidance scale factor, defaults to 4.0. Higher values make the output more closely follow the prompt. * `input_image`: Input image for image-to-image (img2img). When provided, the input image is noised and denoised according to `denoising_strength`. * `edit_image`: Image to be edited for instruction-guided editing. When provided, the model modifies the image according to the `prompt` instruction. * `height`: Height of the output image, defaults to 1024. Must be divisible by 16. * `width`: Width of the output image, defaults to 1024. Must be divisible by 16. * `seed`: Random seed for reproducibility. Set to `None` for random seed. * `denoising_strength`: Denoising strength controlling how much the input image is repainted, defaults to 1.0. Only effective when `input_image` is provided. * `sigmas`: Custom sigma scheduling sequence to override the default scheduling strategy. Required for Turbo models. * `num_inference_steps`: Number of inference steps, defaults to 20. More steps typically yield better quality. * `max_sequence_length`: Maximum sequence length for the text encoder, defaults to 1280. * `max_input_image_pixels`: Maximum pixel area for input images, defaults to 4194304. Images larger than this will be scaled down. * `max_input_image_side_length`: Maximum side length for input images, defaults to 4096. * `max_vlm_input_pil_pixels`: Maximum pixel area for VLM input images, defaults to 147456. Only effective in image editing mode. * `max_vlm_input_pil_side_length`: Maximum side length for VLM input images, defaults to 768. Only effective in image editing mode. * `rand_device`: Device for generating initial noise, defaults to "cpu". * `progress_bar_cmd`: Progress bar display mode, defaults to tqdm. When running low on VRAM, please refer to [VRAM Management](../Pipeline_Usage/VRAM_management.md) to enable VRAM management features. ## Model Training Models in the boogu_image series are trained uniformly via `examples/boogu_image/model_training/train.py`. The script parameters include: * General Training Parameters * Dataset Configuration * `--dataset_base_path`: Root directory of the dataset. * `--dataset_metadata_path`: Path to the dataset metadata file. * `--dataset_repeat`: Number of dataset repeats per epoch. * `--dataset_num_workers`: Number of processes per DataLoader. * `--data_file_keys`: Field names to load from metadata, typically paths to image or video files, separated by `,`. * Model Loading Configuration * `--model_paths`: Paths to load models from, in JSON format. * `--model_id_with_origin_paths`: Model IDs with original paths, separated by commas. * `--extra_inputs`: Additional input parameters required by the model Pipeline, separated by `,`. * `--fp8_models`: Models to load in FP8 format, currently only supported for models whose parameters are not updated by gradients. * `--quant_options`: Dynamically quantize loaded models. Semicolon-separated entries, each `:[/]`, where `` matches an entry in `--model_paths`/`--model_id_with_origin_paths`, `method` is a registered method (e.g. `bitsandbytes_nf4`), and `exclude_modules` optionally lists layers kept in full precision. * Basic Training Configuration * `--learning_rate`: Learning rate. * `--num_epochs`: Number of epochs. * `--trainable_models`: Trainable models, e.g., `dit`, `vae`, `text_encoder`. * `--find_unused_parameters`: Whether unused parameters exist in DDP training. * `--weight_decay`: Weight decay magnitude. * `--task`: Training task, defaults to `sft`. * Output Configuration * `--output_path`: Path to save the model. * `--remove_prefix_in_ckpt`: Remove prefix in the model's state dict. * `--save_steps`: Interval in training steps to save the model. * LoRA Configuration * `--lora_base_model`: Which model to add LoRA to. * `--lora_target_modules`: Which layers to add LoRA to. * `--lora_rank`: Rank of LoRA. * `--lora_checkpoint`: Path to LoRA checkpoint. * `--preset_lora_path`: Path to preset LoRA checkpoint for LoRA differential training. * `--preset_lora_model`: Which model to integrate preset LoRA into, e.g., `dit`. * Gradient Configuration * `--use_gradient_checkpointing`: Whether to enable gradient checkpointing. * `--use_gradient_checkpointing_offload`: Whether to offload gradient checkpointing to CPU memory. * `--gradient_accumulation_steps`: Number of gradient accumulation steps. * Resolution Configuration * `--height`: Height of the image/video. Leave empty to enable dynamic resolution. * `--width`: Width of the image/video. Leave empty to enable dynamic resolution. * `--max_pixels`: Maximum pixel area, images larger than this will be scaled down during dynamic resolution. * `--num_frames`: Number of frames for video (video generation models only). * Boogu-Image Specific Parameters * `--processor_path`: Path to the processor for processing text and image encoder inputs. * `--initialize_model_on_cpu`: Whether to initialize models on CPU. By default, models are initialized on the accelerator device. ```shell modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --local_dir ./data/diffsynth_example_dataset ``` We provide recommended training scripts for each model, please refer to the table in "Model Overview" above. For guidance on writing model training scripts, see [Model Training](../Pipeline_Usage/Model_Training.md); for more advanced training algorithms, see [Training Framework Overview](https://github.com/modelscope/DiffSynth-Studio/tree/main/docs/en/Training/). --- ### En Model Details DiffSynth Music (`docs/en/Model_Details/DiffSynth-Music.md`) # DiffSynth-Music DiffSynth-Music is a controllable music generation model suite developed by the DiffSynth-Studio team, built upon ACE-Step-1.5. It retains the foundational architecture of ACE-Step-1.5 while incorporating several additional modules to support diverse control mechanisms, including beat, vocals, accompaniment, rhythm, and timbre references. ## Installation Before performing model inference and training, please install DiffSynth-Studio first. ```shell git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` For more information on installation, please refer to [Setup Dependencies](../Pipeline_Usage/Setup.md). ## Quick Start Running the following code will load the [DiffSynth-Studio/DiffSynth-Music](https://www.modelscope.cn/models/DiffSynth-Studio/DiffSynth-Music) model for inference. The example covers six use cases: native music synthesis, beats control, vocals control, accompaniment control, prosody control, and audio reference (timbre) control. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` A low VRAM version is available at [examples/diffsynth_music/model_inference_low_vram/DiffSynth-Music.py](/examples/diffsynth_music/model_inference_low_vram/DiffSynth-Music.py). ## Model Overview |Model ID|Inference|Low VRAM Inference|Full Training|Full Training Validation|LoRA Training|LoRA Training Validation| |-|-|-|-|-|-|-| | [DiffSynth-Studio/DiffSynth-Music](https://www.modelscope.cn/models/DiffSynth-Studio/DiffSynth-Music) | [code](/examples/diffsynth_music/model_inference/DiffSynth-Music.py) | [code](/examples/diffsynth_music/model_inference_low_vram/DiffSynth-Music.py) | [code](/examples/diffsynth_music/model_training/full/DiffSynth-Music.sh) | [code](/examples/diffsynth_music/model_training/validate_full/DiffSynth-Music.py) | - | - | ## Model Inference The model is loaded via `DiffSynthMusicPipeline.from_pretrained`, see [Loading Models](../Pipeline_Usage/Model_Inference.md#loading-models) for details. The input parameters for `DiffSynthMusicPipeline` inference include: * `prompt`: The music description, specifying genre, mood and style. * `negative_prompt`: The negative prompt for classifier-free guidance. * `lyrics`: The lyrics. Structure tags such as `[Intro]` and `[Verse]` must each be on their own line. Leave it empty to generate instrumental music. * `bpm`: Beats per minute. Default: 100. * `timesignature`: Time signature. Default: "4". * `keyscale`: Key and scale. Default: "B minor". * `input_audio`: Optional audio tensor used as the starting point, combined with `denoising_strength` for audio-to-audio generation. * `duration`: Duration of the generated audio in seconds. * `num_inference_steps`: Number of flow-matching steps. * `cfg_scale`: Classifier-free guidance scale. * `seed`: Random seed. * `tiled`: Whether to use tiled VAE decoding. * `kv_cache`: Pre-computed KV cache for faster inference. * `target_audio` / `target_track`: Optional track fusion, replacing the given track of the output with the corresponding track of `target_audio`. The `TemplatePipeline` provides three control adapters loaded via `template_inputs` with `model_id`: * `model_id: 0` — template_control: beats / vocals / accompaniment control. The generated audio follows the rhythm of the given audio. Combined with `target_audio`, the target track is fused into the output. * `model_id: 1` — template_prosody: the generated vocals follow the pitch and rhythm (prosody) of the given audio. Use `extract_prosody` to build the input. * `model_id: 2` — template_reference: the timbre of the output is controlled by the reference audio. If you run out of VRAM, please refer to [VRAM Management](../Pipeline_Usage/Model_Inference.md#vram-management). ## Model Training The model supports LoRA training and full training of the DiT (trainable models: `dit`) on `(prompt, lyrics, audio)` data. See [examples/diffsynth_music/model_training](/examples/diffsynth_music/model_training/) for training scripts. --- ### En Model Details ERNIE Image (`docs/en/Model_Details/ERNIE-Image.md`) # ERNIE-Image ERNIE-Image is a powerful image generation model with 8B parameters developed by Baidu, featuring a compact and efficient architecture with strong instruction-following capability. Based on an 8B DiT backbone, it delivers performance comparable to larger (20B+) models in certain scenarios while maintaining parameter efficiency. It offers reliable performance in instruction understanding and execution, text generation (English/Chinese/Japanese), and overall stability. ## Installation Before performing model inference and training, please install DiffSynth-Studio first. ```shell git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` For more information on installation, please refer to [Setup Dependencies](../Pipeline_Usage/Setup.md). ## Quick Start Running the following code will load the [PaddlePaddle/ERNIE-Image](https://www.modelscope.cn/models/PaddlePaddle/ERNIE-Image) model for inference. VRAM management is enabled, the framework automatically controls parameter loading based on available VRAM, requiring a minimum of 3G VRAM. ```python from diffsynth.pipelines.ernie_image import ErnieImagePipeline, ModelConfig import torch vram_config = { "offload_dtype": torch.bfloat16, "offload_device": "cpu", "onload_dtype": torch.bfloat16, "onload_device": "cpu", "preparing_dtype": torch.bfloat16, "preparing_device": "cuda", "computation_dtype": torch.bfloat16, "computation_device": "cuda", } pipe = ErnieImagePipeline.from_pretrained( torch_dtype=torch.bfloat16, device='cuda', model_configs=[ ModelConfig(model_id="PaddlePaddle/ERNIE-Image", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors", **vram_config), ModelConfig(model_id="PaddlePaddle/ERNIE-Image", origin_file_pattern="text_encoder/model.safetensors", **vram_config), ModelConfig(model_id="PaddlePaddle/ERNIE-Image", origin_file_pattern="vae/diffusion_pytorch_model.safetensors", **vram_config), ], tokenizer_config=ModelConfig(model_id="PaddlePaddle/ERNIE-Image", origin_file_pattern="tokenizer/"), vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 0.5, ) image = pipe( prompt="一只黑白相间的中华田园犬", negative_prompt="", height=1024, width=1024, seed=42, num_inference_steps=50, cfg_scale=4.0, ) image.save("output.jpg") ``` ## Model Overview |Model ID|Inference|Low VRAM Inference|Full Training|Full Training Validation|LoRA Training|LoRA Training Validation| |-|-|-|-|-|-|-| |[PaddlePaddle/ERNIE-Image](https://www.modelscope.cn/models/PaddlePaddle/ERNIE-Image)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ernie_image/model_inference/ERNIE-Image.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ernie_image/model_inference_low_vram/ERNIE-Image.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ernie_image/model_training/full/ERNIE-Image.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ernie_image/model_training/validate_full/ERNIE-Image.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ernie_image/model_training/lora/ERNIE-Image.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ernie_image/model_training/validate_lora/ERNIE-Image.py)| |[PaddlePaddle/ERNIE-Image-Turbo](https://www.modelscope.cn/models/PaddlePaddle/ERNIE-Image-Turbo)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ernie_image/model_inference/ERNIE-Image-Turbo.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ernie_image/model_inference_low_vram/ERNIE-Image-Turbo.py)|—|—|—|—| ## Model Inference The model is loaded via `ErnieImagePipeline.from_pretrained`, see [Loading Models](../Pipeline_Usage/Model_Inference.md#loading-models) for details. The input parameters for `ErnieImagePipeline` inference include: * `prompt`: The prompt describing the content to appear in the image. * `negative_prompt`: The negative prompt describing what should not appear in the image, default value is `""`. * `cfg_scale`: Classifier-free guidance parameter, default value is 4.0. * `height`: Image height, must be a multiple of 16, default value is 1024. * `width`: Image width, must be a multiple of 16, default value is 1024. * `seed`: Random seed. Default is `None`, meaning completely random. * `rand_device`: The computing device for generating random Gaussian noise matrices, default is `"cuda"`. When set to `cuda`, different GPUs will produce different results. * `num_inference_steps`: Number of inference steps, default value is 50. If VRAM is insufficient, please enable [VRAM Management](../Pipeline_Usage/VRAM_management.md). We provide recommended low-VRAM configurations for each model in the "Model Overview" table above. ## Model Training ERNIE-Image series models are trained uniformly via [`examples/ernie_image/model_training/train.py`](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ernie_image/model_training/train.py). The script parameters include: * General Training Parameters * Dataset Configuration * `--dataset_base_path`: Root directory of the dataset. * `--dataset_metadata_path`: Path to the dataset metadata file. * `--dataset_repeat`: Number of dataset repeats per epoch. * `--dataset_num_workers`: Number of processes per DataLoader. * `--data_file_keys`: Field names to load from metadata, typically paths to image or video files, separated by `,`. * Model Loading Configuration * `--model_paths`: Paths to load models from, in JSON format. * `--model_id_with_origin_paths`: Model IDs with original paths, e.g., `"PaddlePaddle/ERNIE-Image:transformer/diffusion_pytorch_model*.safetensors"`, separated by commas. * `--extra_inputs`: Additional input parameters required by the model Pipeline, separated by `,`. * `--fp8_models`: Models to load in FP8 format, currently only supported for models whose parameters are not updated by gradients. * `--quant_options`: Dynamically quantize loaded models. Semicolon-separated entries, each `:[/]`, where `` matches an entry in `--model_paths`/`--model_id_with_origin_paths`, `method` is a registered method (e.g. `bitsandbytes_nf4`), and `exclude_modules` optionally lists layers kept in full precision. * Basic Training Configuration * `--learning_rate`: Learning rate. * `--num_epochs`: Number of epochs. * `--trainable_models`: Trainable models, e.g., `dit`, `vae`, `text_encoder`. * `--find_unused_parameters`: Whether unused parameters exist in DDP training. * `--weight_decay`: Weight decay magnitude. * `--task`: Training task, defaults to `sft`. * Output Configuration * `--output_path`: Path to save the model. * `--remove_prefix_in_ckpt`: Remove prefix in the model's state dict. * `--save_steps`: Interval in training steps to save the model. * LoRA Configuration * `--lora_base_model`: Which model to add LoRA to. * `--lora_target_modules`: Which layers to add LoRA to. * `--lora_rank`: Rank of LoRA. * `--lora_checkpoint`: Path to LoRA checkpoint. * `--preset_lora_path`: Path to preset LoRA checkpoint for LoRA differential training. * `--preset_lora_model`: Which model to integrate preset LoRA into, e.g., `dit`. * Gradient Configuration * `--use_gradient_checkpointing`: Whether to enable gradient checkpointing. * `--use_gradient_checkpointing_offload`: Whether to offload gradient checkpointing to CPU memory. * `--gradient_accumulation_steps`: Number of gradient accumulation steps. * Resolution Configuration * `--height`: Height of the image. Leave empty to enable dynamic resolution. * `--width`: Width of the image. Leave empty to enable dynamic resolution. * `--max_pixels`: Maximum pixel area, images larger than this will be scaled down during dynamic resolution. * ERNIE-Image Specific Parameters * `--tokenizer_path`: Path to the tokenizer, leave empty to auto-download from remote. We provide an example image dataset for testing, which can be downloaded with the following command: ```shell modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --local_dir ./data/diffsynth_example_dataset ``` We provide recommended training scripts for each model, please refer to the table in "Model Overview" above. For guidance on writing model training scripts, see [Model Training](../Pipeline_Usage/Model_Training.md); for more advanced training algorithms, see [Training Framework Overview](https://github.com/modelscope/DiffSynth-Studio/tree/main/docs/en/Training/). --- ### En Model Details FLUX.3 (`docs/en/Model_Details/FLUX.3.md`) # FLUX.3 FLUX.3 Image Klein is a 5.6B open-weight image generation model from Black Forest Labs that supports text-to-image generation, editing with up to 10 reference images, and bounding-box layout control in one model. ## Installation Before performing model inference and training, please install DiffSynth-Studio first. ```shell git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` For more information on installation, please refer to [Setup Dependencies](../Pipeline_Usage/Setup.md). ## Model Access `black-forest-labs/flux-3-image-klein` and `black-forest-labs/flux-3-image-klein-base` are access-request-only models on ModelScope. Before downloading them: 1. Request access on the model page ([Klein](https://modelscope.cn/models/black-forest-labs/flux-3-image-klein) or [Klein Base](https://modelscope.cn/models/black-forest-labs/flux-3-image-klein-base)). 2. Log in via the CLI with your ModelScope token: `modelscope login --token ms-xxx`. Get your token at [Settings / Access Token](https://modelscope.cn/my/settings/token). Access must be granted for the same account used to log in. ## Quick Start Running the following code will load the [black-forest-labs/flux-3-image-klein](https://www.modelscope.cn/models/black-forest-labs/flux-3-image-klein) model for inference. VRAM management is enabled, and the framework automatically controls parameter loading based on available VRAM. Minimum 4GB VRAM is required to run. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Model Overview |Model ID|Inference|Low VRAM Inference|Full Training|Full Training Validation|LoRA Training|LoRA Training Validation| |-|-|-|-|-|-|-| |[black-forest-labs/flux-3-image-klein](https://www.modelscope.cn/models/black-forest-labs/flux-3-image-klein)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux3/model_inference/flux-3-image-klein.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux3/model_inference_low_vram/flux-3-image-klein.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux3/model_training/full/flux-3-image-klein.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux3/model_training/validate_full/flux-3-image-klein.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux3/model_training/lora/flux-3-image-klein.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux3/model_training/validate_lora/flux-3-image-klein.py)| |[black-forest-labs/flux-3-image-klein-base](https://www.modelscope.cn/models/black-forest-labs/flux-3-image-klein-base)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux3/model_inference/flux-3-image-klein-base.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux3/model_inference_low_vram/flux-3-image-klein-base.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux3/model_training/full/flux-3-image-klein-base.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux3/model_training/validate_full/flux-3-image-klein-base.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux3/model_training/lora/flux-3-image-klein-base.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux3/model_training/validate_lora/flux-3-image-klein-base.py)| |[DiffSynth-Studio/flux-3-image-klein-entropack](https://www.modelscope.cn/models/DiffSynth-Studio/flux-3-image-klein-entropack)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux3/model_inference/flux-3-image-klein-EntroPack.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux3/model_inference_low_vram/flux-3-image-klein-EntroPack.py)|-|-|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux3/model_training/lora/flux-3-image-klein-entropack.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux3/model_training/validate_lora/flux-3-image-klein-entropack.py)| ## Model Inference The model is loaded via `Flux3ImagePipeline.from_pretrained`, see [Loading Models](../Pipeline_Usage/Model_Inference.md#loading-models) for details. The input parameters for `Flux3ImagePipeline` inference include: * `prompt`: Prompt describing the target image, edit, or layout; defaults to `""`. * `negative_prompt`: Negative prompt used by Base when CFG is enabled; defaults to `""`. * `cfg_scale`: Classifier-free guidance strength, defaults to 1.0. Distilled requires 1.0; the Base examples use 3.5. * `input_image`: Input image for image-to-image generation; generation starts from pure noise when omitted. * `denoising_strength`: Image-to-image denoising strength from 0 to 1, defaults to 1.0. * `edit_image`: Reference image(s) for editing, accepting one `PIL.Image` or a list of images; up to 10 references are supported. * `height`: Output image height, defaults to 1024 and is aligned to a multiple of 16. * `width`: Output image width, defaults to 1024 and is aligned to a multiple of 16. * `seed`: Random seed, defaults to `None`. * `rand_device`: Device used to generate the initial Gaussian noise, defaults to `"cpu"`. * `num_inference_steps`: Number of inference steps, defaults to 16. The Distilled examples use 16 steps and the Base examples use 50. * `alpha`: Rational-shift coefficient for the FLUX.3 flow-matching schedule, defaults to 6.93. * `progress_bar_cmd`: Function wrapping the inference timesteps with a progress bar, defaults to `tqdm`. If VRAM is insufficient, please enable [VRAM Management](../Pipeline_Usage/VRAM_management.md). Recommended configurations are linked in the "Model Overview" table above. ## Model Training Models in the FLUX.3 series are trained uniformly via [`examples/flux3/model_training/train.py`](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux3/model_training/train.py). The script parameters include: * General Training Parameters * Dataset Configuration * `--dataset_base_path`: Root directory of the dataset. * `--dataset_metadata_path`: Path to the dataset metadata file. * `--dataset_repeat`: Number of dataset repeats per epoch. * `--dataset_num_workers`: Number of processes per DataLoader. * `--data_file_keys`: Field names to load from metadata, typically paths to image or video files, separated by `,`. * Model Loading Configuration * `--model_paths`: Paths to load models from, in JSON format. * `--model_id_with_origin_paths`: Model IDs with original paths, separated by commas. * `--extra_inputs`: Additional input parameters required by the model Pipeline, separated by `,`. * `--fp8_models`: Models to load in FP8 format, currently only supported for models whose parameters are not updated by gradients. * `--quant_options`: Dynamically quantize loaded models. Semicolon-separated entries, each `:[/]`, where `` matches an entry in `--model_paths`/`--model_id_with_origin_paths`, `method` is a registered method (e.g. `bitsandbytes_nf4`), and `exclude_modules` optionally lists layers kept in full precision. * Basic Training Configuration * `--learning_rate`: Learning rate. * `--num_epochs`: Number of epochs. * `--trainable_models`: Trainable models, e.g., `dit`, `vae`, `text_encoder`. * `--find_unused_parameters`: Whether unused parameters exist in DDP training. * `--weight_decay`: Weight decay magnitude. * `--task`: Training task, defaults to `sft`. * Output Configuration * `--output_path`: Path to save the model. * `--remove_prefix_in_ckpt`: Remove prefix in the model's state dict. * `--save_steps`: Interval in training steps to save the model. * LoRA Configuration * `--lora_base_model`: Which model to add LoRA to. * `--lora_target_modules`: Which layers to add LoRA to. * `--lora_rank`: Rank of LoRA. * `--lora_checkpoint`: Path to LoRA checkpoint. * `--preset_lora_path`: Path to preset LoRA checkpoint for LoRA differential training. * `--preset_lora_model`: Which model to integrate preset LoRA into, e.g., `dit`. * Gradient Configuration * `--use_gradient_checkpointing`: Whether to enable gradient checkpointing. * `--use_gradient_checkpointing_offload`: Whether to offload gradient checkpointing to CPU memory. * `--gradient_accumulation_steps`: Number of gradient accumulation steps. * Resolution Configuration * `--height`: Height of the image/video. Leave empty to enable dynamic resolution. * `--width`: Width of the image/video. Leave empty to enable dynamic resolution. * `--max_pixels`: Maximum pixel area, images larger than this will be scaled down during dynamic resolution. * `--num_frames`: Number of frames for video (video generation models only). * FLUX.3 Specific Parameters * `--processor_path`: Local processor path or model ID; when omitted, the Base model's `text_encoder/` is used. * `--initialize_model_on_cpu`: Initialize models on CPU to reduce peak VRAM during distributed training startup. We have built a sample image dataset for your testing. You can download this dataset with the following command: ```shell modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --local_dir ./data/diffsynth_example_dataset ``` We provide recommended training scripts for each model, please refer to the table in "Model Overview" above. For guidance on writing model training scripts, see [Model Training](../Pipeline_Usage/Model_Training.md); for more advanced training algorithms, see [Training Framework Overview](https://github.com/modelscope/DiffSynth-Studio/tree/main/docs/en/Training/). --- ### En Model Details FLUX (`docs/en/Model_Details/FLUX.md`) # FLUX FLUX is an image generation model series developed and open-sourced by Black Forest Labs. ## Installation Before using this project for model inference and training, please install DiffSynth-Studio first. ```shell git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` For more information about installation, please refer to [Install Dependencies](../Pipeline_Usage/Setup.md). ## Quick Start Run the following code to quickly load the [black-forest-labs/FLUX.1-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.1-dev) model and perform inference. VRAM management is enabled, and the framework will automatically control model parameter loading based on remaining VRAM. Minimum 8GB VRAM is required to run. ```python import torch from diffsynth.pipelines.flux_image import FluxImagePipeline, ModelConfig vram_config = { "offload_dtype": torch.float8_e4m3fn, "offload_device": "cpu", "onload_dtype": torch.float8_e4m3fn, "onload_device": "cpu", "preparing_dtype": torch.float8_e4m3fn, "preparing_device": "cuda", "computation_dtype": torch.bfloat16, "computation_device": "cuda", } pipe = FluxImagePipeline.from_pretrained( torch_dtype=torch.bfloat16, device="cuda", model_configs=[ ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="flux1-dev.safetensors", **vram_config), ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="text_encoder/model.safetensors", **vram_config), ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="text_encoder_2/*.safetensors", **vram_config), ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="ae.safetensors", **vram_config), ], vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 1, ) prompt = "CG, masterpiece, best quality, solo, long hair, wavy hair, silver hair, blue eyes, blue dress, medium breasts, dress, underwater, air bubble, floating hair, refraction, portrait. The girl's flowing silver hair shimmers with every color of the rainbow and cascades down, merging with the floating flora around her." image = pipe(prompt=prompt, seed=0) image.save("image.jpg") ``` ## Model Overview | Model ID | Extra Parameters | Inference | Low VRAM Inference | Full Training | Validation After Full Training | LoRA Training | Validation After LoRA Training | | - | - | - | - | - | - | - | - | | [black-forest-labs/FLUX.1-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.1-dev) | | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-dev.py) | | [black-forest-labs/FLUX.1-Krea-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.1-Krea-dev) | | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-Krea-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-Krea-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-Krea-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-Krea-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-Krea-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-Krea-dev.py) | | [black-forest-labs/FLUX.1-Kontext-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.1-Kontext-dev) | `kontext_images` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-Kontext-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-Kontext-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-Kontext-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-Kontext-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-Kontext-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-Kontext-dev.py) | | [black-forest-labs/FLUX.1-Fill-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.1-Fill-dev) | `flux_fill_image`, `flux_fill_mask` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-Fill-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-Fill-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-Fill-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-Fill-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-Fill-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-Fill-dev.py) | | [black-forest-labs/FLUX.1-Redux-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.1-Redux-dev) | `flux_redux_image` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-Redux-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-Redux-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-Redux-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-Redux-dev.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-Redux-dev.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-Redux-dev.py) | | [HuanJue/Insert-Anything](https://www.modelscope.cn/models/HuanJue/Insert-Anything) | `insert_anything_source_image`, `insert_anything_source_mask`, `insert_anything_ref_image`, `insert_anything_ref_mask` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/Insert-Anything.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/Insert-Anything.py) | - | - | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/Insert-Anything.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/Insert-Anything.py) | | [alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Beta](https://www.modelscope.cn/models/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Beta) | `controlnet_inputs` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-dev-Controlnet-Inpainting-Beta.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-dev-Controlnet-Inpainting-Beta.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-dev-Controlnet-Inpainting-Beta.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-dev-Controlnet-Inpainting-Beta.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-dev-Controlnet-Inpainting-Beta.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-dev-Controlnet-Inpainting-Beta.py) | | [InstantX/FLUX.1-dev-Controlnet-Union-alpha](https://www.modelscope.cn/models/InstantX/FLUX.1-dev-Controlnet-Union-alpha) | `controlnet_inputs` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-dev-Controlnet-Union-alpha.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-dev-Controlnet-Union-alpha.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-dev-Controlnet-Union-alpha.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-dev-Controlnet-Union-alpha.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-dev-Controlnet-Union-alpha.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-dev-Controlnet-Union-alpha.py) | | [jasperai/Flux.1-dev-Controlnet-Upscaler](https://www.modelscope.cn/models/jasperai/Flux.1-dev-Controlnet-Upscaler) | `controlnet_inputs` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-dev-Controlnet-Upscaler.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-dev-Controlnet-Upscaler.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-dev-Controlnet-Upscaler.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-dev-Controlnet-Upscaler.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-dev-Controlnet-Upscaler.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-dev-Controlnet-Upscaler.py) | | [InstantX/FLUX.1-dev-IP-Adapter](https://www.modelscope.cn/models/InstantX/FLUX.1-dev-IP-Adapter) | `ipadapter_images`, `ipadapter_scale` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-dev-IP-Adapter.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-dev-IP-Adapter.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-dev-IP-Adapter.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-dev-IP-Adapter.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-dev-IP-Adapter.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-dev-IP-Adapter.py) | | [ByteDance/InfiniteYou](https://www.modelscope.cn/models/ByteDance/InfiniteYou) | `infinityou_id_image`, `infinityou_guidance`, `controlnet_inputs` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-dev-InfiniteYou.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-dev-InfiniteYou.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-dev-InfiniteYou.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-dev-InfiniteYou.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-dev-InfiniteYou.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-dev-InfiniteYou.py) | | [DiffSynth-Studio/Eligen](https://www.modelscope.cn/models/DiffSynth-Studio/Eligen) | `eligen_entity_prompts`, `eligen_entity_masks`, `eligen_enable_on_negative`, `eligen_enable_inpaint` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-dev-EliGen.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-dev-EliGen.py) | - | - | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLUX.1-dev-EliGen.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLUX.1-dev-EliGen.py) | | [DiffSynth-Studio/LoRA-Encoder-FLUX.1-Dev](https://www.modelscope.cn/models/DiffSynth-Studio/LoRA-Encoder-FLUX.1-Dev) | `lora_encoder_inputs`, `lora_encoder_scale` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-dev-LoRA-Encoder.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLUX.1-dev-LoRA-Encoder.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLUX.1-dev-LoRA-Encoder.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLUX.1-dev-LoRA-Encoder.py) | - | - | | [DiffSynth-Studio/LoRAFusion-preview-FLUX.1-dev](https://modelscope.cn/models/DiffSynth-Studio/LoRAFusion-preview-FLUX.1-dev) | | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLUX.1-dev-LoRA-Fusion.py) | - | - | - | - | - | | [stepfun-ai/Step1X-Edit](https://www.modelscope.cn/models/stepfun-ai/Step1X-Edit) | `step1x_reference_image` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/Step1X-Edit.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/Step1X-Edit.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/Step1X-Edit.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/Step1X-Edit.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/Step1X-Edit.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/Step1X-Edit.py) | | [ostris/Flex.2-preview](https://www.modelscope.cn/models/ostris/Flex.2-preview) | `flex_inpaint_image`, `flex_inpaint_mask`, `flex_control_image`, `flex_control_strength`, `flex_control_stop` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/FLEX.2-preview.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/FLEX.2-preview.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/FLEX.2-preview.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/FLEX.2-preview.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/FLEX.2-preview.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/FLEX.2-preview.py) | | [DiffSynth-Studio/Nexus-GenV2](https://www.modelscope.cn/models/DiffSynth-Studio/Nexus-GenV2) | `nexus_gen_reference_image` | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference/Nexus-Gen-Editing.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_inference_low_vram/Nexus-Gen-Editing.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/full/Nexus-Gen.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_full/Nexus-Gen.py) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/lora/Nexus-Gen.sh) | [code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/validate_lora/Nexus-Gen.py) | Special Training Scripts: * Differential LoRA Training: [doc](../Training/Differential_LoRA.md) * FP8 Precision Training: [doc](../Training/FP8_Precision.md) * Two-stage Split Training: [doc](../Training/Split_Training.md) * End-to-end Direct Distillation: [doc](../Training/Direct_Distill.md) ## Model Inference Models are loaded via `FluxImagePipeline.from_pretrained`, see [Loading Models](../Pipeline_Usage/Model_Inference.md#loading-models). Input parameters for `FluxImagePipeline` inference include: * `prompt`: Prompt describing the content appearing in the image. * `negative_prompt`: Negative prompt describing content that should not appear in the image, default value is `""`. * `cfg_scale`: Classifier-free guidance parameter, default value is 1. When set to a value greater than 1, CFG is enabled. * `height`: Image height, must be a multiple of 16. * `width`: Image width, must be a multiple of 16. * `seed`: Random seed. Default is `None`, meaning completely random. * `rand_device`: Computing device for generating random Gaussian noise matrix, default is `"cpu"`. When set to `cuda`, different GPUs will produce different generation results. * `num_inference_steps`: Number of inference steps, default value is 30. * `embedded_guidance`: Embedded guidance parameter, default value is 3.5. * `t5_sequence_length`: Sequence length of the T5 text encoder, default is 512. * `tiled`: Whether to enable VAE tiling inference, default is `False`. Setting to `True` can significantly reduce VRAM usage during VAE encoding/decoding stages, producing slight errors and slightly longer inference time. * `tile_size`: Tile size during VAE encoding/decoding stages, default is 128, only effective when `tiled=True`. * `tile_stride`: Tile stride during VAE encoding/decoding stages, default is 64, only effective when `tiled=True`, must be less than or equal to `tile_size`. * `progress_bar_cmd`: Progress bar, default is `tqdm.tqdm`. Can be disabled by setting to `lambda x:x`. * `controlnet_inputs`: ControlNet model inputs, type is `ControlNetInput` list. * `ipadapter_images`: IP-Adapter model input image list. * `ipadapter_scale`: Guidance strength of the IP-Adapter model. * `infinityou_id_image`: InfiniteYou model input image. * `infinityou_guidance`: Guidance strength of the InfiniteYou model. * `kontext_images`: Kontext model input images. * `eligen_entity_prompts`: EliGen partition control prompt list. * `eligen_entity_masks`: EliGen partition control region mask image list. * `eligen_enable_on_negative`: Whether to enable EliGen partition control on the negative side of CFG. * `eligen_enable_inpaint`: Whether to enable EliGen partition control inpainting function. * `lora_encoder_inputs`: LoRA encoder input image list. * `lora_encoder_scale`: Guidance strength of the LoRA encoder. * `step1x_reference_image`: Step1X model reference image. * `flex_inpaint_image`: Flex model image to be inpainted. * `flex_inpaint_mask`: Flex model inpainting mask. * `flex_control_image`: Flex model control image. * `flex_control_strength`: Flex model control strength. * `flex_control_stop`: Flex model control stop timestep. * `nexus_gen_reference_image`: Nexus-Gen model reference image. * `flux_fill_image`: FLUX.1-Fill model image to be inpainted. * `flux_fill_mask`: FLUX.1-Fill model inpainting mask. * `flux_redux_image`: FLUX.1-Redux model reference image. * `insert_anything_source_image`: Insert-Anything model source image, i.e., the target image to be edited. * `insert_anything_source_mask`: Insert-Anything model source image mask, specifying the region to be edited. * `insert_anything_ref_image`: Insert-Anything model reference image, providing the content to be inserted. * `insert_anything_ref_mask`: Insert-Anything model reference image mask, specifying the target object in the reference image. If VRAM is insufficient, please enable [VRAM Management](../Pipeline_Usage/VRAM_management.md). We provide recommended low VRAM configurations for each model in the example code, see the table in the "Model Overview" section above. ## Model Training FLUX series models are uniformly trained through [`examples/flux/model_training/train.py`](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux/model_training/train.py), and the script parameters include: * General Training Parameters * Dataset Basic Configuration * `--dataset_base_path`: Root directory of the dataset. * `--dataset_metadata_path`: Metadata file path of the dataset. * `--dataset_repeat`: Number of times the dataset is repeated in each epoch. * `--dataset_num_workers`: Number of processes for each DataLoader. * `--data_file_keys`: Field names to be loaded from metadata, usually image or video file paths, separated by `,`. * Model Loading Configuration * `--model_paths`: Paths of models to be loaded. JSON format. * `--model_id_with_origin_paths`: Model IDs with original paths, e.g., `"black-forest-labs/FLUX.1-dev:flux1-dev.safetensors"`. Separated by commas. * `--extra_inputs`: Extra input parameters required by the model Pipeline, e.g., `controlnet_inputs` when training ControlNet models, separated by `,`. * `--fp8_models`: Models loaded in FP8 format, consistent with `--model_paths` or `--model_id_with_origin_paths` format. Currently only supports models whose parameters are not updated by gradients (no gradient backpropagation, or gradients only update their LoRA). * `--quant_options`: Dynamically quantize loaded models. Semicolon-separated entries, each `:[/]`, where `` matches an entry in `--model_paths`/`--model_id_with_origin_paths`, `method` is a registered method (e.g. `bitsandbytes_nf4`), and `exclude_modules` optionally lists layers kept in full precision. * Training Basic Configuration * `--learning_rate`: Learning rate. * `--num_epochs`: Number of epochs. * `--trainable_models`: Trainable models, e.g., `dit`, `vae`, `text_encoder`. * `--find_unused_parameters`: Whether there are unused parameters in DDP training. Some models contain redundant parameters that do not participate in gradient calculation, and this setting needs to be enabled to avoid errors in multi-GPU training. * `--weight_decay`: Weight decay size, see [torch.optim.AdamW](https://docs.pytorch.org/docs/stable/generated/torch.optim.AdamW.html). * `--task`: Training task, default is `sft`. Some models support more training modes, please refer to the documentation of each specific model. * Output Configuration * `--output_path`: Model saving path. * `--remove_prefix_in_ckpt`: Remove prefix in the state dict of the model file. * `--save_steps`: Interval of training steps to save the model. If this parameter is left blank, the model is saved once per epoch. * LoRA Configuration * `--lora_base_model`: Which model to add LoRA to. * `--lora_target_modules`: Which layers to add LoRA to. * `--lora_rank`: Rank of LoRA. * `--lora_checkpoint`: Path of the LoRA checkpoint. If this path is provided, LoRA will be loaded from this checkpoint. * `--preset_lora_path`: Preset LoRA checkpoint path. If this path is provided, this LoRA will be loaded in the form of being merged into the base model. This parameter is used for LoRA differential training. * `--preset_lora_model`: Model that the preset LoRA is merged into, e.g., `dit`. * Gradient Configuration * `--use_gradient_checkpointing`: Whether to enable gradient checkpointing. * `--use_gradient_checkpointing_offload`: Whether to offload gradient checkpointing to memory. * `--gradient_accumulation_steps`: Number of gradient accumulation steps. * Image Width/Height Configuration (Applicable to Image Generation and Video Generation Models) * `--height`: Height of image or video. Leave `height` and `width` blank to enable dynamic resolution. * `--width`: Width of image or video. Leave `height` and `width` blank to enable dynamic resolution. * `--max_pixels`: Maximum pixel area of image or video frames. When dynamic resolution is enabled, images with resolution larger than this value will be downscaled, and images with resolution smaller than this value will remain unchanged. * FLUX Specific Parameters * `--tokenizer_1_path`: Path of the CLIP tokenizer, leave blank to automatically download from remote. * `--tokenizer_2_path`: Path of the T5 tokenizer, leave blank to automatically download from remote. * `--align_to_opensource_format`: Whether to align LoRA format to open-source format, only applicable to DiT's LoRA. We have built a sample image dataset for your testing. You can download this dataset with the following command: ```shell modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --local_dir ./data/diffsynth_example_dataset ``` We have written recommended training scripts for each model, please refer to the table in the "Model Overview" section above. For how to write model training scripts, please refer to [Model Training](../Pipeline_Usage/Model_Training.md); for more advanced training algorithms, please refer to [Training Framework Detailed Explanation](https://github.com/modelscope/DiffSynth-Studio/tree/main/docs/en/Training/). --- ### En Model Details FLUX2 (`docs/en/Model_Details/FLUX2.md`) # FLUX.2 FLUX.2 is an image generation model trained and open-sourced by Black Forest Labs. ## Installation Before using this project for model inference and training, please install DiffSynth-Studio first. ```shell git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` For more information about installation, please refer to [Install Dependencies](../Pipeline_Usage/Setup.md). ## Quick Start Run the following code to quickly load the [black-forest-labs/FLUX.2-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.2-dev) model and perform inference. VRAM management is enabled, and the framework will automatically control model parameter loading based on remaining VRAM. Minimum 10GB VRAM is required to run. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Model Overview | Model ID | Inference | Low VRAM Inference | Full Training | Validation After Full Training | LoRA Training | Validation After LoRA Training | | - | - | - | - | - | - | - | |[black-forest-labs/FLUX.2-dev](https://www.modelscope.cn/models/black-forest-labs/FLUX.2-dev)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/FLUX.2-dev.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/FLUX.2-dev.py)|-|-|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/lora/FLUX.2-dev.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_lora/FLUX.2-dev.py)| |[black-forest-labs/FLUX.2-klein-4B](https://www.modelscope.cn/models/black-forest-labs/FLUX.2-klein-4B)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/FLUX.2-klein-4B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/FLUX.2-klein-4B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/FLUX.2-klein-4B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/FLUX.2-klein-4B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/lora/FLUX.2-klein-4B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_lora/FLUX.2-klein-4B.py)| |[black-forest-labs/FLUX.2-klein-9B](https://www.modelscope.cn/models/black-forest-labs/FLUX.2-klein-9B)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/FLUX.2-klein-9B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/FLUX.2-klein-9B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/FLUX.2-klein-9B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/FLUX.2-klein-9B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/lora/FLUX.2-klein-9B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_lora/FLUX.2-klein-9B.py)| |[black-forest-labs/FLUX.2-klein-base-4B](https://www.modelscope.cn/models/black-forest-labs/FLUX.2-klein-base-4B)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/FLUX.2-klein-base-4B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/FLUX.2-klein-base-4B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/FLUX.2-klein-base-4B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/FLUX.2-klein-base-4B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/lora/FLUX.2-klein-base-4B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_lora/FLUX.2-klein-base-4B.py)| |[black-forest-labs/FLUX.2-klein-base-9B](https://www.modelscope.cn/models/black-forest-labs/FLUX.2-klein-base-9B)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/FLUX.2-klein-base-9B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/FLUX.2-klein-base-9B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/FLUX.2-klein-base-9B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/FLUX.2-klein-base-9B.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/lora/FLUX.2-klein-base-9B.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_lora/FLUX.2-klein-base-9B.py)| |[DiffSynth-Studio/Template-KleinBase4B-Aesthetic](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Aesthetic)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-Aesthetic.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Aesthetic.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-Aesthetic.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-Aesthetic.py)|-|-| |[DiffSynth-Studio/Template-KleinBase4B-Brightness](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Brightness)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-Brightness.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Brightness.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-Brightness.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-Brightness.py)|-|-| |[DiffSynth-Studio/Template-KleinBase4B-Age](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Age)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-Age.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Age.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-Age.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-Age.py)|-|-| |[DiffSynth-Studio/Template-KleinBase4B-ControlNet](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-ControlNet)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-ControlNet.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-ControlNet.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-ControlNet.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-ControlNet.py)|-|-| |[DiffSynth-Studio/Template-KleinBase4B-Edit](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Edit)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-Edit.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Edit.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-Edit.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-Edit.py)|-|-| |[DiffSynth-Studio/Template-KleinBase4B-Inpaint](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Inpaint)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-Inpaint.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Inpaint.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-Inpaint.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-Inpaint.py)|-|-| |[DiffSynth-Studio/Template-KleinBase4B-PandaMeme](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-PandaMeme)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-PandaMeme.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-PandaMeme.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-PandaMeme.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-PandaMeme.py)|-|-| |[DiffSynth-Studio/Template-KleinBase4B-Sharpness](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Sharpness)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-Sharpness.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Sharpness.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-Sharpness.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-Sharpness.py)|-|-| |[DiffSynth-Studio/Template-KleinBase4B-SoftRGB](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-SoftRGB)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-SoftRGB.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-SoftRGB.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-SoftRGB.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-SoftRGB.py)|-|-| |[DiffSynth-Studio/Template-KleinBase4B-Upscaler](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-Upscaler)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-Upscaler.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-Upscaler.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-Upscaler.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-Upscaler.py)|-|-| |[DiffSynth-Studio/Template-KleinBase4B-ContentRef](https://www.modelscope.cn/models/DiffSynth-Studio/Template-KleinBase4B-ContentRef)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/Template-KleinBase4B-ContentRef.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/Template-KleinBase4B-ContentRef.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/Template-KleinBase4B-ContentRef.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/Template-KleinBase4B-ContentRef.py)|-|-| |[DiffSynth-Studio/KleinBase4B-i2L-v2](https://www.modelscope.cn/models/DiffSynth-Studio/KleinBase4B-i2L-v2)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference/KleinBase4B-i2L-v2.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_inference_low_vram/KleinBase4B-i2L-v2.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/full/KleinBase4B-i2L-v2.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/validate_full/KleinBase4B-i2L-v2.py)|-|-| Special Training Scripts: * Differential LoRA Training: [doc](../Training/Differential_LoRA.md) * FP8 Precision Training: [doc](../Training/FP8_Precision.md) * Two-stage Split Training: [doc](../Training/Split_Training.md) * End-to-end Direct Distillation: [doc](../Training/Direct_Distill.md) ## Model Inference Models are loaded via `Flux2ImagePipeline.from_pretrained`, see [Loading Models](../Pipeline_Usage/Model_Inference.md#loading-models). Input parameters for `Flux2ImagePipeline` inference include: * `prompt`: Prompt describing the content appearing in the image. * `negative_prompt`: Negative prompt describing content that should not appear in the image, default value is `""`. * `cfg_scale`: Classifier-free guidance parameter, default value is 1. When set to a value greater than 1, CFG is enabled. * `height`: Image height, must be a multiple of 16. * `width`: Image width, must be a multiple of 16. * `seed`: Random seed. Default is `None`, meaning completely random. * `rand_device`: Computing device for generating random Gaussian noise matrix, default is `"cpu"`. When set to `cuda`, different GPUs will produce different generation results. * `num_inference_steps`: Number of inference steps, default value is 30. * `embedded_guidance`: Embedded guidance parameter, default value is 3.5. * `t5_sequence_length`: Sequence length of the T5 text encoder, default is 512. * `tiled`: Whether to enable VAE tiling inference, default is `False`. Setting to `True` can significantly reduce VRAM usage during VAE encoding/decoding stages, producing slight errors and slightly longer inference time. * `tile_size`: Tile size during VAE encoding/decoding stages, default is 128, only effective when `tiled=True`. * `tile_stride`: Tile stride during VAE encoding/decoding stages, default is 64, only effective when `tiled=True`, must be less than or equal to `tile_size`. * `progress_bar_cmd`: Progress bar, default is `tqdm.tqdm`. Can be disabled by setting to `lambda x:x`. If VRAM is insufficient, please enable [VRAM Management](../Pipeline_Usage/VRAM_management.md). We provide recommended low VRAM configurations for each model in the example code, see the table in the "Model Overview" section above. ## Model Training FLUX.2 series models are uniformly trained through [`examples/flux2/model_training/train.py`](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/flux2/model_training/train.py), and the script parameters include: * General Training Parameters * Dataset Basic Configuration * `--dataset_base_path`: Root directory of the dataset. * `--dataset_metadata_path`: Metadata file path of the dataset. * `--dataset_repeat`: Number of times the dataset is repeated in each epoch. * `--dataset_num_workers`: Number of processes for each DataLoader. * `--data_file_keys`: Field names to be loaded from metadata, usually image or video file paths, separated by `,`. * Model Loading Configuration * `--model_paths`: Paths of models to be loaded. JSON format. * `--model_id_with_origin_paths`: Model IDs with original paths, e.g., `"black-forest-labs/FLUX.2-dev:text_encoder/*.safetensors"`. Separated by commas. * `--extra_inputs`: Extra input parameters required by the model Pipeline, e.g., `controlnet_inputs` when training ControlNet models, separated by `,`. * `--fp8_models`: Models loaded in FP8 format, consistent with `--model_paths` or `--model_id_with_origin_paths` format. Currently only supports models whose parameters are not updated by gradients (no gradient backpropagation, or gradients only update their LoRA). * `--quant_options`: Dynamically quantize loaded models. Semicolon-separated entries, each `:[/]`, where `` matches an entry in `--model_paths`/`--model_id_with_origin_paths`, `method` is a registered method (e.g. `bitsandbytes_nf4`), and `exclude_modules` optionally lists layers kept in full precision. * Training Basic Configuration * `--learning_rate`: Learning rate. * `--num_epochs`: Number of epochs. * `--trainable_models`: Trainable models, e.g., `dit`, `vae`, `text_encoder`. * `--find_unused_parameters`: Whether there are unused parameters in DDP training. Some models contain redundant parameters that do not participate in gradient calculation, and this setting needs to be enabled to avoid errors in multi-GPU training. * `--weight_decay`: Weight decay size, see [torch.optim.AdamW](https://docs.pytorch.org/docs/stable/generated/torch.optim.AdamW.html). * `--task`: Training task, default is `sft`. Some models support more training modes, please refer to the documentation of each specific model. * Output Configuration * `--output_path`: Model saving path. * `--remove_prefix_in_ckpt`: Remove prefix in the state dict of the model file. * `--save_steps`: Interval of training steps to save the model. If this parameter is left blank, the model is saved once per epoch. * LoRA Configuration * `--lora_base_model`: Which model to add LoRA to. * `--lora_target_modules`: Which layers to add LoRA to. * `--lora_rank`: Rank of LoRA. * `--lora_checkpoint`: Path of the LoRA checkpoint. If this path is provided, LoRA will be loaded from this checkpoint. * `--preset_lora_path`: Preset LoRA checkpoint path. If this path is provided, this LoRA will be loaded in the form of being merged into the base model. This parameter is used for LoRA differential training. * `--preset_lora_model`: Model that the preset LoRA is merged into, e.g., `dit`. * Gradient Configuration * `--use_gradient_checkpointing`: Whether to enable gradient checkpointing. * `--use_gradient_checkpointing_offload`: Whether to offload gradient checkpointing to memory. * `--gradient_accumulation_steps`: Number of gradient accumulation steps. * Image Width/Height Configuration (Applicable to Image Generation and Video Generation Models) * `--height`: Height of image or video. Leave `height` and `width` blank to enable dynamic resolution. * `--width`: Width of image or video. Leave `height` and `width` blank to enable dynamic resolution. * `--max_pixels`: Maximum pixel area of image or video frames. When dynamic resolution is enabled, images with resolution larger than this value will be downscaled, and images with resolution smaller than this value will remain unchanged. * FLUX.2 Specific Parameters * `--tokenizer_path`: Path of the tokenizer, applicable to text-to-image models, leave blank to automatically download from remote. We have built a sample image dataset for your testing. You can download this dataset with the following command: ```shell modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --local_dir ./data/diffsynth_example_dataset ``` We have written recommended training scripts for each model, please refer to the table in the "Model Overview" section above. For how to write model training scripts, please refer to [Model Training](../Pipeline_Usage/Model_Training.md); for more advanced training algorithms, please refer to [Training Framework Detailed Explanation](https://github.com/modelscope/DiffSynth-Studio/tree/main/docs/en/Training/). --- ### En Model Details Fibo (`docs/en/Model_Details/Fibo.md`) # Fibo 2.0 Fibo 2.0 is an 8B unified image generation and editing model developed by Bria AI. It runs in the latent space of the FLUX.2 VAE, encodes prompts and reference images with Qwen3-VL-4B, and uses a single-stream DiT with 44 blocks. Text-to-image and image editing share one set of weights. Fibo 2.0 Dev is the full, non-distilled base model, trained with classifier-free guidance and intended for fine-tuning. Fibo 2.0 Edit Turbo is a distilled editing-only variant for 4-step, guidance-free inference. ## Installation Before performing model inference and training, please install DiffSynth-Studio first. ```shell git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` For more information on installation, please refer to [Setup Dependencies](../Pipeline_Usage/Setup.md). ## Quick Start Run the following code to quickly load the [briaai/Fibo-2.0-dev](https://www.modelscope.cn/models/briaai/Fibo-2.0-dev) model and perform text-to-image inference. VRAM management is enabled, and the framework will automatically control model parameter loading based on remaining VRAM. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Model Overview |Model ID|Inference|Low VRAM Inference|Full Training|Full Training Validation|LoRA Training|LoRA Training Validation| |-|-|-|-|-|-|-| |[briaai/Fibo-2.0-dev](https://www.modelscope.cn/models/briaai/Fibo-2.0-dev)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/fibo/model_inference/Fibo-2.0-dev.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/fibo/model_inference_low_vram/Fibo-2.0-dev.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/fibo/model_training/full/Fibo-2.0-dev-T2I.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/fibo/model_training/validate_full/Fibo-2.0-dev-T2I.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/fibo/model_training/lora/Fibo-2.0-dev-T2I.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/fibo/model_training/validate_lora/Fibo-2.0-dev-T2I.py)| |[briaai/Fibo-2.0-edit-turbo](https://www.modelscope.cn/models/briaai/Fibo-2.0-edit-turbo)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/fibo/model_inference/Fibo-2.0-edit-turbo.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/fibo/model_inference_low_vram/Fibo-2.0-edit-turbo.py)|-|-|-|-| The training columns above link to the text-to-image scripts. Fibo 2.0 Dev also supports image-editing training, see [Model Training](#model-training). ## Model Inference The model is loaded via `FiboPipeline.from_pretrained`, see [Loading Models](../Pipeline_Usage/Model_Inference.md#loading-models) for details. `from_pretrained` also accepts: * `processor_config`: `ModelConfig` of the Qwen3-VL processor. * `font_config`: `ModelConfig` of the fonts used to draw `text_render` images. It can point to a font directory, a font file, or a list of font files. The input parameters for `FiboPipeline` inference include: * `prompt`: A structured JSON caption, a plain description, or an edit instruction. Edit prompts may also be a JSON caption of the result with an `edit_instruction` field. * `negative_prompt`: Used when `cfg_scale > 1`. Defaults to the null caption Fibo 2.0 was trained with. * `edit_image`: The image or list of reference images to edit. Leave empty for text-to-image. * `mask`: For single-image editing, white marks the region to regenerate. * `text_images`: Images of the text in a caption's `text_render` entries. When omitted, they are drawn from the caption. * `fonts`: Font paths used to draw characters that Pillow's built-in font lacks. * `render_text_images`: Whether to draw text images from the caption, default is `True`. * `height`, `width`: Output size, multiples of 16. When editing and left empty, the trained size for the first image's aspect ratio is used. * `cfg_scale`: Classifier-free guidance scale, default is 5.0. * `num_inference_steps`: Number of inference steps, default is 30. * `denoising_strength`: Denoising strength, range is 0~1, default is 1. Lower values start from a lower noise level. * `seed`, `rand_device`: Random seed and the device used to generate noise. * `progress_bar_cmd`: Progress bar, default is `tqdm.tqdm`. Set to `lambda x:x` to disable it. If VRAM is insufficient, please enable [VRAM Management](../Pipeline_Usage/VRAM_management.md). We provide recommended low VRAM configurations for each model in the example code, see the table in the "Model Overview" section above. ### Image Editing `edit_image` accepts one image, or a list of up to 5 images for multi-reference editing. The example below uses the distilled `Fibo-2.0-edit-turbo` model: ```python from PIL import Image from diffsynth.pipelines.fibo import FiboPipeline, ModelConfig pipe = FiboPipeline.from_pretrained( torch_dtype=torch.bfloat16, device="cuda", model_configs=[ ModelConfig(model_id="briaai/Fibo-2.0-edit-turbo", origin_file_pattern="transformer/*.safetensors"), ModelConfig(model_id="briaai/Fibo-2.0-edit-turbo", origin_file_pattern="text_encoder/model.safetensors"), ModelConfig(model_id="briaai/Fibo-2.0-edit-turbo", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"), ], processor_config=ModelConfig(model_id="briaai/Fibo-2.0-edit-turbo", origin_file_pattern="processor/"), font_config=ModelConfig(model_id="briaai/Fibo-2.0-edit-turbo", origin_file_pattern="fonts/"), ) source = Image.open("photo.jpg").convert("RGB") image = pipe(prompt="replace the jacket with a red leather one", edit_image=source, seed=42, num_inference_steps=4, cfg_scale=1.0) image.save("edited.jpg") ``` `Fibo-2.0-dev` uses 50 steps with `cfg_scale=5.0`; `Fibo-2.0-edit-turbo` uses 4 steps with `cfg_scale=1.0`. ### Masked Editing For single-image editing, pass `mask` together with `edit_image` (white marks the region to regenerate): ```python mask = Image.open("mask.png").convert("L") # white: regenerate, black: keep image = pipe( prompt="replace the suitcase with a vintage brown leather one", edit_image=source, mask=mask, seed=42, num_inference_steps=4, cfg_scale=1.0, ) ``` ### Text Rendering Fibo renders text from a caption's `text_render` entries. Each entry's text is drawn as a small black-on-white image and shown to Qwen3-VL together with the prompt: ```python caption = { "short_description": "A young woman holding a white sign that reads \"ModelScope\".", "objects": [{"description": "a white sign with bold black text", "location": "center"}], "style_medium": "photograph", "text_render": [{"text": "ModelScope"}], } image = pipe( prompt=json.dumps(caption, separators=(",", ":"), ensure_ascii=False), height=1024, width=1024, seed=42, num_inference_steps=50, cfg_scale=5.0, ) ``` `font_config` loads the fonts shipped with the model. To render characters that those fonts lack, pass additional font paths through `fonts`: ```python image = pipe(prompt=prompt, fonts=["/path/to/a/cjk-font.ttf"], ...) ``` Characters not covered by any font are drawn with Pillow's built-in font. Text rendering accuracy is model-dependent, especially for non-Latin scripts. ## Model Training Fibo 2.0 Dev is trained via [`examples/fibo/model_training/train.py`](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/fibo/model_training/train.py). The script parameters include: * General Training Parameters * Dataset Basic Configuration * `--dataset_base_path`: Root directory of the dataset. * `--dataset_metadata_path`: Metadata file path of the dataset. * `--dataset_repeat`: Number of times the dataset is repeated in each epoch. * `--dataset_num_workers`: Number of processes for each DataLoader. * `--data_file_keys`: Field names to be loaded from metadata, usually image or video file paths, separated by `,`. * Model Loading Configuration * `--model_paths`: Paths of models to be loaded. JSON format. * `--model_id_with_origin_paths`: Model IDs with original paths, e.g., `"briaai/Fibo-2.0-dev:transformer/*.safetensors"`. Separated by commas. * `--extra_inputs`: Extra input parameters required by the model Pipeline, e.g., `edit_image` and `mask` when training the image editing model, separated by `,`. * `--fp8_models`: Models loaded in FP8 format, consistent with `--model_paths` or `--model_id_with_origin_paths` format. Currently only supports models whose parameters are not updated by gradients (no gradient backpropagation, or gradients only update their LoRA). * `--quant_options`: Dynamically quantize loaded models. Semicolon-separated entries, each `:[/]`, where `` matches an entry in `--model_paths`/`--model_id_with_origin_paths`, `method` is a registered method (e.g. `bitsandbytes_nf4`), and `exclude_modules` optionally lists layers kept in full precision. * Training Basic Configuration * `--learning_rate`: Learning rate. * `--num_epochs`: Number of epochs. * `--trainable_models`: Trainable models, e.g., `dit`, `vae`, `text_encoder`. * `--find_unused_parameters`: Whether there are unused parameters in DDP training. Some models contain redundant parameters that do not participate in gradient calculation, and this setting needs to be enabled to avoid errors in multi-GPU training. * `--weight_decay`: Weight decay size, see [torch.optim.AdamW](https://docs.pytorch.org/docs/stable/generated/torch.optim.AdamW.html). * `--task`: Training task, default is `sft`. Some models support more training modes, please refer to the documentation of each specific model. * Output Configuration * `--output_path`: Model saving path. * `--remove_prefix_in_ckpt`: Remove prefix in the state dict of the model file. * `--save_steps`: Interval of training steps to save the model. If this parameter is left blank, the model is saved once per epoch. * LoRA Configuration * `--lora_base_model`: Which model to add LoRA to. * `--lora_target_modules`: Which layers to add LoRA to. * `--lora_rank`: Rank of LoRA. * `--lora_checkpoint`: Path of the LoRA checkpoint. If this path is provided, LoRA will be loaded from this checkpoint. * `--preset_lora_path`: Preset LoRA checkpoint path. If this path is provided, this LoRA will be loaded in the form of being merged into the base model. This parameter is used for LoRA differential training. * `--preset_lora_model`: Model that the preset LoRA is merged into, e.g., `dit`. * Gradient Configuration * `--use_gradient_checkpointing`: Whether to enable gradient checkpointing. * `--use_gradient_checkpointing_offload`: Whether to offload gradient checkpointing to memory. * `--gradient_accumulation_steps`: Number of gradient accumulation steps. * Image Width/Height Configuration (Applicable to Image Generation and Video Generation Models) * `--height`: Height of image or video. Leave `height` and `width` blank to enable dynamic resolution. * `--width`: Width of image or video. Leave `height` and `width` blank to enable dynamic resolution. * `--max_pixels`: Maximum pixel area of image or video frames. When dynamic resolution is enabled, images with resolution larger than this value will be downscaled, and images with resolution smaller than this value will remain unchanged. * Fibo 2.0 Specific Parameters * `--processor_path`: Path or `model_id:origin_file_pattern` of the Qwen3-VL processor, leave blank to automatically download from remote. * `--font_path`: Path or `model_id:origin_file_pattern` of the fonts used for text rendering, leave blank to automatically download from remote. * `--initialize_model_on_cpu`: Whether to initialize the model on CPU. We have built sample datasets, including a text-to-image dataset and an image-editing dataset. You can download them with the following command: ```shell modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "fibo/*" --local_dir ./data/diffsynth_example_dataset ``` * `data/diffsynth_example_dataset/fibo/Fibo-2.0-dev`: text-to-image, with `metadata.csv` containing `image` and `prompt`. * `data/diffsynth_example_dataset/fibo/Fibo-2.0-dev-Edit`: image editing, with `metadata.json` containing `image`, `prompt`, `edit_image`, and an optional `mask`. Fibo 2.0 Dev supports both text-to-image and image-editing training. Training reuses the inference Pipeline: the VAE and text encoder are frozen, and the DiT is trained with flow matching. Full training uses `--trainable_models "dit"`, while LoRA training uses `--lora_base_model "dit"`. For image editing, `edit_image` and `mask` are loaded with `UnifiedDataset` and injected through `--extra_inputs`. | Mode | Full Training | LoRA Training | Full Validation | LoRA Validation | |-|-|-|-|-| | Text-to-image | [script](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/fibo/model_training/full/Fibo-2.0-dev-T2I.sh) | [script](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/fibo/model_training/lora/Fibo-2.0-dev-T2I.sh) | [script](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/fibo/model_training/validate_full/Fibo-2.0-dev-T2I.py) | [script](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/fibo/model_training/validate_lora/Fibo-2.0-dev-T2I.py) | | Image editing | [script](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/fibo/model_training/full/Fibo-2.0-dev-Edit.sh) | [script](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/fibo/model_training/lora/Fibo-2.0-dev-Edit.sh) | [script](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/fibo/model_training/validate_full/Fibo-2.0-dev-Edit.py) | [script](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/fibo/model_training/validate_lora/Fibo-2.0-dev-Edit.py) | The full training scripts use `--config_file examples/fibo/model_training/full/accelerate_config.yaml`; the LoRA scripts run on a single GPU by default. `mask` is optional per sample, and rows without `mask` are trained as ordinary editing samples. Fibo 2.0 Edit Turbo is not intended for fine-tuning; train on Fibo 2.0 Dev instead. We have written recommended training scripts for each model, please refer to the tables above. For how to write model training scripts, please refer to [Model Training](../Pipeline_Usage/Model_Training.md); for more advanced training algorithms, please refer to [Training Framework Detailed Explanation](https://github.com/modelscope/DiffSynth-Studio/tree/main/docs/en/Training/). ## License Fibo 2.0 is released under CC BY-NC 4.0 for non-commercial use. Commercial use requires a separate agreement with Bria AI. --- ### En Model Details HiDream O1 Image (`docs/en/Model_Details/HiDream-O1-Image.md`) # HiDream-O1-Image HiDream-O1-Image is an image generation model open-sourced by HiDream.ai, based on the Pixel-Level Unified Transformer (UiT) architecture. This model unifies VAE, DiT, and TextEncoder within a single Qwen3VLModel, performing diffusion denoising directly in pixel patch space without requiring a separate VAE component. ## Installation Before performing model inference and training, please install DiffSynth-Studio first. ```shell git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` For more information on installation, please refer to [Setup Dependencies](../Pipeline_Usage/Setup.md). ## Quick Start Running the following code will quickly load the [HiDream-ai/HiDream-O1-Image](https://modelscope.cn/HiDream-ai/HiDream-O1-Image) model for inference. VRAM management is enabled, the framework automatically controls parameter loading based on available VRAM, requiring a minimum of 3GB VRAM. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Model Overview |Model ID|Inference|Low VRAM Inference|Full Training|Full Training Validation|LoRA Training|LoRA Training Validation| |-|-|-|-|-|-|-| |[HiDream-ai/HiDream-O1-Image](https://modelscope.cn/HiDream-ai/HiDream-O1-Image)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/hidream_o1_image/model_inference/HiDream-O1-Image.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/hidream_o1_image/model_inference_low_vram/HiDream-O1-Image.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/hidream_o1_image/model_training/full/HiDream-O1-Image.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/hidream_o1_image/model_training/validate_full/HiDream-O1-Image.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/hidream_o1_image/model_training/lora/HiDream-O1-Image.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/hidream_o1_image/model_training/validate_lora/HiDream-O1-Image.py)| |[HiDream-ai/HiDream-O1-Image-Dev](https://modelscope.cn/HiDream-ai/HiDream-O1-Image-Dev)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/hidream_o1_image/model_inference/HiDream-O1-Image-Dev.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/hidream_o1_image/model_inference_low_vram/HiDream-O1-Image-Dev.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/hidream_o1_image/model_training/full/HiDream-O1-Image-Dev.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/hidream_o1_image/model_training/validate_full/HiDream-O1-Image-Dev.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/hidream_o1_image/model_training/lora/HiDream-O1-Image-Dev.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/hidream_o1_image/model_training/validate_lora/HiDream-O1-Image-Dev.py)| |[DiffSynth-Studio/HidreamO1-i2L-v2](https://www.modelscope.cn/models/DiffSynth-Studio/HidreamO1-i2L-v2)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/hidream_o1_image/model_inference/HidreamO1-i2L-v2.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/hidream_o1_image/model_inference_low_vram/HidreamO1-i2L-v2.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/hidream_o1_image/model_training/full/HidreamO1-i2L-v2.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/hidream_o1_image/model_training/validate_full/HidreamO1-i2L-v2.py)|-|-| ## Model Inference The model is loaded via `HiDreamO1ImagePipeline.from_pretrained`, see [Loading Models](../Pipeline_Usage/Model_Inference.md#loading-models) for details. The input parameters for `HiDreamO1ImagePipeline` inference include: * `prompt`: Text prompt. * `negative_prompt`: Negative prompt, defaults to `" "`. * `cfg_scale`: Classifier-Free Guidance scale, defaults to 4.0. For the Dev model, it is recommended to set to 1.0. * `height`: Output image height, defaults to 2048. * `width`: Output image width, defaults to 2048. * `seed`: Random seed, defaults to random. * `rand_device`: Noise generation device, defaults to `"cpu"`. * `num_inference_steps`: Number of inference steps, defaults to 50 for Full model and 28 for Dev model. * `model_type`: Model type, `"full"` for Full model, `"dev"` for distilled Dev model. * `shift`: Timestep shift parameter affecting sigma computation, defaults to 3.0. * `noise_scale`: Noise scaling factor, defaults to 8.0. For the Dev model, it is recommended to set to 7.5. * `edit_image`: List of reference images for image editing. Defaults to None (text-to-image mode). * `keep_original_aspect`: Whether to preserve the original aspect ratio of reference images, defaults to True. > **VRAM Note**: HiDream-O1-Image has a large parameter count (~8B). When generating 2048x2048 images, it is recommended to enable VRAM management (vram_config) or use the low VRAM inference scripts. ## Model Training Models in the hidream_o1_image series are trained uniformly via `examples/hidream_o1_image/model_training/train.py`. The script parameters include: * General Training Parameters * Dataset Configuration * `--dataset_base_path`: Root directory of the dataset. * `--dataset_metadata_path`: Path to the dataset metadata file. * `--dataset_repeat`: Number of dataset repeats per epoch. * `--dataset_num_workers`: Number of processes per DataLoader. * `--data_file_keys`: Field names to load from metadata, typically paths to image or video files, separated by `,`. * Model Loading Configuration * `--model_paths`: Paths to load models from, in JSON format. * `--model_id_with_origin_paths`: Model IDs with original paths, separated by commas. * `--extra_inputs`: Additional input parameters required by the model Pipeline, separated by `,`. * `--fp8_models`: Models to load in FP8 format, currently only supported for models whose parameters are not updated by gradients. * `--quant_options`: Dynamically quantize loaded models. Semicolon-separated entries, each `:[/]`, where `` matches an entry in `--model_paths`/`--model_id_with_origin_paths`, `method` is a registered method (e.g. `bitsandbytes_nf4`), and `exclude_modules` optionally lists layers kept in full precision. * Basic Training Configuration * `--learning_rate`: Learning rate. * `--num_epochs`: Number of epochs. * `--trainable_models`: Trainable models, e.g., `dit`, `vae`, `text_encoder`. * `--find_unused_parameters`: Whether unused parameters exist in DDP training. * `--weight_decay`: Weight decay magnitude. * `--task`: Training task, defaults to `sft`. * Output Configuration * `--output_path`: Path to save the model. * `--remove_prefix_in_ckpt`: Remove prefix in the model's state dict. * `--save_steps`: Interval in training steps to save the model. * LoRA Configuration * `--lora_base_model`: Which model to add LoRA to. * `--lora_target_modules`: Which layers to add LoRA to. * `--lora_rank`: Rank of LoRA. * `--lora_checkpoint`: Path to LoRA checkpoint. * `--preset_lora_path`: Path to preset LoRA checkpoint for LoRA differential training. * `--preset_lora_model`: Which model to integrate preset LoRA into, e.g., `dit`. * Gradient Configuration * `--use_gradient_checkpointing`: Whether to enable gradient checkpointing. * `--use_gradient_checkpointing_offload`: Whether to offload gradient checkpointing to CPU memory. * `--gradient_accumulation_steps`: Number of gradient accumulation steps. * Resolution Configuration * `--height`: Height of the image/video. Leave empty to enable dynamic resolution. * `--width`: Width of the image/video. Leave empty to enable dynamic resolution. * `--max_pixels`: Maximum pixel area, images larger than this will be scaled down during dynamic resolution. * `--num_frames`: Number of frames for video (video generation models only). * HiDream-O1-Image Specific Parameters * `--processor_config`: Path to the processor configuration file, used for loading AutoProcessor for text tokenization. * `--noise_scale`: Noise scaling factor, defaults to 8.0. * `--initialize_model_on_cpu`: Whether to initialize the model on CPU, which can help reduce peak GPU VRAM usage. ```shell modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --local_dir ./data/diffsynth_example_dataset ``` We provide recommended training scripts for each model, please refer to the table in "Model Overview" above. For guidance on writing model training scripts, see [Model Training](../Pipeline_Usage/Model_Training.md); for more advanced training algorithms, see [Training Framework Overview](https://github.com/modelscope/DiffSynth-Studio/tree/main/docs/en/Training/). --- ### En Model Details Ideogram 4 (`docs/en/Model_Details/Ideogram-4.md`) # Ideogram 4 Ideogram 4 is an image generation model open-sourced by Ideogram. DiffSynth-Studio supports inference, low VRAM inference, full training, and LoRA training for both the FP8 quantized version and the BF16 repackaged version. ## Installation Before performing model inference and training, please install DiffSynth-Studio first. ```shell git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` For more information about installation, please refer to [Install Dependencies](../Pipeline_Usage/Setup.md). ## Quick Start Running the following code will load the [ideogram-ai/ideogram-4-fp8](https://www.modelscope.cn/models/ideogram-ai/ideogram-4-fp8) model for inference. A minimum of 24GB VRAM is required to run. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Model Overview |Model ID|Inference|Low VRAM Inference|Full Training|Full Training Validation|LoRA Training|LoRA Training Validation| |-|-|-|-|-|-|-| |[ideogram-ai/ideogram-4-fp8](https://www.modelscope.cn/models/ideogram-ai/ideogram-4-fp8)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ideogram4/model_inference/ideogram-4-fp8.py)|-|-|-|-|-| |[DiffSynth-Studio/ideogram-4-bf16-repackage](https://www.modelscope.cn/models/DiffSynth-Studio/ideogram-4-bf16-repackage)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ideogram4/model_inference/ideogram-4-bf16-repackage.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ideogram4/model_inference_low_vram/ideogram-4-bf16-repackage.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ideogram4/model_training/full/Ideogram-4-bf16-repackage.sh)|-|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ideogram4/model_training/lora/Ideogram-4-bf16-repackage.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ideogram4/model_training/validate_lora/Ideogram-4-bf16-repackage.py)| ## Model Inference The model is loaded via `Ideogram4Pipeline.from_pretrained`, see [Loading Models](../Pipeline_Usage/Model_Inference.md#loading-models) for details. The input parameters for `Ideogram4Pipeline` inference include: * `prompt`: Prompt describing the content appearing in the image. Ideogram 4 supports structured JSON format prompts, including high-level description, style description, and compositional deconstruction. * `negative_prompt`: Negative prompt describing content that should not appear in the image, default value is `""`. * `cfg_scale`: Classifier-free guidance parameter, default value is 7.0. * `input_image`: Input image for image-to-image generation, used in conjunction with `denoising_strength`. * `denoising_strength`: Denoising strength, range is 0~1, default value is 1. When the value approaches 0, the generated image is similar to the input image; when the value approaches 1, the generated image differs more from the input image. When `input_image` parameter is not provided, do not set this to a non-1 value. * `height`: Image height, must be a multiple of 16, default value is 1024. * `width`: Image width, must be a multiple of 16, default value is 1024. * `seed`: Random seed. Default is `None`, meaning completely random. * `rand_device`: Computing device for generating random Gaussian noise matrix, default is `"cpu"`. * `num_inference_steps`: Number of inference steps, default value is 50. ## Model Training Models in the ideogram4 series are trained uniformly via `examples/ideogram4/model_training/train.py`. The script parameters include: * General Training Parameters * Dataset Configuration * `--dataset_base_path`: Root directory of the dataset. * `--dataset_metadata_path`: Path to the dataset metadata file. * `--dataset_repeat`: Number of dataset repeats per epoch. * `--dataset_num_workers`: Number of processes per DataLoader. * `--data_file_keys`: Field names to load from metadata, typically paths to image or video files, separated by `,`. * Model Loading Configuration * `--model_paths`: Paths to load models from, in JSON format. * `--model_id_with_origin_paths`: Model IDs with original paths, separated by commas. * `--extra_inputs`: Additional input parameters required by the model Pipeline, separated by `,`. * `--fp8_models`: Models to load in FP8 format, currently only supported for models whose parameters are not updated by gradients. * `--quant_options`: Dynamically quantize loaded models. Semicolon-separated entries, each `:[/]`, where `` matches an entry in `--model_paths`/`--model_id_with_origin_paths`, `method` is a registered method (e.g. `bitsandbytes_nf4`), and `exclude_modules` optionally lists layers kept in full precision. * Basic Training Configuration * `--learning_rate`: Learning rate. * `--num_epochs`: Number of epochs. * `--trainable_models`: Trainable models, e.g., `dit`, `vae`, `text_encoder`. * `--find_unused_parameters`: Whether unused parameters exist in DDP training. * `--weight_decay`: Weight decay magnitude. * `--task`: Training task, defaults to `sft`. * Output Configuration * `--output_path`: Path to save the model. * `--remove_prefix_in_ckpt`: Remove prefix in the model's state dict. * `--save_steps`: Interval in training steps to save the model. * LoRA Configuration * `--lora_base_model`: Which model to add LoRA to. * `--lora_target_modules`: Which layers to add LoRA to. * `--lora_rank`: Rank of LoRA. * `--lora_checkpoint`: Path to LoRA checkpoint. * `--preset_lora_path`: Path to preset LoRA checkpoint for LoRA differential training. * `--preset_lora_model`: Which model to integrate preset LoRA into, e.g., `dit`. * Gradient Configuration * `--use_gradient_checkpointing`: Whether to enable gradient checkpointing. * `--use_gradient_checkpointing_offload`: Whether to offload gradient checkpointing to CPU memory. * `--gradient_accumulation_steps`: Number of gradient accumulation steps. * Resolution Configuration * `--height`: Height of the image/video. Leave empty to enable dynamic resolution. * `--width`: Width of the image/video. Leave empty to enable dynamic resolution. * `--max_pixels`: Maximum pixel area, images larger than this will be scaled down during dynamic resolution. * `--num_frames`: Number of frames for video (video generation models only). * Ideogram-4 Specific Parameters * `--tokenizer_path`: Path to tokenizer. Defaults to downloading from `ideogram-ai/ideogram-4-fp8`. ```shell modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --local_dir ./data/diffsynth_example_dataset ``` We provide recommended training scripts for each model, please refer to the table in "Model Overview" above. For guidance on writing model training scripts, see [Model Training](../Pipeline_Usage/Model_Training.md); for more advanced training algorithms, see [Training Framework Overview](https://github.com/modelscope/DiffSynth-Studio/tree/main/docs/en/Training/). --- ### En Model Details Image Quality Metrics (`docs/en/Model_Details/Image-Quality-Metrics.md`) # Image Quality Evaluation Metrics DiffSynth-Studio provides a suite of image quality evaluation metrics and reward models in `diffsynth.metrics` to assess text alignment, aesthetic quality, human preference, and image distribution quality of generated images. Example code for these metrics can be found in [`examples/image_quality_metric/`](../../../examples/image_quality_metric/). ## Installation Before using this project for model inference and training, please install DiffSynth-Studio first. ```shell git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` For more information about installation, please refer to [Install Dependencies](../Pipeline_Usage/Setup.md). ## Quick Start Run the following code to quickly load PickScore and score an image against a prompt. The default models will be downloaded from ModelScope to `./models`. ```python from diffsynth.metrics import PickScoreMetric, ModelConfig from modelscope import dataset_snapshot_download from PIL import Image dataset_snapshot_download( "DiffSynth-Studio/diffsynth_example_dataset", allow_file_pattern="flux/FLUX.1-dev/*", local_dir="./data/diffsynth_example_dataset", ) image = Image.open("data/diffsynth_example_dataset/flux/FLUX.1-dev/1.jpg").convert("RGB") prompt = "a dog" metric = PickScoreMetric.from_pretrained( model_config=ModelConfig(model_id="DiffSynth-Studio/ImageMetrics", origin_file_pattern="PickScore/model.safetensors"), device="cuda" ) score = metric.compute(prompt, image)[0] print(f"PickScore score:: {score:.3f}") ``` ## Metrics Overview | Metric | Input | Output | Example Code | | --- | --- | --- | --- | | PickScore | prompt + PIL Image | Preference Score | [code](../../../examples/image_quality_metric/pickscore.py) | | ImageReward | prompt + PIL Image | Preference Score | [code](../../../examples/image_quality_metric/image_reward.py) | | HPSv2 | prompt + PIL Image | Preference Score | [code](../../../examples/image_quality_metric/hpsv2.py) | | HPSv3 | prompt + PIL Image | Preference Score | [code](../../../examples/image_quality_metric/hpsv3.py) | | CLIP Score | prompt + PIL Image | Text-Image Similarity | [code](../../../examples/image_quality_metric/clipscore.py) | | UnifiedReward 2.0 | prompt + PIL Image | multi-dimension scores | [code](../../../examples/image_quality_metric/unified_reward_2.py) | | Qwen-Image-Bench | prompt + PIL Image | Overall score and multi-level dimension scores | [code](../../../examples/image_quality_metric/qwen_image_bench.py) | | UnifiedReward Edit | editing instruction + source image + edited image | Image editing quality score | [code](../../../examples/image_quality_metric/unified_reward_edit.py) | | Aesthetic | PIL Image | Aesthetic Score | [code](../../../examples/image_quality_metric/aesthetic.py) | | FID | reference image directory + generated image directory | Distribution Distance | [code](../../../examples/image_quality_metric/fid.py) | ### Text-Image Alignment and Preference Evaluation Applicable metrics: **PickScore**, **ImageReward**, **HPSv2**, **HPSv3**, **CLIP Score**, **UnifiedReward 2.0**, **Qwen-Image-Bench** These models are used to evaluate whether an image follows the prompt and aligns with human visual preferences. They must receive both the `prompt` and the `image` simultaneously. **Basic Scoring** ```python score = metric.compute(prompt, image)[0] ``` **Batch Scoring** If you need to evaluate multiple images, you can directly pass a list: ```python scores = metric.compute("a cute cat", [image1, image2, image3]) scores = metric.compute(["a cat", "a dog"], [image_cat, image_dog]) ``` When prompt is a single string, the same prompt will be applied to every image. When prompt is a list of strings, the number of prompts must exactly match the number of images. ### Multi-Dimensional Image Quality Evaluation Applicable metrics: **UnifiedReward 2.0**, **Qwen-Image-Bench** These metrics also receive a `prompt` and an `image`, but in addition to the primary score, `evaluate()` returns more detailed evaluation dimensions. They are useful when you need to analyze text-image alignment, visual coherence, style, or multi-level quality dimensions. **Qwen-Image-Bench** ```python from diffsynth.metrics import ModelConfig, QwenImageBenchMetric metric = QwenImageBenchMetric.from_pretrained( model_config=ModelConfig( model_id="Qwen/Qwen-Image-Bench", origin_file_pattern="model-*.safetensors", ), processor_config=ModelConfig( model_id="Qwen/Qwen-Image-Bench", origin_file_pattern="", ), device="cuda", ) details = metric.evaluate(prompt, image)[0] score = details["total_score"] print(details["level1_scores"]) print(details["level2_scores"]) ``` If you only need the primary score, you can also call `metric.compute(prompt, image)`. ### Image Editing Quality Evaluation Applicable metric: **UnifiedReward Edit** UnifiedReward Edit evaluates whether an edited image follows the editing instruction and whether it is over-edited. The input usually includes an editing instruction, a source image, and edited image candidates. It supports three tasks: * `edit_pointwise_score`: scores a single edited result with `[source_image, edited_image]`. * `edit_pairwise_rank`: compares two edited results and returns the winner with `[source_image, edited_image_1, edited_image_2]`. * `edit_pairwise_score`: returns separate scores for two edited results with `[source_image, edited_image_1, edited_image_2]`. ```python from diffsynth.metrics import ModelConfig, UnifiedRewardEditMetric metric = UnifiedRewardEditMetric.from_pretrained( model_config=ModelConfig( model_id="DiffSynth-Studio/ImageMetrics", origin_file_pattern="UnifiedReward-Edit-qwen3vl-8b/model-*.safetensors", ), processor_config=ModelConfig( model_id="DiffSynth-Studio/ImageMetrics", origin_file_pattern="UnifiedReward-Edit-qwen3vl-8b/", ), device="cuda", ) details = metric.evaluate( instruction, [source_image, edited_image], task="edit_pointwise_score", )[0] print(details["score"], details["editing_success"], details["overediting"]) ``` ### Pure Image Aesthetics Evaluation Applicable metric: **Aesthetic** This model solely evaluates aesthetic features such as the composition, color, and clarity of the image itself. It does not require a prompt. ```python from diffsynth.metrics import AestheticMetric metric = AestheticMetric.from_pretrained(device="cuda") score = metric.compute(image)[0] ``` ### Dataset Distribution Evaluation Applicable metric: **FID** (Fréchet Inception Distance) FID does not score individual images; instead, it compares the overall feature distribution distance between a real reference image set and a generated image set. A lower score indicates that the generated distribution is closer to the real distribution. ```python from diffsynth.metrics import FIDMetric reference_dir = "path/to/real_reference_images" generated_dir = "path/to/model_generated_images" metric = FIDMetric.from_pretrained(device="cuda", batch_size=16) fid_score = metric.compute(reference_dir, generated_dir) print(f"FID: {fid_score:.3f}") ``` The baseline for FID is not fixed or unique. For general image generation, COCO Validation is commonly used; for specific domains (such as medical images or e-commerce products), a `reference_dir` composed of real data from that specific domain should be provided. ## Important Notes * The scores from PickScore, ImageReward, HPSv2, HPSv3, CLIPScore, UnifiedReward 2.0, Qwen-Image-Bench, UnifiedReward Edit, and Aesthetic are suitable for relative comparison within the same metric. It is not recommended to directly compare the numerical values across different metrics. * HPSv3, UnifiedReward 2.0, UnifiedReward Edit, and Qwen-Image-Bench are based on multimodal large models, requiring significantly more VRAM than CLIP-based metrics. * FID is sensitive to the choice of reference, the reference sample size, and the generated sample size. --- ### En Model Details JoyAI Image (`docs/en/Model_Details/JoyAI-Image.md`) # JoyAI-Image JoyAI-Image is a unified multi-modal foundation model open-sourced by JD.com, supporting image understanding, text-to-image generation, and instruction-guided image editing. ## Installation Before performing model inference and training, please install DiffSynth-Studio first. ```shell git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` For more information on installation, please refer to [Setup Dependencies](../Pipeline_Usage/Setup.md). ## Quick Start Running the following code will load the [jd-opensource/JoyAI-Image-Edit](https://modelscope.cn/models/jd-opensource/JoyAI-Image-Edit) model for inference. VRAM management is enabled, the framework automatically controls parameter loading based on available VRAM, requiring a minimum of 4GB VRAM. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Model Overview |Model ID|Inference|Low VRAM Inference|Full Training|Full Training Validation|LoRA Training|LoRA Training Validation| |-|-|-|-|-|-|-| |[jd-opensource/JoyAI-Image-Edit](https://modelscope.cn/models/jd-opensource/JoyAI-Image-Edit)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/joyai_image/model_inference/JoyAI-Image-Edit.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/joyai_image/model_inference_low_vram/JoyAI-Image-Edit.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/joyai_image/model_training/full/JoyAI-Image-Edit.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/joyai_image/model_training/validate_full/JoyAI-Image-Edit.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/joyai_image/model_training/lora/JoyAI-Image-Edit.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/joyai_image/model_training/validate_lora/JoyAI-Image-Edit.py)| ## Model Inference The model is loaded via `JoyAIImagePipeline.from_pretrained`, see [Loading Models](../Pipeline_Usage/Model_Inference.md#loading-models) for details. The input parameters for `JoyAIImagePipeline` inference include: * `prompt`: Text prompt describing the desired image editing effect. * `negative_prompt`: Negative prompt specifying what should not appear in the result, defaults to empty string. * `cfg_scale`: Classifier-free guidance scale factor, defaults to 5.0. Higher values make the output more closely follow the prompt. * `edit_image`: Image to be edited. * `denoising_strength`: Denoising strength controlling how much the input image is repainted, defaults to 1.0. * `height`: Height of the output image, defaults to 1024. Must be divisible by 16. * `width`: Width of the output image, defaults to 1024. Must be divisible by 16. * `seed`: Random seed for reproducibility. Set to `None` for random seed. * `max_sequence_length`: Maximum sequence length for the text encoder, defaults to 4096. * `num_inference_steps`: Number of inference steps, defaults to 30. More steps typically yield better quality. * `tiled`: Whether to enable tiling for reduced VRAM usage, defaults to False. * `tile_size`: Tile size, defaults to (30, 52). * `tile_stride`: Tile stride, defaults to (15, 26). * `shift`: Shift parameter for the scheduler, controlling the Flow Match scheduling curve, defaults to 4.0. * `progress_bar_cmd`: Progress bar display mode, defaults to tqdm. ## Model Training Models in the joyai_image series are trained uniformly via `examples/joyai_image/model_training/train.py`. The script parameters include: * General Training Parameters * Dataset Configuration * `--dataset_base_path`: Root directory of the dataset. * `--dataset_metadata_path`: Path to the dataset metadata file. * `--dataset_repeat`: Number of dataset repeats per epoch. * `--dataset_num_workers`: Number of processes per DataLoader. * `--data_file_keys`: Field names to load from metadata, typically paths to image or video files, separated by `,`. * Model Loading Configuration * `--model_paths`: Paths to load models from, in JSON format. * `--model_id_with_origin_paths`: Model IDs with original paths, separated by commas. * `--extra_inputs`: Additional input parameters required by the model Pipeline, separated by `,`. * `--fp8_models`: Models to load in FP8 format, currently only supported for models whose parameters are not updated by gradients. * `--quant_options`: Dynamically quantize loaded models. Semicolon-separated entries, each `:[/]`, where `` matches an entry in `--model_paths`/`--model_id_with_origin_paths`, `method` is a registered method (e.g. `bitsandbytes_nf4`), and `exclude_modules` optionally lists layers kept in full precision. * Basic Training Configuration * `--learning_rate`: Learning rate. * `--num_epochs`: Number of epochs. * `--trainable_models`: Trainable models, e.g., `dit`, `vae`, `text_encoder`. * `--find_unused_parameters`: Whether unused parameters exist in DDP training. * `--weight_decay`: Weight decay magnitude. * `--task`: Training task, defaults to `sft`. * Output Configuration * `--output_path`: Path to save the model. * `--remove_prefix_in_ckpt`: Remove prefix in the model's state dict. * `--save_steps`: Interval in training steps to save the model. * LoRA Configuration * `--lora_base_model`: Which model to add LoRA to. * `--lora_target_modules`: Which layers to add LoRA to. * `--lora_rank`: Rank of LoRA. * `--lora_checkpoint`: Path to LoRA checkpoint. * `--preset_lora_path`: Path to preset LoRA checkpoint for LoRA differential training. * `--preset_lora_model`: Which model to integrate preset LoRA into, e.g., `dit`. * Gradient Configuration * `--use_gradient_checkpointing`: Whether to enable gradient checkpointing. * `--use_gradient_checkpointing_offload`: Whether to offload gradient checkpointing to CPU memory. * `--gradient_accumulation_steps`: Number of gradient accumulation steps. * Resolution Configuration * `--height`: Height of the image/video. Leave empty to enable dynamic resolution. * `--width`: Width of the image/video. Leave empty to enable dynamic resolution. * `--max_pixels`: Maximum pixel area, images larger than this will be scaled down during dynamic resolution. * `--num_frames`: Number of frames for video (video generation models only). * JoyAI-Image Specific Parameters * `--processor_path`: Path to the processor for processing text and image encoder inputs. * `--initialize_model_on_cpu`: Whether to initialize models on CPU. By default, models are initialized on the accelerator device. ```shell modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --local_dir ./data/diffsynth_example_dataset ``` We provide recommended training scripts for each model, please refer to the table in "Model Overview" above. For guidance on writing model training scripts, see [Model Training](../Pipeline_Usage/Model_Training.md); for more advanced training algorithms, see [Training Framework Overview](https://github.com/modelscope/DiffSynth-Studio/tree/main/docs/en/Training/). --- ### En Model Details Krea 2 (`docs/en/Model_Details/Krea-2.md`) # Krea-2 Krea-2 is an image generation model developed by the Krea team. ## Installation Before performing model inference and training, please install DiffSynth-Studio first. ```shell git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` For more information on installation, please refer to [Setup Dependencies](../Pipeline_Usage/Setup.md). ## Quick Start Running the following code will load the [krea/Krea-2-Raw](https://www.modelscope.cn/models/krea/Krea-2-Raw) model for inference. VRAM management is enabled, the framework automatically controls parameter loading based on available VRAM, requiring a minimum of 24GB VRAM. ```python from diffsynth.pipelines.krea2 import Krea2Pipeline, ModelConfig import torch vram_config = { "offload_dtype": "disk", "offload_device": "disk", "onload_dtype": torch.float8_e4m3fn, "onload_device": "cpu", "preparing_dtype": torch.float8_e4m3fn, "preparing_device": "cuda", "computation_dtype": torch.bfloat16, "computation_device": "cuda", } pipe = Krea2Pipeline.from_pretrained( torch_dtype=torch.bfloat16, device="cuda", model_configs=[ ModelConfig(model_id="krea/Krea-2-Raw", origin_file_pattern="raw.safetensors", **vram_config), ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern="*.safetensors", **vram_config), ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="vae/diffusion_pytorch_model.safetensors", **vram_config), ], tokenizer_config=ModelConfig(model_id="Qwen/Qwen3-VL-4B-Instruct", origin_file_pattern=""), vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 1, ) prompt = "A cat standing on a stone." image = pipe(prompt, seed=0, num_inference_steps=52, cfg_scale=4.5) image.save("image.jpg") ``` ## Model Overview |Model ID|Inference|Low VRAM Inference|Full Training|Full Training Validation|LoRA Training|LoRA Training Validation| |-|-|-|-|-|-|-| |[krea/Krea-2-Raw](https://www.modelscope.cn/models/krea/Krea-2-Raw)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/krea2/model_inference/Krea-2-Raw.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/krea2/model_inference_low_vram/Krea-2-Raw.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/krea2/model_training/full/Krea-2-Raw.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/krea2/model_training/validate_full/Krea-2-Raw.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/krea2/model_training/lora/Krea-2-Raw.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/krea2/model_training/validate_lora/Krea-2-Raw.py)| |[krea/Krea-2-Turbo](https://www.modelscope.cn/models/krea/Krea-2-Turbo)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/krea2/model_inference/Krea-2-Turbo.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/krea2/model_inference_low_vram/Krea-2-Turbo.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/krea2/model_training/full/Krea-2-Turbo.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/krea2/model_training/validate_full/Krea-2-Turbo.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/krea2/model_training/lora/Krea-2-Turbo.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/krea2/model_training/validate_lora/Krea-2-Turbo.py)| ## Model Inference The model is loaded via `Krea2Pipeline.from_pretrained`, see [Loading Models](../Pipeline_Usage/Model_Inference.md#loading-models) for details. The input parameters for `Krea2Pipeline` inference include: * `prompt`: Prompt describing the content of the image to generate, default value is `""`. * `negative_prompt`: Negative prompt describing content that should not appear in the image, default value is `""`. * `cfg_scale`: Classifier-free guidance parameter, default value is 3.5. * `height`: Image height, must be a multiple of 16, default value is 1024. * `width`: Image width, must be a multiple of 16, default value is 1024. * `seed`: Random seed, default is `None`, meaning completely random. * `rand_device`: Computing device for generating random Gaussian noise matrix, default is `"cpu"`. * `num_inference_steps`: Number of inference steps, default value is 52. * `mu`: Timestep dynamic shift parameter, default is `None`. * `progress_bar_cmd`: Progress bar, default is `tqdm.tqdm`. Can be disabled by setting to `lambda x:x`. If VRAM is insufficient, please enable [VRAM Management](../Pipeline_Usage/VRAM_management.md). We provide recommended low VRAM configurations for each model in the example code, see the table in the "Model Overview" section above. ## Model Training Models in the Krea-2 series are trained uniformly via [`examples/krea2/model_training/train.py`](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/krea2/model_training/train.py). The script parameters include: * General Training Parameters * Dataset Configuration * `--dataset_base_path`: Root directory of the dataset. * `--dataset_metadata_path`: Path to the dataset metadata file. * `--dataset_repeat`: Number of dataset repeats per epoch. * `--dataset_num_workers`: Number of processes per DataLoader. * `--data_file_keys`: Field names to load from metadata, typically paths to image or video files, separated by `,`. * Model Loading Configuration * `--model_paths`: Paths to load models from, in JSON format. * `--model_id_with_origin_paths`: Model IDs with original paths, separated by commas. * `--extra_inputs`: Additional input parameters required by the model Pipeline, separated by `,`. * `--fp8_models`: Models to load in FP8 format, currently only supported for models whose parameters are not updated by gradients. * `--quant_options`: Dynamically quantize loaded models. Semicolon-separated entries, each `:[/]`, where `` matches an entry in `--model_paths`/`--model_id_with_origin_paths`, `method` is a registered method (e.g. `bitsandbytes_nf4`), and `exclude_modules` optionally lists layers kept in full precision. * Basic Training Configuration * `--learning_rate`: Learning rate. * `--num_epochs`: Number of epochs. * `--trainable_models`: Trainable models, e.g., `dit`, `vae`, `text_encoder`. * `--find_unused_parameters`: Whether unused parameters exist in DDP training. * `--weight_decay`: Weight decay magnitude. * `--task`: Training task, defaults to `sft`. * Output Configuration * `--output_path`: Path to save the model. * `--remove_prefix_in_ckpt`: Remove prefix in the model's state dict. * `--save_steps`: Interval in training steps to save the model. * LoRA Configuration * `--lora_base_model`: Which model to add LoRA to. * `--lora_target_modules`: Which layers to add LoRA to. * `--lora_rank`: Rank of LoRA. * `--lora_checkpoint`: Path to LoRA checkpoint. * `--preset_lora_path`: Path to preset LoRA checkpoint for LoRA differential training. * `--preset_lora_model`: Which model to integrate preset LoRA into, e.g., `dit`. * Gradient Configuration * `--use_gradient_checkpointing`: Whether to enable gradient checkpointing. * `--use_gradient_checkpointing_offload`: Whether to offload gradient checkpointing to CPU memory. * `--gradient_accumulation_steps`: Number of gradient accumulation steps. * Resolution Configuration * `--height`: Height of the image. Leave empty to enable dynamic resolution. * `--width`: Width of the image. Leave empty to enable dynamic resolution. * `--max_pixels`: Maximum pixel area, images larger than this will be scaled down during dynamic resolution. * Krea-2 Specific Parameters * `--tokenizer_path`: Path to the tokenizer, leave blank to automatically download from remote. * `--initialize_model_on_cpu`: Whether to initialize models on CPU. * `--align_to_opensource_format`: Whether to align the LoRA format to the opensource format, useful for compatibility with other frameworks. We have built a sample dataset for your testing. You can download it with the following command: ```shell modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "krea2/*" --local_dir ./data/diffsynth_example_dataset ``` We provide recommended training scripts for each model, please refer to the table in "Model Overview" above. For guidance on writing model training scripts, see [Model Training](../Pipeline_Usage/Model_Training.md); for more advanced training algorithms, see [Training Framework Overview](https://github.com/modelscope/DiffSynth-Studio/tree/main/docs/en/Training/). ## License > **⚠️ Notice**: **Krea-2** weights (Raw and Turbo) are released under the [Krea 2 Community License](https://www.krea.ai/krea-2-licensing), **not** the Apache 2.0 license that governs DiffSynth-Studio itself. --- ### En Model Details LTX 2 (`docs/en/Model_Details/LTX-2.md`) # LTX-2 LTX-2 is a series of audio-video generation models developed by Lightricks. ## Installation Before using this project for model inference and training, please install DiffSynth-Studio first. ```shell git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` For more information about installation, please refer to [Installation Dependencies](../Pipeline_Usage/Setup.md). ## Quick Start Run the following code to quickly load the [Lightricks/LTX-2.3](https://www.modelscope.cn/models/Lightricks/LTX-2.3) model and perform inference. VRAM management has been enabled, and the framework will automatically control model parameter loading based on remaining VRAM. It can run with a minimum of 8GB VRAM. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Model Overview |Model ID|Additional Parameters|Inference|Low VRAM Inference|Full Training|Validation After Full Training|LoRA Training|Validation After LoRA Training| |-|-|-|-|-|-|-|-| |[jd-opensource/JoyAI-Echo](https://modelscope.cn/models/jd-opensource/JoyAI-Echo)||[code](/examples/ltx2/model_inference/JoyAI-Echo-T2AV.py)|[code](/examples/ltx2/model_inference_low_vram/JoyAI-Echo-T2AV.py)|[code](/examples/ltx2/model_training/full/JoyAI-Echo-T2AV-splited.sh)|[code](/examples/ltx2/model_training/validate_full/JoyAI-Echo-T2AV.py)|[code](/examples/ltx2/model_training/lora/JoyAI-Echo-T2AV-splited.sh)|[code](/examples/ltx2/model_training/validate_lora/JoyAI-Echo-T2AV.py)| |[Lightricks/LTX-2.3: OneStagePipeline-I2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.3)|`input_images`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2.3-I2AV-OneStage.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2.3-I2AV-OneStage.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/full/LTX-2.3-I2AV-splited.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/validate_full/LTX-2.3-I2AV.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/lora/LTX-2.3-I2AV-splited.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/validate_lora/LTX-2.3-I2AV.py)| |[Lightricks/LTX-2.3: TwoStagePipeline-I2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.3)|`input_images`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2.3-I2AV-TwoStage.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2.3-I2AV-TwoStage.py)|-|-|-|-| |[Lightricks/LTX-2.3: DistilledPipeline-I2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.3)|`input_images`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2.3-I2AV-DistilledPipeline.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2.3-I2AV-DistilledPipeline.py)|-|-|-|-| |[Lightricks/LTX-2.3: OneStagePipeline-T2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.3)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2.3-T2AV-OneStage.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2.3-T2AV-OneStage.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/full/LTX-2.3-T2AV-splited.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/validate_full/LTX-2.3-T2AV.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/lora/LTX-2.3-T2AV-splited.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/validate_lora/LTX-2.3-T2AV.py)| |[Lightricks/LTX-2.3: TwoStagePipeline-T2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.3)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2.3-T2AV-TwoStage.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2.3-T2AV-TwoStage.py)|-|-|-|-| |[Lightricks/LTX-2.3: DistilledPipeline-T2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.3)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2.3-T2AV-DistilledPipeline.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2.3-T2AV-DistilledPipeline.py)|-|-|-|-| |[Lightricks/LTX-2.3: A2V](https://www.modelscope.cn/models/Lightricks/LTX-2.3)|`retake_audio`,`audio_sample_rate`,`retake_audio_regions`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2.3-A2V-TwoStage.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2.3-A2V-TwoStage.py)|-|-|-|-| |[Lightricks/LTX-2.3: Retake](https://www.modelscope.cn/models/Lightricks/LTX-2.3)|`retake_video`,`retake_video_regions`,`retake_audio`,`audio_sample_rate`,`retake_audio_regions`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2.3-T2AV-TwoStage-Retake.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2.3-T2AV-TwoStage-Retake.py)|-|-|-|-| |[Lightricks/LTX-2.3-22b-IC-LoRA-Union-Control](https://www.modelscope.cn/models/Lightricks/LTX-2.3-22b-IC-LoRA-Union-Control)|`in_context_videos`,`in_context_downsample_factor`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2.3-T2AV-IC-LoRA-Union-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2.3-T2AV-IC-LoRA-Union-Control.py)|-|-|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/lora/LTX-2.3-T2AV-IC-LoRA-splited.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/validate_lora/LTX-2.3-T2AV-IC-LoRA.py)| |[Lightricks/LTX-2.3-22b-IC-LoRA-Motion-Track-Control](https://www.modelscope.cn/models/Lightricks/LTX-2.3-22b-IC-LoRA-Motion-Track-Control)|`in_context_videos`,`in_context_downsample_factor`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2.3-T2AV-IC-LoRA-Motion-Track-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2.3-T2AV-IC-LoRA-Motion-Track-Control.py)|-|-|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/lora/LTX-2.3-T2AV-IC-LoRA-splited.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/validate_lora/LTX-2.3-T2AV-IC-LoRA.py)| |[Lightricks/LTX-2: OneStagePipeline-T2AV](https://www.modelscope.cn/models/Lightricks/LTX-2)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2-T2AV-OneStage.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-OneStage.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/full/LTX-2-T2AV-splited.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/validate_full/LTX-2-T2AV.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/lora/LTX-2-T2AV-splited.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/validate_lora/LTX-2-T2AV.py)| |[Lightricks/LTX-2-19b-IC-LoRA-Union-Control](https://www.modelscope.cn/models/Lightricks/LTX-2-19b-IC-LoRA-Union-Control)|`in_context_videos`,`in_context_downsample_factor`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2-T2AV-IC-LoRA-Union-Control.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-IC-LoRA-Union-Control.py)|-|-|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/lora/LTX-2-T2AV-IC-LoRA-splited.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/validate_lora/LTX-2-T2AV-IC-LoRA.py)| |[Lightricks/LTX-2-19b-IC-LoRA-Detailer](https://www.modelscope.cn/models/Lightricks/LTX-2-19b-IC-LoRA-Detailer)|`in_context_videos`,`in_context_downsample_factor`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2-T2AV-IC-LoRA-Detailer.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-IC-LoRA-Detailer.py)|-|-|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/lora/LTX-2-T2AV-IC-LoRA-splited.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/validate_lora/LTX-2-T2AV-IC-LoRA.py)| |[Lightricks/LTX-2: TwoStagePipeline-T2AV](https://www.modelscope.cn/models/Lightricks/LTX-2)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2-T2AV-TwoStage.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-TwoStage.py)|-|-|-|-| |[Lightricks/LTX-2: DistilledPipeline-T2AV](https://www.modelscope.cn/models/Lightricks/LTX-2)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2-T2AV-DistilledPipeline.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-DistilledPipeline.py)|-|-|-|-| |[Lightricks/LTX-2: OneStagePipeline-I2AV](https://www.modelscope.cn/models/Lightricks/LTX-2)|`input_images`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2-I2AV-OneStage.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2-I2AV-OneStage.py)|-|-|-|-| |[Lightricks/LTX-2: TwoStagePipeline-I2AV](https://www.modelscope.cn/models/Lightricks/LTX-2)|`input_images`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2-I2AV-TwoStage.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2-I2AV-TwoStage.py)|-|-|-|-| |[Lightricks/LTX-2: DistilledPipeline-I2AV](https://www.modelscope.cn/models/Lightricks/LTX-2)|`input_images`|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2-I2AV-DistilledPipeline.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2-I2AV-DistilledPipeline.py)|-|-|-|-| |[Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-In](https://www.modelscope.cn/models/Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-In)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2-T2AV-Camera-Control-Dolly-In.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-Camera-Control-Dolly-In.py)|-|-|-|-| |[Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-Out](https://www.modelscope.cn/models/Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-Out)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2-T2AV-Camera-Control-Dolly-Out.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-Camera-Control-Dolly-Out.py)|-|-|-|-| |[Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-Left](https://www.modelscope.cn/models/Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-Left)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2-T2AV-Camera-Control-Dolly-Left.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-Camera-Control-Dolly-Left.py)|-|-|-|-| |[Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-Right](https://www.modelscope.cn/models/Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-Right)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2-T2AV-Camera-Control-Dolly-Right.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-Camera-Control-Dolly-Right.py)|-|-|-|-| |[Lightricks/LTX-2-19b-LoRA-Camera-Control-Jib-Up](https://www.modelscope.cn/models/Lightricks/LTX-2-19b-LoRA-Camera-Control-Jib-Up)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2-T2AV-Camera-Control-Jib-Up.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-Camera-Control-Jib-Up.py)|-|-|-|-| |[Lightricks/LTX-2-19b-LoRA-Camera-Control-Jib-Down](https://www.modelscope.cn/models/Lightricks/LTX-2-19b-LoRA-Camera-Control-Jib-Down)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2-T2AV-Camera-Control-Jib-Down.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-Camera-Control-Jib-Down.py)|-|-|-|-| |[Lightricks/LTX-2-19b-LoRA-Camera-Control-Static](https://www.modelscope.cn/models/Lightricks/LTX-2-19b-LoRA-Camera-Control-Static)||[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference/LTX-2-T2AV-Camera-Control-Static.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_inference_low_vram/LTX-2-T2AV-Camera-Control-Static.py)|-|-|-|-| |[Lightricks/LTX-2.5: OneStagePipeline-T2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.5)|-|[code](/examples/ltx2/model_inference/LTX-2.5-T2AV-OneStage.py)|[code](/examples/ltx2/model_inference_low_vram/LTX-2.5-T2AV-OneStage.py)|[code](/examples/ltx2/model_training/full/LTX-2.5-T2AV-splited.sh)|[code](/examples/ltx2/model_training/validate_full/LTX-2.5-T2AV.py)|[code](/examples/ltx2/model_training/lora/LTX-2.5-T2AV-splited.sh)|[code](/examples/ltx2/model_training/validate_lora/LTX-2.5-T2AV.py)| |[Lightricks/LTX-2.5: TwoStagePipeline-T2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.5)|-|[code](/examples/ltx2/model_inference/LTX-2.5-T2AV-TwoStage.py)|[code](/examples/ltx2/model_inference_low_vram/LTX-2.5-T2AV-TwoStage.py)|-|-|-|-| |[Lightricks/LTX-2.5: OneStagePipeline-I2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.5)|`input_images`,`input_images_indexes`|[code](/examples/ltx2/model_inference/LTX-2.5-I2AV-OneStage.py)|[code](/examples/ltx2/model_inference_low_vram/LTX-2.5-I2AV-OneStage.py)|-|-|-|-| |[Lightricks/LTX-2.5: TwoStagePipeline-I2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.5)|`input_images`,`input_images_indexes`|[code](/examples/ltx2/model_inference/LTX-2.5-I2AV-TwoStage.py)|[code](/examples/ltx2/model_inference_low_vram/LTX-2.5-I2AV-TwoStage.py)|-|-|-|-| |[Lightricks/LTX-2.5: TwoStagePipeline-A2V](https://www.modelscope.cn/models/Lightricks/LTX-2.5)|`retake_audio`,`audio_sample_rate`,`stage2_lora_config`|[code](/examples/ltx2/model_inference/LTX-2.5-A2V-TwoStage.py)|[code](/examples/ltx2/model_inference_low_vram/LTX-2.5-A2V-TwoStage.py)|-|-|-|-| |[Lightricks/LTX-2.5: TwoStagePipeline-Retake](https://www.modelscope.cn/models/Lightricks/LTX-2.5)|`retake_video`,`retake_video_regions`,`stage2_lora_config`|[code](/examples/ltx2/model_inference/LTX-2.5-T2AV-TwoStage-Retake.py)|[code](/examples/ltx2/model_inference_low_vram/LTX-2.5-T2AV-TwoStage-Retake.py)|-|-|-|-| |[Lightricks/LTX-2.5: T2A](https://www.modelscope.cn/models/Lightricks/LTX-2.5)|`audio_only=True`|[code](/examples/ltx2/model_inference/LTX-2.5-T2A.py)|[code](/examples/ltx2/model_inference_low_vram/LTX-2.5-T2A.py)|-|-|-|-| |[Lightricks/LTX-2.5: DistilledPipeline-T2AV](https://www.modelscope.cn/models/Lightricks/LTX-2.5)|`auto_duration`|[code](/examples/ltx2/model_inference/LTX-2.5-T2AV-DistilledPipeline.py)|[code](/examples/ltx2/model_inference_low_vram/LTX-2.5-T2AV-DistilledPipeline.py)|-|-|-|-| |[Lightricks/LTX-2.5-22b-IC-LoRA-Pixel-Spatial-Upscaler](https://www.modelscope.cn/models/Lightricks/LTX-2.5-22b-IC-LoRA-Pixel-Spatial-Upscaler)|`in_context_videos`,`in_context_downsample_factor`|[code](/examples/ltx2/model_inference/LTX-2.5-IC-LoRA-Pixel-Spatial-Upscaler.py)|[code](/examples/ltx2/model_inference_low_vram/LTX-2.5-IC-LoRA-Pixel-Spatial-Upscaler.py)|-|-|-|-| |[Lightricks/LTX-2.5: INT8-ConvRot](https://www.modelscope.cn/models/Lightricks/LTX-2.5)|-|[code](/examples/ltx2/model_inference/LTX-2.5-T2AV-INT8-ConvRot.py)|[code](/examples/ltx2/model_inference_low_vram/LTX-2.5-T2AV-INT8-ConvRot.py)|-|-|[code](/examples/ltx2/model_training/lora/LTX-2.5-Int8-ConvRot-splited.sh)|[code](/examples/ltx2/model_training/validate_lora/LTX-2.5-Int8-ConvRot.py)| ## Model Inference Models are loaded through `LTX2AudioVideoPipeline.from_pretrained`, see [Loading Models](../Pipeline_Usage/Model_Inference.md#loading-models) for details. Input parameters for `LTX2AudioVideoPipeline` inference include: * `prompt`: Prompt describing the content appearing in the video. * `negative_prompt`: Negative prompt describing content that should not appear in the video, default value is `""`. * `denoising_strength`: Denoising strength, range is 0～1, default value is 1.0. * `input_images`: List of input images for image-to-video generation. * `input_images_indexes`: Frame index list of input images in the video. * `input_images_strength`: Strength of input images, default value is 1.0. * `seed`: Random seed. Default is `None`, which means completely random. * `rand_device`: Computing device for generating random Gaussian noise matrix, default is `"cpu"`. When set to `cuda`, different results will be generated on different GPUs. * `height`: Video height, must be a multiple of 32 (single-stage) or 64 (two-stage). * `width`: Video width, must be a multiple of 32 (single-stage) or 64 (two-stage). * `num_frames`: Number of video frames, default value is 121, must be a multiple of 8 + 1. * `auto_duration`: Predict the clip duration from the prompt. Defaults to `False`. When enabled, `num_frames` is not required and the Duration Head must be loaded. * `auto_duration_min_seconds` / `auto_duration_max_seconds`: Lower and upper bounds (seconds) for the predicted duration. Default to 1.0 and 20.0. * `audio_only`: Whether to generate audio only (T2A). Defaults to `False`; when `True`, the video VAE and latent upsampler are not required. * `cfg_scale`: Classifier-free guidance parameter, default value is 3.0. * `num_inference_steps`: Number of inference steps, default value is 40. * `tiled`: Whether to enable VAE tiling inference, default is `True`. When set to `True`, it can significantly reduce VRAM usage during VAE encoding/decoding stages, with slight errors and minor inference time extension. * `tile_size_in_pixels`: Pixel tiling size during VAE encoding/decoding stages, default is 512. * `tile_overlap_in_pixels`: Pixel tiling overlap size during VAE encoding/decoding stages, default is 128. * `tile_size_in_frames`: Frame tiling size during VAE encoding/decoding stages, default is 128. * `tile_overlap_in_frames`: Frame tiling overlap size during VAE encoding/decoding stages, default is 24. * `use_two_stage_pipeline`: Whether to use two-stage pipeline, default is `False`. * `use_distilled_pipeline`: Whether to use distilled pipeline, default is `False`. * `progress_bar_cmd`: Progress bar, default is `tqdm.tqdm`. Can be set to `lambda x:x` to hide the progress bar. If VRAM is insufficient, please enable [VRAM Management](../Pipeline_Usage/VRAM_management.md). We provide recommended low VRAM configurations for each model in the example code, see the table in the previous "Supported Inference Scripts" section. ## Model Training LTX-2 series models are uniformly trained through [`examples/ltx2/model_training/train.py`](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/ltx2/model_training/train.py), and the script parameters include: * General Training Parameters * Dataset Basic Configuration * `--dataset_base_path`: Root directory of the dataset. * `--dataset_metadata_path`: Metadata file path of the dataset. * `--dataset_repeat`: Number of times the dataset is repeated in each epoch. * `--dataset_num_workers`: Number of processes for each DataLoader. * `--data_file_keys`: Field names to be loaded from metadata, usually image or video file paths, separated by `,`. * Model Loading Configuration * `--model_paths`: Paths of models to be loaded. JSON format. * `--model_id_with_origin_paths`: Model IDs with original paths, e.g., `"Wan-AI/Wan2.1-T2V-1.3B:diffusion_pytorch_model*.safetensors"`. Separated by commas. * `--extra_inputs`: Extra input parameters required by the model Pipeline, e.g., extra parameters when training image editing models, separated by `,`. * `--fp8_models`: Models loaded in FP8 format, consistent with `--model_paths` or `--model_id_with_origin_paths` format. Currently only supports models whose parameters are not updated by gradients (no gradient backpropagation, or gradients only update their LoRA). * `--quant_options`: Dynamically quantize loaded models. Semicolon-separated entries, each `:[/]`, where `` matches an entry in `--model_paths`/`--model_id_with_origin_paths`, `method` is a registered method (e.g. `bitsandbytes_nf4`), and `exclude_modules` optionally lists layers kept in full precision. * Training Basic Configuration * `--learning_rate`: Learning rate. * `--num_epochs`: Number of epochs. * `--trainable_models`: Trainable models, e.g., `dit`, `vae`, `text_encoder`. * `--find_unused_parameters`: Whether there are unused parameters in DDP training. Some models contain redundant parameters that do not participate in gradient calculation, and this setting needs to be enabled to avoid errors in multi-GPU training. * `--weight_decay`: Weight decay size, see [torch.optim.AdamW](https://docs.pytorch.org/docs/stable/generated/torch.optim.AdamW.html). * `--task`: Training task, default is `sft`. Some models support more training modes, please refer to the documentation of each specific model. * Output Configuration * `--output_path`: Model saving path. * `--remove_prefix_in_ckpt`: Remove prefix in the state dict of the model file. * `--save_steps`: Interval of training steps to save the model. If this parameter is left blank, the model is saved once per epoch. * LoRA Configuration * `--lora_base_model`: Which model to add LoRA to. * `--lora_target_modules`: Which layers to add LoRA to. * `--lora_rank`: Rank of LoRA. * `--lora_checkpoint`: Path of the LoRA checkpoint. If this path is provided, LoRA will be loaded from this checkpoint. * `--preset_lora_path`: Preset LoRA checkpoint path. If this path is provided, this LoRA will be loaded in the form of being merged into the base model. This parameter is used for LoRA differential training. * `--preset_lora_model`: Model that the preset LoRA is merged into, e.g., `dit`. * Gradient Configuration * `--use_gradient_checkpointing`: Whether to enable gradient checkpointing. * `--use_gradient_checkpointing_offload`: Whether to offload gradient checkpointing to memory. * `--gradient_accumulation_steps`: Number of gradient accumulation steps. * Video Width/Height Configuration * `--height`: Height of the video. Leave `height` and `width` blank to enable dynamic resolution. * `--width`: Width of the video. Leave `height` and `width` blank to enable dynamic resolution. * `--max_pixels`: Maximum pixel area of video frames. When dynamic resolution is enabled, video frames with resolution larger than this value will be downscaled, and video frames with resolution smaller than this value will remain unchanged. * `--num_frames`: Number of frames in the video. * LTX-2 Series Specific Parameters * `--tokenizer_path`: Path of the tokenizer, applicable to text-to-video models, leave blank to automatically download from remote. * `--frame_rate`: frame rate of the training videos. We have built a sample video dataset for your testing. You can download this dataset with the following command: ```shell modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --local_dir ./data/diffsynth_example_dataset ``` We have written recommended training scripts for each model, please refer to the table in the "Model Overview" section above. For how to write model training scripts, please refer to [Model Training](../Pipeline_Usage/Model_Training.md); for more advanced training algorithms, please refer to [Training Framework Detailed Explanation](https://github.com/modelscope/DiffSynth-Studio/tree/main/docs/en/Training/). --- ### En Model Details LingBot Video (`docs/en/Model_Details/LingBot-Video.md`) # LingBot-Video LingBot-Video is a flow-matching video generation model developed by the LingBot team; a single model handles text-to-video, image-to-video and text-to-image tasks. Huge thanks to [NancyFyong](https://github.com/NancyFyong) for the outstanding contribution to the integration of this model! ## Installation Before performing model inference and training, please install DiffSynth-Studio first. ```shell git clone https://github.com/modelscope/DiffSynth-Studio.git cd DiffSynth-Studio pip install -e . ``` For more information on installation, please refer to [Setup Dependencies](../Pipeline_Usage/Setup.md). ## Quick Start Running the following code will load the [Robbyant/lingbot-video-dense-1.3b](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b) model for inference. VRAM management is enabled, the framework automatically controls parameter loading based on available VRAM, requiring a minimum of 6GB VRAM. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Model Overview |Model ID|Inference|Low VRAM Inference|Full Training|Full Training Validation|LoRA Training|LoRA Training Validation| |-|-|-|-|-|-|-| |[Robbyant/lingbot-video-dense-1.3b: T2V](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-dense-1.3b_t2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-dense-1.3b_t2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/full/lingbot-video-dense-1.3b_t2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_full/lingbot-video-dense-1.3b_t2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/lora/lingbot-video-dense-1.3b_t2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_lora/lingbot-video-dense-1.3b_t2v.py)| |[Robbyant/lingbot-video-dense-1.3b: TI2V](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-dense-1.3b_ti2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-dense-1.3b_ti2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/full/lingbot-video-dense-1.3b_ti2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_full/lingbot-video-dense-1.3b_ti2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/lora/lingbot-video-dense-1.3b_ti2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_lora/lingbot-video-dense-1.3b_ti2v.py)| |[Robbyant/lingbot-video-dense-1.3b: T2I](https://modelscope.cn/models/Robbyant/lingbot-video-dense-1.3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-dense-1.3b_t2i.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-dense-1.3b_t2i.py)|-|-|-|-| |[Robbyant/lingbot-video-moe-30b-a3b: T2V](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_t2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_t2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_t2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_t2v.py)| |[Robbyant/lingbot-video-moe-30b-a3b: TI2V](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_ti2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_ti2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/full/lingbot-video-moe-30b-a3b_ti2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_full/lingbot-video-moe-30b-a3b_ti2v.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/lora/lingbot-video-moe-30b-a3b_ti2v.sh)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/validate_lora/lingbot-video-moe-30b-a3b_ti2v.py)| |[Robbyant/lingbot-video-moe-30b-a3b: T2I](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2i.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2i.py)|-|-|-|-| |[Robbyant/lingbot-video-moe-30b-a3b: T2V + Refinement](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_t2v_refiner.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_t2v_refiner.py)|-|-|-|-| |[Robbyant/lingbot-video-moe-30b-a3b: TI2V + Refinement](https://modelscope.cn/models/Robbyant/lingbot-video-moe-30b-a3b)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference/lingbot-video-moe-30b-a3b_ti2v_refiner.py)|[code](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_inference_low_vram/lingbot-video-moe-30b-a3b_ti2v_refiner.py)|-|-|-|-| ## Model Inference The model is loaded via `LingBotVideoPipeline.from_pretrained`, see [Loading Models](../Pipeline_Usage/Model_Inference.md#loading-models) for details. The input parameters for `LingBotVideoPipeline` inference include: * `prompt`: Structured-JSON caption (`dict`) or a plain string describing the content. LingBot-Video is trained on structured captions; the pipeline normalises a `dict` automatically. Released structured captions ship in the example dataset (see [Prompt rewriting](#prompt-rewriting) below). * `negative_prompt`: Negative prompt describing content that should not appear. `pipe.default_negative_prompt` ships the official T2V/V2V/TI2V negative prompt; `pipe.default_negative_prompt_image` is the T2I variant with temporal terms removed. * `input_image`: First-frame PIL image for image-to-video (TI2V). The frame is VAE-encoded to a clean latent pinned into the first temporal slot after every scheduler step, so the model only generates the frames that follow. Leave `None` for T2V / V2V / T2I. * `input_video`: Input video (a list of frames or a `VideoData`) for video-to-video generation, used together with `denoising_strength`. * `denoising_strength`: Denoising strength in `[0, 1]`, default `1.0`. Lower values keep more of the input video structure. Only effective when `input_video` is provided. * `height`: Video / image height, default `480`. Must be a multiple of 16. * `width`: Video / image width, default `480`. Must be a multiple of 16. * `num_frames`: Number of frames, default `81`. Must satisfy `4k+1` (the VAE compresses time by 4×). Use `num_frames=1` for text-to-image. * `cfg_scale`: Classifier-free guidance scale, default `3.0`. * `num_inference_steps`: Number of inference steps, default `40`. * `sigma_shift`: Flow-matching timestep shift, default `3.0`. * `t_thresh`: Refinement start sigma, default `None` (plain generation). When set, the schedule is truncated so that sampling starts at `sigma=t_thresh` and `input_video` is noised to exactly that level. Only meaningful together with `input_video`; TI2V additionally re-pins the clean first-frame latent after every step. The official refiner setting is `0.85`. * `sigma_tail_steps`: Number of extra low-noise steps appended to the tail of the refinement schedule, default `2`. Only effective when `t_thresh` is set. * `seed`: Random seed. Default is `None`, meaning completely random. * `rand_device`: Device for generating the initial noise, default `"cpu"`. * `progress_bar_cmd`: Progress bar, default `tqdm`. Can be disabled by setting to `lambda x: x`. If VRAM is insufficient, please enable [VRAM Management](../Pipeline_Usage/VRAM_management.md). We provide recommended low-VRAM configurations for each task in the example code, see the table in the "Model Overview" section above. ### Two-stage refinement The MoE refiner performs a short second pass at a higher resolution: the official setup generates at 480×832 with 40 steps, then refines at 1088×1920 with 8 steps. Load the pipeline with the `refiner/` shards instead of `transformer/`, feed the base clip back in through `input_video` at the higher resolution, and set `t_thresh`: ```python input_video = VideoData("video_base.mp4", height=1088, width=1920) video = pipe( prompt=caption, negative_prompt=pipe.default_negative_prompt, input_video=input_video, height=1088, width=1920, num_frames=81, num_inference_steps=8, cfg_scale=3.0, t_thresh=0.85, sigma_tail_steps=2, seed=0, ) ``` The upscaled clip is VAE-encoded and noised back to `sigma=t_thresh`, so the pass keeps the structure of the base clip and regenerates detail at the target resolution. Pass the same caption as the base pass and keep the same aspect ratio. The refinement resolution dominates the cost — at 1088×1920 the sequence is ~5× longer than at 480×832 — so run this pass with VRAM management enabled. ### Prompt rewriting LingBot-Video is trained on **structured-JSON captions**, not free-form prose. Feeding a flat sentence is out-of-distribution and visibly degrades quality. The pipeline accepts a caption as a `dict` (the format used at training time) or a plain string, and normalises the `dict` internally. Released structured captions ship in the example dataset (`t2v_example_*.json`, `ti2v_example.json`, `t2i_example.json` under `DiffSynth-Studio/diffsynth_example_dataset`, downloaded automatically by the inference example scripts). Load one with `json.load` and pass the resulting `dict` to the pipeline, or use one as a template. To turn a brief idea into a structured caption, use the two-stage rewriter shipped under `examples/lingbot_video/model_training/scripts/prompt_rewriter.py` — stage 1 expands the idea into a natural-language caption, stage 2 maps it into structured JSON. The rewriter is a **separate VLM + stage-2 LoRA adapter** and is not downloaded automatically: | Role | Model ID | Size | |-|-|-| | Rewriter base VLM (stage 1 + 2) | [`Qwen/Qwen3.6-27B`](https://modelscope.cn/models/Qwen/Qwen3.6-27B) | ~55 GB | | Rewriter stage-2 LoRA adapter | [`Robbyant/lingbot-video-rewriter-lora`](https://modelscope.cn/models/Robbyant/lingbot-video-rewriter-lora) | ~0.5 GB | ```python import os os.environ["REWRITER_BASE_MODEL"] = "./models/Qwen/Qwen3.6-27B" os.environ["REWRITER_ADAPTER"] = "./models/Robbyant/lingbot-video-rewriter-lora" # Run from the repo root so the package-style import resolves. from examples.lingbot_video.model_training.scripts.prompt_rewriter import rewrite_prompt caption = rewrite_prompt("a puppy running across a meadow", mode="t2v", duration=5) video = pipe(prompt=caption, height=480, width=832, num_frames=81, cfg_scale=3.0) ``` Instead of the env vars you can pass `base=` / `adapter=` to `rewrite_prompt`, or drive a hosted / OpenAI-compatible endpoint by passing a custom object exposing `generate(text, image, use_lora)` as `backend=`. ## Model Training Models in the LingBot-Video series are trained uniformly via [`examples/lingbot_video/model_training/train.py`](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/train.py). The script parameters include: * General Training Parameters * Dataset Configuration * `--dataset_base_path`: Root directory of the dataset. * `--dataset_metadata_path`: Path to the dataset metadata file. * `--dataset_repeat`: Number of dataset repeats per epoch. * `--dataset_num_workers`: Number of processes per DataLoader. * `--data_file_keys`: Field names to load from metadata, typically paths to image or video files, separated by `,`. * Model Loading Configuration * `--model_paths`: Paths to load models from, in JSON format. * `--model_id_with_origin_paths`: Model IDs with original paths, separated by commas. * `--extra_inputs`: Additional input parameters required by the model Pipeline, separated by `,`. * `--fp8_models`: Models to load in FP8 format, currently only supported for models whose parameters are not updated by gradients. * `--quant_options`: Dynamically quantize loaded models. Semicolon-separated entries, each `:[/]`, where `` matches an entry in `--model_paths`/`--model_id_with_origin_paths`, `method` is a registered method (e.g. `bitsandbytes_nf4`), and `exclude_modules` optionally lists layers kept in full precision. * Basic Training Configuration * `--learning_rate`: Learning rate. * `--num_epochs`: Number of epochs. * `--trainable_models`: Trainable models, e.g., `dit`, `vae`, `text_encoder`. * `--find_unused_parameters`: Whether unused parameters exist in DDP training. * `--weight_decay`: Weight decay magnitude. * `--task`: Training task, defaults to `sft`. * Output Configuration * `--output_path`: Path to save the model. * `--remove_prefix_in_ckpt`: Remove prefix in the model's state dict. * `--save_steps`: Interval in training steps to save the model. * LoRA Configuration * `--lora_base_model`: Which model to add LoRA to. * `--lora_target_modules`: Which layers to add LoRA to. * `--lora_rank`: Rank of LoRA. * `--lora_checkpoint`: Path to LoRA checkpoint. * `--preset_lora_path`: Path to preset LoRA checkpoint for LoRA differential training. * `--preset_lora_model`: Which model to integrate preset LoRA into, e.g., `dit`. * Gradient Configuration * `--use_gradient_checkpointing`: Whether to enable gradient checkpointing. * `--use_gradient_checkpointing_offload`: Whether to offload gradient checkpointing to CPU memory. * `--gradient_accumulation_steps`: Number of gradient accumulation steps. * Resolution Configuration * `--height`: Height of the video. Must be divisible by 16. * `--width`: Width of the video. Must be divisible by 16. * `--max_pixels`: Maximum pixel area, images larger than this will be scaled down during dynamic resolution. * `--num_frames`: Number of frames in the video. Must satisfy `4k+1`. * LingBot-Video Specific Parameters * `--processor_path`: Path to the Qwen3-VL processor directory (or `model_id:origin_file_pattern`). Used to tokenize prompts. * `--first_frame_as_condition`: Enable image-to-video (TI2V) LoRA / full training. Each clip is conditioned on its own first frame: the frame is VAE-encoded to a clean latent pinned into the first temporal slot (and fed to the Qwen3-VL text encoder as vision input), and excluded from the flow-matching loss. * `--max_timestep_boundary`: Max timestep boundary as a fraction of the training schedule, in `[0, 1]`. * `--min_timestep_boundary`: Min timestep boundary as a fraction of the training schedule, in `[0, 1]`. * `--initialize_model_on_cpu`: Whether to initialize models on CPU. We provide a sample dataset for your testing. You can download it with the following command: ```shell modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "lingbot_video/lingbot-video-dense-1.3b_t2v/*" --local_dir ./data/diffsynth_example_dataset ``` Training captions should be **structured-JSON captions** (the same in-distribution format used at inference). If your dataset stores raw prose, rewrite it once offline with [`examples/lingbot_video/model_training/scripts/rewrite_captions.py`](https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/lingbot_video/model_training/scripts/rewrite_captions.py) before training. We provide recommended training scripts for each task, please refer to the table in "Model Overview" above. For guidance on writing model training scripts, see [Model Training](../Pipeline_Usage/Model_Training.md); for more advanced training algorithms, see [Training Framework Overview](https://github.com/modelscope/DiffSynth-Studio/tree/main/docs/en/Training/). --- --- METRICS --- - Files Extracted: 47 - Estimated Token Budget: ~126665 tokens - Canonical Reference: https://codewiki.google/github.com/modelscope/DiffSynth-Studio