## File: README.md ๐Ÿค— Diffusers is the go-to library for state-of-the-art pretrained diffusion models for generating images, audio, and even 3D structures of molecules. Whether you're looking for a simple inference solution or training your own diffusion models, ๐Ÿค— Diffusers is a modular toolbox that supports both. Our library is designed with a focus on [usability over performance](https://huggingface.co/docs/diffusers/conceptual/philosophy#usability-over-performance), [simple over easy](https://huggingface.co/docs/diffusers/conceptual/philosophy#simple-over-easy), and [customizability over abstractions](https://huggingface.co/docs/diffusers/conceptual/philosophy#tweakable-contributorfriendly-over-abstraction). ๐Ÿค— Diffusers offers three core components: - State-of-the-art [diffusion pipelines](https://huggingface.co/docs/diffusers/api/pipelines/overview) that can be run in inference with just a few lines of code. - Interchangeable noise [schedulers](https://huggingface.co/docs/diffusers/api/schedulers/overview) for different diffusion speeds and output quality. - Pretrained [models](https://huggingface.co/docs/diffusers/api/models/overview) that can be used as building blocks, and combined with schedulers, for creating your own end-to-end diffusion systems. ## Installation We recommend installing ๐Ÿค— Diffusers in a virtual environment from PyPI or Conda. For more details about installing [PyTorch](https://pytorch.org/get-started/locally/), please refer to their official documentation. ### PyTorch With `pip` (official package): ```bash pip install --upgrade diffusers[torch] ``` With `conda` (maintained by the community): ```sh conda install -c conda-forge diffusers ``` ### Apple Silicon (M1/M2) support Please refer to the [How to use Stable Diffusion in Apple Silicon](https://huggingface.co/docs/diffusers/optimization/mps) guide. ## Quickstart Generating outputs is super easy with ๐Ÿค— Diffusers. To generate an image from text, use the `from_pretrained` method to load any pretrained diffusion model (browse the [Hub](https://huggingface.co/models?library=diffusers&sort=downloads) for 30,000+ checkpoints): ```python from diffusers import DiffusionPipeline import torch pipeline = DiffusionPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5", dtype=torch.float16) pipeline.to("cuda") pipeline("An image of a squirrel in Picasso style").images[0] ``` You can also dig into the models and schedulers toolbox to build your own diffusion system: ```python from diffusers import DDPMScheduler, UNet2DModel from PIL import Image import torch scheduler = DDPMScheduler.from_pretrained("google/ddpm-cat-256") model = UNet2DModel.from_pretrained("google/ddpm-cat-256").to("cuda") scheduler.set_timesteps(50) sample_size = model.config.sample_size noise = torch.randn((1, 3, sample_size, sample_size), device="cuda") input = noise for t in scheduler.timesteps: with torch.no_grad(): noisy_residual = model(input, t).sample prev_noisy_sample = scheduler.step(noisy_residual, t, input).prev_sample input = prev_noisy_sample image = (input / 2 + 0.5).clamp(0, 1) image = image.cpu().permute(0, 2, 3, 1).numpy()[0] image = Image.fromarray((image * 255).round().astype("uint8")) image ``` Check out the [Quickstart](https://huggingface.co/docs/diffusers/quicktour) to launch your diffusion journey today! ## How to navigate the documentation | **Documentation** | **What can I learn?** | |---------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [Tutorial](https://huggingface.co/docs/diffusers/tutorials/tutorial_overview) | A basic crash course for learning how to use the library's most important features like using models and schedulers to build your own diffusion system, and training your own diffusion model. | | [Loading](https://huggingface.co/docs/diffusers/using-diffusers/loading) | Guides for how to load and configure all the components (pipelines, models, and schedulers) of the library, as well as how to use different schedulers. | | [Pipelines for inference](https://huggingface.co/docs/diffusers/using-diffusers/overview_techniques) | Guides for how to use pipelines for different inference tasks, batched generation, controlling generated outputs and randomness, and how to contribute a pipeline to the library. | | [Optimization](https://huggingface.co/docs/diffusers/optimization/fp16) | Guides for how to optimize your diffusion model to run faster and consume less memory. | | [Training](https://huggingface.co/docs/diffusers/training/overview) | Guides for how to train a diffusion model for different tasks with different training techniques. | ## Contribution We โค๏ธ contributions from the open-source community! If you want to contribute to this library, please check out our [Contribution guide](https://huggingface.co/docs/diffusers/main/en/conceptual/contribution). If you are using an AI agent, please point it at the project conventions in [`.ai/`](https://github.com/huggingface/diffusers/tree/main/.ai) first (run `make claude` or `make codex`) โ€” see [Coding with AI agents](https://huggingface.co/docs/diffusers/main/en/conceptual/contribution#coding-with-ai-agents). You can look out for [issues](https://github.com/huggingface/diffusers/issues) you'd like to tackle to contribute to the library. - See [Good first issues](https://github.com/huggingface/diffusers/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) for general opportunities to contribute - See [New model/pipeline](https://github.com/huggingface/diffusers/issues?q=is%3Aopen+is%3Aissue+label%3A%22New+pipeline%2Fmodel%22) to contribute exciting new diffusion models / diffusion pipelines - See [New scheduler](https://github.com/huggingface/diffusers/issues?q=is%3Aopen+is%3Aissue+label%3A%22New+scheduler%22) Also, say ๐Ÿ‘‹ in our public Discord channel [](https://discord.gg/G7tWnz98XR). We discuss the hottest trends about diffusion models, help each other with contributions, personal projects or just hang out โ˜•. ## Popular Tasks & Pipelines | Task | Pipeline | ๐Ÿค— Hub | | --- | --- | --- | | Unconditional Image Generation | [ DDPM ](https://huggingface.co/docs/diffusers/api/pipelines/ddpm) | [ google/ddpm-ema-church-256 ](https://huggingface.co/google/ddpm-ema-church-256) | | Text-to-Image | [Stable Diffusion Text-to-Image](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/text2img) | [ stable-diffusion-v1-5/stable-diffusion-v1-5 ](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5) | | Text-to-Image | [unCLIP](https://huggingface.co/docs/diffusers/api/pipelines/unclip) | [ kakaobrain/karlo-v1-alpha ](https://huggingface.co/kakaobrain/karlo-v1-alpha) | | Text-to-Image | [DeepFloyd IF](https://huggingface.co/docs/diffusers/api/pipelines/deepfloyd_if) | [ DeepFloyd/IF-I-XL-v1.0 ](https://huggingface.co/DeepFloyd/IF-I-XL-v1.0) | | Text-to-Image | [Kandinsky](https://huggingface.co/docs/diffusers/api/pipelines/kandinsky) | [ kandinsky-community/kandinsky-2-2-decoder ](https://huggingface.co/kandinsky-community/kandinsky-2-2-decoder) | | Text-guided Image-to-Image | [ControlNet](https://huggingface.co/docs/diffusers/api/pipelines/controlnet) | [ lllyasviel/sd-controlnet-canny ](https://huggingface.co/lllyasviel/sd-controlnet-canny) | | Text-guided Image-to-Image | [InstructPix2Pix](https://huggingface.co/docs/diffusers/api/pipelines/pix2pix) | [ timbrooks/instruct-pix2pix ](https://huggingface.co/timbrooks/instruct-pix2pix) | | Text-guided Image-to-Image | [Stable Diffusion Image-to-Image](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/img2img) | [ stable-diffusion-v1-5/stable-diffusion-v1-5 ](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5) | | Text-guided Image Inpainting | [Stable Diffusion Inpainting](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/inpaint) | [ stable-diffusion-v1-5/stable-diffusion-inpainting ](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-inpainting) | | Image Variation | [Stable Diffusion Image Variation](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/image_variation) | [ lambdalabs/sd-image-variations-diffusers ](https://huggingface.co/lambdalabs/sd-image-variations-diffusers) | | Super Resolution | [Stable Diffusion Upscale](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/upscale) | [ stabilityai/stable-diffusion-x4-upscaler ](https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler) | | Super Resolution | [Stable Diffusion Latent Upscale](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/latent_upscale) | [ stabilityai/sd-x2-latent-upscaler ](https://huggingface.co/stabilityai/sd-x2-latent-upscaler) | ## Popular libraries using ๐Ÿงจ Diffusers - https://github.com/microsoft/TaskMatrix - https://github.com/invoke-ai/InvokeAI - https://github.com/InstantID/InstantID - https://github.com/apple/ml-stable-diffusion - https://github.com/Sanster/lama-cleaner - https://github.com/IDEA-Research/Grounded-Segment-Anything - https://github.com/ashawkey/stable-dreamfusion - https://github.com/deep-floyd/IF - https://github.com/bentoml/BentoML - https://github.com/bmaltais/kohya_ss - +14,000 other amazing GitHub repositories ๐Ÿ’ช Thank you for using us โค๏ธ. ## Credits This library concretizes previous work by many different authors and would not have been possible without their great research and implementations. We'd like to thank, in particular, the following implementations which have helped us in our development and without which the API could not have been as polished today: - @CompVis' latent diffusion models library, available [here](https://github.com/CompVis/latent-diffusion) - @hojonathanho original DDPM implementation, available [here](https://github.com/hojonathanho/diffusion) as well as the extremely useful translation into PyTorch by @pesser, available [here](https://github.com/pesser/pytorch_diffusion) - @ermongroup's DDIM implementation, available [here](https://github.com/ermongroup/ddim) - @yang-song's Score-VE and Score-VP implementations, available [here](https://github.com/yang-song/score_sde_pytorch) We also want to thank @heejkoo for the very helpful overview of papers, code and resources on diffusion models, available [here](https://github.com/heejkoo/Awesome-Diffusion-Models) as well as @crowsonkb and @rromb for useful discussions and insights. ## Citation ```bibtex @misc{von-platen-etal-2022-diffusers, author = {Patrick von Platen and Suraj Patil and Anton Lozhkov and Pedro Cuenca and Nathan Lambert and Kashif Rasul and Mishig Davaadorj and Dhruv Nair and Sayak Paul and William Berman and Yiyi Xu and Steven Liu and Thomas Wolf}, title = {Diffusers: State-of-the-art diffusion models}, year = {2022}, publisher = {GitHub}, journal = {GitHub repository}, howpublished = {\url{https://github.com/huggingface/diffusers}} } ``` --- ## File: docs/source/en/advanced_inference/outpaint.md # Outpainting Outpainting extends an image beyond its original boundaries, allowing you to add, replace, or modify visual elements in an image while preserving the original image. Like [inpainting](../using-diffusers/inpaint), you want to fill the white area (in this case, the area outside of the original image) with new visual elements while keeping the original image (represented by a mask of black pixels). There are a couple of ways to outpaint, such as with a [ControlNet](https://hf.co/blog/OzzyGT/outpainting-controlnet) or with [Differential Diffusion](https://hf.co/blog/OzzyGT/outpainting-differential-diffusion). This guide will show you how to outpaint with an inpainting model, ControlNet, and a ZoeDepth estimator. Before you begin, make sure you have the [controlnet_aux](https://github.com/huggingface/controlnet_aux) library installed so you can use the ZoeDepth estimator. ```py !pip install -q controlnet_aux ``` ## Image preparation Start by picking an image to outpaint with and remove the background with a Space like [BRIA-RMBG-1.4](https://hf.co/spaces/briaai/BRIA-RMBG-1.4). For example, remove the background from this image of a pair of shoes.
original image
background removed
[Stable Diffusion XL (SDXL)](../api/pipelines/stable_diffusion/stable_diffusion_xl) models work best with 1024x1024 images, but you can resize the image to any size as long as your hardware has enough memory to support it. The transparent background in the image should also be replaced with a white background. Create a function (like the one below) that scales and pastes the image onto a white background. ```py import random import requests import torch from controlnet_aux import ZoeDetector from PIL import Image, ImageOps from diffusers import ( AutoencoderKL, ControlNetModel, StableDiffusionXLControlNetPipeline, StableDiffusionXLInpaintPipeline, ) def scale_and_paste(original_image): aspect_ratio = original_image.width / original_image.height if original_image.width > original_image.height: new_width = 1024 new_height = round(new_width / aspect_ratio) else: new_height = 1024 new_width = round(new_height * aspect_ratio) resized_original = original_image.resize((new_width, new_height), Image.LANCZOS) white_background = Image.new("RGBA", (1024, 1024), "white") x = (1024 - new_width) // 2 y = (1024 - new_height) // 2 white_background.paste(resized_original, (x, y), resized_original) return resized_original, white_background original_image = Image.open( requests.get( "https://huggingface.co/datasets/stevhliu/testing-images/resolve/main/no-background-jordan.png", stream=True, ).raw ).convert("RGBA") resized_img, white_bg_image = scale_and_paste(original_image) ``` To avoid adding unwanted extra details, use the ZoeDepth estimator to provide additional guidance during generation and to ensure the shoes remain consistent with the original image. ```py zoe = ZoeDetector.from_pretrained("lllyasviel/Annotators") image_zoe = zoe(white_bg_image, detect_resolution=512, image_resolution=1024) image_zoe ``` ## Outpaint Once your image is ready, you can generate content in the white area around the shoes with [controlnet-inpaint-dreamer-sdxl](https://hf.co/destitech/controlnet-inpaint-dreamer-sdxl), a SDXL ControlNet trained for inpainting. Load the inpainting ControlNet, ZoeDepth model, VAE and pass them to the [`StableDiffusionXLControlNetPipeline`]. Then you can create an optional `generate_image` function (for convenience) to outpaint an initial image. ```py controlnets = [ ControlNetModel.from_pretrained( "destitech/controlnet-inpaint-dreamer-sdxl", dtype=torch.float16, variant="fp16" ), ControlNetModel.from_pretrained( "diffusers/controlnet-zoe-depth-sdxl-1.0", dtype=torch.float16 ), ] vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", dtype=torch.float16).to("cuda") pipeline = StableDiffusionXLControlNetPipeline.from_pretrained( "SG161222/RealVisXL_V4.0", dtype=torch.float16, variant="fp16", controlnet=controlnets, vae=vae ).to("cuda") def generate_image(prompt, negative_prompt, inpaint_image, zoe_image, seed: int = None): if seed is None: seed = random.randint(0, 2**32 - 1) generator = torch.Generator(device="cpu").manual_seed(seed) image = pipeline( prompt, negative_prompt=negative_prompt, image=[inpaint_image, zoe_image], guidance_scale=6.5, num_inference_steps=25, generator=generator, controlnet_conditioning_scale=[0.5, 0.8], control_guidance_end=[0.9, 0.6], ).images[0] return image prompt = "nike air jordans on a basketball court" negative_prompt = "" temp_image = generate_image(prompt, negative_prompt, white_bg_image, image_zoe, 908097) ``` Paste the original image over the initial outpainted image. You'll improve the outpainted background in a later step. ```py x = (1024 - resized_img.width) // 2 y = (1024 - resized_img.height) // 2 temp_image.paste(resized_img, (x, y), resized_img) temp_image ``` > [!TIP] > Now is a good time to free up some memory if you're running low! > > ```py > pipeline=None > torch.cuda.empty_cache() > ``` Now that you have an initial outpainted image, load the [`StableDiffusionXLInpaintPipeline`] with the [RealVisXL](https://hf.co/SG161222/RealVisXL_V4.0) model to generate the final outpainted image with better quality. ```py pipeline = StableDiffusionXLInpaintPipeline.from_pretrained( "OzzyGT/RealVisXL_V4.0_inpainting", dtype=torch.float16, variant="fp16", vae=vae, ).to("cuda") ``` Prepare a mask for the final outpainted image. To create a more natural transition between the original image and the outpainted background, blur the mask to help it blend better. ```py mask = Image.new("L", temp_image.size) mask.paste(resized_img.split()[3], (x, y)) mask = ImageOps.invert(mask) final_mask = mask.point(lambda p: p > 128 and 255) mask_blurred = pipeline.mask_processor.blur(final_mask, blur_factor=20) mask_blurred ``` Create a better prompt and pass it to the `generate_outpaint` function to generate the final outpainted image. Again, paste the original image over the final outpainted background. ```py def generate_outpaint(prompt, negative_prompt, image, mask, seed: int = None): if seed is None: seed = random.randint(0, 2**32 - 1) generator = torch.Generator(device="cpu").manual_seed(seed) image = pipeline( prompt, negative_prompt=negative_prompt, image=image, mask_image=mask, guidance_scale=10.0, strength=0.8, num_inference_steps=30, generator=generator, ).images[0] return image prompt = "high quality photo of nike air jordans on a basketball court, highly detailed" negative_prompt = "" final_image = generate_outpaint(prompt, negative_prompt, temp_image, mask_blurred, 7688778) x = (1024 - resized_img.width) // 2 y = (1024 - resized_img.height) // 2 final_image.paste(resized_img, (x, y), resized_img) final_image ``` --- ## File: docs/source/en/api/loaders/ip_adapter.md # IP-Adapter [IP-Adapter](https://hf.co/papers/2308.06721) is a lightweight adapter that enables prompting a diffusion model with an image. This method decouples the cross-attention layers of the image and text features. The image features are generated from an image encoder. > [!TIP] > Learn how to load and use an IP-Adapter checkpoint and image in the [IP-Adapter](../../using-diffusers/ip_adapter) guide,. ## IPAdapterMixin [[autodoc]] loaders.ip_adapter.IPAdapterMixin ## SD3IPAdapterMixin [[autodoc]] loaders.ip_adapter.SD3IPAdapterMixin - all - is_ip_adapter_active ## IPAdapterMaskProcessor [[autodoc]] image_processor.IPAdapterMaskProcessor --- ## File: docs/source/en/api/loaders/lora.md # LoRA LoRA is a fast and lightweight training method that inserts and trains a significantly smaller number of parameters instead of all the model parameters. This produces a smaller file (~100 MBs) and makes it easier to quickly train a model to learn a new concept. LoRA weights are typically loaded into the denoiser, text encoder or both. The denoiser usually corresponds to a UNet ([`UNet2DConditionModel`], for example) or a Transformer ([`SD3Transformer2DModel`], for example). There are several classes for loading LoRA weights: - [`StableDiffusionLoraLoaderMixin`] provides functions for loading and unloading, fusing and unfusing, enabling and disabling, and more functions for managing LoRA weights. This class can be used with any model. - [`StableDiffusionXLLoraLoaderMixin`] is a [Stable Diffusion (SDXL)](../../api/pipelines/stable_diffusion/stable_diffusion_xl) version of the [`StableDiffusionLoraLoaderMixin`] class for loading and saving LoRA weights. It can only be used with the SDXL model. - [`SD3LoraLoaderMixin`] provides similar functions for [Stable Diffusion 3](https://huggingface.co/blog/sd3). - [`FluxLoraLoaderMixin`] provides similar functions for [Flux](https://huggingface.co/docs/diffusers/main/en/api/pipelines/flux). - [`CogVideoXLoraLoaderMixin`] provides similar functions for [CogVideoX](https://huggingface.co/docs/diffusers/main/en/api/pipelines/cogvideox). - [`Mochi1LoraLoaderMixin`] provides similar functions for [Mochi](https://huggingface.co/docs/diffusers/main/en/api/pipelines/mochi). - [`AuraFlowLoraLoaderMixin`] provides similar functions for [AuraFlow](https://huggingface.co/fal/AuraFlow). - [`LTXVideoLoraLoaderMixin`] provides similar functions for [LTX-Video](https://huggingface.co/docs/diffusers/main/en/api/pipelines/ltx_video). - [`SanaLoraLoaderMixin`] provides similar functions for [Sana](https://huggingface.co/docs/diffusers/main/en/api/pipelines/sana). - [`HeliosLoraLoaderMixin`] provides similar functions for [HunyuanVideo](https://huggingface.co/docs/diffusers/main/en/api/pipelines/helios). - [`HunyuanVideoLoraLoaderMixin`] provides similar functions for [HunyuanVideo](https://huggingface.co/docs/diffusers/main/en/api/pipelines/hunyuan_video). - [`Lumina2LoraLoaderMixin`] provides similar functions for [Lumina2](https://huggingface.co/docs/diffusers/main/en/api/pipelines/lumina2). - [`WanLoraLoaderMixin`] provides similar functions for [Wan](https://huggingface.co/docs/diffusers/main/en/api/pipelines/wan). - [`SkyReelsV2LoraLoaderMixin`] provides similar functions for [SkyReels-V2](https://huggingface.co/docs/diffusers/main/en/api/pipelines/skyreels_v2). - [`CogView4LoraLoaderMixin`] provides similar functions for [CogView4](https://huggingface.co/docs/diffusers/main/en/api/pipelines/cogview4). - [`AmusedLoraLoaderMixin`] is for the [`AmusedPipeline`]. - [`AnimaLoraLoaderMixin`] provides similar functions for [Anima](https://huggingface.co/docs/diffusers/main/en/api/pipelines/anima). - [`AceStepLoraLoaderMixin`] provides similar functions for [ACE-Step](https://huggingface.co/docs/diffusers/main/en/api/pipelines/ace_step). - [`HiDreamImageLoraLoaderMixin`] provides similar functions for [HiDream Image](https://huggingface.co/docs/diffusers/main/en/api/pipelines/hidream) - [`QwenImageLoraLoaderMixin`] provides similar functions for [Qwen Image](https://huggingface.co/docs/diffusers/main/en/api/pipelines/qwen). - [`ZImageLoraLoaderMixin`] provides similar functions for [Z-Image](https://huggingface.co/docs/diffusers/main/en/api/pipelines/zimage). - [`Flux2LoraLoaderMixin`] provides similar functions for [Flux2](https://huggingface.co/docs/diffusers/main/en/api/pipelines/flux2). - [`ErnieImageLoraLoaderMixin`] provides similar functions for [Ernie-Image](https://huggingface.co/docs/diffusers/main/en/api/pipelines/ernie_image). - [`LTX2LoraLoaderMixin`] provides similar functions for [Flux2](https://huggingface.co/docs/diffusers/main/en/api/pipelines/ltx2). - [`MiniMaxH3LoraLoaderMixin`] provides similar functions for [MiniMax-H3](https://huggingface.co/docs/diffusers/main/en/api/pipelines/minimax_h3). - [`LoraBaseMixin`] provides a base class with several utility methods to fuse, unfuse, unload, LoRAs and more. > [!TIP] > To learn more about how to load LoRA weights, see the [LoRA](../../tutorials/using_peft_for_inference) loading guide. ## LoraBaseMixin [[autodoc]] loaders.lora_base.LoraBaseMixin ## StableDiffusionLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.StableDiffusionLoraLoaderMixin ## StableDiffusionXLLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.StableDiffusionXLLoraLoaderMixin ## SD3LoraLoaderMixin [[autodoc]] loaders.lora_pipeline.SD3LoraLoaderMixin ## FluxLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.FluxLoraLoaderMixin ## Flux2LoraLoaderMixin [[autodoc]] loaders.lora_pipeline.Flux2LoraLoaderMixin ## ErnieImageLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.ErnieImageLoraLoaderMixin ## LTX2LoraLoaderMixin [[autodoc]] loaders.lora_pipeline.LTX2LoraLoaderMixin ## CogVideoXLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.CogVideoXLoraLoaderMixin ## Mochi1LoraLoaderMixin [[autodoc]] loaders.lora_pipeline.Mochi1LoraLoaderMixin ## AuraFlowLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.AuraFlowLoraLoaderMixin ## LTXVideoLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.LTXVideoLoraLoaderMixin ## SanaLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.SanaLoraLoaderMixin ## HeliosLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.HeliosLoraLoaderMixin ## HunyuanVideoLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.HunyuanVideoLoraLoaderMixin ## Lumina2LoraLoaderMixin [[autodoc]] loaders.lora_pipeline.Lumina2LoraLoaderMixin ## CogView4LoraLoaderMixin [[autodoc]] loaders.lora_pipeline.CogView4LoraLoaderMixin ## WanLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.WanLoraLoaderMixin ## SkyReelsV2LoraLoaderMixin [[autodoc]] loaders.lora_pipeline.SkyReelsV2LoraLoaderMixin ## AmusedLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.AmusedLoraLoaderMixin ## AnimaLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.AnimaLoraLoaderMixin ## AceStepLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.AceStepLoraLoaderMixin ## HiDreamImageLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.HiDreamImageLoraLoaderMixin ## QwenImageLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.QwenImageLoraLoaderMixin ## ZImageLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.ZImageLoraLoaderMixin ## CosmosLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.CosmosLoraLoaderMixin ## KandinskyLoraLoaderMixin [[autodoc]] loaders.lora_pipeline.KandinskyLoraLoaderMixin ## Ideogram4LoraLoaderMixin [[autodoc]] loaders.lora_pipeline.Ideogram4LoraLoaderMixin ## Krea2LoraLoaderMixin [[autodoc]] loaders.lora_pipeline.Krea2LoraLoaderMixin ## MiniMaxH3LoraLoaderMixin [[autodoc]] loaders.lora_pipeline.MiniMaxH3LoraLoaderMixin ## LoraBaseMixin [[autodoc]] loaders.lora_base.LoraBaseMixin --- ## File: docs/source/en/api/loaders/peft.md # PEFT Diffusers supports loading adapters such as [LoRA](../../tutorials/using_peft_for_inference) with the [PEFT](https://huggingface.co/docs/peft/index) library with the [`~loaders.peft.PeftAdapterMixin`] class. This allows modeling classes in Diffusers like [`UNet2DConditionModel`], [`SD3Transformer2DModel`] to operate with an adapter. > [!TIP] > Refer to the [Inference with PEFT](../../tutorials/using_peft_for_inference.md) tutorial for an overview of how to use PEFT in Diffusers for inference. ## PeftAdapterMixin [[autodoc]] loaders.peft.PeftAdapterMixin --- ## File: docs/source/en/api/loaders/single_file.md # Single files The [`~loaders.FromSingleFileMixin.from_single_file`] method allows you to load: * a model stored in a single file, which is useful if you're working with models from the diffusion ecosystem, like Automatic1111, and commonly rely on a single-file layout to store and share models * a model stored in their originally distributed layout, which is useful if you're working with models finetuned with other services, and want to load it directly into Diffusers model objects and pipelines > [!TIP] > Read the [Model files and layouts](../../using-diffusers/other-formats) guide to learn more about the Diffusers-multifolder layout versus the single-file layout, and how to load models stored in these different layouts. ## Supported pipelines - [`StableDiffusionPipeline`] - [`StableDiffusionImg2ImgPipeline`] - [`StableDiffusionInpaintPipeline`] - [`StableDiffusionControlNetPipeline`] - [`StableDiffusionControlNetImg2ImgPipeline`] - [`StableDiffusionControlNetInpaintPipeline`] - [`StableDiffusionUpscalePipeline`] - [`StableDiffusionXLPipeline`] - [`StableDiffusionXLImg2ImgPipeline`] - [`StableDiffusionXLInpaintPipeline`] - [`StableDiffusionXLInstructPix2PixPipeline`] - [`StableDiffusionXLControlNetPipeline`] - [`StableDiffusionXLKDiffusionPipeline`] - [`StableDiffusion3Pipeline`] - [`LatentConsistencyModelPipeline`] - [`LatentConsistencyModelImg2ImgPipeline`] - [`StableDiffusionControlNetXSPipeline`] - [`StableDiffusionXLControlNetXSPipeline`] - [`LEditsPPPipelineStableDiffusion`] - [`LEditsPPPipelineStableDiffusionXL`] - [`PIAPipeline`] ## Supported models - [`UNet2DConditionModel`] - [`StableCascadeUNet`] - [`AutoencoderKL`] - [`ControlNetModel`] - [`SD3Transformer2DModel`] - [`FluxTransformer2DModel`] ## FromSingleFileMixin [[autodoc]] loaders.single_file.FromSingleFileMixin ## FromOriginalModelMixin [[autodoc]] loaders.single_file_model.FromOriginalModelMixin --- ## File: docs/source/en/api/loaders/textual_inversion.md # Textual Inversion Textual Inversion is a training method for personalizing models by learning new text embeddings from a few example images. The file produced from training is extremely small (a few KBs) and the new embeddings can be loaded into the text encoder. [`TextualInversionLoaderMixin`] provides a function for loading Textual Inversion embeddings from Diffusers and Automatic1111 into the text encoder and loading a special token to activate the embeddings. > [!TIP] > To learn more about how to load Textual Inversion embeddings, see the [Textual Inversion](../../using-diffusers/textual_inversion_inference) loading guide. ## TextualInversionLoaderMixin [[autodoc]] loaders.textual_inversion.TextualInversionLoaderMixin --- ## File: docs/source/en/api/loaders/transformer_sd3.md # SD3Transformer2D This class is useful when *only* loading weights into a [`SD3Transformer2DModel`]. If you need to load weights into the text encoder or a text encoder and SD3Transformer2DModel, check [`SD3LoraLoaderMixin`](lora#diffusers.loaders.SD3LoraLoaderMixin) class instead. The [`SD3Transformer2DLoadersMixin`] class currently only loads IP-Adapter weights, but will be used in the future to save weights and load LoRAs. > [!TIP] > To learn more about how to load LoRA weights, see the [LoRA](../../tutorials/using_peft_for_inference) loading guide. ## SD3Transformer2DLoadersMixin [[autodoc]] loaders.transformer_sd3.SD3Transformer2DLoadersMixin - all - _load_ip_adapter_weights --- ## File: docs/source/en/api/loaders/unet.md # UNet Some training methods - like LoRA and Custom Diffusion - typically target the UNet's attention layers, but these training methods can also target other non-attention layers. Instead of training all of a model's parameters, only a subset of the parameters are trained, which is faster and more efficient. This class is useful if you're *only* loading weights into a UNet. If you need to load weights into the text encoder or a text encoder and UNet, try using the [`~loaders.StableDiffusionLoraLoaderMixin.load_lora_weights`] function instead. The [`UNet2DConditionLoadersMixin`] class provides functions for loading and saving weights, fusing and unfusing LoRAs, disabling and enabling LoRAs, and setting and deleting adapters. > [!TIP] > To learn more about how to load LoRA weights, see the [LoRA](../../tutorials/using_peft_for_inference) guide. ## UNet2DConditionLoadersMixin [[autodoc]] loaders.unet.UNet2DConditionLoadersMixin --- ## File: docs/source/en/api/models/allegro_transformer3d.md # AllegroTransformer3DModel A Diffusion Transformer model for 3D data from [Allegro](https://github.com/rhymes-ai/Allegro) was introduced in [Allegro: Open the Black Box of Commercial-Level Video Generation Model](https://huggingface.co/papers/2410.15458) by RhymesAI. The model can be loaded with the following code snippet. ```python from diffusers import AllegroTransformer3DModel transformer = AllegroTransformer3DModel.from_pretrained("rhymes-ai/Allegro", subfolder="transformer", dtype=torch.bfloat16).to("cuda") ``` ## AllegroTransformer3DModel [[autodoc]] AllegroTransformer3DModel ## Transformer2DModelOutput [[autodoc]] models.modeling_outputs.Transformer2DModelOutput