### Adetailer # ADetailer `sd-cli` can run a YOLOv8 object detector on an existing or newly generated image and perform a cropped inpaint pass for every detected object. The first implementation supports YOLOv8 detection checkpoints. YOLOv8 segmentation and MediaPipe models are not supported yet. ## Convert a detector Ultralytics checkpoints must be converted before use. The converter fuses BatchNorm into convolution layers and writes a safetensors file with the weight names expected by the native GGML implementation. ```bash python scripts/convert_yolov8_to_safetensors.py face_yolov8n.pt face_yolov8n.safetensors ``` The converter requires Python packages `ultralytics`, `torch`, and `safetensors`. Only YOLOv8 detection checkpoints are accepted. PyTorch checkpoints use pickle internally, so only convert `.pt` files from a trusted source. ## Repair an existing image Use the dedicated `adetailer` mode to detect and repair objects in an existing image: ```bash ./bin/sd-cli \ -M adetailer \ -m model.safetensors \ -i input.png \ -o repaired.png \ -p "detailed portrait photo" \ --negative-prompt "deformed face" \ --steps 24 \ --cfg-scale 6 \ --strength 0.4 \ --sampling-method dpm++2m \ --scheduler karras \ --ad-model face_yolov8n.safetensors \ --extra-ad-args "confidence=0.3,inpaint_padding=32,mask_blur=4" ``` This mode reuses the normal image-generation options for the detail pass: - `--init-img`, `--output`, `--prompt`, and `--negative-prompt` - `--steps`, `--cfg-scale`, `--sampling-method`, and `--scheduler` - `--strength`, `--seed`, LoRA settings, VAE tiling, and backend assignments - `--width` and `--height`, which also resize the input when specified `--ad-prompt` and `--ad-negative-prompt` optionally override the normal prompts. Values provided in `--extra-ad-args`, such as `steps`, `cfg_scale`, `denoising_strength`, or `inpaint_width`, take precedence over inherited values. ## Repair generated images ADetailer can also run automatically after normal image generation: ```bash ./bin/sd-cli \ -m model.safetensors \ -p "portrait photo" \ --ad-model face_yolov8n.safetensors \ --ad-prompt "[PROMPT], detailed face" \ --ad-negative-prompt "" \ --extra-ad-args "confidence=0.3,denoising_strength=0.4,inpaint_width=512,inpaint_height=512" ``` An empty ADetailer prompt inherits the main prompt. `[PROMPT]` inserts the main prompt, `[SEP]` assigns different prompts to consecutive masks, and `[SKIP]` skips the corresponding mask. All settings other than the detector path and prompts are passed through `--extra-ad-args` as a comma-separated `key=value` list: | Key | Default | Description | | --- | ---: | --- | | `input_size` | `640` | Square YOLO input size; must be a multiple of 32 | | `confidence` | `0.3` | Detection confidence threshold | | `nms` | `0.45` | NMS IoU threshold | | `max_detections` | `100` | Maximum detections retained after NMS | | `mask_k_largest` | `0` | Keep only the largest K masks; zero keeps all | | `mask_min_ratio` | `0` | Minimum bbox area relative to the image | | `mask_max_ratio` | `1` | Maximum bbox area relative to the image | | `dilate_erode` | `4` | Positive values dilate; negative values erode | | `x_offset`, `y_offset` | `0` | Mask offset in pixels; positive Y moves upward | | `mask_mode` | `none` | `none`, `merge`, or `merge_invert` | | `merge_masks`, `invert_mask` | `false` | Boolean alternatives to `mask_mode` | | `mask_blur` | `4` | Final composite feather radius | | `inpaint_padding` | `32` | Padding around the detected region | | `inpaint_width`, `inpaint_height` | mode-specific | `512x512` after generation; input/output size in `adetailer` mode | | `denoising_strength` | mode-specific | `0.4` after generation; inherits `--strength` in `adetailer` mode | | `steps` | `0` | Detail steps; zero inherits the main generation | | `cfg_scale` | `-1` | Detail CFG; a negative value inherits the main generation | | `sample_method` | inherited | Detail sampler name | | `scheduler` | inherited | Detail scheduler name | | `sort_by` | `none` | `none`, `left_to_right`, `center_to_edge`, or `area` | Multiple masks are processed serially. Each completed inpaint becomes the input for the next mask, and the seed is incremented by the mask index. Use `mask_mode=merge` to process all detections in one inpaint pass. The detector uses the `detector` backend module. For example, keep detection on the CPU while diffusion runs on CUDA: ```bash --backend "diffusion=cuda0,detector=cpu" ``` --- ### Anima # How to Use ## Download weights - Download Anima - safetensors: https://huggingface.co/circlestone-labs/Anima/tree/main/split_files/diffusion_models - gguf: https://huggingface.co/Bedovyy/Anima-GGUF/tree/main - gguf Anima2: https://huggingface.co/JusteLeo/Anima2-GGUF/tree/main - Download vae - safetensors: https://huggingface.co/circlestone-labs/Anima/tree/main/split_files/vae - Download Qwen3-0.6B-Base - safetensors: https://huggingface.co/circlestone-labs/Anima/tree/main/split_files/text_encoders - gguf: https://huggingface.co/mradermacher/Qwen3-0.6B-Base-GGUF/tree/main ## Examples ```sh .\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\anima-preview.safetensors --vae ..\models\vae\qwen_image_vae.safetensors --llm ..\models\text_encoders\qwen_3_06b_base.safetensors -p "a lovely cat holding a sign says 'anima.cpp'" --cfg-scale 6.0 --sampling-method euler -v --offload-to-cpu --diffusion-fa ``` anima image example --- ### Animatediff # AnimateDiff (SD 1.5) AnimateDiff adds motion to a frozen Stable Diffusion 1.5 checkpoint by injecting a temporal-attention module at 20 UNet slots. The base SD 1.5 model, VAE, and text encoder are unchanged; only the motion module produces the temporal residual that turns a batch of independent frames into a coherent animation. Reference: Guo et al., "AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning" (https://arxiv.org/abs/2307.04725). ## Download weights - Motion module (v3, recommended) - fp16 safetensors: https://huggingface.co/conrevo/AnimateDiff-A1111/resolve/main/motion_module/mm_sd15_v3.safetensors - original checkpoint: https://huggingface.co/guoyww/animatediff/resolve/main/v3_sd15_mm.ckpt - SD 1.5 base model - any SD 1.5 checkpoint works. `realisticVisionV60B1` and `toonyou_beta3` are the ones used in guoyww's reference configs. - Domain Adapter LoRA (optional, v3 only, sharpens the base UNet's output toward the motion module's trained distribution) - ckpt: https://huggingface.co/guoyww/animatediff/resolve/main/v3_sd15_adapter.ckpt - place under your `--lora-model-dir` and reference in the prompt as ``. The motion module is `~836 MB` and loads alongside the SD 1.5 UNet via `--motion-module`. ## Motion module versions Per [animatediff.net/models](https://animatediff.net/models): | Module | Base | Native res | Character | |---------------------|------|------------|-----------| | `mm_sd_v14.ckpt` | 1.5 | 256x256 | earliest, more jittery | | `mm_sd_v15.ckpt` | 1.5 | 256x256 | improved stability over v1.4 | | `mm_sd_v15_v2.ckpt` | 1.5 | 384x384 | significantly better motion dynamics | | `v3_sd15_mm.ckpt` | 1.5 | 512x512 | smoothest, highest quality; pairs with a Domain Adapter LoRA | | `mm_sdxl_v10_beta` | SDXL | 512x512 | experimental, not yet supported here | Match your `-H -W` to the module's native resolution for best results. v3 is trained at 512x512 - going smaller (e.g. 384x384) still works but the motion character is closer to v2. ## Examples Generate an 8-frame animation at 512x512, seed 42, 20 steps. The sampler / scheduler / CFG values below match what mm_sd15_v3 was trained with; using SD 1.5 defaults (euler_a, low CFG) produces noise-like output. ``` .\bin\Release\sd-cli.exe -M vid_gen \ --model ..\models\checkpoints\realisticVisionV60B1.safetensors \ --motion-module ..\models\animatediff\mm_sd15_v3.safetensors \ --offload-to-cpu --diffusion-fa \ -p "a red apple on a wooden table" \ --cfg-scale 8.0 --sampling-method euler --scheduler discrete \ -H 512 -W 512 --video-frames 8 --fps 8 --steps 20 -s 42 \ -o out.avi ``` Generate at the motion module's native 16-frame context (recommended for best temporal quality). Needs more VRAM at 512x512, so drop to 384x384 or use layer streaming: ``` .\bin\Release\sd-cli.exe -M vid_gen \ --model ..\models\checkpoints\realisticVisionV60B1.safetensors \ --motion-module ..\models\animatediff\mm_sd15_v3.safetensors \ --offload-to-cpu --diffusion-fa \ -p "photo of coastline, rocks, storm weather, wind, waves, lightning" \ --cfg-scale 8.0 --sampling-method euler --scheduler discrete \ -H 384 -W 384 --video-frames 16 --fps 8 --steps 20 -s 42 \ -o out.avi ``` Low-VRAM streaming (verified with a 2 GiB cap on RTX 3060): ``` .\bin\Release\sd-cli.exe -M vid_gen \ --model ..\models\checkpoints\realisticVisionV60B1.safetensors \ --motion-module ..\models\animatediff\mm_sd15_v3.safetensors \ --max-vram 2.0 --stream-layers --diffusion-fa \ -p "photo of coastline, rocks, storm weather, wind, waves, lightning" \ --cfg-scale 8.0 --sampling-method euler --scheduler discrete \ -H 384 -W 384 --video-frames 8 --fps 8 --steps 20 -s 42 \ -o out.avi ``` ## Reference-quality reproduction Using guoyww's official reference configs on this impl (RealisticVision v6.0 base + `mm_sd15_v3` or `mm_sd_v15_v2` + native resolution + 16 frames + euler + 25 steps + CFG 8 + linear beta schedule) reproduces the reference AnimateDiff output style. ### v3 (512x512, `mm_sd15_v3`) | Prompt | Sample | |---------------------------------------|--------| | B&W man on stormy coastline | | | Close-up rabbit macro shot | | | Coastline, storm, waves, lightning | | | Old house, storm, forest, night | | ### v2 (384x384, `mm_sd_v15_v2.ckpt`) | Prompt | Sample | |---------------------------------------|--------| | B&W man on stormy coastline | | | Close-up rabbit macro shot | | | Coastline, storm, waves, lightning | | | Old house, storm, forest, night | | Motion is strong for scenes with motion cues in the prompt (storm/waves/wind) and subtle for static subjects (close-up macro), matching reference behavior. v2 has an additional motion module at the UNet middle block that v3 dropped; this impl auto-detects the topology from the checkpoint. ### v3 + Domain Adapter LoRA Attaching the `v3_sd15_adapter` LoRA sharpens the base UNet output toward the training distribution the motion module was fine-tuned against. Same prompt, seed, config as above: Individual fur strands, glowing inner-ear, and richer forest detail become visible compared to the no-LoRA rendering. ``` sd-cli -M vid_gen --model realisticVisionV60B1.safetensors \ --motion-module mm_sd15_v3.safetensors \ --lora-model-dir ./loras \ -p "close up photo of a rabbit ..." ... ``` ## img2video Pass a pre-rendered image via `-i / --init-img` to animate FROM it. All N output frames start from the encoded init latent, then per-frame noise is added at `--strength`. Character identity, composition, and quality are anchored by the init image; the motion module adds subtle motion on top. Left: init image rendered with `-M img_gen`. Right: 8-frame vid_gen output. ``` sd-cli -M img_gen ... -o init.png # any high-quality still sd-cli -M vid_gen --motion-module mm_sd15_v3.safetensors \ -i init.png --strength 0.75 \ --cfg-scale 7.0 --sampling-method euler --scheduler karras \ -H 512 -W 512 --video-frames 8 --steps 25 -s 42 \ -p "..." -o out.avi ``` `--strength` controls how far the motion module is allowed to deviate from the init image (higher = more motion, lower = more static). ## Notes - The motion module was trained at `video_length=16`. Running with `--video-frames 16` gives the best coherence; F=8 works but shows a shorter motion arc. Frame counts up to 32 are supported by the positional encoding but exceed the trained regime and produce more static output. - At `--video-frames 1` the motion module is skipped entirely and the output is bit-identical to `-M img_gen`. This avoids the single-token temporal-attention degeneracy that would otherwise emit an untrained-magnitude residual on a single-frame sample. - The base UNet is frozen, so character identity and style follow the SD 1.5 checkpoint you pass to `--model`. LoRAs and prompt weighting attach to the base model in the usual way. - No mid_block motion module in v3. `mm_sdxl_v10_beta` (SDXL variant) is not supported yet. - Output is written as MJPEG AVI. Use `--fps` to set playback speed. --- ### Backend # Backend selection `stable-diffusion.cpp` has two backend assignments: - `--backend` selects the runtime backend used to execute model graphs. - `--params-backend` selects where model parameters are kept. If `--params-backend` is not set, parameters use the same backend as their module runtime backend. ## Syntax A backend assignment can be a single backend name: ```shell sd-cli -m model.safetensors -p "a cat" --backend cpu ``` This applies to every module that does not have a more specific assignment. Assignments can also target individual modules: ```shell sd-cli -m model.safetensors -p "a cat" --backend te=cpu,vae=cuda0,diffusion=vulkan0 ``` The same syntax is used for parameter placement: ```shell sd-cli -m model.safetensors -p "a cat" --backend cuda0 --params-backend te=cpu,vae=cpu ``` `--params-backend` also accepts the special value `disk`: ```shell sd-cli -m model.safetensors -p "a cat" --backend cuda0 --params-backend disk ``` `--max-vram` can target resolved backend/device names: ```shell sd-cli -m model.safetensors -p "a cat" --backend diffusion=cuda0,vae=vulkan0 --max-vram cuda0=6,vulkan0=2 ``` The budget applies to every module running on that backend. Module names are case-insensitive. Hyphens and underscores in module names are ignored, so `clip_vision`, `clip-vision`, and `clipvision` are equivalent. `all=`, `default=`, and `*=` can be used to set the default backend inside a mixed assignment: ```shell sd-cli -m model.safetensors -p "a cat" --backend all=cuda0,te=cpu ``` ## Multiple devices per module (layer split) A `--backend` module assignment can list several devices separated by `&`: ```shell sd-cli -m model.safetensors -p "a cat" --backend "diffusion=cuda0&cuda1" ``` The module's transformer blocks are then distributed across the listed devices in contiguous ranges sized proportionally to each device's free memory (minus a compute-buffer headroom of about 2 GiB per device), and the module's graphs are executed with a `ggml_backend_sched` that runs each block on the device holding its weights, copying the residual stream at the range boundaries. The first device in the list is the module's main device: it also holds the non-block tensors (embeddings, final norms, small sub-runners such as CLIP models or projectors) and the graph inputs/outputs. Layer split is supported for the `diffusion` and `te` modules. For `te` it applies to the dominant text encoder (`t5xxl` or the LLM); other modules accept only a single device. If the module has no recognizable transformer blocks, the assignment falls back to the first listed device. `--params-backend` accepts no device lists. If the module has no explicit params assignment, each block range's parameters are loaded directly to (and, with `--params-backend diffusion=disk`, released directly from) its own device; an explicit assignment such as `te=cpu` keeps the parameters on that backend and stages each range to its device on demand. Layer split cannot be combined with `--max-vram` graph-cut segmentation or `--stream-layers` for the split module; those are single-device mechanisms and are disabled for it. Use `--list-devices` to see the device names available on the system. ### Row split (`--split-mode row`) `--split-mode` selects how a multi-device module distributes its weights: `layer` (the default, described above) or `row`. It accepts a single mode or per-module assignments: ```shell sd-cli -m model.safetensors -p "a cat" --backend "diffusion=cuda0&cuda1" --split-mode row sd-cli -m model.safetensors -p "a cat" --backend "diffusion=cuda0&cuda1,te=cuda0&cuda1" --split-mode diffusion=row,te=layer ``` In row mode the module keeps executing on its main (first listed) device, but its transformer-block matmul weights are allocated in the backend's row-split buffer type, which slices each weight's rows across the listed devices in proportion to free memory and runs those matmuls on all devices in parallel. Compared to a layer split this uses all GPUs within every layer (instead of sequentially device by device) at the cost of a cross-device reduction per matmul - usually the faster option when the devices have fast interconnect. Row split requires backend support for split buffers and is currently available on CUDA only; on other backends (or when the listed devices belong to different backend registries) the module falls back to a layer split. Embeddings, normalization weights, biases and other non-block tensors stay in regular buffers on the main device. Direct ("immediately") LoRA application cannot patch row-split tensors; with `--split-mode row` the automatic LoRA mode selects runtime application, and an explicit `--lora-apply-mode immediately` skips the split tensors with a warning. ## Automatic placement (`--auto-fit`) `--auto-fit` derives the `diffusion` / `te` / `vae` placements from the model metadata and the per-device memory budgets, then feeds them into the same backend assignment mechanism described above (the chosen specs are printed). `--backend` and `--params-backend` are ignored while auto-fit is enabled. ```shell sd-cli -m model.safetensors -p "a cat" --auto-fit sd-cli -m model.safetensors -p "a cat" --auto-fit --max-vram cuda0=8,cuda1=14 sd-cli -m model.safetensors -p "a cat" --auto-fit --split-mode row ``` Budgets reuse `--max-vram`: a positive per-device value caps what auto-fit plans with on that device, a negative value means "free memory minus that many GiB", and with no budget set each device's free memory minus a 512 MiB margin is used. (The same values still drive graph-cut segmented execution for modules that end up on a single device.) When everything fits resident, components are simply spread across the available GPUs. When it does not, auto-fit switches to time-share mode: the heavy components get `disk` params residency (loaded for their phase, freed after), and a component too large for any single device is split across all GPUs with the layer/row split mechanism (`--split-mode` selects which, layer by default). Components that fit nowhere fall back to the CPU. If a VAE decode still runs out of memory, tiling is enabled and the decode retried once. ## Modules | Module | Purpose | Accepted names | | --- | --- | --- | | `diffusion` | UNet, DiT, MMDiT, Flux, Wan, Qwen Image, and other diffusion models | `diffusion`, `model`, `unet`, `dit` | | `te` | Text encoders and conditioners | `te`, `clip`, `text`, `textencoder`, `textencoders`, `conditioner`, `cond`, `llm`, `t5`, `t5xxl` | | `clip_vision` | CLIP vision encoder | `clip_vision`, `clipvision`, `clip-vision`, `vision` | | `vae` | VAE and TAE | `vae`, `firststage`, `autoencoder`, `tae` | | `controlnet` | ControlNet | `controlnet`, `control` | | `photomaker` | PhotoMaker ID encoder and PhotoMaker LoRA | `photomaker`, `photomakerid`, `pmid`, `photo` | | `upscaler` | ESRGAN upscaler | `upscaler`, `esrgan`, `hires` | | `detector` | ADetailer YOLOv8 detector | `detector`, `adetailer`, `yolo` | `te` is the preferred module name for text encoders. `clip` is kept as an accepted alias because many existing commands and model names use CLIP terminology. ## Backend names Backend names are resolved against the GGML backend device list. Matching is case-insensitive and accepts exact names or unique prefixes, so common values include names such as: - `cpu` - `cuda0` - `vulkan0` - `metal` The special values `auto`, `default`, and an empty backend name select the default backend. The default preference is GPU, then integrated GPU, then CPU. The special value `gpu` selects the first GPU backend, falling back to the first integrated GPU backend. The special value `disk` is accepted only by `--params-backend`. `--backend disk` is invalid because `disk` is a parameter residency mode, not a runtime compute backend. ## Runtime backend vs. parameter backend The runtime backend controls where graph execution runs. The parameter backend controls where model weights are allocated or whether they are reloaded from disk on demand. For example: ```shell sd-cli -m model.safetensors -p "a cat" --backend cuda0 --params-backend cpu ``` This runs all modules on `cuda0`, but stores parameters in CPU RAM. During execution, parameters are moved to the runtime backend as needed. For example: ```shell sd-cli -m model.safetensors -p "a cat" --backend cuda0 --params-backend disk ``` This runs all modules on `cuda0`, reloads parameters from the model file as needed, and releases those parameter buffers after use. `disk` is never selected implicitly. If `--params-backend` is not set, parameters use the runtime backend. Per-module assignments can be mixed: ```shell sd-cli -m model.safetensors -p "a cat" --backend diffusion=cuda0,te=cpu,vae=cpu --params-backend diffusion=cuda0,te=cpu,vae=cpu ``` This keeps text encoding and VAE execution on CPU while the diffusion model runs on GPU. ## Backend sharing and lifetime Backends are managed by `SDBackendManager`. Within one manager, backend instances are cached by resolved backend device name. If multiple modules request the same backend, they share the same `ggml_backend_t`. For example: ```shell --backend te=cpu,vae=cpu ``` uses one shared CPU backend for both `te` and `vae` runtime execution. Runtime and parameter assignments also share the same backend cache. If `--backend diffusion=cuda0` and `--params-backend diffusion=cuda0` resolve to the same device, both use the same backend instance. `--params-backend disk` does not create a separate backend instance. Parameters are loaded lazily using the module runtime backend. `SDBackendManager` owns the backend instances and frees them when the context or upscaler is destroyed. Model runners receive non-owning runtime and parameter backend pointers and do not free them. ## Compatibility flags The example CLI/server still accepts these older CPU placement flags as compatibility aliases: - `--clip-on-cpu` - `--vae-on-cpu` - `--control-net-cpu` - `--offload-to-cpu` `--clip-on-cpu`, `--vae-on-cpu`, and `--control-net-cpu` are deprecated. The example argument layer prepends `te=cpu`, `vae=cpu`, and `controlnet=cpu` to `--backend` before creating the context. `--offload-to-cpu` prepends a CPU default to the parameter assignment in the caller before creating the context: ```shell --params-backend '*=cpu' ``` Because this default is inserted first, later explicit `--params-backend` entries can still override it, for example `--offload-to-cpu --params-backend te=disk` keeps non-TE parameters on CPU and reloads TE parameters from disk. Library callers should set `backend` and `params_backend` directly. The old CPU/offload fields are no longer part of the C API. Explicit `--backend` and `--params-backend` assignments are preferred for new commands. --- ### Boogu Image # How to Use Boogu Image uses a Boogu diffusion transformer, the FLUX VAE, and Qwen3-VL as the LLM text and vision encoder. ## Download weights - Download Boogu Image - safetensors: https://huggingface.co/Comfy-Org/Boogu-Image/tree/main/diffusion_models - Download vae - safetensors: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/ae.safetensors - Download Qwen3-VL 8B - gguf: https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-GGUF/tree/main - For image editing with GGUF text encoders, also download the matching mmproj file and pass it with `--llm_vision`. ## Examples ### Boogu Image Base ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\boogu_image_base_bf16.safetensors --llm ..\models\text_encoders\Qwen3VL-8B-Instruct-Q4_K_M.gguf --vae ..\models\vae\ae.sft -p "a lovely cat" --diffusion-fa -v --offload-to-cpu ``` Boogu Image Base example ### Boogu Image Edit ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\boogu_image_edit_bf16.safetensors --llm ..\models\text_encoders\Qwen3VL-8B-Instruct-Q4_K_M.gguf --llm_vision ..\models\text_encoders\mmproj-Qwen3VL-8B-Instruct-F16.gguf --vae ..\models\vae\ae.sft --diffusion-fa -v --offload-to-cpu -r ..\assets\flux\flux1-dev-q8_0.png -p "change 'flux.cpp' to 'boogu.cpp'" ``` Boogu Image Edit example --- ### Build # Build from scratch ## Get the Code ``` git clone --recursive https://github.com/leejet/stable-diffusion.cpp cd stable-diffusion.cpp ``` - If you have already cloned the repository, you can use the following command to update the repository to the latest code. ``` cd stable-diffusion.cpp git pull origin master git submodule init git submodule update ``` ## WebP and WebM Support in Examples The example applications (`examples/cli` and `examples/server`) use `libwebp` to support WebP image I/O, and `examples/cli` can also use `libwebm` for `.webm` video output. Both are enabled by default. WebM output currently reuses `libwebp` to encode each frame as VP8 before muxing with `libwebm`. If you do not want WebP/WebM support, you can disable them at configure time: ```shell mkdir build && cd build cmake .. -DSD_WEBP=OFF -DSD_WEBM=OFF cmake --build . --config Release ``` If the submodules are not available, you can also link against system packages instead: ```shell mkdir build && cd build cmake .. -DSD_USE_SYSTEM_WEBP=ON -DSD_USE_SYSTEM_WEBM=ON cmake --build . --config Release ``` ## Build (CPU only) If you don't have a GPU or CUDA installed, you can build a CPU-only version. ```shell mkdir build && cd build cmake .. cmake --build . --config Release ``` ## Build with OpenBLAS ```shell mkdir build && cd build cmake .. -DGGML_OPENBLAS=ON cmake --build . --config Release ``` ## Build with CUDA This provides GPU acceleration using NVIDIA GPU. Make sure to have the CUDA toolkit installed. You can download it from your Linux distro's package manager (e.g. `apt install nvidia-cuda-toolkit`) or from here: [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads). Recommended to have at least 4 GB of VRAM. ```shell mkdir build && cd build cmake .. -DSD_CUDA=ON cmake --build . --config Release ``` ## Build with HipBLAS This provides GPU acceleration using AMD GPU. Make sure to have the ROCm toolkit installed. To build for another GPU architecture than installed in your system, set `$GFX_NAME` manually to the desired architecture (replace first command). This is also necessary if your GPU is not officially supported by ROCm, for example you have to set `$GFX_NAME` manually to `gfx1030` for consumer RDNA2 cards. Windows User Refer to [docs/hipBLAS_on_Windows.md](docs%2FhipBLAS_on_Windows.md) for a comprehensive guide. ```shell mkdir build && cd build if command -v rocminfo; then export GFX_NAME=$(rocminfo | awk '/ *Name: +gfx[1-9]/ {print $2; exit}'); else echo "rocminfo missing!"; fi if [ -z "${GFX_NAME}" ]; then echo "Error: Couldn't detect GPU!"; else echo "Building for GPU: ${GFX_NAME}"; fi cmake .. -G "Ninja" -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DSD_HIPBLAS=ON -DCMAKE_BUILD_TYPE=Release -DGPU_TARGETS=$GFX_NAME -DAMDGPU_TARGETS=$GFX_NAME -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON -DCMAKE_POSITION_INDEPENDENT_CODE=ON cmake --build . --config Release ``` ## Build with MUSA This provides GPU acceleration using Moore Threads GPU. Make sure to have the MUSA toolkit installed. ```shell mkdir build && cd build cmake .. -DCMAKE_C_COMPILER=/usr/local/musa/bin/clang -DCMAKE_CXX_COMPILER=/usr/local/musa/bin/clang++ -DSD_MUSA=ON -DCMAKE_BUILD_TYPE=Release cmake --build . --config Release ``` ## Build with Metal Using Metal makes the computation run on the GPU. Currently, there are some issues with Metal when performing operations on very large matrices, making it highly inefficient at the moment. Performance improvements are expected in the near future. ```shell mkdir build && cd build cmake .. -DSD_METAL=ON cmake --build . --config Release ``` ## Build with Vulkan Install Vulkan SDK from https://www.lunarg.com/vulkan-sdk/. On Ubuntu, install the Vulkan development packages and SPIR-V headers: ```shell sudo apt-get install build-essential libvulkan-dev glslc spirv-headers ``` ```shell mkdir build && cd build cmake .. -DSD_VULKAN=ON cmake --build . --config Release ``` ## Build with OpenCL (for Adreno GPU) Currently, it supports only Adreno GPUs and is primarily optimized for Q4_0 type To build for Windows ARM please refers to [Windows 11 Arm64](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/OPENCL.md#windows-11-arm64) Building for Android: Android NDK: Download and install the Android NDK from the [official Android developer site](https://developer.android.com/ndk/downloads). Setup OpenCL Dependencies for NDK: You need to provide OpenCL headers and the ICD loader library to your NDK sysroot. * OpenCL Headers: ```bash # In a temporary working directory git clone https://github.com/KhronosGroup/OpenCL-Headers cd OpenCL-Headers # Replace with your actual NDK installation path # e.g., cp -r CL /path/to/android-ndk-r26c/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include sudo cp -r CL /toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include cd .. ``` * OpenCL ICD Loader: ```shell # In the same temporary working directory git clone https://github.com/KhronosGroup/OpenCL-ICD-Loader cd OpenCL-ICD-Loader mkdir build_ndk && cd build_ndk # Replace in the CMAKE_TOOLCHAIN_FILE and OPENCL_ICD_LOADER_HEADERS_DIR cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_TOOLCHAIN_FILE=/build/cmake/android.toolchain.cmake \ -DOPENCL_ICD_LOADER_HEADERS_DIR=/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include \ -DANDROID_ABI=arm64-v8a \ -DANDROID_PLATFORM=24 \ -DANDROID_STL=c++_shared ninja # Replace # e.g., cp libOpenCL.so /path/to/android-ndk-r26c/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android sudo cp libOpenCL.so /toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android cd ../.. ``` Build `stable-diffusion.cpp` for Android with OpenCL: ```shell mkdir build-android && cd build-android # Replace with your actual NDK installation path # e.g., -DCMAKE_TOOLCHAIN_FILE=/path/to/android-ndk-r26c/build/cmake/android.toolchain.cmake cmake .. -G Ninja \ -DCMAKE_TOOLCHAIN_FILE=/build/cmake/android.toolchain.cmake \ -DANDROID_ABI=arm64-v8a \ -DANDROID_PLATFORM=android-28 \ -DGGML_OPENMP=OFF \ -DSD_OPENCL=ON ninja ``` *(Note: Don't forget to include `LD_LIBRARY_PATH=/vendor/lib64` in your command line before running the binary)* ## Build with SYCL Using SYCL makes the computation run on the Intel GPU. Please make sure you have installed the related driver and [Intel® oneAPI Base toolkit](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit.html) before start. More details and steps can refer to [llama.cpp SYCL backend](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/SYCL.md#linux). ```shell # Export relevant ENV variables source /opt/intel/oneapi/setvars.sh # Option 1: Use FP32 (recommended for better performance in most cases) cmake .. -DSD_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx # Option 2: Use FP16 cmake .. -DSD_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DGGML_SYCL_F16=ON cmake --build . --config Release ``` --- ### Caching ## Caching Caching methods accelerate diffusion inference by reusing intermediate computations when changes between steps are small. ### Cache Modes | Mode | Target | Description | |------|--------|-------------| | `ucache` | UNET models | Condition-level caching with error tracking | | `easycache` | DiT models | Condition-level cache | | `dbcache` | DiT models | Block-level L1 residual threshold | | `taylorseer` | DiT models | Taylor series approximation | | `cache-dit` | DiT models | Combined DBCache + TaylorSeer | | `spectrum` | UNET and DiT models | Chebyshev + Taylor output forecasting | ### UCache (UNET Models) UCache caches the residual difference (output - input) and reuses it when input changes are below threshold. ```bash sd-cli -m model.safetensors -p "a cat" --cache-mode ucache --cache-option "threshold=1.5" ``` #### Parameters | Parameter | Description | Default | |-----------|-------------|---------| | `threshold` | Error threshold for reuse decision | 1.0 | | `start` | Start caching at this percent of steps | 0.15 | | `end` | Stop caching at this percent of steps | 0.95 | | `decay` | Error decay rate (0-1) | 1.0 | | `relative` | Scale threshold by output norm (0/1) | 1 | | `reset` | Reset error after computing (0/1) | 1 | #### Reset Parameter The `reset` parameter controls error accumulation behavior: - `reset=1` (default): Resets accumulated error after each computed step. More aggressive caching, works well with most samplers. - `reset=0`: Keeps error accumulated. More conservative, recommended for `euler_a` sampler. ### EasyCache (DiT Models) Condition-level caching for DiT models. Caches and reuses outputs when input changes are below threshold. ```bash --cache-mode easycache --cache-option "threshold=0.3" ``` #### Parameters | Parameter | Description | Default | |-----------|-------------|---------| | `threshold` | Input change threshold for reuse | 0.2 | | `start` | Start caching at this percent of steps | 0.15 | | `end` | Stop caching at this percent of steps | 0.95 | ### Cache-DIT (DiT Models) For DiT models like FLUX and QWEN, use block-level caching modes. #### DBCache Caches blocks based on L1 residual difference threshold: ```bash --cache-mode dbcache --cache-option "threshold=0.25,warmup=4" ``` #### TaylorSeer Uses Taylor series approximation to predict block outputs: ```bash --cache-mode taylorseer ``` #### Cache-DIT (Combined) Combines DBCache and TaylorSeer: ```bash --cache-mode cache-dit ``` #### Parameters | Parameter | Description | Default | |-----------|-------------|---------| | `Fn` | Front blocks to always compute | 8 | | `Bn` | Back blocks to always compute | 0 | | `threshold` | L1 residual difference threshold | 0.08 | | `warmup` | Steps before caching starts | 8 | #### SCM Options Steps Computation Mask controls which steps can be cached: ```bash --scm-mask "1,1,1,1,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,1" ``` Mask values: `1` = compute, `0` = can cache. | Policy | Description | |--------|-------------| | `dynamic` | Check threshold before caching | | `static` | Always cache on cacheable steps | ```bash --scm-policy dynamic ``` ### Spectrum (UNET and DiT Models) Spectrum uses Chebyshev polynomial fitting blended with Taylor extrapolation to predict denoised outputs, skipping entire forward passes. Based on the paper [Spectrum: Adaptive Spectral Feature Forecasting for Efficient Diffusion Sampling](https://github.com/tingyu215/Spectrum). ```bash sd-cli -m model.safetensors -p "a cat" --cache-mode spectrum ``` #### Parameters | Parameter | Description | Default | |-----------|-------------|---------| | `w` | Chebyshev vs Taylor blend weight (0=Taylor, 1=Chebyshev) | 0.40 | | `m` | Chebyshev polynomial degree | 3 | | `lam` | Ridge regression regularization | 1.0 | | `window` | Initial window size (compute every N steps) | 2 | | `flex` | Window growth per computed step after warmup | 0.50 | | `warmup` | Steps to always compute before caching starts | 4 | | `stop` | Stop caching at this fraction of total steps | 0.9 | ### Performance Tips - Start with default thresholds and adjust based on output quality - Lower threshold = better quality, less speedup - Higher threshold = more speedup, potential quality loss - More steps generally means more caching opportunities --- ### Chroma # How to Use You can run Chroma using stable-diffusion.cpp with a GPU that has 6GB or even 4GB of VRAM, without needing to offload to RAM. ## Download weights - Download Chroma - If you don't want to do the conversion yourself, download the preconverted gguf model from [silveroxides/Chroma-GGUF](https://huggingface.co/silveroxides/Chroma-GGUF) - Otherwise, download chroma's safetensors from [lodestones/Chroma](https://huggingface.co/lodestones/Chroma) - Download vae from https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/ae.safetensors - Download t5xxl from https://huggingface.co/comfyanonymous/flux_text_encoders/blob/main/t5xxl_fp16.safetensors ## Convert Chroma weights You can download the preconverted gguf weights from [silveroxides/Chroma-GGUF](https://huggingface.co/silveroxides/Chroma-GGUF), this way you don't have to do the conversion yourself. ``` .\bin\Release\sd-cli.exe -M convert -m ..\models\unet\chroma-unlocked-v40.safetensors -o ..\models\chroma-unlocked-v40-q8_0.gguf -v --type q8_0 ``` ## Run ### Example For example: ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\chroma-unlocked-v40-q8_0.gguf --vae ..\models\ae.sft --t5xxl ..\models\t5xxl_fp16.safetensors -p "a lovely cat holding a sign says 'chroma.cpp'" --cfg-scale 4.0 --sampling-method euler -v --model-args chroma_use_dit_mask=false --clip-on-cpu ``` --- ### Chroma Radiance # How to Use ## Download weights - Download Chroma1-Radiance - safetensors: https://huggingface.co/lodestones/Chroma1-Radiance/tree/main - gguf: https://huggingface.co/silveroxides/Chroma1-Radiance-GGUF/tree/main - Download t5xxl - safetensors: https://huggingface.co/comfyanonymous/flux_text_encoders/blob/main/t5xxl_fp16.safetensors ## Examples ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\Chroma1-Radiance-v0.4-Q8_0.gguf --t5xxl ..\models\clip\t5xxl_fp16.safetensors -p "a lovely cat holding a sign says 'chroma radiance cpp'" --cfg-scale 4.0 --sampling-method euler -v ``` Chroma1-Radiance --- ### Distilled Sd # Running distilled models: SSD1B, Vega and SDx.x with tiny U-Nets ## Preface These models feature a reduced U-Net architecture. Unlike standard SDXL models, the SSD-1B and Vega U-Net contains only one middle block and fewer attention layers in its up- and down-blocks, resulting in significantly smaller file sizes. Using these models can reduce inference time by more than 33%. For more details, refer to Segmind's paper: https://arxiv.org/abs/2401.02677v1. Similarly, SD1.x- and SD2.x-style models with a tiny U-Net consist of only 6 U-Net blocks, leading to very small files and time savings of up to 50%. For more information, see the paper: https://arxiv.org/pdf/2305.15798.pdf. ## SSD1B Note that not all of these models follow the standard parameter naming conventions. However, several useful SSD-1B models are available online, such as: * https://huggingface.co/segmind/SSD-1B/resolve/main/SSD-1B-A1111.safetensors * https://huggingface.co/hassenhamdi/SSD-1B-fp8_e4m3fn/resolve/main/SSD-1B_fp8_e4m3fn.safetensors Useful LoRAs are also available: * https://huggingface.co/seungminh/lora-swarovski-SSD-1B/resolve/main/pytorch_lora_weights.safetensors * https://huggingface.co/kylielee505/mylcmlorassd/resolve/main/pytorch_lora_weights.safetensors ## Vega Segmind's Vega model is available online here: * https://huggingface.co/segmind/Segmind-Vega/resolve/main/segmind-vega.safetensors VegaRT is an example for an LCM-LoRA: * https://huggingface.co/segmind/Segmind-VegaRT/resolve/main/pytorch_lora_weights.safetensors Both files can be used out-of-the-box, unlike the models described in next sections. ## SD1.x, SD2.x with tiny U-Nets These models require conversion before use. You will need a Python script provided by the diffusers team, available on GitHub: * https://raw.githubusercontent.com/huggingface/diffusers/refs/heads/main/scripts/convert_diffusers_to_original_stable_diffusion.py ### SD2.x NotaAI provides the following model online: * https://huggingface.co/nota-ai/bk-sdm-v2-tiny Creating a .safetensors file involves two steps. First, run this short Python script to download the model from Hugging Face: ```python from diffusers import StableDiffusionPipeline pipe = StableDiffusionPipeline.from_pretrained("nota-ai/bk-sdm-v2-tiny",cache_dir="./") ``` Second, create the .safetensors file by running: ```bash python convert_diffusers_to_original_stable_diffusion.py \ --model_path models--nota-ai--bk-sdm-v2-tiny/snapshots/68277af553777858cd47e133f92e4db47321bc74 \ --checkpoint_path bk-sdm-v2-tiny.safetensors --half --use_safetensors ``` This will generate the **file bk-sdm-v2-tiny.safetensors**, which is now ready for use with sd.cpp. ### SD1.x Several Tiny SD 1.x models are available online, such as: * https://huggingface.co/segmind/tiny-sd * https://huggingface.co/segmind/portrait-finetuned * https://huggingface.co/nota-ai/bk-sdm-tiny These models also require conversion, partly because some tensors are stored in a non-contiguous manner. To create a usable checkpoint file, follow these simple steps: Download and prepare the model using Python: ##### Download the model using Python on your computer, for example this way: ```python import torch from diffusers import StableDiffusionPipeline pipe = StableDiffusionPipeline.from_pretrained("segmind/tiny-sd") unet=pipe.unet for param in unet.parameters(): param.data = param.data.contiguous() # <- important here pipe.save_pretrained("segmindtiny-sd", safe_serialization=True) ``` ##### Run the conversion script: ```bash python convert_diffusers_to_original_stable_diffusion.py \ --model_path ./segmindtiny-sd \ --checkpoint_path ./segmind_tiny-sd.safetensors --half --use_safetensors ``` The file segmind_tiny-sd.safetensors will be generated and is now ready for use with sd.cpp. You can follow a similar process for the other models mentioned above. ### SDXS-512-DreamShaper Another very tiny and **incredibly fast** model is SDXS by IDKiro et al. The authors refer to it as *"Real-Time One-Step Latent Diffusion Models with Image Conditions"*. For details read the paper: https://arxiv.org/pdf/2403.16627 . Once again the authors removed some more blocks of U-Net part and unlike other SD1 models they use an adjusted _AutoEncoderTiny_ instead of default _AutoEncoderKL_ for the VAE part. ##### Some ready-to-run SDXS-512 model files are available online, such as: * https://huggingface.co/akleine/sdxs-512 * https://huggingface.co/concedo/sdxs-512-tinySDdistilled-GGUF ##### Run the model as follows: ```bash ~/stable-diffusion.cpp/build/bin/sd-cli -m sdxs.safetensors -p "portrait of a lovely cat" \ --cfg-scale 1 --steps 1 ``` Both options: ``` --cfg-scale 1 ``` and ``` --steps 1 ``` are mandatory here. ### SDXS-512-0.9 Even though the name "SDXS-512-0.9" is similar to "SDXS-512-DreamShaper", it is *completely different* but also **incredibly fast**. Sometimes it is preferred, so try it yourself. ##### Download a ready-to-run file from here: * https://huggingface.co/akleine/sdxs-09 For the use of this model, both options ``` --cfg-scale 1 ``` and ``` --steps 1 ``` are again absolutely necessary. --- ### Docker # Docker ## Run CLI ```shell docker run --rm -v /path/to/models:/models -v /path/to/output/:/output ghcr.io/leejet/stable-diffusion.cpp:master [args...] # For example # docker run --rm -v ./models:/models -v ./build:/output ghcr.io/leejet/stable-diffusion.cpp:master -m /models/sd-v1-4.ckpt -p "a lovely cat" -v -o /output/output.png ``` ## Run server ```shell docker run --rm --init -v /path/to/models:/models -v /path/to/output/:/output -p "1234:1234" --entrypoint "/sd-server" ghcr.io/leejet/stable-diffusion.cpp:master [args...] # For example # docker run --rm --init -v ./models:/models -v ./build:/output -p "1234:1234" --entrypoint "/sd-server" ghcr.io/leejet/stable-diffusion.cpp:master -m /models/sd-v1-4.ckpt -p "a lovely cat" -v -o /output/output.png ``` ## Building using Docker ```shell docker build -f docker/Dockerfile -t sd . ``` ## Building variants using Docker Vulkan: ```shell docker build -f docker/Dockerfile.vulkan -t sd . ``` ## Run locally built image's CLI ```shell docker run --rm -v /path/to/models:/models -v /path/to/output/:/output sd [args...] # For example # docker run --rm -v ./models:/models -v ./build:/output sd -m /models/sd-v1-4.ckpt -p "a lovely cat" -v -o /output/output.png ``` --- ### Edit # Image Editing Image editing in `stable-diffusion.cpp` allows you to use reference images to guide the generation process, enabling tasks like identity preservation, style transfer, or layout modification. ## Supported Models Depending on the architecture, different models handle reference images differently. | Model | Default Preset | | :--- | :--- | | [**FLUX.1-Kontext-dev**](./kontext.md) | `flux_kontext` | | [**LongCat Image Edit**](./longcat_image.md) | `longcat` | | [**Qwen Image Edit**](./qwen_image_edit.md) | `qwen` | | **Qwen Image LAYERED** | `qwen_layered` | | [**Flux.2 [Dev] / Flux.2 [Klein]**](./flux2.md) | `flux2` | | [**Boogu Image Edit**](./boogu_image.md) | `z_image_omni` | | **Krea2 (Community Edit LoRAs)** | `krea2_ostris_edit` | | [**Mage-Flow-Edit**](./mage_flow.md#image-editing) | `mage_flow` | | **Anima (Community Edit LoRAs)** | `cosmos_reference` | Stable-diffusion.spp also supports basic Unet-based editing models like instruct-pix2pix or CosXL-Edit. This document is not about those. --- ## Configuring Reference Modes (`--ref-image-args`) Different DiT-based editing models require different configurations to process reference images correctly (e.g., whether to use a Vision Language Model (VLM) encoder or pass VAE-encoded images directly to the DiT). To simplify this, we provide **Presets**. By default, the system automatically selects the best preset based on the model architecture. However, you can override this using the `--ref-image-args` argument. ### Usage The `--ref-image-args` argument accepts a comma-separated list of key-value pairs: **Using a preset:** `--ref-image-args "preset=qwen_layered"` **Using a preset with a specific override:** `--ref-image-args "preset=krea2_edit,force_ref_timestep_zero=true"` ### Available Presets | Preset | Primary Use Case | | :--- | :--- | | `flux_kontext` | FLUX.1 Kontext | | `longcat` | LongCat Image Edit | | `flux2` | FLUX.2 models | | `qwen` | Qwen Image Edit | | `qwen_layered` | Qwen Image Layered | | `z_image_omni` | Boogu, Z-Image Omni | | `krea2_ostris_edit` | Most Krea2 Community edit LoRAs (trained with Ostris script) | | `mage_flow` | Mage-Flow-Edit | | `krea2_edit` | Specifically for [lbouaraba/krea2edit](https://huggingface.co/conradlocke/krea2-identity-edit). (or similar) | | `cosmos_reference` | For Anima | | `default` | Uses the automatic detection based on model architecture. | --- ## Advanced Parameter Reference If presets are insufficient, you can manually configure the following parameters via `--ref-image-args`: | Key | Type | Description | Allowed Values | | :--- | :--- | :--- | :--- | | `preset` | string | Overrides the automatic preset. | (See the Presets table above) | | `pass_to_vlm` | bool | Whether reference images are passed to the VLM encoder. | `true`, `false` | | `pass_to_dit` | bool | Whether VAE-encoded references are passed directly to the DiT. | `true`, `false` | | `ref_index_mode` | string | Behavior of the RoPE index. | `fixed`, `increase`, `decrease` | | `force_ref_timestep_zero` | bool | Forces timestep=0 for reference tokens. | `true`, `false` (Krea2 only) | | `resize_before_vae` | bool | Whether reference images are resized before VAE encoding. | `true`, `false` | | `vae_input_max_pixels` | int | Maximum pixel area for VAE reference inputs. | Integer | | `vlm_resize_mode` | string | How to resize VLM reference inputs. | `longest_side`, `area`, `none` | | `vlm_max_size` | int | Maximum VLM input size; interpreted according to `vlm_resize_mode`. | Integer | | `vlm_min_size` | int | Minimum VLM input size; interpreted according to `vlm_resize_mode`. | Integer | | `vlm_size` | int | Shortcut to set both VLM min and max size to the same value. | Integer | ### Preset Default Values For a technical overview of how each preset is configured, see the table below. | Preset | VLM | RoPE Index | Cond Resize | Special Notes | | :--- | :---: | :---: | :---: | :--- | | `flux_kontext` | No | `fixed` | `none` | | | `longcat` | Yes | `fixed` | `area` | | | `flux2` | No | `increase` | `none` | | | `qwen` | Yes | `increase` | `area` | | | `qwen_layered` | Yes | `decrease` | `area` | | | `mage_flow` | Yes | `increase` | `longest` | `vlm_max_size = 384`, VAE input resized to target | | `z_image_omni` | Yes | `fixed` | `area` | | | `krea2_ostris_edit`| Yes | `increase` | `area` | `force_ref_timestep_zero = true` | | `krea2_edit` | Yes | `increase` | `longest` | `vlm_size = 768` | | `cosmos_reference` | No | `fixed` | `none` | `resize_before_vae = false` | **Additional Default Notes:** - **VLM Input Sizes:** For most presets, `vlm_max_size` and `vlm_min_size` are set to `-1`, meaning the values are model-dependent and handled automatically. In `area` mode they represent pixel area; in `longest_side` mode they represent a side length in pixels. - **VAE Input Size:** `vae_input_max_pixels` defaults to $1024 \times 1024$ pixels (`1048576`). --- ### Ernie Image # How to Use You can run ERNIE-Image with stable-diffusion.cpp on GPUs with 4GB of VRAM — or even less. ## Download weights - Download ERNIE-Image-Turbo - safetensors: https://huggingface.co/Comfy-Org/ERNIE-Image/tree/main/diffusion_models - gguf: https://huggingface.co/unsloth/ERNIE-Image-Turbo-GGUF/tree/main - Download ERNIE-Image - safetensors: https://huggingface.co/Comfy-Org/ERNIE-Image/tree/main/diffusion_models - gguf: https://huggingface.co/unsloth/ERNIE-Image-GGUF/tree/main - Download vae - safetensors: https://huggingface.co/Comfy-Org/ERNIE-Image/tree/main/vae - Download ministral 3b - safetensors: https://huggingface.co/Comfy-Org/ERNIE-Image/tree/main/text_encoders - gguf: https://huggingface.co/unsloth/Ministral-3-3B-Instruct-2512-GGUF/tree/main ## Examples ### ERNIE-Image-Turbo ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\ernie-image-turbo.safetensors --vae ..\models\vae\flux2_ae.safetensors --llm ..\models\text_encoders\ministral-3-3b.safetensors -p "a lovely cat" --cfg-scale 1.0 --steps 8 -v --offload-to-cpu --diffusion-fa ``` ERNIE-Image Turbo example ### ERNIE-Image ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\ernie-image-UD-Q4_K_M.gguf --vae ..\models\vae\flux2_ae.safetensors --llm ..\models\text_encoders\ministral-3-3b.safetensors -p "a lovely cat" --cfg-scale 5.0 -v --offload-to-cpu --diffusion-fa ``` ERNIE-Image example --- ### Esrgan ## Using ESRGAN to upscale results You can use ESRGAN—such as the model [RealESRGAN_x4plus_anime_6B.pth](https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth)—to upscale the generated images and improve their overall resolution and clarity. - Specify the model path using the `--upscale-model PATH` parameter. example: ```bash sd-cli -m ../models/v1-5-pruned-emaonly.safetensors -p "a lovely cat" --upscale-model ../models/RealESRGAN_x4plus_anime_6B.pth ``` --- ### Flux # How to Use You can run Flux using stable-diffusion.cpp with a GPU that has 6GB or even 4GB of VRAM, without needing to offload to RAM. ## Download weights - Download flux - If you don't want to do the conversion yourself, download the preconverted gguf model from [FLUX.1-dev-gguf](https://huggingface.co/leejet/FLUX.1-dev-gguf) or [FLUX.1-schnell](https://huggingface.co/leejet/FLUX.1-schnell-gguf) - Otherwise, download flux-dev from https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/flux1-dev.safetensors or flux-schnell from https://huggingface.co/black-forest-labs/FLUX.1-schnell/blob/main/flux1-schnell.safetensors - Download vae from https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/ae.safetensors - Download clip_l from https://huggingface.co/comfyanonymous/flux_text_encoders/blob/main/clip_l.safetensors - Download t5xxl from https://huggingface.co/comfyanonymous/flux_text_encoders/blob/main/t5xxl_fp16.safetensors ## Convert flux weights You can download the preconverted gguf weights from [FLUX.1-dev-gguf](https://huggingface.co/leejet/FLUX.1-dev-gguf) or [FLUX.1-schnell](https://huggingface.co/leejet/FLUX.1-schnell-gguf), this way you don't have to do the conversion yourself. For example: ``` .\bin\Release\sd-cli.exe -M convert -m ..\models\unet\flux1-dev.sft -o ..\models\flux1-dev-q8_0.gguf -v --type q8_0 ``` ## Run - `--cfg-scale` is recommended to be set to 1. ### Flux-dev For example: ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\flux1-dev-q8_0.gguf --vae ..\models\ae.sft --clip_l ..\models\clip_l.safetensors --t5xxl ..\models\t5xxl_fp16.safetensors -p "a lovely cat holding a sign says 'flux.cpp'" --cfg-scale 1.0 --sampling-method euler -v --clip-on-cpu ``` Using formats of different precisions will yield results of varying quality. | Type | q8_0 | q4_0 | q4_k | q3_k | q2_k | |---- | ---- |---- |---- |---- |---- | | **Memory** | 12068.09 MB | 6394.53 MB | 6395.17 MB | 4888.16 MB | 3735.73 MB | | **Result** | | | | || ### Flux-schnell ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\flux1-schnell-q8_0.gguf --vae ..\models\ae.sft --clip_l ..\models\clip_l.safetensors --t5xxl ..\models\t5xxl_fp16.safetensors -p "a lovely cat holding a sign says 'flux.cpp'" --cfg-scale 1.0 --sampling-method euler -v --steps 4 --clip-on-cpu ``` | q8_0 | | ---- | | | ## Run with LoRA Since many flux LoRA training libraries have used various LoRA naming formats, it is possible that not all flux LoRA naming formats are supported. It is recommended to use LoRA with naming formats compatible with ComfyUI. ### Flux-dev q8_0 with LoRA - LoRA model from https://huggingface.co/XLabs-AI/flux-lora-collection/tree/main (using comfy converted version!!!) ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\flux1-dev-q8_0.gguf --vae ...\models\ae.sft --clip_l ..\models\clip_l.safetensors --t5xxl ..\models\t5xxl_fp16.safetensors -p "a lovely cat holding a sign says 'flux.cpp'" --cfg-scale 1.0 --sampling-method euler -v --lora-model-dir ../models --clip-on-cpu ``` --- ### Flux2 # How to Use ## Flux.2-dev ### Download weights - Download FLUX.2-dev - gguf: https://huggingface.co/city96/FLUX.2-dev-gguf/tree/main - Download vae - safetensors: https://huggingface.co/black-forest-labs/FLUX.2-dev/tree/main - Download FLUX.2-small-decoder (full_encoder_small_decoder.safetensors) as an alternative VAE option - safetensors: https://huggingface.co/black-forest-labs/FLUX.2-small-decoder/tree/main - Download Mistral-Small-3.2-24B-Instruct-2506-GGUF - gguf: https://huggingface.co/unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF/tree/main ### Examples ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\flux2-dev-Q4_K_S.gguf --vae ..\models\vae\flux2_ae.safetensors --llm ..\models\text_encoders\Mistral-Small-3.2-24B-Instruct-2506-Q4_K_M.gguf -r .\kontext_input.png -p "change 'flux.cpp' to 'flux2-dev.cpp'" --cfg-scale 1.0 --sampling-method euler -v --diffusion-fa --offload-to-cpu ``` flux2 example ## Flux.2 klein 4B / Flux.2 klein base 4B ### Download weights - Download FLUX.2-klein-4B - safetensors: https://huggingface.co/black-forest-labs/FLUX.2-klein-4B - gguf: https://huggingface.co/leejet/FLUX.2-klein-4B-GGUF/tree/main - Download FLUX.2-klein-base-4B - safetensors: https://huggingface.co/black-forest-labs/FLUX.2-klein-base-4B - gguf: https://huggingface.co/leejet/FLUX.2-klein-base-4B-GGUF/tree/main - Download vae - safetensors: https://huggingface.co/black-forest-labs/FLUX.2-dev/tree/main - Download FLUX.2-small-decoder (full_encoder_small_decoder.safetensors) as an alternative VAE option - safetensors: https://huggingface.co/black-forest-labs/FLUX.2-small-decoder/tree/main - Download Qwen3 4b - safetensors: https://huggingface.co/Comfy-Org/flux2-klein-4B/tree/main/split_files/text_encoders - gguf: https://huggingface.co/unsloth/Qwen3-4B-GGUF/tree/main ### Examples ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\flux-2-klein-4b.safetensors --vae ..\models\vae\flux2_ae.safetensors --llm ..\models\text_encoders\qwen_3_4b.safetensors -p "a lovely cat" --cfg-scale 1.0 --steps 4 -v --offload-to-cpu --diffusion-fa ``` flux2-klein-4b ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\flux-2-klein-4b.safetensors --vae ..\models\vae\flux2_ae.safetensors --llm ..\models\text_encoders\qwen_3_4b.safetensors -r .\kontext_input.png -p "change 'flux.cpp' to 'klein.cpp'" --cfg-scale 1.0 --sampling-method euler -v --diffusion-fa --offload-to-cpu --steps 4 ``` flux2-klein-4b-edit ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\flux-2-klein-base-4b.safetensors --vae ..\models\vae\flux2_ae.safetensors --llm ..\models\text_encoders\qwen_3_4b.safetensors -p "a lovely cat" --cfg-scale 4.0 --steps 20 -v --offload-to-cpu --diffusion-fa ``` flux2-klein-base-4b ## Flux.2 klein 9B / Flux.2 klein base 9B ### Download weights - Download FLUX.2-klein-9B - safetensors: https://huggingface.co/black-forest-labs/FLUX.2-klein-9B - gguf: https://huggingface.co/leejet/FLUX.2-klein-9B-GGUF/tree/main - Download FLUX.2-klein-base-9B - safetensors: https://huggingface.co/black-forest-labs/FLUX.2-klein-base-9B - gguf: https://huggingface.co/leejet/FLUX.2-klein-base-9B-GGUF/tree/main - Download vae - safetensors: https://huggingface.co/black-forest-labs/FLUX.2-dev/tree/main - Download Qwen3 8B - safetensors: https://huggingface.co/Comfy-Org/flux2-klein-9B/tree/main/split_files/text_encoders - gguf: https://huggingface.co/unsloth/Qwen3-8B-GGUF/tree/main ### Examples ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\flux-2-klein-9b.safetensors --vae ..\models\vae\flux2_ae.safetensors --llm ..\models\text_encoders\qwen_3_8b.safetensors -p "a lovely cat" --cfg-scale 1.0 --steps 4 -v --offload-to-cpu --diffusion-fa ``` flux2-klein-9b ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\flux-2-klein-9b.safetensors --vae ..\models\vae\flux2_ae.safetensors --llm ..\models\text_encoders\qwen_3_8b.safetensors -r .\kontext_input.png -p "change 'flux.cpp' to 'klein.cpp'" --cfg-scale 1.0 --sampling-method euler -v --diffusion-fa --offload-to-cpu --steps 4 ``` flux2-klein-9b-edit ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\flux-2-klein-base-9b.safetensors --vae ..\models\vae\flux2_ae.safetensors --llm ..\models\text_encoders\qwen_3_8b.safetensors -p "a lovely cat" --cfg-scale 4.0 --steps 20 -v --offload-to-cpu --diffusion-fa ``` flux2-klein-base-9b --- ### Hidream O1 Image # How to Use ## Download weights - Download HiDream-O1-Image-Dev - safetensors: https://huggingface.co/Comfy-Org/HiDream-O1-Image/tree/main/checkpoints - Download HiDream-O1-Image - safetensors: https://huggingface.co/Comfy-Org/HiDream-O1-Image/tree/main/checkpoints ## Examples ### HiDream-O1-Image-Dev ``` .\bin\Release\sd-cli.exe -m ..\models\diffusion_models\hidream_o1_image_dev_bf16.safetensors -p "a lovely cat holding a sign says 'hidream o1 cpp'" --cfg-scale 1.0 -v -H 1024 -W 1024 ``` HiDream-O1-Image-Dev example --- ### HipBLAS On Windows # Using hipBLAS on Windows To get hipBLAS in `stable-diffusion.cpp` working on Windows, go through this guide section by section. ## Build Tools for Visual Studio 2022 Skip this step if you already have Build Tools installed. To install Build Tools, go to [Visual Studio Downloads](https://visualstudio.microsoft.com/vs/), download `Visual Studio 2022 and other Products` and run the installer. ## CMake Skip this step if you already have CMake installed: running `cmake --version` should output `cmake version x.y.z`. Download latest `Windows x64 Installer` from [Download | CMake](https://cmake.org/download/) and run it. ## ROCm Skip this step if you already have Build Tools installed. The [validation tools](https://rocm.docs.amd.com/en/latest/reference/validation_tools.html) not support on Windows. So you should confirm the Version of `ROCM` by yourself. Fortunately, `AMD` provides complete help documentation, you can use the help documentation to install [ROCM](https://rocm.docs.amd.com/en/latest/deploy/windows/quick_start.html) >**If you encounter an error, if it is [AMD ROCm Windows Installation Error 215](https://github.com/RadeonOpenCompute/ROCm/issues/2363), don't worry about this error. ROCM has been installed correctly, but the vs studio plugin installation failed, we can ignore it.** Then we must set `ROCM` as environment variables before running cmake. Usually if you install according to the official tutorial and do not modify the ROCM path, then there is a high probability that it is here `C:\Program Files\AMD\ROCm\7.1.1\bin` This is what I use to set the clang: ```Commandline set CC=C:\Program Files\AMD\ROCm\7.1.1\bin\clang.exe set CXX=C:\Program Files\AMD\ROCm\7.1.1\bin\clang++.exe ``` ## Ninja Skip this step if you already have Ninja installed: running `ninja --version` should output `1.11.1`. Download latest `ninja-win.zip` from [GitHub Releases Page](https://github.com/ninja-build/ninja/releases/tag/v1.11.1) and unzip. Then set as environment variables. I unzipped it in `C:\Program Files\ninja`, so I set it like this: ```Commandline set ninja=C:\Program Files\ninja\ninja.exe ``` ## Building stable-diffusion.cpp The thing different from the regular CPU build is `-DSD_HIPBLAS=ON` , `-G "Ninja"`, `-DCMAKE_C_COMPILER=clang`, `-DCMAKE_CXX_COMPILER=clang++`, `-DAMDGPU_TARGETS=gfx1150;gfx1151;gfx1200;gfx1201;gfx1100;gfx1101;gfx1102;gfx1030;gfx1031;gfx1032` >**Notice**: check the `clang` and `clang++` information: ```Commandline clang --version clang++ --version ``` If you see like this, we can continue: ``` clang version 17.0.0 (git@github.amd.com:Compute-Mirrors/llvm-project e3201662d21c48894f2156d302276eb1cf47c7be) Target: x86_64-pc-windows-msvc Thread model: posix InstalledDir: C:\Program Files\AMD\ROCm\7.1.1\bin ``` ``` clang version 17.0.0 (git@github.amd.com:Compute-Mirrors/llvm-project e3201662d21c48894f2156d302276eb1cf47c7be) Target: x86_64-pc-windows-msvc Thread model: posix InstalledDir: C:\Program Files\AMD\ROCm\7.1.1\bin ``` >**Notice** that the GPU targets are now compatible with multiple GPU architectures (ROCm 7.1.1 targets). You can change them to match your GPU architecture. Click here to see your architecture [LLVM Target](https://rocm.docs.amd.com/en/latest/release/windows_support.html#windows-supported-gpus) Examples: - AMD Radeon™ RX 7900 XTX Graphics: `gfx1100` - AMD Radeon™ RX 7900 XT Graphics: `gfx1101` - AMD Radeon™ RX 7900 GRE Graphics: `gfx1102` option: ```commandline mkdir build cd build cmake .. -G "Ninja" -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DSD_HIPBLAS=ON -DCMAKE_BUILD_TYPE=Release -DAMDGPU_TARGETS="gfx1150;gfx1151;gfx1200;gfx1201;gfx1100;gfx1101;gfx1102;gfx1030;gfx1031;gfx1032" cmake --build . --config Release ``` If everything went OK, `build\bin\sd-cli.exe` file should appear. --- ### Hunyuan Video # HunyuanVideo 1.5 HunyuanVideo 1.5 uses a HunyuanVideo diffusion transformer, a causal video VAE, Qwen2.5-VL 7B for the main text conditioning, and ByT5 Small GlyphXL for glyph-aware text conditioning. ## Download weights - Download HunyuanVideo 1.5 - safetensors: https://huggingface.co/Comfy-Org/HunyuanVideo_1.5_repackaged/tree/main/split_files/diffusion_models - Download vae - safetensors: https://huggingface.co/Comfy-Org/HunyuanVideo_1.5_repackaged/tree/main/split_files/vae - Download qwen_2.5_vl 7b - safetensors: https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/tree/main/split_files/text_encoders - gguf: https://huggingface.co/mradermacher/Qwen2.5-VL-7B-Instruct-GGUF/tree/main - Download byt5 small glyphxl - safetensros: https://huggingface.co/Comfy-Org/HunyuanVideo_1.5_repackaged/tree/main/split_files/text_encoders ## Text-to-video example ```shell .\bin\Release\sd-cli.exe -M vid_gen --diffusion-model ..\models\diffusion_models\hunyuanvideo1.5_720p_t2v_fp16.safetensors --vae ..\models\vae\hunyuanvideo15_vae_fp16.safetensors --llm ..\models\text_encoders\qwen_2.5_vl_7b.safetensors --t5xxl ..\models\text_encoders\byt5_small_glyphxl_fp16.safetensors -p "a lovely cat" --cfg-scale 6.0 --sampling-method euler -v -W 1280 -H 720 --offload-to-cpu --diffusion-fa --video-frames 33 --vae-tiling ``` --- ### Ideogram4 # How to Use ## Download weights - Download Ideogram4 - safetensors: https://huggingface.co/ideogram-ai/ideogram-4-fp8/tree/main/transformer - Download Ideogram4 uncond - safetensors: https://huggingface.co/ideogram-ai/ideogram-4-fp8/tree/main/unconditional_transformer - Download vae - safetensors: https://huggingface.co/black-forest-labs/FLUX.2-dev/tree/main - Download Qwen3-VL-8B-Instruct - gguf: https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-GGUF/tree/main ## Convert weights fp8 scale -> bf16 ``` python .\convert_fp8_scale_to_bf16.py --input .\ideogram4_fp8.safetensors --output ideogram4_bf16.safetensors python .\convert_fp8_scale_to_bf16.py --input .\ideogram4_uncond_fp8.safetensors --output ideogram4_uncond_bf16.safetensors ``` bf16 -> q8 ``` .\bin\Release\sd-cli.exe -M convert -m ideogram4_bf16.safetensors -o ideogram4-Q8_0.gguf --tensor-type-rules "^layers.*adaln_modulation.*weight=q8_0,layers.*attention.o.*weight=q8_0,layers.*attention.qkv.*weight=q8_0,layers.*feed_forward.*weight=q8_0" -v .\bin\Release\sd-cli.exe -M convert -m ideogram4_uncond_bf16.safetensors -o ideogram4_uncond-Q8_0.gguf --tensor-type-rules "^layers.*adaln_modulation.*weight=q8_0,layers.*attention.o.*weight=q8_0,layers.*attention.qkv.*weight=q8_0,layers.*feed_forward.*weight=q8_0" -v ``` If you want lower VRAM usage, you can change the quantization from q8_0 to a lower-level quantization, such as q4_0. ## Examples ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ideogram4 image example --- ### Imatrix # Importance Matrix (imatrix) Quantization ## What is an Importance Matrix? Quantization reduces the precision of a model's weights, decreasing its size and computational requirements. However, this can lead to a loss of quality. An importance matrix helps mitigate this by identifying which weights are *most* important for the model's performance. During quantization, these important weights are preserved with higher precision, while less important weights are quantized more aggressively. This allows for better overall quality at a given quantization level. This originates from work done with language models in [llama.cpp](https://github.com/ggml-org/llama.cpp/blob/master/tools/imatrix/README.md). ## Usage The imatrix feature involves two main steps: *training* the matrix and *using* it during quantization. ### Training the Importance Matrix To generate an imatrix, run stable-diffusion.cpp with the `--imat-out` flag, specifying the output filename. This process runs alongside normal image generation. ```bash sd.exe [same exact parameters as normal generation] --imat-out imatrix.dat ``` * **`[same exact parameters as normal generation]`**: Use the same command-line arguments you would normally use for image generation (e.g., prompt, dimensions, sampling method, etc.). * **`--imat-out imatrix.dat`**: Specifies the output file for the generated imatrix. You can generate multiple images at once using the `-b` flag to speed up the training process. ### Continuing Training an Existing Matrix If you want to refine an existing imatrix, use the `--imat-in` flag *in addition* to `--imat-out`. This will load the existing matrix and continue training it. ```bash sd.exe [same exact parameters as normal generation] --imat-out imatrix.dat --imat-in imatrix.dat ``` With that, you can train and refine the imatrix while generating images like you'd normally do. ### Using Multiple Matrices You can load and merge multiple imatrices together: ```bash sd.exe [same exact parameters as normal generation] --imat-out imatrix.dat --imat-in imatrix.dat --imat-in imatrix2.dat ``` ### Quantizing with an Importance Matrix To quantize a model using a trained imatrix, use the `-M convert` option (or equivalent quantization command) and the `--imat-in` flag, specifying the imatrix file. ```bash sd.exe -M convert [same exact parameters as normal quantization] --imat-in imatrix.dat ``` * **`[same exact parameters as normal quantization]`**: Use the same command-line arguments you would normally use for quantization (e.g., target quantization method, input/output filenames). * **`--imat-in imatrix.dat`**: Specifies the imatrix file to use during quantization. You can specify multiple `--imat-in` flags to combine multiple matrices. ## Important Considerations * The quality of the imatrix depends on the prompts and settings used during training. Use prompts and settings representative of the types of images you intend to generate for the best results. * Experiment with different training parameters (e.g., number of images, prompt variations) to optimize the imatrix for your specific use case. * The performance impact of training an imatrix during image generation or using an imatrix for quantization is negligible. * Using already quantized models to train the imatrix seems to be working fine. --- ### Int8 Convrot # INT8 Convrot Safetensors sd.cpp can load and execute ComfyUI `int8_tensorwise` safetensors with `convrot` metadata directly. The stored INT8 weights are not converted to another weight type at load time. ## Checkpoint format Each quantized linear module contains the following tensors: - `.weight`: an I8 weight matrix. - `.weight_scale`: one floating-point scale for each output row. ComfyUI's two-dimensional `[out_features, 1]` representation is normalized to a one-dimensional tensor while loading. - `.comfy_quant`: a U8 tensor containing the JSON quantization configuration. A supported configuration has this form: ```json { "format": "int8_tensorwise", "convrot": true, "convrot_groupsize": 256 } ``` The convrot group size must be a power of four and must divide the input feature dimension. The commonly used configuration is H256, with `convrot_groupsize` set to `256`. ## How INT8 convrot works Convrot combines an offline rotation of the weights with the same rotation of the activations at runtime. The rotation uses a normalized regular Hadamard matrix constructed recursively from ```text [ 1 1 1 -1 ] [ 1 1 -1 1 ] H4 = [ 1 -1 1 1 ] / 2 [-1 1 1 1 ] ``` For a group size `G = 4^n`, the transform is the normalized Kronecker power of `H4`. It is applied independently to every contiguous group of `G` input features. The resulting block-diagonal rotation matrix `R` is orthogonal and symmetric, so `R R^T = I`. For an original floating-point linear layer ```text Y = X W^T + b ``` the checkpoint stores a rotated weight matrix `W_rot = W R`, quantized per output row. At runtime sd.cpp computes `X_rot = X R`. Ignoring quantization error, ```text X_rot W_rot^T = X R (W R)^T = X R R^T W^T = X W^T ``` The rotation therefore preserves the linear operation. Its purpose is to spread isolated large values across each feature group, reducing the effect of outliers on tensorwise INT8 quantization. ### Weight quantization The rotated weights are quantized offline with one scale per output row: ```text s_w[o] = max_i(abs(W_rot[o, i])) / 127 Q_w[o, i] = clamp(round(W_rot[o, i] / s_w[o]), -127, 127) ``` `Q_w` is stored in `.weight`, and `s_w` is stored in `.weight_scale`. ### Runtime activation quantization For every activation row, sd.cpp applies the group-wise Hadamard rotation and then calculates one dynamic scale across the entire rotated row: ```text s_x[r] = max_i(abs(X_rot[r, i])) / 127 Q_x[r, i] = clamp(round(X_rot[r, i] / s_x[r]), -127, 127) ``` The matrix multiplication accumulates into signed 32-bit integers: ```text A[r, o] = sum_i(Q_x[r, i] * Q_w[o, i]) ``` The floating-point output is reconstructed as ```text Y[r, o] ~= A[r, o] * s_x[r] * s_w[o] + b[o] ``` The packed runtime activation tensor contains the I8 activation rows and their floating-point row scales. Linear layers that share the same input and convrot group size reuse this packed tensor, avoiding repeated rotation and activation quantization within the graph. ## Backend support - CPU provides the portable regular Hadamard, activation quantization, INT8 matrix multiplication, and scale restoration implementations. - NVIDIA CUDA devices with compute capability 7.5 or newer use the native accelerated path. For H256, CUDA fuses the rotation, row-wise maximum reduction, and activation quantization. It uses cuBLAS for I8 x I8 to I32 GEMM and a CUDA kernel for scale restoration and bias addition. - Vulkan and other GPU backends do not currently have dedicated INT8 convrot kernels. They use the backend scheduler to fall back to CPU, which is expected to be substantially slower than the CUDA path. LoRA adapters are applied at runtime without modifying the INT8 weights. The INT8 convrot path computes the base linear output, while LoRA, LoHa, LoKr, and raw weight-difference adapters compute their output corrections from the original, unrotated activation and add them to the base output. `--lora-apply-mode auto` selects this path for models containing INT8 tensorwise weights. If `immediately` is requested, sd.cpp falls back to runtime application because merging an adapter would require dequantizing and rotating its weight update, then recalculating the per-row scales and requantizing the result. The dedicated CUDA convrot activation path currently requires a group size of `256`; other supported group sizes use CPU execution. ## Example ComfyUI INT8 convrot safetensors can be passed to `--diffusion-model` without conversion: ```powershell .\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\krea2_turbo_int8_convrot.safetensors --llm ..\models\text_encoders\Qwen3-VL-4B-Instruct-Q4_K_M.gguf --vae ..\models\vae\wan_2.1_vae.safetensors -p "a lovely cat holding a sign says 'krea2.cpp'" --steps 8 --cfg-scale 1 --diffusion-fa -v --offload-to-cpu ``` --- ### Ip Adapter # IP-Adapter stable-diffusion.cpp supports [IP-Adapter](https://github.com/tencent-ailab/IP-Adapter) image-prompt conditioning for SD 1.5 and SDXL. Given a reference image, IP-Adapter transfers the subject and appearance of that image into the generation, alongside the text prompt. IP-Adapter encodes the reference image with a CLIP-Vision (ViT-H/14) encoder, projects the embedding into a few image tokens, and injects them through a decoupled cross-attention added to every attn2 layer of the UNet. It composes with Control Net, so a reference image (appearance) and an OpenPose hint (pose) can be combined in a single generation. Both the classic adapters and the higher-fidelity **Plus** adapters are supported; see [Plus variants](#plus-variants) below. The variant is detected from the weight file, so the same options work for both. ## Required weights 1. A base SD 1.5 or SDXL model. 2. A CLIP-Vision (ViT-H/14) image encoder, passed with `--clip_vision` (for example `clip_vision_h.safetensors`). 3. An IP-Adapter weight file, passed with `--ip-adapter`. The `vit-h` variants reuse the same ViT-H encoder as above. From [h94/IP-Adapter](https://huggingface.co/h94/IP-Adapter): - SD 1.5: `models/ip-adapter_sd15.safetensors` - SDXL: `sdxl_models/ip-adapter_sdxl_vit-h.safetensors` - SD 1.5 Plus: `models/ip-adapter-plus_sd15.safetensors` - SDXL Plus: `sdxl_models/ip-adapter-plus_sdxl_vit-h.safetensors` The Plus files (`ip-adapter-plus_*`) are used exactly like the classic ones; see [Plus variants](#plus-variants). ## Options - `--ip-adapter ` path to the IP-Adapter weight file. - `--ip-adapter-image ` path to the reference image. - `--ip-adapter-strength ` strength of the IP-Adapter injection (default 1.0). Lower values let the text prompt dominate; 0.6 to 0.8 is a good starting range. ## Example (SD 1.5) ``` sd-cli -m ..\models\sd_v1.5.safetensors --clip_vision ..\models\clip_vision_h.safetensors --ip-adapter ..\models\ip-adapter_sd15.safetensors --ip-adapter-image ..\assets\reference.png --ip-adapter-strength 0.8 -p "a woman, best quality" -n "lowres, bad anatomy" --cfg-scale 7 --steps 30 --sampling-method dpm++2m --scheduler karras -W 512 -H 512 ``` ## Example (SDXL) ``` sd-cli -m ..\models\sdxl.safetensors --clip_vision ..\models\clip_vision_h.safetensors --ip-adapter ..\models\ip-adapter_sdxl_vit-h.safetensors --ip-adapter-image ..\assets\reference.png --ip-adapter-strength 0.8 -p "a woman, best quality" -n "lowres, bad anatomy" --cfg-scale 6 --steps 25 --sampling-method dpm++2m --scheduler karras -W 1024 -H 1024 --diffusion-fa --vae-tiling ``` The SDXL VAE decode at 1024x1024 is memory heavy; add `--vae-tiling` (and `--offload-to-cpu`) on GPUs with limited VRAM. ## Plus variants The Plus adapters (`ip-adapter-plus_sd15`, `ip-adapter-plus_sdxl_vit-h`) replace the small linear image projection with a Resampler (a Perceiver-style module with learned latent queries). Instead of pooling the CLIP-Vision output into one vector, the Resampler attends over the full grid of penultimate CLIP-Vision hidden states and emits more image tokens (16 instead of 4). The result transfers finer detail and layout from the reference, at a small extra cost in the image-projection step. No extra flags are needed. The variant is detected from the weight file (the Resampler's `image_proj.latents` tensor), and every Resampler dimension is read from the tensor shapes, so the same `--ip-adapter`, `--ip-adapter-image`, and `--ip-adapter-strength` options apply. Plus composes with Control Net in the same way as the classic adapters. ``` sd-cli -m ..\models\sd_v1.5.safetensors --clip_vision ..\models\clip_vision_h.safetensors --ip-adapter ..\models\ip-adapter-plus_sd15.safetensors --ip-adapter-image ..\assets\reference.png --ip-adapter-strength 0.8 -p "a woman, best quality" -n "lowres, bad anatomy" --cfg-scale 7 --steps 30 --sampling-method dpm++2m --scheduler karras -W 512 -H 512 ``` The startup log line `IP-Adapter: 16 image tokens` (versus `4` for the classic adapters) confirms a Plus file was loaded. ## Combining with Control Net Add the usual Control Net options to keep the reference appearance while controlling the pose: ``` sd-cli -m ..\models\sdxl.safetensors --clip_vision ..\models\clip_vision_h.safetensors --ip-adapter ..\models\ip-adapter_sdxl_vit-h.safetensors --ip-adapter-image ..\assets\character.png --ip-adapter-strength 0.9 --control-net ..\models\OpenPoseXL2.safetensors --control-image ..\assets\pose.png --control-strength 0.8 -p "a character, side view" --cfg-scale 6 --steps 25 -W 1024 -H 1024 --diffusion-fa --vae-tiling ``` --- ### Kontext # How to Use You can run Kontext using stable-diffusion.cpp with a GPU that has 6GB or even 4GB of VRAM, without needing to offload to RAM. ## Download weights - Download Kontext - If you don't want to do the conversion yourself, download the preconverted gguf model from [FLUX.1-Kontext-dev-GGUF](https://huggingface.co/QuantStack/FLUX.1-Kontext-dev-GGUF) - Otherwise, download FLUX.1-Kontext-dev from https://huggingface.co/black-forest-labs/FLUX.1-Kontext-dev/blob/main/flux1-kontext-dev.safetensors - Download vae from https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/ae.safetensors - Download clip_l from https://huggingface.co/comfyanonymous/flux_text_encoders/blob/main/clip_l.safetensors - Download t5xxl from https://huggingface.co/comfyanonymous/flux_text_encoders/blob/main/t5xxl_fp16.safetensors ## Convert Kontext weights You can download the preconverted gguf weights from [FLUX.1-Kontext-dev-GGUF](https://huggingface.co/QuantStack/FLUX.1-Kontext-dev-GGUF), this way you don't have to do the conversion yourself. ``` .\bin\Release\sd-cli.exe -M convert -m ..\models\unet\flux1-kontext-dev.safetensors -o ..\models\flux1-kontext-dev-q8_0.gguf -v --type q8_0 ``` ## Run - `--cfg-scale` is recommended to be set to 1. ### Example For example: ``` .\bin\Release\sd-cli.exe -r .\flux1-dev-q8_0.png --diffusion-model ..\models\flux1-kontext-dev-q8_0.gguf --vae ..\models\ae.sft --clip_l ..\models\clip_l.safetensors --t5xxl ..\models\t5xxl_fp16.safetensors -p "change 'flux.cpp' to 'kontext.cpp'" --cfg-scale 1.0 --sampling-method euler -v --clip-on-cpu ``` | ref_image | prompt | output | | ---- | ---- |---- | | | change 'flux.cpp' to 'kontext.cpp' | | --- ### Krea2 # How to Use Krea2 uses a Krea2 diffusion transformer, the Wan2.1 VAE, and Qwen3-VL 4B as the LLM text encoder. ## Download weights - Download Krea2 Raw - safetensors: https://huggingface.co/krea/Krea-2-Raw/tree/main - gguf: https://huggingface.co/realrebelai/KREA-2_GGUFs/tree/main/BASE - Download Krea2 Turbo - safetensors: https://huggingface.co/krea/Krea-2-Turbo/tree/main - gguf: https://huggingface.co/realrebelai/KREA-2_GGUFs/tree/main/TURBO - Download vae - safetensors: https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/blob/main/split_files/vae/wan_2.1_vae.safetensors - Download Qwen3-VL 4B - safetensors: https://huggingface.co/Comfy-Org/Krea-2/tree/main/text_encoders - gguf: https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct-GGUF/tree/main ## Examples ### Krea2 ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\Krea-2-Raw-Q8_0.gguf --llm ..\models\text_encoders\Qwen3-VL-4B-Instruct-Q4_K_M.gguf --vae ..\models\vae\wan_2.1_vae.safetensors -p "a lovely cat holding a sign says 'krea2.cpp'" --diffusion-fa -v --offload-to-cpu ``` Krea2 Raw example --- ### Lcm ## LCM/LCM-LoRA - Download LCM-LoRA form https://huggingface.co/latent-consistency/lcm-lora-sdv1-5 - Specify LCM-LoRA by adding `` to prompt - It's advisable to set `--cfg-scale` to `1.0` instead of the default `7.0`. For `--steps`, a range of `2-8` steps is recommended. For `--sampling-method`, `lcm`/`euler_a` is recommended. Here's a simple example: ``` ./bin/sd-cli -m ../models/v1-5-pruned-emaonly.safetensors -p "a lovely cat" --steps 4 --lora-model-dir ../models -v --cfg-scale 1 ``` | without LCM-LoRA (--cfg-scale 7) | with LCM-LoRA (--cfg-scale 1) | | ---- |---- | | | | --- ### Lens # How to Use Lens uses a Lens diffusion transformer, the FLUX.2 VAE, and GPT-OSS-20B as the LLM text encoder. ## Download weights - Download Lens - safetensors: https://huggingface.co/Comfy-Org/Lens/tree/main/diffusion_models - Download Lens Turbo - safetensors: https://huggingface.co/Comfy-Org/Lens/tree/main/diffusion_models - Download vae - safetensors: https://huggingface.co/black-forest-labs/FLUX.2-dev/tree/main - Download GPT-OSS-20B - gguf: https://huggingface.co/unsloth/gpt-oss-20b-GGUF/tree/main ## Examples ### Lens ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\lens_bf16.safetensors --llm "..\models\text_encoders\gpt-oss-20b-UD-Q8_K_XL.gguf" --vae ..\models\vae\flux2_ae.safetensors --cfg-scale 5.0 -p "A crystal dragon soaring through an aurora borealis sky, its entire body made of transparent faceted crystal refracting the green and purple aurora light into rainbow spectra, ice particles trailing from its wings, high fantasy digital art" --diffusion-fa -v ``` Lens example ### Lens Turbo ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\lens_turbo_bf16.safetensors --llm "..\models\text_encoders\gpt-oss-20b-UD-Q8_K_XL.gguf" --vae ..\models\vae\flux2_ae.safetensors --cfg-scale 1.0 -p "A crystal dragon soaring through an aurora borealis sky, its entire body made of transparent faceted crystal refracting the green and purple aurora light into rainbow spectra, ice particles trailing from its wings, high fantasy digital art" --diffusion-fa -v --steps 4 ``` Lens Turbo example --- ### Lingbot Video # How to Use Lingbot Video uses a Lingbot diffusion transformer, the Wan2.1 VAE, and Qwen3-VL 4B as the LLM text encoder. ## Download weights - Download lingbot-video-dense-1.3b - safetensors: https://huggingface.co/robbyant/lingbot-video-dense-1.3b/tree/main/transformer - Download lingbot-video-moe-30b-a3b - safetensors: https://huggingface.co/robbyant/lingbot-video-moe-30b-a3b/tree/main/transformer - Download vae - safetensors: https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/blob/main/split_files/vae/wan_2.1_vae.safetensors - Download Qwen3-VL 4B - safetensors: https://huggingface.co/Comfy-Org/Krea-2/tree/main/text_encoders - gguf: https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct-GGUF/tree/main ## Notes - Use `-M vid_gen`. - T2V uses the text prompt only. - I2V uses `-i` as the first video frame. The same image is also passed to the Qwen3-VL prompt enhancer when vision weights are available. - Video frames are aligned to Wan-style temporal compression, so use frame counts like 33, 49, or 81. ## Examples ### LingBot-Video T2V ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ### Longcat Image # How to Use LongCat-Image uses a LongCat diffusion transformer, the FLUX VAE, and Qwen2.5-VL as the LLM text encoder. ## Download weights - Download LongCat Image - safetensors: https://huggingface.co/Comfy-Org/LongCat-Image/tree/main/split_files/diffusion_models - gguf: https://huggingface.co/vantagewithai/LongCat-Image-GGUF/tree/main/comfy - Download LongCat Image Edit - LongCat Image Edit Turbo: https://huggingface.co/meituan-longcat/LongCat-Image-Edit-Turbo - gguf: https://huggingface.co/vantagewithai/LongCat-Image-Edit-GGUF/tree/main - Download vae - safetensors: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/ae.safetensors - Download qwen_2.5_vl 7b - safetensors: https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/tree/main/split_files/text_encoders - gguf: https://huggingface.co/mradermacher/Qwen2.5-VL-7B-Instruct-GGUF/tree/main - For image editing with GGUF text encoders, also download the matching mmproj file and pass it with `--llm_vision`. ## Run LongCat uses quoted text for character-level text rendering. Put target text inside single quotes, double quotes, or Chinese quotes. ### LongCat Image ``` .\bin\Release\sd-cli.exe --diffusion-model ..\models\diffusion_models\LongCat-Image-Q4_K_M.gguf --vae ..\models\vae\ae.sft --llm ..\models\text_encoders\Qwen2.5-VL-7B-Instruct-Q8_0.gguf -p "a lovely cat holding a sign says 'longcat.cpp'" --cfg-scale 5.0 --sampling-method euler --flow-shift 3 -v --offload-to-cpu --diffusion-fa ``` longcat example --- ### Lora ## LoRA - You can specify the directory where the lora weights are stored via `--lora-model-dir`. If not specified, the default is the current working directory. - LoRA is specified via prompt, just like [stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features#lora). Here's a simple example: ``` ./bin/sd-cli -m ../models/v1-5-pruned-emaonly.safetensors -p "a lovely cat" --lora-model-dir ../models ``` `../models/marblesh.safetensors` or `../models/marblesh.ckpt` will be applied to the model # Lora Apply Mode There are two ways to apply LoRA: **immediately** and **at_runtime**. You can specify it using the `--lora-apply-mode` parameter. By default, the mode is selected automatically: * If the model weights contain any quantized parameters, the **at_runtime** mode is used; * Otherwise, the **immediately** mode is used. The **immediately** mode may have precision and compatibility issues with quantized parameters, but it usually offers faster inference speed and, in some cases, lower memory usage. In contrast, the **at_runtime** mode provides better compatibility and higher precision, but inference may be slower and memory usage may be higher in some cases. ---