candle Documentation

File: README.md

candle

Candle is a minimalist ML framework for Rust with a focus on performance (including GPU support)
and ease of use. Try our online demos:
whisper,
LLaMA2,
T5,
yolo,
Segment
Anything.

Get started

Make sure that you have candle-core correctly installed as described in Installation.

Let's see how to run a simple matrix multiplication.
Write the following to your myapp/src/main.rs file:

rust

use candle_core::{Device, Tensor};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let device = Device::Cpu;

    let a = Tensor::randn(0f32, 1., (2, 3), &device)?;
    let b = Tensor::randn(0f32, 1., (3, 4), &device)?;

    let c = a.matmul(&b)?;
    println!("{c}");
    Ok(())
}

cargo run should display a tensor of shape Tensor[[2, 4], f32].

Having installed candle with Cuda support, simply define the device to be on GPU:

diff

- let device = Device::Cpu;
+ let device = Device::new_cuda(0)?;

For more advanced examples, please have a look at the following section.

Check out our examples

These online demos run entirely in your browser:

yolo: pose estimation and
object recognition.
whisper: speech recognition.
LLaMA2: text generation.
T5: text generation.
Phi-1.5, and Phi-2: text generation.
Segment Anything Model: Image segmentation.
BLIP: image captioning.

We also provide some command line based examples using state of the art models:

LLaMA v1, v2, and v3: general LLM, includes
the SOLAR-10.7B variant.
Falcon: general LLM.
Codegeex4: Code completion, code interpreter, web search, function calling, repository-level
GLM4: Open Multilingual Multimodal Chat LMs by THUDM
Gemma v1 and v2: 2b and 7b+/9b general LLMs from Google Deepmind.
RecurrentGemma: 2b and 7b
Griffin based models from Google that mix attention with a RNN like state.
Phi-1, Phi-1.5, Phi-2, and Phi-3: 1.3b,
2.7b, and 3.8b general LLMs with performance on par with 7b models.
StableLM-3B-4E1T: a 3b general LLM
pre-trained on 1T tokens of English and code datasets. Also supports
StableLM-2, a 1.6b LLM trained on 2T tokens, as well as the code variants.
Mamba: an inference only
implementation of the Mamba state space model.
Mistral7b-v0.1: a 7b general LLM with
better performance than all publicly available 13b models as of 2023-09-28.
Mixtral8x7b-v0.1: a sparse mixture of
experts 8x7b general LLM with better performance than a Llama 2 70B model with
much faster inference.
StarCoder and
StarCoder2: LLM specialized to code generation.
Qwen1.5: Bilingual (English/Chinese) LLMs.
RWKV v5 and v6: An RNN with transformer level LLM
performance.
Replit-code-v1.5: a 3.3b LLM specialized for code completion.
Yi-6B / Yi-34B: two bilingual
(English/Chinese) general LLMs with 6b and 34b parameters.
Quantized LLaMA: quantized version of
the LLaMA model using the same quantization techniques as
llama.cpp.
Quantized Qwen3 MoE: support gguf quantized models of Qwen3 MoE models.
Stable Diffusion: text to
image generative model, support for the 1.5, 2.1, SDXL 1.0 and Turbo versions.
Wuerstchen: another text to
image generative model.
yolo-v3 and
yolo-v8: object detection and pose
estimation models.
segment-anything: image
segmentation model with prompt.
SegFormer: transformer based semantic segmentation model.
Whisper: speech recognition model.
EnCodec: high-quality audio compression
model using residual vector quantization.
MetaVoice: foundational model for
text-to-speech.
Parler-TTS: large text-to-speech
model.
T5, Bert,
JinaBert : useful for sentence embeddings.
DINOv2: computer vision model trained
using self-supervision (can be used for imagenet classification, depth
evaluation, segmentation).
VGG,
RepVGG: computer vision models.
BLIP: image to text model, can be used to
generate captions for an image.
CLIP: multi-model vision and language
model.
TrOCR: a transformer OCR model, with
dedicated submodels for hand-writing and printed recognition.
Marian-MT: neural machine translation
model, generates the translated text from the input text.
Moondream: tiny computer-vision model
that can answer real-world questions about images.

Run them using commands like:

text

cargo run --example quantized --release

In order to use CUDA add --features cuda to the example command line. If
you have cuDNN installed, use --features cudnn for even more speedups.

There are also some wasm examples for whisper and
llama2.c. You can either build them with
trunk or try them online:
whisper,
llama2,
T5,
Phi-1.5, and Phi-2,
Segment Anything Model.

For LLaMA2, run the following command to retrieve the weight files and start a
test server:

bash

# install target platform 'wasm32-unknown-unknown'
rustup target add wasm32-unknown-unknown

cd candle-wasm-examples/llama2-c
wget https://huggingface.co/spaces/lmz/candle-llama2/resolve/main/model.bin
wget https://huggingface.co/spaces/lmz/candle-llama2/resolve/main/tokenizer.json
trunk serve --release --port 8081

And then head over to
http://localhost:8081/.

Useful External Resources

candle-tutorial: A
very detailed tutorial showing how to convert a PyTorch model to Candle.
candle-lora: Efficient and
ergonomic LoRA implementation for Candle. candle-lora has
out-of-the-box LoRA support for many models from Candle, which can be found
here.
candle-video: Rust library for text-to-video generation (LTX-Video and related models) built on Candle, focused on fast, Python-free inference.
optimisers: A collection of optimisers
including SGD with momentum, AdaGrad, AdaDelta, AdaMax, NAdam, RAdam, and RMSprop.
candle-vllm: Efficient platform for inference and
serving local LLMs including an OpenAI compatible API server.
candle-ext: An extension library to Candle that provides PyTorch functions not currently available in Candle.
candle-coursera-ml: Implementation of ML algorithms from Coursera's Machine Learning Specialization course.
kalosm: A multi-modal meta-framework in Rust for interfacing with local pre-trained models with support for controlled generation, custom samplers, in-memory vector databases, audio transcription, and more.
candle-sampling: Sampling techniques for Candle.
gpt-from-scratch-rs: A port of Andrej Karpathy's Let's build GPT tutorial on YouTube showcasing the Candle API on a toy problem.
candle-einops: A pure rust implementation of the python einops library.
atoma-infer: A Rust library for fast inference at scale, leveraging FlashAttention2 for efficient attention computation, PagedAttention for efficient KV-cache memory management, and multi-GPU support. It is OpenAI api compatible.
llms-from-scratch-rs: A comprehensive Rust translation of the code from Sebastian Raschka's Build an LLM from Scratch book.
vllm.rs: A minimalist vLLM implementation in Rust based on Candle.

If you have an addition to this list, please submit a pull request.

Features

Simple syntax, looks and feels like PyTorch.
- Model training.
- Embed user-defined ops/kernels, such as flash-attention v2.
Backends.
- Optimized CPU backend with optional MKL support for x86 and Accelerate for macs.
- CUDA backend for efficiently running on GPUs, multiple GPU distribution via NCCL.
- WASM support, run your models in a browser.
Included models.
- Language Models.
  - LLaMA v1, v2, and v3 with variants such as SOLAR-10.7B.
  - Falcon.
  - StarCoder, StarCoder2.
  - Phi 1, 1.5, 2, and 3.
  - Mamba, Minimal Mamba
  - Gemma v1 2b and 7b+, v2 2b and 9b.
  - Mistral 7b v0.1.
  - Mixtral 8x7b v0.1.
  - StableLM-3B-4E1T, StableLM-2-1.6B, Stable-Code-3B.
  - Replit-code-v1.5-3B.
  - Bert.
  - Yi-6B and Yi-34B.
  - Qwen1.5, Qwen1.5 MoE, Qwen3 MoE.
  - RWKV v5 and v6.
- Quantized LLMs.
  - Llama 7b, 13b, 70b, as well as the chat and code variants.
  - Mistral 7b, and 7b instruct.
  - Mixtral 8x7b.
  - Zephyr 7b a and b (Mistral-7b based).
  - OpenChat 3.5 (Mistral-7b based).
  - Qwen3 MoE (16B-A3B, 32B-A3B)
- Text to text.
  - T5 and its variants: FlanT5, UL2, MADLAD400 (translation), CoEdit (Grammar correction).
  - Marian MT (Machine Translation).
- Text to image.
  - Stable Diffusion v1.5, v2.1, XL v1.0.
  - Wurstchen v2.
- Image to text.
  - BLIP.
  - TrOCR.
- Audio.
  - Whisper, multi-lingual speech-to-text.
  - EnCodec, audio compression model.
  - MetaVoice-1B, text-to-speech model.
  - Parler-TTS, text-to-speech model.
- Computer Vision Models.
  - DINOv2, ConvMixer, EfficientNet, ResNet, ViT, VGG, RepVGG, ConvNeXT,
    ConvNeXTv2, MobileOne, EfficientVit (MSRA), MobileNetv4, Hiera, FastViT.
  - yolo-v3, yolo-v8.
  - Segment-Anything Model (SAM).
  - SegFormer.
File formats: load models from safetensors, npz, ggml, or PyTorch files.
Serverless (on CPU), small and fast deployments.
Quantization support using the llama.cpp quantized types.

How to use

Cheatsheet:

	Using PyTorch	Using Candle
Creation	`torch.Tensor([[1, 2], [3, 4]])`	`Tensor::new(&[[1f32, 2.], [3., 4.]], &Device::Cpu)?`
Creation	`torch.zeros((2, 2))`	`Tensor::zeros((2, 2), DType::F32, &Device::Cpu)?`
Indexing	`tensor[:, :4]`	`tensor.i((.., ..4))?`
Operations	`tensor.view((2, 2))`	`tensor.reshape((2, 2))?`
Operations	`a.matmul(b)`	`a.matmul(&b)?`
Arithmetic	`a + b`	`&a + &b`
Device	`tensor.to(device="cuda")`	`tensor.to_device(&Device::new_cuda(0)?)?`
Dtype	`tensor.to(dtype=torch.float16)`	`tensor.to_dtype(&DType::F16)?`
Saving	`torch.save({"A": A}, "model.bin")`	`candle::safetensors::save(&HashMap::from([("A", A)]), "model.safetensors")?`
Loading	`weights = torch.load("model.bin")`	`candle::safetensors::load("model.safetensors", &device)`

Structure

candle-core: Core ops, devices, and Tensor struct definition
candle-nn: Tools to build real models
candle-examples: Examples of using the library in realistic settings
candle-kernels: CUDA custom kernels
candle-datasets: Datasets and data loaders.
candle-transformers: transformers-related utilities.
candle-flash-attn: Flash attention v2 layer.
candle-onnx: ONNX model evaluation.

FAQ

Why should I use Candle?

Candle's core goal is to make serverless inference possible. Full machine learning frameworks like PyTorch
are very large, which makes creating instances on a cluster slow. Candle allows deployment of lightweight
binaries.

Secondly, Candle lets you remove Python from production workloads. Python overhead can seriously hurt performance,
and the GIL is a notorious source of headaches.

Finally, Rust is cool! A lot of the HF ecosystem already has Rust crates, like safetensors and tokenizers.

Other ML frameworks

dfdx is a formidable crate, with shapes being included
in types. This prevents a lot of headaches by getting the compiler to complain about shape mismatches right off the bat.
However, we found that some features still require nightly, and writing code can be a bit daunting for non rust experts.

We're leveraging and contributing to other core crates for the runtime so hopefully both crates can benefit from each
other.
burn is a general crate that can leverage multiple backends so you can choose the best
engine for your workload.
tch-rs Bindings to the torch library in Rust. Extremely versatile, but they
bring in the entire torch library into the runtime. The main contributor of tch-rs is also involved in the development
of candle.

Common Errors

Missing symbols when compiling with the mkl feature.

If you get some missing symbols when compiling binaries/tests using the mkl
or accelerate features, e.g. for mkl you get:

text

= note: /usr/bin/ld: (....o): in function `blas::sgemm':
          .../blas-0.22.0/src/lib.rs:1944: undefined reference to `sgemm_' collect2: error: ld returned 1 exit status

  = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo

or for accelerate:

text

Undefined symbols for architecture arm64:
            "_dgemm_", referenced from:
                candle_core::accelerate::dgemm::h1b71a038552bcabe in libcandle_core...
            "_sgemm_", referenced from:
                candle_core::accelerate::sgemm::h2cf21c592cba3c47 in libcandle_core...
          ld: symbol(s) not found for architecture arm64

This is likely due to a missing linker flag that was needed to enable the mkl library. You
can try adding the following for mkl at the top of your binary:

rust

extern crate intel_mkl_src;

or for accelerate:

rust

extern crate accelerate_src;

text

Error: request error: https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/tokenizer.json: status code 401

This is likely because you're not permissioned for the LLaMA-v2 model. To fix
this, you have to register on the huggingface-hub, accept the LLaMA-v2 model
conditions, and set up your
authentication token. See issue
#350 for more details.

Docker build

When building CUDA kernels inside a Dockerfile, nvidia-smi cannot be used to auto-detect compute capability.

You must explicitly set CUDA_COMPUTE_CAP, for example:

text

FROM nvidia/cuda:12.9.0-devel-ubuntu22.04

# Install git and curl
RUN set -eux; \
  apt-get update; \
  apt-get install -y curl git ca-certificates;

# Install Rust
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y

# Clone candle repo
RUN git clone https://github.com/huggingface/candle.git

# Set compute capability for the build
ARG CUDA_COMPUTE_CAP=90
ENV CUDA_COMPUTE_CAP=${CUDA_COMPUTE_CAP}

# Build with explicit compute cap
WORKDIR /app
COPY . .
RUN cargo build --release features cuda

Compiling with flash-attention fails

text

/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:

This is a bug in gcc-11 triggered by the Cuda compiler. To fix this, install a different, supported gcc version - for example gcc-10, and specify the path to the compiler in the NVCC_CCBIN environment variable.

text

env NVCC_CCBIN=/usr/lib/gcc/x86_64-linux-gnu/10 cargo ...

Linking error on windows when running rustdoc or mdbook tests

text

Couldn't compile the test.
---- .\candle-book\src\inference\hub.md - Using_the_hub::Using_in_a_real_model_ (line 50) stdout ----
error: linking with `link.exe` failed: exit code: 1181
//very long chain of linking
 = note: LINK : fatal error LNK1181: cannot open input file 'windows.0.48.5.lib'

Make sure you link all native libraries that might be located outside a project target, e.g., to run mdbook tests, you should run:

text

mdbook test candle-book -L .\target\debug\deps\ `
-L native=$env:USERPROFILE\.cargo\registry\src\index.crates.io-6f17d22bba15001f\windows_x86_64_msvc-0.42.2\lib `
-L native=$env:USERPROFILE\.cargo\registry\src\index.crates.io-6f17d22bba15001f\windows_x86_64_msvc-0.48.5\lib

Extremely slow model load time with WSL

This may be caused by the models being loaded from /mnt/c, more details on
stackoverflow.

Tracking down errors

You can set RUST_BACKTRACE=1 to be provided with backtraces when a candle
error is generated.

CudaRC error

If you encounter an error like this one called Result::unwrap()on anErr value: LoadLibraryExW { source: Os { code: 126, kind: Uncategorized, message: "The specified module could not be found." } } on windows. To fix copy and rename these 3 files (make sure they are in path). The paths depend on your cuda version.
c:\Windows\System32\nvcuda.dll -> cuda.dll
c:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\bin\cublas64_12.dll -> cublas.dll
c:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\bin\curand64_10.dll -> curand.dll

File: candle-book/src/guide/mnist/intro.md

Candle MNIST Tutorial

Introduction

This tutorial provides an introduction to Candle by implementing and training a neural network for MNIST digit classification from scratch.

Throughout this tutorial, you will learn the basics of:

Tensor operations and model construction
Creating and implementing neural network layers
Parameter initialization
Training loop implementation
Saving and loading trained models

Getting Started

Before proceeding, please ensure that you have properly installed Candle by following the instructions in the Installation guide.

File: candle-book/src/guide/mnist/modeling.md

Candle MNIST Tutorial

Modeling

Open src/main.rs in your project folder and insert the following code:

rust

use candle_core::{Device, Result, Tensor};

struct Model {
    first: Tensor,
    second: Tensor,
}

impl Model {
    fn forward(&self, image: &Tensor) -> Result<Tensor> {
        let x = image.matmul(&self.first)?;
        let x = x.relu()?;
        x.matmul(&self.second)
    }
}

fn main() -> Result<()> {
    // Use Device::new_cuda(0)?; to utilize GPU acceleration.
    let device = Device::Cpu;

    let first = Tensor::randn(0f32, 1.0, (784, 100), &device)?;
    let second = Tensor::randn(0f32, 1.0, (100, 10), &device)?;
    let model = Model { first, second };

    let dummy_image = Tensor::randn(0f32, 1.0, (1, 784), &device)?;

    let digit = model.forward(&dummy_image)?;
    println!("Digit {digit:?} digit");
    Ok(())
}

Execute the program with:

bash

$ cargo run --release

> Digit Tensor[dims 1, 10; f32] digit

Since random inputs are provided, expect an incoherent output.

Implementing a `Linear` Layer

To create a more sophisticated layer type, add a bias to the weight to construct the standard Linear layer.

Replace the entire content of src/main.rs with:

rust

use candle_core::{Device, Result, Tensor};

struct Linear {
    weight: Tensor,
    bias: Tensor,
}

impl Linear {
    fn forward(&self, x: &Tensor) -> Result<Tensor> {
        let x = x.matmul(&self.weight)?;
        x.broadcast_add(&self.bias)
    }
}

struct Model {
    first: Linear,
    second: Linear,
}

impl Model {
    fn forward(&self, image: &Tensor) -> Result<Tensor> {
        let x = self.first.forward(image)?;
        let x = x.relu()?;
        self.second.forward(&x)
    }
}

fn main() -> Result<()> {
    // Use Device::new_cuda(0)?; for GPU acceleration.
    // Use Device::Cpu; for CPU computation.
    let device = Device::cuda_if_available(0)?;

    // Initialize model parameters
    let weight = Tensor::randn(0f32, 1.0, (784, 100), &device)?;
    let bias = Tensor::randn(0f32, 1.0, (100, ), &device)?;
    let first = Linear { weight, bias };
    let weight = Tensor::randn(0f32, 1.0, (100, 10), &device)?;
    let bias = Tensor::randn(0f32, 1.0, (10, ), &device)?;
    let second = Linear { weight, bias };
    let model = Model { first, second };

    let dummy_image = Tensor::randn(0f32, 1.0, (1, 784), &device)?;

    // Perform inference
    let digit = model.forward(&dummy_image)?;
    println!("Digit {digit:?} digit");
    Ok(())
}

Execute again with:

bash

$ cargo run --release

> Digit Tensor[dims 1, 10; f32] digit

Utilizing `candle_nn`

Many classical layers (such as Linear) are already implemented in candle-nn.

This Linear implementation follows PyTorch conventions for improved compatibility with existing models, utilizing the transpose of weights rather than direct weights.

Let's simplify our implementation. First, add candle-nn as a dependency:

bash

$ cargo add --git https://github.com/huggingface/candle.git candle-nn

Now, replace the entire content of src/main.rs with:

rust

use candle_core::{Device, Result, Tensor};
use candle_nn::{Linear, Module};

struct Model {
    first: Linear,
    second: Linear,
}

impl Model {
    fn forward(&self, image: &Tensor) -> Result<Tensor> {
        let x = self.first.forward(image)?;
        let x = x.relu()?;
        self.second.forward(&x)
    }
}

fn main() -> Result<()> {
    // Use Device::new_cuda(0)?; for GPU acceleration.
    let device = Device::Cpu;

    // Note the dimension change: (784, 100) -> (100, 784)
    let weight = Tensor::randn(0f32, 1.0, (100, 784), &device)?;
    let bias = Tensor::randn(0f32, 1.0, (100, ), &device)?;
    let first = Linear::new(weight, Some(bias));
    let weight = Tensor::randn(0f32, 1.0, (10, 100), &device)?;
    let bias = Tensor::randn(0f32, 1.0, (10, ), &device)?;
    let second = Linear::new(weight, Some(bias));
    let model = Model { first, second };

    let dummy_image = Tensor::randn(0f32, 1.0, (1, 784), &device)?;

    let digit = model.forward(&dummy_image)?;
    println!("Digit {digit:?} digit");
    Ok(())
}

Execute the final version:

bash

$ cargo run --release

> Digit Tensor[dims 1, 10; f32] digit

File: candle-book/src/guide/mnist/saving_loading.md

Candle MNIST Tutorial

Saving and Loading Models

After training a model, it is useful to save and subsequently load the model parameters. In Candle, this functionality is managed through the VarMap data structure, with parameters stored on disk using the safetensors format.

Saving Model Parameters

Let's modify our training_loop function to include functionality for saving weights:

text

/* Detailed source-code truncated for AI context efficiency. */

bash

$ cargo run --release

> 1 train loss:  2.40485 test acc:  0.11%
> 2 train loss:  2.34161 test acc:  0.14%
> 3 train loss:  2.28841 test acc:  0.17%
> 4 train loss:  2.24158 test acc:  0.19%
> 5 train loss:  2.19898 test acc:  0.23%
> 6 train loss:  2.15927 test acc:  0.26%
> 7 train loss:  2.12161 test acc:  0.29%
> 8 train loss:  2.08549 test acc:  0.32%
> 9 train loss:  2.05053 test acc:  0.35%

Loading Model Parameters

Now that we have saved our model parameters, we can modify the code to load them. The primary change required is to make the varmap variable mutable:

text

/* Detailed source-code truncated for AI context efficiency. */

bash

$ cargo run --release

> 1 train loss:  2.01645 test acc:  0.38%
> 2 train loss:  1.98300 test acc:  0.41%
> 3 train loss:  1.95008 test acc:  0.44%
> 4 train loss:  1.91754 test acc:  0.47%
> 5 train loss:  1.88534 test acc:  0.50%
> 6 train loss:  1.85349 test acc:  0.53%
> 7 train loss:  1.82198 test acc:  0.56%
> 8 train loss:  1.79077 test acc:  0.59%
> 9 train loss:  1.75989 test acc:  0.61%

Note that loading the weights will fail if the specified file does not exist or is incompatible with the current model architecture. Implementing file existence checks and appropriate error handling is left to the user.

File: candle-book/src/guide/mnist/training.md

Candle MNIST Tutorial

Training Implementation

First, let's create a utility function make_linear that accepts a VarBuilder and returns an initialized linear layer. The VarBuilder constructs a VarMap, which is the data structure that stores our trainable parameters.

rust

use candle_core::{Device, Result, Tensor};
use candle_nn::{Linear, Module, VarBuilder, VarMap};

fn make_linear(vs: VarBuilder, in_dim: usize, out_dim: usize) -> Result<Linear> {
    let ws = vs.get_with_hints(
        (out_dim, in_dim),
        "weight",
        candle_nn::init::DEFAULT_KAIMING_NORMAL,
    )?;
    let bound = 1. / (in_dim as f64).sqrt();
    let bs = vs.get_with_hints(
        out_dim,
        "bias",
        candle_nn::Init::Uniform {
            lo: -bound,
            up: bound,
        },
    )?;
    Ok(Linear::new(ws, Some(bs)))
}

Next, let's implement a new method for our model class to accept a VarBuilder and initialize the model. We use VarBuilder::pp to "push prefix" so that the parameter names are organized hierarchically: the first layer weights as first.weight and first.bias, and the second layer weights as second.weight and second.bias.

rust

impl Model {
    fn new(vs: VarBuilder) -> Result<Self> {
        const IMAGE_DIM: usize = 784;
        const HIDDEN_DIM: usize = 100;
        const LABELS: usize = 10;

        let first = make_linear(vs.pp("first"), IMAGE_DIM, HIDDEN_DIM)?;
        let second = make_linear(vs.pp("second"), HIDDEN_DIM, LABELS)?;

        Ok(Self { first, second })
    }

    fn forward(&self, image: &Tensor) -> Result<Tensor> {
        let x = self.first.forward(image)?;
        let x = x.relu()?;
        self.second.forward(&x)
    }
}

Now, let's add the candle-datasets package to our project to access the MNIST dataset:

bash

$ cargo add --git https://github.com/huggingface/candle.git candle-datasets

With the dataset available, we can implement our training loop:

text

/* Detailed source-code truncated for AI context efficiency. */

Finally, let's implement our main function:

rust

pub fn main() -> anyhow::Result<()> {
    let m = candle_datasets::vision::mnist::load()?;
    return training_loop(m);
}

Let's execute the training process:

bash

$ cargo run --release

> 1 train loss:  2.35449 test acc:  0.12%
> 2 train loss:  2.30760 test acc:  0.15%
> ...

File: candle-book/src/guide/hello_world.md

Hello world!

We will now create the hello world of the ML world, building a model capable of solving MNIST dataset.

Open src/main.rs and fill in this content:

rust

# extern crate candle_core;
use candle_core::{Device, Result, Tensor};

struct Model {
    first: Tensor,
    second: Tensor,
}

impl Model {
    fn forward(&self, image: &Tensor) -> Result<Tensor> {
        let x = image.matmul(&self.first)?;
        let x = x.relu()?;
        x.matmul(&self.second)
    }
}

fn main() -> Result<()> {
    // Use Device::new_cuda(0)?; to use the GPU.
    let device = Device::Cpu;

    let first = Tensor::randn(0f32, 1.0, (784, 100), &device)?;
    let second = Tensor::randn(0f32, 1.0, (100, 10), &device)?;
    let model = Model { first, second };

    let dummy_image = Tensor::randn(0f32, 1.0, (1, 784), &device)?;

    let digit = model.forward(&dummy_image)?;
    println!("Digit {digit:?} digit");
    Ok(())
}

Everything should now run with:

bash

cargo run --release

Using a `Linear` layer.

Now that we have this, we might want to complexify things a bit, for instance by adding bias and creating
the classical Linear layer. We can do as such

rust

# extern crate candle_core;
# use candle_core::{Device, Result, Tensor};
struct Linear{
    weight: Tensor,
    bias: Tensor,
}
impl Linear{
    fn forward(&self, x: &Tensor) -> Result<Tensor> {
        let x = x.matmul(&self.weight)?;
        x.broadcast_add(&self.bias)
    }
}

struct Model {
    first: Linear,
    second: Linear,
}

impl Model {
    fn forward(&self, image: &Tensor) -> Result<Tensor> {
        let x = self.first.forward(image)?;
        let x = x.relu()?;
        self.second.forward(&x)
    }
}

This will change the model running code into a new function

rust

# extern crate candle_core;
# use candle_core::{Device, Result, Tensor};
# struct Linear{
#     weight: Tensor,
#     bias: Tensor,
# }
# impl Linear{
#     fn forward(&self, x: &Tensor) -> Result<Tensor> {
#         let x = x.matmul(&self.weight)?;
#         x.broadcast_add(&self.bias)
#     }
# }
# 
# struct Model {
#     first: Linear,
#     second: Linear,
# }
# 
# impl Model {
#     fn forward(&self, image: &Tensor) -> Result<Tensor> {
#         let x = self.first.forward(image)?;
#         let x = x.relu()?;
#         self.second.forward(&x)
#     }
# }
fn main() -> Result<()> {
    // Use Device::new_cuda(0)?; to use the GPU.
    // Use Device::Cpu; to use the CPU.
    let device = Device::cuda_if_available(0)?;

    // Creating a dummy model
    let weight = Tensor::randn(0f32, 1.0, (784, 100), &device)?;
    let bias = Tensor::randn(0f32, 1.0, (100, ), &device)?;
    let first = Linear{weight, bias};
    let weight = Tensor::randn(0f32, 1.0, (100, 10), &device)?;
    let bias = Tensor::randn(0f32, 1.0, (10, ), &device)?;
    let second = Linear{weight, bias};
    let model = Model { first, second };

    let dummy_image = Tensor::randn(0f32, 1.0, (1, 784), &device)?;

    // Inference on the model
    let digit = model.forward(&dummy_image)?;
    println!("Digit {digit:?} digit");
    Ok(())
}

Now it works, it is a great way to create your own layers.
But most of the classical layers are already implemented in candle-nn.

Using `candle_nn`.

For instance Linear is already there.
This Linear is coded with PyTorch layout in mind, to reuse better existing models out there, so it uses the transpose of the weights and not the weights directly.

So instead we can simplify our example:

bash

cargo add --git https://github.com/huggingface/candle.git candle-nn

And rewrite our examples using it

rust

# extern crate candle_core;
# extern crate candle_nn;
use candle_core::{Device, Result, Tensor};
use candle_nn::{Linear, Module};

struct Model {
    first: Linear,
    second: Linear,
}

impl Model {
    fn forward(&self, image: &Tensor) -> Result<Tensor> {
        let x = self.first.forward(image)?;
        let x = x.relu()?;
        self.second.forward(&x)
    }
}

fn main() -> Result<()> {
    // Use Device::new_cuda(0)?; to use the GPU.
    let device = Device::Cpu;

    // This has changed (784, 100) -> (100, 784) !
    let weight = Tensor::randn(0f32, 1.0, (100, 784), &device)?;
    let bias = Tensor::randn(0f32, 1.0, (100, ), &device)?;
    let first = Linear::new(weight, Some(bias));
    let weight = Tensor::randn(0f32, 1.0, (10, 100), &device)?;
    let bias = Tensor::randn(0f32, 1.0, (10, ), &device)?;
    let second = Linear::new(weight, Some(bias));
    let model = Model { first, second };

    let dummy_image = Tensor::randn(0f32, 1.0, (1, 784), &device)?;

    let digit = model.forward(&dummy_image)?;
    println!("Digit {digit:?} digit");
    Ok(())
}

Feel free to modify this example to use Conv2d to create a classical convnet instead.

Now that we have the running dummy code we can get to more advanced topics:

File: candle-book/src/guide/installation.md

Installation

1. Create a new rust app or library

bash

cargo new myapp
cd myapp

2. Add the correct candle version

Standard

bash

cargo add --git https://github.com/huggingface/candle.git candle-core

CUDA

First, make sure that Cuda is correctly installed.

nvcc --version should print information about your Cuda compiler driver.
nvidia-smi --query-gpu=compute_cap --format=csv should print your GPUs compute capability, e.g. something
like:

bash

compute_cap
8.9

You can also compile the Cuda kernels for a specific compute cap using the
CUDA_COMPUTE_CAP=<compute cap> environment variable.

If any of the above commands errors out, please make sure to update your Cuda version.

Add the candle-core crate with the cuda feature:

bash

cargo add --git https://github.com/huggingface/candle.git candle-core --features "cuda"

MKL

You can also see the mkl feature which can get faster inference on CPU.

Add the candle-core crate with the mkl feature:

bash

cargo add --git https://github.com/huggingface/candle.git candle-core --features "mkl"

Metal

Metal is exclusive to MacOS.

Add the candle-core crate with the metal feature:

bash

cargo add --git https://github.com/huggingface/candle.git candle-core --features "metal"

3. Building

Run cargo build to make sure everything can be correctly built.

bash

cargo build

File: README.md Copy Chapter

candle

Get started Copy Chapter

Check out our examples Copy Chapter

Useful External Resources Copy Chapter

Features Copy Chapter

How to use Copy Chapter

Structure Copy Chapter

FAQ Copy Chapter

Why should I use Candle? Copy Section

Other ML frameworks Copy Section

Common Errors Copy Section

Missing symbols when compiling with the mkl feature.

Cannot run the LLaMA examples: access to source requires login credentials

Docker build

Compiling with flash-attention fails

Linking error on windows when running rustdoc or mdbook tests

Extremely slow model load time with WSL

Tracking down errors

CudaRC error

File: candle-book/src/guide/mnist/intro.md Copy Chapter

Candle MNIST Tutorial

Introduction Copy Chapter

Getting Started Copy Chapter

File: candle-book/src/guide/mnist/modeling.md Copy Chapter

Candle MNIST Tutorial

Modeling Copy Chapter

Implementing a `Linear` Layer Copy Chapter

Utilizing `candle_nn` Copy Chapter

File: candle-book/src/guide/mnist/saving_loading.md Copy Chapter

Candle MNIST Tutorial

Saving and Loading Models Copy Chapter

Saving Model Parameters Copy Section

Loading Model Parameters Copy Section

File: candle-book/src/guide/mnist/training.md Copy Chapter

Candle MNIST Tutorial

Training Implementation Copy Chapter

File: candle-book/src/guide/hello_world.md Copy Chapter

Hello world!

Using a `Linear` layer. Copy Chapter

Using `candle_nn`. Copy Chapter

File: candle-book/src/guide/installation.md Copy Chapter

Installation

1. Create a new rust app or library Copy Chapter

2. Add the correct candle version Copy Chapter

Standard Copy Section

CUDA Copy Section

MKL Copy Section

Metal Copy Section

3. Building Copy Chapter

Explore Related Technologies & Topics

File: README.md

Get started

Check out our examples

Useful External Resources

Features

How to use

Structure

FAQ

Why should I use Candle?

Other ML frameworks

Common Errors

File: candle-book/src/guide/mnist/intro.md

Introduction

Getting Started

File: candle-book/src/guide/mnist/modeling.md

Modeling

Implementing a `Linear` Layer

Utilizing `candle_nn`

File: candle-book/src/guide/mnist/saving_loading.md

Saving and Loading Models

Saving Model Parameters

Loading Model Parameters

File: candle-book/src/guide/mnist/training.md

Training Implementation

File: candle-book/src/guide/hello_world.md

Using a `Linear` layer.

Using `candle_nn`.

File: candle-book/src/guide/installation.md

1. Create a new rust app or library

2. Add the correct candle version

Standard

CUDA

MKL

Metal

3. Building