cpp Agent Skills & AI Rules

# PyTorch

> PyTorch documentation.


---

# Tensor Accessors

For element-wise operations in custom kernels, use *accessors* to avoid
dynamic dispatch overhead.

## CPU Accessors

```
torch::Tensor foo = torch::rand({12, 12});

// Create accessor - validates type and dimensions once
auto foo_a = foo.accessor<float, 2>();

float trace = 0;
for (int i = 0; i < foo_a.size(0); i++) {
 trace += foo_a[i][i];
}
```

## CUDA Packed Accessors

For CUDA kernels, use *packed accessors* which copy metadata instead of
pointing to it:

```
__global__ void kernel(torch::PackedTensorAccessor64<float, 2> foo, float* trace) {
 int i = threadIdx.x;
 gpuAtomicAdd(trace, foo[i][i]);
}

torch::Tensor foo = torch::rand({12, 12}).cuda();
auto foo_a = foo.packed_accessor64<float, 2>();

float trace = 0;
kernel<<<1, 12>>>(foo_a, &trace);
```

Tip

Use `PackedTensorAccessor32` and `packed_accessor32` for 32-bit indexing,
which is faster on CUDA but may overflow for large tensors.

---

# Tensor Creation

Factory functions create new tensors with different initialization patterns.
All factory functions follow a general schema:

```
torch::<function-name>(<function-specific-options>, <sizes>, <tensor-options>)
```

## Available Factory Functions

- `torch::zeros` - Tensor filled with zeros
- `torch::ones` - Tensor filled with ones
- `torch::empty` - Uninitialized tensor
- `torch::full` - Tensor filled with a single value
- `torch::rand` - Uniform random values on [0, 1)
- `torch::randn` - Standard normal distribution
- `torch::randint` - Random integers in a range
- `torch::arange` - Sequence of integers
- `torch::linspace` - Linearly spaced values
- `torch::logspace` - Logarithmically spaced values
- `torch::eye` - Identity matrix
- `torch::randperm` - Random permutation of integers

## Specifying a Size

Functions that do not require specific arguments can be invoked with just a
size. For example, the following line creates a vector with 5 components:

```
torch::Tensor tensor = torch::ones(5);
```

An `IntArrayRef` is constructed by specifying the size along each dimension in
curly braces. For example, `{2, 3}` for a matrix with two rows and three
columns, `{3, 4, 5}` for a three-dimensional tensor:

```
torch::Tensor tensor = torch::randn({3, 4, 5});
assert(tensor.sizes() == std::vector<int64_t>{3, 4, 5});
```

You can also pass an `std::vector<int64_t>` instead of curly braces.
Use `tensor.size(i)` to access a single dimension.

## Passing Function-Specific Parameters

Some factory functions accept additional parameters. For example, `randint`
takes an upper bound on the value for the integers it generates:

```
torch::Tensor tensor = torch::randint(/*high=*/10, {5, 5});

// With a lower bound
torch::Tensor tensor = torch::randint(/*low=*/3, /*high=*/10, {5, 5});
```

Tip

The size always follows the function-specific arguments.

Attention

Some functions like `arange` do not need a size at all, since it is fully
determined by the function-specific arguments (the range bounds).

## Configuring Properties with TensorOptions

`TensorOptions` configures the data type, layout, device, and
`requires_grad` of a new tensor. The construction axes are:

- `dtype`: the data type of the elements (e.g. `kFloat32`, `kInt64`)
- `layout`: either `kStrided` (dense) or `kSparse`
- `device`: a compute device (e.g. `kCPU`, `kCUDA`)
- `requires_grad`: whether to track gradients

Allowed values:

- `dtype`: `kUInt8`, `kInt8`, `kInt16`, `kInt32`, `kInt64`,
`kFloat32`, `kFloat64`
- `layout`: `kStrided`, `kSparse`
- `device`: `kCPU`, or `kCUDA` (with an optional device index)
- `requires_grad`: `true` or `false`

Tip

Rust-style shorthands exist for dtypes, like `kF32` instead of
`kFloat32`. See
[torch/types.h](https://github.com/pytorch/pytorch/blob/main/torch/csrc/api/include/torch/types.h)
for the full list.

Here is an example of creating a `TensorOptions` object:

```
auto options =
 torch::TensorOptions()
 .dtype(torch::kFloat64)
 .layout(torch::kStrided)
 .device(torch::kCUDA, 1)
 .requires_grad(true);

torch::Tensor tensor = torch::full({3, 4}, /*value=*/123, options);

assert(tensor.dtype() == torch::kFloat64);
assert(tensor.layout() == torch::kStrided);
assert(tensor.device().type() == torch::kCUDA);
assert(tensor.device().index() == 1);
assert(tensor.requires_grad());
```

**Defaults:** Any axis you omit takes its default value: `kFloat32` for dtype,
`kStrided` for layout, `kCPU` for device, and `false` for
`requires_grad`. This means you can omit `TensorOptions` entirely:

```
// A 32-bit float, strided, CPU tensor that does not require a gradient.
torch::Tensor tensor = torch::randn({3, 4});
```

**Shorthand syntax:** For each axis there is a free function in the `torch::`
namespace (`torch::dtype()`, `torch::device()`, `torch::layout()`,
`torch::requires_grad()`). Each returns a `TensorOptions` object that can
be further refined with builder methods:

```
// These are equivalent:
torch::ones(10, torch::TensorOptions().dtype(torch::kFloat32))
torch::ones(10, torch::dtype(torch::kFloat32))

// Chaining:
torch::ones(10, torch::dtype(torch::kFloat32).layout(torch::kStrided))
```

**Implicit construction:** `TensorOptions` is implicitly constructible from
individual values, so when only one axis differs from the default you can
write:

```
torch::ones(10, torch::kFloat32)
```

Putting it all together, a C++ tensor creation call mirrors the Python
equivalent closely:

```
# Python
torch.randn(3, 4, dtype=torch.float32, device=torch.device('cuda', 1), requires_grad=True)
```

```
// C++
torch::randn({3, 4}, torch::dtype(torch::kFloat32).device(torch::kCUDA, 1).requires_grad(true))
```

## Using Externally Created Data

If you already have tensor data allocated in memory (CPU or CUDA), use
`from_blob` to view that memory as a `Tensor`:

```
float data[] = {1, 2, 3, 4, 5, 6};
torch::Tensor tensor = torch::from_blob(data, {2, 3});
```

Note

Tensors created with `from_blob` cannot be resized because ATen does not
own the memory.

## Tensor Conversion

Use `to()` to convert tensors between dtypes and devices. The conversion
creates a new tensor and does not occur in-place:

```
torch::Tensor source = torch::randn({2, 3}, torch::kInt64);

// Convert dtype
torch::Tensor float_tensor = source.to(torch::kFloat32);

// Move to GPU (default CUDA device)
torch::Tensor gpu_tensor = float_tensor.to(torch::kCUDA);

// Specific GPU device
torch::Tensor gpu1_tensor = float_tensor.to(torch::Device(torch::kCUDA, 1));

// Async copy
torch::Tensor async_tensor = gpu_tensor.to(torch::kCPU, /*non_blocking=*/true);
```

Attention

The result of the conversion is a new tensor pointing to new memory,
unrelated to the source tensor.

## Scalars and Zero-Dimensional Tensors

`Scalar` represents a single dynamically-typed number. Like a `Tensor`,
`Scalar` is dynamically typed and can hold any of ATen's number types.
Scalars can be implicitly constructed from C++ number types:

```
namespace torch {
Tensor addmm(Scalar beta, const Tensor & self,
 Scalar alpha, const Tensor & mat1,
 const Tensor & mat2);
Scalar sum(const Tensor & self);
} // namespace torch

// Usage
torch::Tensor a = ...;
torch::Tensor b = ...;
torch::Tensor c = ...;
torch::Tensor r = torch::addmm(1.0, a, .5, b, c);
```

Zero-dimensional tensors hold a single value and can reference elements in
larger tensors:

```
torch::Tensor matrix = torch::rand({10, 20});
matrix[1][2] = 4; // matrix[1][2] is a zero-dimensional tensor
```

---

# ATen: Tensor Library

ATen (A Tensor Library) is the foundational tensor and mathematical operation
library on which all of PyTorch is built. It provides the core `Tensor` class
and hundreds of mathematical operations that work on tensors.

**When to use ATen directly:**

- When writing low-level tensor operations or custom kernels
- When you need direct access to tensor data and metadata
- When working with the PyTorch internals or extending PyTorch

**Basic usage:**

```
#include <ATen/ATen.h>

// Create tensors
at::Tensor a = at::ones({2, 3});
at::Tensor b = at::randn({2, 3});

// Perform operations
at::Tensor c = a + b;
at::Tensor d = at::matmul(a.t(), b);

// Move to GPU
if (at::cuda::is_available()) {
 at::Tensor gpu_tensor = c.to(at::kCUDA);
}
```

For most applications, prefer using the higher-level `torch::` namespace
(see [Neural Network Modules (torch::nn)](../nn/index.html), [Optimizers (torch::optim)](../optim/index.html)) which provides a more user-friendly API.

## Header Files

The following headers are part of the ATen public API:

- `ATen/ATen.h` - Main ATen header
- `ATen/Backend.h` - Backend enumeration
- `ATen/core/Tensor.h` - Tensor class
- `ATen/core/ivalue.h` - IValue type (see [C10: Core Utilities](../c10/index.html))
- `ATen/core/ScalarType.h` - Data type definitions
- `ATen/TensorOptions.h` - Tensor creation options
- `ATen/Scalar.h` - Scalar type
- `ATen/Layout.h` - Tensor layout
- `ATen/DeviceGuard.h` - Device context management
- `ATen/native/TensorShape.h` - Tensor shape operations
- `ATen/cuda/CUDAContext.h` - CUDA context (see [CUDA Support](../cuda/index.html))
- `ATen/cudnn/Descriptors.h` - cuDNN descriptors
- `ATen/mkl/Descriptors.h` - MKL descriptors

Note

The core `at::Tensor` class is defined in a generated header file
(`TensorBody.h`) that only exists after building PyTorch. The documentation
below describes the API manually.

## ATen Categories

- [Tensor Class](tensor.html)
- [Tensor Creation](creation.html)
- [Tensor Indexing](indexing.html)
- [Tensor Accessors](accessors.html)

---

# Tensor Indexing

The PyTorch C++ API provides tensor indexing similar to Python. Use
`torch::indexing` namespace for index types:

```
using namespace torch::indexing;
```

The main difference from Python is that instead of using the `[]` operator,
the C++ API uses the `index` and `index_put_` methods:

- `torch::Tensor::index` -- read elements
- `torch::Tensor::index_put_` -- write elements

## Index Types

The `TensorIndex` class accepts six types of indices via implicit constructors:

| Type | C++ | Python equivalent |
| --- | --- | --- |
| None (unsqueeze) | `None` | `None` |
| Ellipsis | `Ellipsis` or `"..."` | `...` |
| Integer | `0`, `1`, `-1` | `0`, `1`, `-1` |
| Boolean | `true`, `false` | `True`, `False` |
| Slice | `Slice(start, stop, step)` | `start:stop:step` |
| Tensor | `torch::tensor({0, 2})` | `torch.tensor([0, 2])` |

## Getter Operations

| Python | C++ |
| --- | --- |
| `tensor[0]` | `tensor.index({0})` |
| `tensor[-1]` | `tensor.index({-1})` |
| `tensor[1, 2]` | `tensor.index({1, 2})` |
| `tensor[1, :, 3]` | `tensor.index({1, Slice(), 3})` |
| `tensor[None]` | `tensor.index({None})` |
| `tensor[:, None]` | `tensor.index({Slice(), None})` |
| `tensor[...]` | `tensor.index({Ellipsis})` or `tensor.index({"..."})` |
| `tensor[..., 0]` | `tensor.index({Ellipsis, 0})` |
| `tensor[1::2]` | `tensor.index({Slice(1, None, 2)})` |
| `tensor[True]` | `tensor.index({true})` |
| `tensor[torch.tensor([1, 2])]` | `tensor.index({torch::tensor({1, 2})})` |
| `tensor[bool_mask]` | `tensor.index({bool_mask})` |
| `tensor[:, torch.tensor([[0,1],[4,3]])]` | `tensor.index({Slice(), torch::tensor({{0,1},{4,3}})})` |
| `tensor[cond > 0]` | `tensor.index({cond > 0})` |

## Setter Operations

| Python | C++ |
| --- | --- |
| `tensor[0] = 1` | `tensor.index_put_({0}, 1)` |
| `tensor[1, 2] = 1` | `tensor.index_put_({1, 2}, 1)` |
| `tensor[1] = torch.arange(5)` | `tensor.index_put_({1}, torch::arange(5))` |
| `tensor[1::2] = 1` | `tensor.index_put_({Slice(1, None, 2)}, 1)` |
| `tensor[0, 1::2] = torch.tensor([3., 4.])` | `tensor.index_put_({0, Slice(1, None, 2)}, torch::tensor({3., 4.}))` |
| `tensor[...] = 0` | `tensor.index_put_({Ellipsis}, 0)` |
| `tensor[None] = value` | `tensor.index_put_({None}, value)` |
| `tensor[bool_mask] = 0` | `tensor.index_put_({bool_mask}, 0)` |
| `tensor[torch.tensor([0, 2])] = value` | `tensor.index_put_({torch::tensor({0, 2})}, value)` |
| `tensor[1:2, torch.tensor([1,2])] = 0` | `tensor.index_put_({Slice(1, 2), torch::tensor({1, 2})}, 0)` |

The `index_put_` method also accepts an optional `accumulate` parameter.
When `true`, values are added to existing values instead of replacing them:

```
tensor.index_put_({mask}, values, /*accumulate=*/true);
```

## Slice Syntax

The `Slice` constructor signature is:

```
Slice(
 std::optional<c10::SymInt> start = std::nullopt,
 std::optional<c10::SymInt> stop = std::nullopt,
 std::optional<c10::SymInt> step = std::nullopt);
```

Pass `None` for open-ended bounds:

| Python | C++ |
| --- | --- |
| `:` or `::` | `Slice()` or `Slice(None, None)` |
| `1:` | `Slice(1, None)` |
| `:3` | `Slice(None, 3)` |
| `1:3` | `Slice(1, 3)` |
| `1:3:2` | `Slice(1, 3, 2)` |
| `::2` | `Slice(None, None, 2)` |

## Full Example

```
#include <torch/torch.h>

using namespace torch::indexing;

auto tensor = torch::arange(2 * 3 * 4).reshape({2, 3, 4});

// Basic indexing
auto row = tensor.index({0}); // tensor[0]
auto elem = tensor.index({1, 2, 3}); // tensor[1, 2, 3]

// Slicing
auto sliced = tensor.index({Slice(), Slice(0, 2)}); // tensor[:, 0:2]

// None (unsqueeze) and Ellipsis
auto unsqueezed = tensor.index({None}); // tensor[None]
auto last_dim = tensor.index({Ellipsis, -1}); // tensor[..., -1]

// Boolean mask indexing
auto mask = tensor > 10;
auto selected = tensor.index({mask}); // tensor[tensor > 10]

// Integer tensor (fancy) indexing
auto idx = torch::tensor({0, 2});
auto gathered = tensor.index({Slice(), idx}); // tensor[:, [0, 2]]

// Setting values
tensor.index_put_({0, Slice(), 0}, 99); // tensor[0, :, 0] = 99
tensor.index_put_({mask}, 0); // tensor[tensor > 10] = 0
```

---

# Tensor Class

The `at::Tensor` class is the primary tensor class in ATen, representing
a multi-dimensional array with a specific data type and device.

## Tensor

class at::Tensor

The primary tensor class in ATen. Represents a multi-dimensional array
with a specific data type and device.

Tensor()

Default constructor. Creates an undefined tensor.

int64_t dim() const

Returns the number of dimensions of the tensor.

int64_t size(int64_t dim) const

Returns the size of the tensor at the given dimension.

IntArrayRef sizes() const

Returns the sizes of all dimensions.

IntArrayRef strides() const

Returns the strides of all dimensions.

ScalarType scalar_type() const

Returns the data type of the tensor.

Device device() const

Returns the device where the tensor is stored.

bool is_cuda() const

Returns true if the tensor is on a CUDA device.

bool is_cpu() const

Returns true if the tensor is on CPU.

bool requires_grad() const

Returns true if gradients need to be computed for this tensor.

Tensor &requires_grad_(bool requires_grad = true)

Sets whether gradients should be computed for this tensor.

Tensor to(Device device) const

Returns a tensor on the specified device.

Tensor to(ScalarType dtype) const

Returns a tensor with the specified data type.

Tensor contiguous() const

Returns a contiguous tensor with the same data.

void *data_ptr() const

Returns a pointer to the underlying data.

**Example:**

```
#include <ATen/ATen.h>

at::Tensor a = at::ones({2, 2}, at::kInt);
at::Tensor b = at::randn({2, 2});
auto c = a + b.to(at::kInt);
```

## TensorOptions

class at::TensorOptions

A class to specify options for tensor creation, including dtype, device,
layout, and requires_grad.

TensorOptions()

Default constructor.

TensorOptions dtype(ScalarType dtype) const

Returns options with the specified data type.

TensorOptions device(Device device) const

Returns options with the specified device.

TensorOptions layout(Layout layout) const

Returns options with the specified layout.

TensorOptions requires_grad(bool requires_grad) const

Returns options with the specified requires_grad setting.

**Example:**

```
auto options = at::TensorOptions()
 .dtype(at::kFloat)
 .device(at::kCUDA, 0)
 .requires_grad(true);

at::Tensor tensor = at::zeros({3, 4}, options);
```

## Scalar

class at::Scalar

Represents a scalar value that can be converted to various numeric types.

Scalar(int64_t v)

Construct from an integer.

Scalar(double v)

Construct from a double.

template<typename T>
T to() const

Convert to the specified type.

bool isIntegral(bool includeBool = false) const

Returns true if the scalar is an integral type.

bool isFloatingPoint() const

Returns true if the scalar is a floating point type.

## ScalarType

enum class at::ScalarType

Enumeration of data types supported by tensors.

enumerator Byte

8-bit unsigned integer (uint8_t)

enumerator Char

8-bit signed integer (int8_t)

enumerator Short

16-bit signed integer (int16_t)

enumerator Int

32-bit signed integer (int32_t)

enumerator Long

64-bit signed integer (int64_t)

enumerator Half

16-bit floating point (float16)

enumerator Float

32-bit floating point (float)

enumerator Double

64-bit floating point (double)

enumerator Bool

Boolean type

enumerator BFloat16

Brain floating point (bfloat16)

Convenience constants:

- `at::kByte`, `at::kChar`, `at::kShort`, `at::kInt`, `at::kLong`
- `at::kHalf`, `at::kFloat`, `at::kDouble`, `at::kBFloat16`
- `at::kBool`

## DeviceGuard

class DeviceGuard

RAII guard that sets a certain default device in its constructor, and changes it back to the device that was originally active upon destruction.

The device is always reset to the one that was active at the time of construction of the guard. Even if you `set_device` after construction, the destructor will still reset the device to the one that was active at construction time.

This device guard does NOT have an uninitialized state; it is guaranteed to reset a device on exit. If you are in a situation where you *might* want to setup a guard (i.e., are looking for the moral equivalent of std::optional<DeviceGuard>), see [OptionalDeviceGuard](../c10/guards.html#PyTorchclassc10_1_1_optional_device_guard).

Public Functions

explicit DeviceGuard() = delete

No default constructor; see Note [Omitted default constructor from RAII].

inline explicit DeviceGuard([Device](../c10/device.html#_CPPv4N3c106DeviceE) device)

Set the current device to the passed [Device](../c10/device.html#PyTorchstructc10_1_1_device).

inline explicit DeviceGuard([Device](../c10/device.html#_CPPv4N3c106DeviceE) device, const impl::DeviceGuardImplInterface *impl)

This constructor is for testing only.

~DeviceGuard() = default

DeviceGuard(const DeviceGuard&) = delete

Copy is disallowed.

DeviceGuard &operator=(const DeviceGuard&) = delete

DeviceGuard(DeviceGuard &&other) = delete

Move is disallowed, as [DeviceGuard](../c10/guards.html#PyTorchclassc10_1_1_device_guard) does not have an uninitialized state, which is required for moves on types with nontrivial destructors.

DeviceGuard &operator=(DeviceGuard &&other) = delete

inline void reset_device(at::Device device)

Sets the device to the given one.

The specified device must be consistent with the device type originally specified during guard construction.

TODO: The consistency check here is inconsistent with StreamGuard's behavior with set_stream, where a stream on a different device than the original one isn't an error; we just reset the stream and then switch devices.

inline void reset_device(at::Device device, const impl::DeviceGuardImplInterface *impl)

This method is for testing only.

inline void set_index(DeviceIndex index)

Sets the device index to the given one.

The device type is inferred from the original device type the guard was constructed with.

inline [Device](../c10/device.html#_CPPv4N3c106DeviceE) original_device() const

Returns the device that was set at the time the guard was constructed.

inline [Device](../c10/device.html#_CPPv4N3c106DeviceE) current_device() const

Returns the most recent device that was set using this device guard, either from construction, or via set_device.

**Example:**

```
{
 c10::DeviceGuard guard(at::Device(at::kCUDA, 1));
 // Operations here run on CUDA device 1
 auto tensor = at::zeros({2, 2});
}
// Previous device is restored
```

## Layout

enum class at::Layout

Enumeration of tensor memory layouts.

enumerator Strided

Dense tensor with strides.

enumerator Sparse

Sparse tensor (COO format).

enumerator SparseCsr

Sparse tensor in CSR format.

enumerator SparseCsc

Sparse tensor in CSC format.

---

# Custom Autograd Functions

PyTorch allows you to define custom autograd functions with custom forward
and backward implementations.

## Function Base Class

template<class T>
struct Function

To use custom autograd operations, implement a Function subclass with static forward and backward functions:

`forward` can take as many arguments as you want and should return either a variable list or a Variable. Use of any direct Variable arguments will be registered in the graph but no vectors/sets or any other data structures will be traversed. You can use std::optional<Tensor> as one of the arguments and it will be registered as a variable in the graph if the argument has a value. It should take a pointer to `torch::autograd::AutogradContext` as the first argument. Variables can be saved in the `ctx` using `ctx->save_for_backward` (see `torch::autograd::AutogradContext::save_for_backward`) and other data can be saved in the `ctx->saved_data` map (see `torch::autograd::AutogradContext::saved_data`) in the form of `<std::string, at::IValue>` pairs.

`backward` should take a pointer to `torch::autograd::AutogradContext` and a variable list containing as many Variables as there were outputs from `forward` as arguments. It should return as many Variables as there were inputs with each of them containing the gradient w.r.t. its corresponding input. Variables saved in `forward` can be accessed with `ctx->get_saved_variables` (see `torch::autograd::AutogradContext::get_saved_variables`) and other saved data can be accessed from `ctx->saved_data`. To enable compiled autograd support (torch.compile for backward) for your custom autograd operation, you can set MyFunction::is_traceable (see Function::istraceable notes below).

For example:

```
class MyFunction : public Function<MyFunction> {
 public:
 static constexpr bool is_traceable = true;

 static variable_list forward(AutogradContext *ctx, int n, Variable var) {
 // Save data for backward in context
 ctx->saved_data["n"] = n;
 var.mul_(n);
 // Mark var as modified by inplace operation
 ctx->mark_dirty({var});
 return {var};
 }

 static variable_list backward(AutogradContext *ctx, variable_list
 grad_output) {
 // Use data saved in forward
 auto n = ctx->saved_data["n"].toInt();
 return {grad_output[0]*n};
 }
};
```

To use `MyFunction`:

```
Variable x;
auto y = MyFunction::apply(6, x);
// Example backward call
y[0].sum().backward();
```

Public Static Functions

template<typename X = T, typename ...Args>
static auto apply(Args&&... args) -> std::enable_if_t<std::is_same_v<X, T>, forward_t<X, Args...>>

Public Static Attributes

static constexpr bool is_traceable = false

## AutogradContext

struct AutogradContext

Context to save information during `forward` that can be accessed in `backward` in custom autograd operations (see `torch::autograd::Function` for details).

Public Functions

AutogradContext() = default

AutogradContext(const AutogradContext &other) = delete

AutogradContext &operator=(const AutogradContext &other) = delete

AutogradContext(AutogradContext &&other) = delete

AutogradContext &operator=(AutogradContext &&other) = delete

~AutogradContext() = default

AutogradContext(PackedArgs &packed_args)

void save_for_backward(variable_list to_save)

Saves the list of variables for a future call to `backward`.

This should be called at most once from inside of `forward`.

void mark_dirty(const variable_list &inputs)

Marks variables in the list as modified in an in-place operation.

This should be called at most once from inside of `forward` and all arguments should be inputs.

void mark_non_differentiable(const variable_list &outputs)

Marks outputs in the list as not requiring gradients.

This should be called at most once from inside of `forward` and all arguments should be outputs.

void set_materialize_grads(bool value)

variable_list get_saved_variables() const

Get the list of variables that were saved in `forward` using `save_for_backward()`.

Before returning them to the user, a check is made to ensure that they were not modified by any in-place operations.

const std::unordered_set<at::TensorImpl*> &get_and_bump_dirty() const

const std::unordered_set<at::TensorImpl*> &get_non_differentiable() const

bool needs_input_grad(size_t output_edge_index) const

Expose the Node's `task_should_compute_output` method to the cpp custom autograd Function as `needs_input_grad`.

bool needs_input_grad(std::initializer_list<IndexRange> idxs) const

Public Members

ska::flat_hash_map<std::string, at::IValue> saved_data

Can be used to save non-variable data for `backward`.

## Creating Custom Functions

To create a custom autograd function, inherit from `torch::autograd::Function`
and implement the static `forward` and `backward` methods:

**Example:**

```
class MyReLU : public torch::autograd::Function<MyReLU> {
 public:
 static torch::Tensor forward(
 torch::autograd::AutogradContext* ctx,
 torch::Tensor input) {
 ctx->save_for_backward({input});
 return input.clamp_min(0);
 }

 static torch::autograd::variable_list backward(
 torch::autograd::AutogradContext* ctx,
 torch::autograd::variable_list grad_outputs) {
 auto saved = ctx->get_saved_variables();
 auto input = saved[0];
 auto grad_output = grad_outputs[0];
 auto grad_input = grad_output * (input > 0).to(grad_output.dtype());
 return {grad_input};
 }
};

// Usage
auto output = MyReLU::apply(input);
```

## Custom Kernels and AutoDispatchBelowADInplaceOrView

For users implementing custom kernels who want to redispatch below `Autograd` dispatch
keys, use `at::AutoDispatchBelowADInplaceOrView` instead of `InferenceMode`:

```
class ROIAlignFunction : public torch::autograd::Function<ROIAlignFunction> {
 public:
 static torch::autograd::variable_list forward(
 torch::autograd::AutogradContext* ctx,
 const torch::autograd::Variable& input,
 const torch::autograd::Variable& rois,
 double spatial_scale,
 int64_t pooled_height,
 int64_t pooled_width,
 int64_t sampling_ratio,
 bool aligned) {
 ctx->saved_data["spatial_scale"] = spatial_scale;
 ctx->saved_data["pooled_height"] = pooled_height;
 ctx->saved_data["pooled_width"] = pooled_width;
 ctx->saved_data["sampling_ratio"] = sampling_ratio;
 ctx->saved_data["aligned"] = aligned;
 ctx->saved_data["input_shape"] = input.sizes();
 ctx->save_for_backward({rois});

 at::AutoDispatchBelowADInplaceOrView guard;
 auto result = roi_align(
 input, rois, spatial_scale, pooled_height,
 pooled_width, sampling_ratio, aligned);
 return {result};
 }
};
```

For customized inplace and view kernels, see the
[custom kernel tutorial](https://pytorch.org/tutorials/advanced/cpp_extension.html#backward-pass)
for more details.

---

# Gradient Computation

PyTorch provides functions for computing gradients of tensors with respect
to graph leaves.

## Gradient Functions

void torch::autograd::backward(const variable_list &tensors, const variable_list &grad_tensors = {}, std::optional<bool> retain_graph = std::nullopt, bool create_graph = false, const variable_list &inputs = {})

Computes the sum of gradients of given tensors with respect to graph leaves.

The graph is differentiated using the chain rule. If any of `tensors`
are non-scalar (i.e. their data has more than one element) and require
gradient, then the Jacobian-vector product would be computed, in this case
the function additionally requires specifying `grad_tensors`. It should be a
sequence of matching length, that contains the "vector" in the
Jacobian-vector product, usually the gradient of the differentiated function
w.r.t. corresponding tensors (`torch::Tensor()` is an acceptable value for
all tensors that don't need gradient tensors).

This function accumulates gradients in the leaves -- you might need to zero
them before calling it.

:param tensors: Tensors of which the derivative will be computed.
:param grad_tensors: The "vector" in the Jacobian-vector product, usually
gradients w.r.t. each element of corresponding tensors.
`torch::Tensor()` values can be specified for scalar Tensors or ones
that don't require grad. If a `torch::Tensor()` value would be
acceptable for all grad_tensors, then this argument is optional.
:param retain_graph: If `false`, the graph used to compute the grad will
be freed. Note that in nearly all cases setting this option to `true`
is not needed and often can be worked around in a much more efficient
way. Defaults to the value of `create_graph`.
:param create_graph: If `true`, graph of the derivative will be
constructed, allowing to compute higher order derivative products.
Defaults to `false`.
:param inputs: Inputs w.r.t. which the gradient will be accumulated into
`at::Tensor::grad`. All other Tensors will be ignored. If not
provided, the gradient is accumulated into all the leaf Tensors that
were used to compute `tensors`.

variable_list torch::autograd::grad(const variable_list &outputs, const variable_list &inputs, const variable_list &grad_outputs = {}, std::optional<bool> retain_graph = std::nullopt, bool create_graph = false, bool allow_unused = false)

Computes and returns the sum of gradients of outputs with respect to the inputs.

`grad_outputs` should be a sequence of length matching `output` containing the "vector" in Jacobian-vector product, usually the pre-computed gradients w.r.t. each of the outputs. If an output doesn't require_grad, then the gradient can be `torch::Tensor()`).

Parameters:

- **outputs** - outputs of the differentiated function.
- **inputs** - Inputs w.r.t. which the gradient will be returned (and not accumulated into `at::Tensor::grad`).
- **grad_outputs** - The "vector" in the Jacobian-vector product. Usually gradients w.r.t. each output. `torch::Tensor()` values can be specified for scalar Tensors or ones that don't require grad. If a `torch::Tensor()` value would be acceptable for all grad_tensors, then this argument is optional. Default: `{}`.
- **retain_graph** - If `false`, the graph used to compute the grad will be freed. Note that in nearly all cases setting this option to `true` is not needed and often can be worked around in a much more efficient way. Defaults to the value of `create_graph`.
- **create_graph** - If `true`, graph of the derivative will be constructed, allowing to compute higher order derivative products. Default: `false`.
- **allow_unused** - If `false`, specifying inputs that were not used when computing outputs (and therefore their grad is always zero) is an error. Defaults to `false`.

**Example:**

```
#include <torch/torch.h>

auto x = torch::randn({2, 2}, torch::requires_grad());
auto y = x * x;
auto z = y.sum();

// Compute gradients
z.backward();
std::cout << x.grad() << std::endl;

// Or use grad() for specific outputs
auto grads = torch::autograd::grad({z}, {x});
```

## Tensor Gradient Methods

Tensors have built-in methods for gradient computation:

```
// Enable gradient tracking
auto x = torch::randn({2, 2}).requires_grad_(true);

// Check if gradient is required
bool needs_grad = x.requires_grad();

// Access the gradient after backward
auto grad = x.grad();

// Detach from computation graph
auto x_detached = x.detach();
```

---

# Autograd: Automatic Differentiation

PyTorch's autograd system provides automatic differentiation for all operations
on tensors. It records operations on tensors to build a computational graph,
then computes gradients automatically via backpropagation.

**When to use Autograd:**

- When training neural networks (gradients are computed automatically)
- When implementing custom backward passes for specialized operations
- When you need fine-grained control over gradient computation

**Basic usage:**

```
#include <torch/torch.h>

// Create tensor with gradient tracking
auto x = torch::randn({2, 2}, torch::requires_grad());
auto y = x * x;
auto z = y.sum();

// Compute gradients via backpropagation
z.backward();
std::cout << x.grad() << std::endl; // dz/dx = 2x

// Disable gradient tracking for inference
{
 torch::NoGradGuard no_grad;
 auto result = model->forward(input); // No gradients computed
}
```

## Header Files

- `torch/csrc/autograd/autograd.h` - High-level autograd API
- `torch/csrc/autograd/function.h` - Custom autograd functions
- `torch/csrc/autograd/grad_mode.h` - Gradient computation modes
- `torch/csrc/api/include/torch/autograd.h` - C++ Frontend autograd

## Autograd Categories

- [Gradient Computation](gradient.html)
- [Custom Autograd Functions](custom_functions.html)
- [Gradient Modes](modes.html)

---

# Gradient Modes

PyTorch provides RAII guards to control gradient computation behavior.

## NoGradGuard

class torch::NoGradGuard

RAII guard that disables gradient computation within its scope.

NoGradGuard()

Disables gradient computation.

~NoGradGuard()

Restores previous gradient mode.

**Example:**

```
{
 torch::NoGradGuard no_grad;
 // No gradients computed in this scope
 auto result = model->forward(input);
}
```

## InferenceMode

`c10::InferenceMode` is a RAII guard analogous to `NoGradMode` designed for use
when you are certain your operations will have no interactions with autograd
(e.g., model inference). Compared to `NoGradMode`, code run under this mode gets
better performance by disabling autograd-related work like view tracking and version
counter bumps. However, tensors created inside `InferenceMode` have more limitations
when interacting with the autograd system.

class c10::InferenceMode

RAII guard that enables inference mode for optimized inference.
This is more efficient than NoGradGuard for inference-only workloads.

explicit InferenceMode(bool enabled = true)

Enables or disables inference mode.

**Inference Tensors:**

`InferenceMode` can be enabled for a given block of code. Inside `InferenceMode`,
all newly allocated (non-view) tensors are marked as inference tensors. Inference tensors:

- Do not have a version counter, so an error will be raised if you try to read their version
(e.g., because you saved this tensor for backward).
- Are immutable outside `InferenceMode`. An error will be raised if you try to:

- Mutate their data outside InferenceMode.
- Mutate them to `requires_grad=True` outside InferenceMode.
- To work around this, make a clone outside `InferenceMode` to get a normal tensor before mutating.

A non-view tensor is an inference tensor if and only if it was allocated inside `InferenceMode`.
A view tensor is an inference tensor if and only if it is a view of an inference tensor.

**Performance Guarantees:**

Inside an `InferenceMode` block:

- Like `NoGradMode`, all operations do not record `grad_fn` even if their inputs have
`requires_grad=True`. This applies to both inference tensors and normal tensors.
- View operations on inference tensors do not perform view tracking. View and non-view
inference tensors are indistinguishable.
- Inplace operations on inference tensors are guaranteed not to do a version bump.

For more implementation details, see the [RFC-0011-InferenceMode](https://github.com/pytorch/rfcs/pull/17).

**Basic Example:**

```
{
 c10::InferenceMode guard;
 // Optimized inference without gradient tracking
 auto result = model->forward(input);
}
```

**Inference Workload Example:**

```
c10::InferenceMode guard;
model.load_jit(saved_model);
auto inputs = preprocess_tensors(data);
auto out = model.forward(inputs);
auto outputs = postprocess_tensors(out);
```

**Nested InferenceMode:**

Unlike some other guards, `InferenceMode` can be nested with different enabled/disabled states:

```
{
 c10::InferenceMode guard(true);
 // InferenceMode is on
 {
 c10::InferenceMode guard(false);
 // InferenceMode is off
 }
 // InferenceMode is on
}
// InferenceMode is off
```

## InferenceMode vs NoGradMode

`InferenceMode` is preferred over `NoGradMode` for pure inference workloads because
it provides better performance. Key differences:

- Both guards affect tensor execution to skip work not related to inference, but
`InferenceMode` also affects tensor creation while `NoGradMode` doesn't.
- Tensors created inside `InferenceMode` are marked as inference tensors with
certain limitations that apply after exiting `InferenceMode`.
- `InferenceMode` can be nested with enabled/disabled states.

## Migrating from AutoNonVariableTypeMode

The legacy `AutoNonVariableTypeMode` guard (now renamed to
`AutoDispatchBelowADInplaceOrView`) was commonly used for inference workloads
but is unsafe -- it can silently bypass safety checks and produce wrong results.

- **For inference-only workloads** (e.g. loading a pretrained JIT model and
running inference in C++ runtime), use `c10::InferenceMode` as a drop-in
replacement. It preserves the performance characteristics while providing
correctness guarantees.
- **For custom autograd kernels** that need to redispatch below the Autograd
dispatch key, use `AutoDispatchBelowADInplaceOrView` instead:

```
class ROIAlignFunction : public torch::autograd::Function<ROIAlignFunction> {
 public:
 static torch::autograd::variable_list forward(
 torch::autograd::AutogradContext* ctx,
 const torch::autograd::Variable& input,
 const torch::autograd::Variable& rois,
 double spatial_scale, int64_t pooled_height,
 int64_t pooled_width, int64_t sampling_ratio, bool aligned) {
 ctx->saved_data["spatial_scale"] = spatial_scale;
 ctx->save_for_backward({rois});
 at::AutoDispatchBelowADInplaceOrView guard;
 auto result = roi_align(input, rois, spatial_scale,
 pooled_height, pooled_width, sampling_ratio, aligned);
 return {result};
 }
};
```

---

# Device and DeviceType

PyTorch provides device abstractions for writing code that works across
CPU, CUDA, and other backends.

## Device

struct Device

Represents a compute device on which a tensor is located.

A device is uniquely identified by a type, which specifies the type of machine it is (e.g. CPU or CUDA GPU), and a device index or ordinal, which identifies the specific compute device when there is more than one of a certain type. The device index is optional, and in its defaulted state represents (abstractly) "the current device". Further, there are two constraints on the value of the device index, if one is explicitly stored:

1. A negative index represents the current device, a non-negative index represents a specific, concrete device,
2. When the device type is CPU, the device index must be zero.

Public Types

using Type = DeviceType

Public Functions

inline Device(DeviceType type, DeviceIndex index = -1)

Constructs a new `Device` from a `DeviceType` and an optional device index.

Device(const std::string &device_string)

Constructs a `Device` from a string description, for convenience.

The string supplied must follow the following schema: `(cpu|cuda)[:<device-index>]` where `cpu` or `cuda` specifies the device type, and `:<device-index>` optionally specifies a device index.

inline bool operator==(const Device &other) const noexcept

Returns true if the type and index of this `Device` matches that of `other`.

inline bool operator!=(const Device &other) const noexcept

Returns true if the type or index of this `Device` differs from that of `other`.

inline void set_index(DeviceIndex index)

Sets the device index.

inline DeviceType type() const noexcept

Returns the type of device this is.

inline DeviceIndex index() const noexcept

Returns the optional index.

inline bool has_index() const noexcept

Returns true if the device has a non-default index.

inline bool is_cuda() const noexcept

Return true if the device is of CUDA type.

inline bool is_privateuseone() const noexcept

Return true if the device is of PrivateUse1 type.

inline bool is_mps() const noexcept

Return true if the device is of MPS type.

inline bool is_hip() const noexcept

Return true if the device is of HIP type.

inline bool is_ve() const noexcept

Return true if the device is of VE type.

inline bool is_xpu() const noexcept

Return true if the device is of XPU type.

inline bool is_ipu() const noexcept

Return true if the device is of IPU type.

inline bool is_xla() const noexcept

Return true if the device is of XLA type.

inline bool is_mtia() const noexcept

Return true if the device is of MTIA type.

inline bool is_hpu() const noexcept

Return true if the device is of HPU type.

inline bool is_lazy() const noexcept

Return true if the device is of Lazy type.

inline bool is_vulkan() const noexcept

Return true if the device is of Vulkan type.

inline bool is_metal() const noexcept

Return true if the device is of Metal type.

inline bool is_maia() const noexcept

Return true if the device is of MAIA type.

inline bool is_meta() const noexcept

Return true if the device is of META type.

inline bool is_cpu() const noexcept

Return true if the device is of CPU type.

inline bool supports_as_strided() const noexcept

Return true if the device supports arbitrary strides.

std::string str() const

Same string as returned from operator<<.

**Example:**

```
c10::Device cpu_device(c10::kCPU);
c10::Device cuda_device(c10::kCUDA, 0); // CUDA device 0

if (cuda_device.is_cuda()) {
 std::cout << "Using CUDA device " << cuda_device.index() << std::endl;
}
```

## DeviceType

enum class c10::DeviceType

Enumeration of supported device types.

enumerator CPU = 0

CPU device.

enumerator CUDA = 1

NVIDIA CUDA GPU.

enumerator HIP = 6

AMD HIP GPU.

enumerator XLA = 9

XLA / TPU.

enumerator Vulkan = 10

Vulkan GPU.

enumerator Metal = 11

Apple Metal GPU.

enumerator XPU = 12

Intel XPU GPU.

enumerator MPS = 13

Apple Metal Performance Shaders.

enumerator Meta = 14

Meta tensors (shape only, no data).

enumerator HPU = 15

Habana HPU.

enumerator Lazy = 17

Lazy tensors.

enumerator IPU = 18

Graphcore IPU.

enumerator MTIA = 19

Meta training and inference accelerator.

enumerator PrivateUse1 = 20

Custom backend registered via `c10::register_privateuse1_backend()`.

Convenience constants:

- `c10::kCPU`, `c10::kCUDA`, `c10::kHIP`
- `c10::kXLA`, `c10::kVulkan`, `c10::kMetal`
- `c10::kXPU`, `c10::kMPS`, `c10::kMeta`
- `c10::kHPU`, `c10::kLazy`, `c10::kIPU`, `c10::kMTIA`
- `c10::kPrivateUse1`

---

# Device Guards

C10 provides device-agnostic RAII guards for managing the current device
context. These guards work across all backends (CUDA, XPU, etc.) and
automatically restore the previous device when they go out of scope.

For backend-specific guards, see [CUDA Guards](../cuda/guards.html) and [XPU Support](../xpu/index.html).

## DeviceGuard

class DeviceGuard

RAII guard that sets a certain default device in its constructor, and changes it back to the device that was originally active upon destruction.

The device is always reset to the one that was active at the time of construction of the guard. Even if you `set_device` after construction, the destructor will still reset the device to the one that was active at construction time.

This device guard does NOT have an uninitialized state; it is guaranteed to reset a device on exit. If you are in a situation where you *might* want to setup a guard (i.e., are looking for the moral equivalent of std::optional<DeviceGuard>), see OptionalDeviceGuard.

Public Functions

explicit DeviceGuard() = delete

No default constructor; see Note [Omitted default constructor from RAII].

inline explicit DeviceGuard([Device](device.html#_CPPv4N3c106DeviceE) device)

Set the current device to the passed [Device](device.html#PyTorchstructc10_1_1_device).

inline explicit DeviceGuard([Device](device.html#_CPPv4N3c106DeviceE) device, const [impl](../aten/tensor.html#_CPPv4N3c1011DeviceGuard11DeviceGuardE6DevicePKN4impl24DeviceGuardImplInterfaceE)::DeviceGuardImplInterface *impl)

This constructor is for testing only.

~DeviceGuard() = default

DeviceGuard(const [DeviceGuard](../aten/tensor.html#_CPPv4N3c1011DeviceGuard11DeviceGuardERK11DeviceGuard)&) = delete

Copy is disallowed.

[DeviceGuard](../aten/tensor.html#_CPPv4N3c1011DeviceGuardE) &operator=(const [DeviceGuard](../aten/tensor.html#_CPPv4N3c1011DeviceGuardE)&) = delete

DeviceGuard([DeviceGuard](../aten/tensor.html#_CPPv4N3c1011DeviceGuard11DeviceGuardERR11DeviceGuard) &&other) = delete

Move is disallowed, as DeviceGuard does not have an uninitialized state, which is required for moves on types with nontrivial destructors.

[DeviceGuard](../aten/tensor.html#_CPPv4N3c1011DeviceGuardE) &operator=([DeviceGuard](../aten/tensor.html#_CPPv4N3c1011DeviceGuardE) &&other) = delete

inline void reset_device(at::Device device)

Sets the device to the given one.

The specified device must be consistent with the device type originally specified during guard construction.

TODO: The consistency check here is inconsistent with StreamGuard's behavior with set_stream, where a stream on a different device than the original one isn't an error; we just reset the stream and then switch devices.

inline void reset_device(at::Device device, const [impl](../aten/tensor.html#_CPPv4N3c1011DeviceGuard12reset_deviceEN2at6DeviceEPKN4impl24DeviceGuardImplInterfaceE)::DeviceGuardImplInterface *impl)

This method is for testing only.

inline void set_index(DeviceIndex index)

Sets the device index to the given one.

The device type is inferred from the original device type the guard was constructed with.

inline [Device](device.html#_CPPv4N3c106DeviceE) original_device() const

Returns the device that was set at the time the guard was constructed.

inline [Device](device.html#_CPPv4N3c106DeviceE) current_device() const

Returns the most recent device that was set using this device guard, either from construction, or via set_device.

**Example:**

```
#include <c10/core/DeviceGuard.h>

{
 c10::DeviceGuard guard(c10::Device(c10::kCUDA, 1));
 // All operations here run on CUDA device 1
}
// Previous device is restored
```

## OptionalDeviceGuard

class OptionalDeviceGuard

A OptionalDeviceGuard is an RAII class that sets a device to some value on initialization, and resets the device to its original value on destruction.

Morally, a OptionalDeviceGuard is equivalent to std::optional<DeviceGuard>, but with extra constructors and methods as appropriate.

Besides its obvious use (optionally applying a DeviceGuard), OptionalDeviceGuard is often also used for the following idiom:

OptionalDeviceGuard g; for (const auto& t : tensors) { g.set_device(t.device()); do_something_with(t); }

This usage is marginally more efficient than constructing a DeviceGuard every iteration of the for loop, as it avoids an unnecessary device reset.

Unlike DeviceGuard, a OptionalDeviceGuard may be uninitialized. This occurs when you use the nullary constructor, or pass a nullopt to the constructor. Uninitialized OptionalDeviceGuards do *nothing*; they do not know what the original device was and they do not reset on destruction. This is why original_device() and current_device() return std::optional<Device> rather than [Device](device.html#PyTorchstructc10_1_1_device) (as they do in DeviceGuard), and also is why we didn't just provide OptionalDeviceGuard by default and hide DeviceGuard from users.

The semantics of an OptionalDeviceGuard are exactly explained by thinking of it as an std::optional<DeviceGuard>. In particular, an initialized OptionalDeviceGuard doesn't restore device to its value at construction; it restores device to its value *at initialization*. So if you have the program:

```
setDevice(1);
OptionalDeviceGuard g;
setDevice(2);
g.reset_device(Device(DeviceType::CUDA, 3)); // initializes!
```

 On destruction, g will reset device to 2, rather than 1.

An uninitialized OptionalDeviceGuard is distinct from a (initialized) DeviceGuard whose original_device_ and current_device_ match, since the DeviceGuard will still reset the device to original_device_.

Public Functions

explicit OptionalDeviceGuard() = default

Create an uninitialized guard. Set the guard later using reset_device.

inline explicit OptionalDeviceGuard([Device](device.html#_CPPv4N3c106DeviceE) device)

Initialize the guard, setting the current device to the passed [Device](device.html#PyTorchstructc10_1_1_device).

inline explicit OptionalDeviceGuard(std::optional<[Device](device.html#_CPPv4N3c106DeviceE)> device)

Initialize the guard if a [Device](device.html#PyTorchstructc10_1_1_device) is passed; otherwise leave the guard uninitialized.

inline explicit OptionalDeviceGuard([Device](device.html#_CPPv4N3c106DeviceE) device, const impl::DeviceGuardImplInterface *impl)

Constructor for testing only.

~OptionalDeviceGuard() = default

OptionalDeviceGuard(const OptionalDeviceGuard&) = delete

Copy is disallowed.

OptionalDeviceGuard &operator=(const OptionalDeviceGuard&) = delete

OptionalDeviceGuard(OptionalDeviceGuard &&other) = delete

Move is disallowed See Note [Explicit initialization of optional fields] and // Note [Move construction for RAII guards is tricky] for rationale.

OptionalDeviceGuard &operator=(OptionalDeviceGuard &&other) = delete

inline void reset_device(at::Device device)

Sets the device to the given one.

The specified device must be consistent with the device type originally specified during guard construction.

inline void reset_device(at::Device device, const impl::DeviceGuardImplInterface *impl)

For testing only.

inline std::optional<[Device](device.html#_CPPv4N3c106DeviceE)> original_device() const

Returns the device that was set at the time the guard was constructed.

inline std::optional<[Device](device.html#_CPPv4N3c106DeviceE)> current_device() const

Returns the most recent device that was set using this device guard, either from construction, or via reset_device.

**Example:**

```
#include <c10/core/DeviceGuard.h>

c10::OptionalDeviceGuard guard;
if (use_gpu) {
 guard.reset_device(c10::Device(c10::kCUDA, 0));
}
// Guard only restores device