pyo3 Documentation

Rust bindings for the Python interpreter

16,008 stars Rust Rust 1 file · ~14,055 tokens #binding#ffi#python#python-c-api#rust
RAW Doc

File: README.md

PyO3







Rust bindings for Python, including tools for creating native Python extension modules. Running and interacting with Python code from a Rust binary is also supported.

Usage

Requires Rust 1.83 or greater.

PyO3 supports the following Python distributions:

  • CPython 3.9 or greater
  • PyPy 7.3 (Python 3.11+)
  • GraalPy 25.0 or greater (Python 3.12+)

You can use PyO3 to write a native Python module in Rust, or to embed Python in a Rust binary. The following sections explain each of these in turn.

Using Rust from Python

PyO3 can be used to generate a native Python module. The easiest way to try this out for the first time is to use maturin. maturin is a tool for building and publishing Rust-based Python packages with minimal configuration. The following steps install maturin, use it to generate and build a new Python package, and then launch Python to import and execute a function from the package.

First, follow the commands below to create a new directory containing a new Python virtualenv, and install maturin into the virtualenv using Python's package manager, pip:

bash
# (replace string_sum with the desired package name)
$ mkdir string_sum
$ cd string_sum
$ python -m venv .env
$ source .env/bin/activate
$ pip install maturin

Still inside this string_sum directory, now run maturin init. This will generate the new package source. When given the choice of bindings to use, select pyo3 bindings:

bash
$ maturin init
✔ 🤷 What kind of bindings to use? · pyo3
  ✨ Done! New project created string_sum

The most important files generated by this command are Cargo.toml and lib.rs, which will look roughly like the following:

Cargo.toml

toml
[package]
name = "string_sum"
version = "0.1.0"
edition = "2021"

[lib]
# The name of the native library. This is the name which will be used in Python to import the
# library (i.e. `import string_sum`). If you change this, you must also change the name of the
# `#[pymodule]` in `src/lib.rs`.
name = "string_sum"
# "cdylib" is necessary to produce a shared library for Python to import from.
#
# Downstream Rust code (including code in `bin/`, `examples/`, and `tests/`) will not be able
# to `use string_sum;` unless the "rlib" or "lib" crate type is also included, e.g.:
# crate-type = ["cdylib", "rlib"]
crate-type = ["cdylib"]

[dependencies]
pyo3 = "0.29.2"

src/lib.rs

rust
/// A Python module implemented in Rust. The name of this module must match
/// the `lib.name` setting in the `Cargo.toml`, else Python will not be able to
/// import the module.
#[pyo3::pymodule]
mod string_sum {
  use pyo3::prelude::*;

  /// Formats the sum of two numbers as string.
  #[pyfunction]
  fn sum_as_string(a: usize, b: usize) -> PyResult<String> {
    Ok((a + b).to_string())
  }
}

Finally, run maturin develop. This will build the package and install it into the Python virtualenv previously created and activated. The package is then ready to be used from python:

bash
$ maturin develop
# lots of progress output as maturin runs the compilation...
$ python
>>> import string_sum
>>> string_sum.sum_as_string(5, 20)
'25'

When checking runtime performance, run maturin develop --release to build with optimizations.

To make changes to the package, just edit the Rust source code and then re-run maturin develop to recompile.

To run this all as a single copy-and-paste, use the bash script below (replace string_sum in the first command with the desired package name):

bash
mkdir string_sum && cd "$_"
python -m venv .env
source .env/bin/activate
pip install maturin
maturin init --bindings pyo3
maturin develop

If you want to be able to run cargo test or use this project in a Cargo workspace and are running into linker issues, there are some workarounds in the FAQ.

As well as with maturin, it is possible to build using setuptools-rust or manually. Both offer more flexibility than maturin but require more configuration to get started.

Using Python from Rust

To embed Python into a Rust binary, you need to ensure that your Python installation contains a shared library. The following steps demonstrate how to ensure this (for Ubuntu), and then give some example code which runs an embedded Python interpreter.

To install the Python shared library on Ubuntu:

bash
sudo apt install python3-dev

To install the Python shared library on RPM based distributions (e.g. Fedora, Red Hat, SuSE), install the python3-devel package.

Start a new project with cargo new and add pyo3 to the Cargo.toml like this:

toml
[dependencies.pyo3]
version = "0.29.2"
# Enabling this cargo feature will cause PyO3 to start a Python interpreter on first call to `Python::attach`
features = ["auto-initialize"]

Example program displaying the value of sys.version and the current user name:

rust
use pyo3::prelude::*;
use pyo3::types::IntoPyDict;

fn main() -> PyResult<()> {
    Python::attach(|py| {
        let sys = py.import("sys")?;
        let version: String = sys.getattr("version")?.extract()?;

        let locals = [("os", py.import("os")?)].into_py_dict(py)?;
        let code = c"os.getenv('USER') or os.getenv('USERNAME') or 'Unknown'";
        let user: String = py.eval(code, None, Some(&locals))?.extract()?;

        println!("Hello {}, I'm Python {}", user, version);
        Ok(())
    })
}

The guide has a section with lots of examples
about this topic.

Tools and libraries

  • maturin Build and publish crates with pyo3, rust-cpython or cffi bindings as well as rust binaries as python packages
  • setuptools-rust Setuptools plugin for Rust support.
  • pyo3-built Simple macro to expose metadata obtained with the built crate as a PyDict
  • rust-numpy Rust binding of NumPy C-API
  • dict-derive Derive FromPyObject to automatically transform Python dicts into Rust structs
  • pyo3-log Bridge from Rust to Python logging
  • pythonize Serde serializer for converting Rust objects to JSON-compatible Python objects
  • pyo3-async-runtimes Utilities for interoperability with Python's Asyncio library and Rust's async runtimes.
  • rustimport Directly import Rust files or crates from Python, without manual compilation step. Provides pyo3 integration by default and generates pyo3 binding code automatically.
  • pyo3-arrow Lightweight Apache Arrow integration for pyo3.
  • pyo3-bytes Integration between bytes and pyo3.
  • pyo3-object_store Integration between object_store and pyo3.

Examples

  • anise A modern, high-performance toolkit for spacecraft mission design, notably used to help softly land Firefly Blue Ghost on the Moon on 02 Feb 2025.
  • arro3 A minimal Python library for Apache Arrow, connecting to the Rust arrow crate.
  • bed-reader Read and write the PLINK BED format, simply and efficiently.
    • Shows Rayon/ndarray::parallel (including capturing errors, controlling thread num), Python types to Rust generics, Github Actions
  • blake3-py Python bindings for the BLAKE3 cryptographic hash function.
    • Parallelized builds on GitHub Actions for MacOS, Linux, Windows, including free-threaded 3.13t wheels.
  • cellular_raza A cellular agent-based simulation framework for building complex models from a clean slate.
  • connector-x Fastest library to load data from DB to DataFrames in Rust and Python.
  • cryptography Python cryptography library with some functionality in Rust.
  • css-inline CSS inlining for Python implemented in Rust.
  • datafusion-python A Python library that binds to Apache Arrow in-memory query engine DataFusion.
  • deltalake-python Native Delta Lake Python binding based on delta-rs with Pandas integration.
  • fastbloom A fast bloom filter | counting bloom filter implemented by Rust for Rust and Python!
  • fastuuid Python bindings to Rust's UUID library.
  • fast-paseto High-performance PASETO (Platform-Agnostic Security Tokens) implementation with Python bindings.
  • feos Lightning fast thermodynamic modeling in Rust with fully developed Python interface.
  • finalytics Investment Analysis library in Rust | Python.
  • forust A lightweight gradient boosted decision tree library written in Rust.
  • geo-index A Rust crate and Python library for packed, immutable, zero-copy spatial indexes.
  • granian A Rust HTTP server for Python applications.
  • haem A Python library for working on Bioinformatics problems.
  • hifitime A high fidelity time management library for engineering and scientific applications where general relativity and time dilation matter.
  • html2text-rs Python library for converting HTML to markup or plain text.
  • html-py-ever Using html5ever through kuchiki to speed up html parsing and css-selecting.
  • hudi-rs The native Rust implementation for Apache Hudi, with C++ & Python API bindings.
  • inline-python Inline Python code directly in your Rust code.
  • johnnycanencrypt OpenPGP library with Yubikey support.
  • jsonschema A high-performance JSON Schema validator for Python.
  • mocpy Astronomical Python library offering data structures for describing any arbitrary coverage regions on the unit sphere.
  • obstore The simplest, highest-throughput Python interface to Amazon S3, Google Cloud Storage, Azure Storage, & other S3-compliant APIs, powered by Rust.
  • opendal A data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way.
  • orjson Fast Python JSON library.
  • ormsgpack Fast Python msgpack library.
  • pdfcrate An ergonomic, high-level PDF generation library for Rust and Python — a Prawn-style layout API for composing documents, not low-level PDF plumbing.
  • polars Fast multi-threaded DataFrame library in Rust | Python | Node.js.
  • pycrdt Python bindings for the Rust CRDT implementation Yrs.
  • pydantic-core Core validation logic for pydantic written in Rust.
  • primp The fastest python HTTP client that can impersonate web browsers by mimicking their headers and TLS/JA3/JA4/HTTP2 fingerprints.
  • quebec A database-backed background job queue for Python, inspired by Rails' Solid Queue.
  • radiate: A high-performance evolution engine for genetic programming and evolutionary algorithms.
  • rateslib A fixed income library for Python using Rust extensions.
  • river Online machine learning in python, the computationally heavy statistics algorithms are implemented in Rust.
  • robyn A Super Fast Async Python Web Framework with a Rust runtime.
  • rust-python-coverage Example PyO3 project with automated test coverage for Rust and Python.
  • rnet Asynchronous Python HTTP Client with Black Magic
  • sail Unifying stream, batch, and AI workloads with Apache Spark compatibility.
  • tibs A sleek Python library for binary data.
  • tiktoken A fast BPE tokeniser for use with OpenAI's models.
  • tokenizers Python bindings to the Hugging Face tokenizers (NLP) written in Rust.
  • tzfpy A fast package to convert longitude/latitude to timezone name.
  • toml-rs A High-Performance TOML v1.0.0 and v1.1.0 parser for Python written in Rust.
  • utiles Fast Python web-map tile utilities

Articles and other media

Contributing

Everyone is welcomed to contribute to PyO3! There are many ways to support the project, such as:

  • help PyO3 users with issues on GitHub and Discord
  • improve documentation
  • write features and bugfixes
  • publish blogs and examples of how to use PyO3

Our contributing notes and architecture guide have more resources if you wish to volunteer time for PyO3 and are searching where to start.

If you don't have time to contribute yourself but still wish to support the project's future success, some of our maintainers have GitHub sponsorship pages:

License

PyO3 is licensed under the Apache-2.0 license or the MIT license, at your option.

Python is licensed under the Python License.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in PyO3 by you, as defined in the Apache License, shall be dual-licensed as above, without any additional terms or conditions.

File: examples/decorator/README.md

decorator

A project showcasing the example from the Emulating callable objects chapter of the guide.

Building and Testing

To build this package, first install maturin:

shell
pip install maturin

To build and test use maturin develop:

shell
pip install -r requirements-dev.txt
maturin develop
pytest

Alternatively, install nox and run the tests inside an isolated environment:

shell
nox

Copying this example

Use cargo-generate:

bash
$ cargo install cargo-generate
$ cargo generate --git https://github.com/PyO3/pyo3 examples/decorator

(cargo generate will take a little while to clone the PyO3 repo first; be patient when waiting for the command to run.)

File: examples/getitem/README.md

getitem

A project showcasing how to create a __getitem__ override that also showcases how to deal with multiple incoming types

Relevant Documentation

Some of the relevant documentation links for this example:

Building and Testing

To build this package, first install maturin:

shell
pip install maturin

To build and test use maturin develop:

shell
pip install -r requirements-dev.txt
maturin develop
pytest

Alternatively, install nox and run the tests inside an isolated environment:

shell
nox

Copying this example

Use cargo-generate:

bash
$ cargo install cargo-generate
$ cargo generate --git https://github.com/PyO3/pyo3 examples/decorator

(cargo generate will take a little while to clone the PyO3 repo first; be patient when waiting for the command to run.)

File: examples/maturin-starter/README.md

maturin-starter

An example of a basic Python extension module built using PyO3 and maturin.

Building and Testing

To build this package, first install maturin:

shell
pip install maturin

To build and test use maturin develop:

shell
pip install -r requirements-dev.txt
maturin develop && pytest

Alternatively, install nox and run the tests inside an isolated environment:

shell
nox

Copying this example

Use cargo-generate:

bash
$ cargo install cargo-generate
$ cargo generate --git https://github.com/PyO3/pyo3 examples/maturin-starter

(cargo generate will take a little while to clone the PyO3 repo first; be patient when waiting for the command to run.)

File: examples/plugin/README.md

plugin

An example of a Rust app that uses Python for a plugin. A Python extension module built using PyO3 and maturin is used to provide
interface types that can be used to exchange data between Rust and Python. This also deals with how to separately test and load python modules.

Building and Testing

Host application

To run the app itself, you only need to run

shell
cargo run

It will build the app, as well as the plugin API, then run the app, load the plugin and show it working.

Plugin API testing

The plugin API is in a separate crate plugin_api, so you can test it separately from the main app.

To build the API only package, install and build with maturin:

shell
pip install maturin
cd plugin_api
maturin build

Alternatively, install nox and run the tests inside an isolated environment:

shell
nox

Copying this example

Use cargo-generate:

bash
$ cargo install cargo-generate
$ cargo generate --git https://github.com/PyO3/pyo3 examples/plugin

(cargo generate will take a little while to clone the PyO3 repo first; be patient when waiting for the command to run.)

File: examples/setuptools-rust-starter/README.md

setuptools-rust-starter

An example of a basic Python extension module built using PyO3 and setuptools_rust.

Building and Testing

To build this package, first install setuptools_rust:

shell
pip install setuptools_rust

To build and test use python setup.py develop:

shell
pip install -r requirements-dev.txt
python setup.py develop && pytest

Alternatively, install nox and run the tests inside an isolated environment:

shell
nox

Copying this example

Use cargo-generate:

bash
$ cargo install cargo-generate
$ cargo generate --git https://github.com/PyO3/pyo3 examples/setuptools-rust-starter

(cargo generate will take a little while to clone the PyO3 repo first; be patient when waiting for the command to run.)

File: examples/word-count/README.md

word-count

Demonstrates searching for a file in plain python, with rust singlethreaded and with rust multithreaded.

Build

shell
pip install .

Usage

python
from word_count import search_py, search, search_sequential

search_py("foo bar", "foo")
search("foo bar", "foo")
search_sequential("foo bar", "foo")

Testing

To test install nox globally and run

shell
nox

Benchmark

To test install nox globally and run

shell
nox -s bench

Copying this example

Use cargo-generate:

bash
$ cargo install cargo-generate
$ cargo generate --git https://github.com/PyO3/pyo3 examples/word-count

(cargo generate will take a little while to clone the PyO3 repo first; be patient when waiting for the command to run.)

File: examples/README.md

PyO3 Examples

These example crates are a collection of toy extension modules built with PyO3. They are all tested using nox in PyO3's CI.

Below is a brief description of each of these:

Example Description
decorator A project showcasing the example from the Emulating callable objects chapter of the guide.
maturin-starter A template project which is configured to use maturin for development.
setuptools-rust-starter A template project which is configured to use setuptools_rust for development.
plugin Illustrates how to use Python as a scripting language within a Rust application

Note that there are also other examples in the pyo3-ffi/examples
directory that illustrate how to create rust extensions using raw FFI calls into
the CPython C API instead of using PyO3's abstractions.

Creating new projects from these examples

To copy an example, use cargo-generate. Follow the commands below, replacing <example> with the example to start from:

bash
$ cargo install cargo-generate
$ cargo generate --git https://github.com/PyO3/pyo3 examples/<example>

(cargo generate will take a little while to clone the PyO3 repo first; be patient when waiting for the command to run.)

File: guide/src/advanced/sharing-types.md

Sharing `#[pyclass]` types between multiple PyO3 extension modules

[!WARNING]
This is an advanced topic which requires reaching deeply into unsafe code.
PyO3 does not have a stable API for doing this, and the approach documented here may break when updating to newer versions of PyO3.
See issue #1444 for ongoing discussion about this topic.

Some Python extension modules such as NumPy expose a C API which can be consumed by other extension modules to build functionality which directly exchanges native data without needing to go via Python objects.
This allows for higher performance than continually moving data in and out of Python objects.

It is a common request for PyO3 extension modules to be able to share #[pyclass] types (and other native data) across multiple crate/package boundaries in a similar fashion to NumPy.
Because PyO3 extension modules are each compiled as individual cdylib binaries, they cannot depend on each other in the typical Rust way of adding a Cargo dependency (which typically just includes the dependency statically inside the final compiled binary).

Instead, the correct way to share #[pyclass] data between separate PyO3 extension modules is to use #[repr(C)] Rust types and an FFI-like API to exchange data across the cdylib boundaries.
The solution in this subchapter will explain how to do this correctly.

Quick summary

The solution which this subchapter describes shares types across extension module boundaries by defining three Rust crates (see also the example project on GitHub).

  1. A crate base-package-core contains:

    1. #[repr(C)] Rust types and APIs to be shared with downstream extension modules (not #[pyclass] types or #[pymethods]).
    2. An API struct which contains function pointers to the API functions, version information, and any additional data.
    3. A global static variable which stores the API struct, and a function to initialise this at runtime by importing from the base_package Python package.
      The implementations of the APIs defined in step 1 will delegate to the function pointers in the API struct.

  2. A crate base-package which is a PyO3 project containing #[pyclass] thin wrappers around the types exposed in base-package-core.

    1. Implement the API functionality defined in base-package-core and populate the API struct accordingly.
    2. As part of the Python module, export the API struct inside a Python capsule object.
      The initialisation function defined in base-package-core will import this capsule and populate the global static copy of the API struct backing the shared APIs.

  3. A crate derived-package which is a PyO3 project depending on base-package-core (not base-package).

    1. As part of the Python module, import the API struct from the capule exposed by the base_package Python package and store it in derived-package's copy of the global static variable defined in base-package-core.
    2. Use the APIs from base-package-core to implement functionality which directly uses the shared types.

The sections below go into more detail about how to implement various parts of this solution and why the solution is architected this way.

Technical background & limitations

The Python ecosystem provides a well-established mechanism for sharing C APIs between native extension modules using Python capsule objects.
This avoids traditional "dynamic linking" between native extension modules, which would rely on system-specific behavior to locate and load base_package during import of derived_package and to resolve any errors in version mismatch between the two packages.

The capsule mechanism works as follows:

  1. base_package defines a #[repr(C)] "API" struct which is exported in a Python capsule at runtime.
  2. derived_package delegates to Python's extension loading mechanism to locate base_package and load the API from the capsule.
  3. derived_package then contains its own logic to check a compatible version of base_package was loaded; this is necessary to ensure safe exchange of the API struct.

When sharing types between multiple PyO3 extension modules through a capsule, the complexity arises from two main sources:

  • Each Rust extension module may be built with completely separate Rust toolchains and build settings.
    • This means anything which is implementation-defined, such as the layout of #[repr(Rust)] structs, the implementation of std, and even optimizations, might disagree between the two extension modules.
  • Each Rust extension module contains a full statically-linked copy of its own dependencies.
    • Any static global variables which are compiled in the base-package will have a completely independent copy in the derived-package.
      This includes all std globals such as the global allocator and panic hook.
    • Any dependency version mismatches might mean that bugs in dependencies of base-package may not reproduce in the copy in derived-package, (e.g. if the common dependency base-package-core depends on foo 0.1, it is possible base-package will compile with foo 0.1.1 and base-package-core will compile with foo 0.1.2).

Practically speaking, this introduces the following limitations on extension modules wanting to share data in this way:

  • Extensions must take extreme care to ensure that only #[repr(C)] types are shared across the package boundary.

    In particular the default #[repr(Rust)] layout has no stability guarantee; two extension modules sharing a #[repr(Rust)] data type is undefined behavior.
    It is also very easy to accidentally share #[repr(Rust)] types, see the safety note on the PyCapsule type documentation for cases to consider when sharing data.

    [!WARNING] > Beware that PyO3's error type, PyErr, is not #[repr(C)] and cannot be shared across the package boundary > This is an easy mistake to make when exposing fallible APIs which cross the boundary. > There is a later section on error handling which suggests alternative strategies

  • APIs which rely on global variables will not work as expected across the package boundary.
    For example:

    • The #[global_allocator] used by each extension will likely be different - each will need to ensure that any allocations are freed by the same allocator.
    • The std::io locks (e.g. for stdout) will not be shared, so concurrent output from the two extensions may interleave in unexpected ways.

  • PyO3 currently stores #[pyclass] types as global variables in static storage in each cdylib crate which compiles them.
    This means that directly sharing a #[pyclass] type across multiple cdylib crates will currently silently create multiple distinct Python types.
    To avoid this, the shared types cannot be #[pyclass], instead the package exporting the type to Python needs to make private #[pyclass] which wraps the shared type.
    PyO3 may remove this limitation in future.

Using a capsule to create a shared API { #capsules }

Let's start with the example of NumPy as an extension which wants to offer a C API for other native extensions to consume.
The section above already established that Python native extensions can reuse data using the Python capsule type.
It is an opaque wrapper which can be used to exchange arbitrary native data between Python extension modules.
PyO3 provides the PyCapsule type to create and consume capsules from Rust code.

NumPy creates a capsule which contains a pointer to the structure mapping the implementation of the "NumPy C API".
While the exact contents of this structure are generated, the resulting API structure looks something like the following C code:

c
// The API is defined as a fully type-erased array of "void pointers".
//
// This copy of the array is not public API, but internal to the NumPy implementation.
void* PyArray_API[] = {
   // Some fields contain function pointers, type erased
   (void *) PyArray_GetNDArrayCVersion,
   // Some fields are empty
   NULL,
   // Some fields point to Python type objects
   (void *) &PyArray_Type,

   // ... the real API is a few hundred elements long. contents and length are version specific.
};

To consume this array from downstream C projects, NumPy also defines a C header file which uses C macros to define the downstream API in terms of cast indexing into this API structure:

c
static void* PyArray_API[] = NULL;

// cast function pointers back to their correct type
#define PyArray_GetNDArrayCVersion (*(unsigned int (*)(void))PyArray_API[0])

// cast type objects back to `PyTypeObject *`
#define PyArray_Type (* (PyTypeObject *)PyArray_API[2])

// downstream packages must call this function before using any of the other APIS
static int PyArray_ImportNumPyAPI(void)
{
    PyArray_API = /* ... */;
}

To expose an API from Rust code, we'll need to take a similar approach.
We have the choice of either matching NumPy and using an array of opaque pointers, or using a more typed API struct ([as long as it is #[repr(C)] - see the safety docs on the PyCapsule type).
A typed struct helps to avoid mistakes in casting fields to the wrong type incorrectly, however additional care needs to be taken to ensure that the layout of the struct does not change incompatibly across non-breaking versions of the API.
This is discussed further in the next section.

The snippets below sketch out what the NumPy approach looks like in Rust, using either an array of opaque pointers or a typed API struct.
In both cases the public API functions are thin wrappers around the function pointers in the API struct, to provide ergonomics similar to the C macros in the NumPy example.

{{#tabs }}
{{#tab name="Using a pointer array" }}

If using a pointer array, base-package-core will define the pointer array, BaseApi, which contains function pointers and other data as opaque *mut c_void pointers:

text
/* Detailed source-code truncated for AI context efficiency. */

{{#endtab }}
{{#tab name="Using a typed struct" }}

If using a typed struct, base-package-core will define BaseApi as a #[repr(C)] struct with typed fields:

rust
// The API struct #[repr(C)] to ensure a stable layout across package boundaries.
// IMPORTANT: all fields must also be `repr(C)`, function pointers must be `extern "C"`,
// and fields can only be added to the end of the struct in non-breaking versions of the API.
#[repr(C)]
pub struct BaseApi {
    get_api_version: extern "C" fn() -> ApiVersion,
}

/// Global variable which will be used by the API functions
static BASE_API: PyOnceLock = PyOnceLock::new();

impl BaseApi {
    /// Internal method used by the public methods to read pointers from the API
    fn get(py: Python<'_>) -> &'static Self {
        BASE_API.get(py).expect("`base_package` not yet imported")
    }
}

/// Version identifiers for the API, returned by `get_api_version`.
#[repr(C)]
pub struct ApiVersion {
    /* details omitted for now, see below section regarding API versioning */
}

/// Downstream packages must call this method to initialise the library API before
/// calling other functions.
pub fn import_base_package(py: Python<'_>) -> PyResult<()> {
    BASE_API.get_or_try_init(py, || /* details of importing omitted for now, see future sections */)?;
}

// Public functions exported by `base_package-core` are thin wrappers around the function pointers
// in the API struct.

/// Returns the version of the `base_package` API loaded. `ApiVersion` is a `#[repr(C)]` struct
/// so can be safely shared across package boundaries.
#[inline]
pub fn get_api_version(py: Python<'_>) -> ApiVersion {
    (BASE_API.get(py).get_api_version)()
}

{{#endtab }}
{{#endtabs }}

The consumers of the API struct will be compiled against a specific version of the base-package-core crate.
It will only be safe to use the API struct if the version in the API struct matches the version expected by the consumer.
Regardless of the choice made, due to backwards compatibility, the API struct can only grow over time, except when the version signals a breaking change.

The example project demonstrates how to do this with Rust.

API versioning

To safely consume the API struct from downstream packages it is first necessary to perform a version check.
This version check needs to establish that the ABI (Application Binary Interface), i.e. the layout of the API struct, matches the expectations of the consumer.

This means that the ApiVersion type exposed by base-package in the example uses four fields: the three major, minor, and patch version fields from semver, plus an additional abi_version field which is only incremented for breaking changes to the ABI.
Having the abi_version field allows for consumers potentially be compatible even across semver-breaking versions of the API.
This means that e.g. derived-package compiled with version 0.0.3 of base-package could potentially be compatible with base-package version 0.0.2 if the API struct layout did not change between these versions, and the abi_version field was not incremented.

To make the version check straightforward, it is recommended to place the version information at the start of the API struct.
This allows the consumer to first read the capsule data as an ApiVersion structure, and only if the version check passes, reinterpret the rest of the data as the full API struct.

To demonstrate, the following code shows the approximate implementation of the import_base_package function from the example project, which performs the version check and if successful reads the API struct from the capsule:

text
/* Detailed source-code truncated for AI context efficiency. */

Creating a shared `#[pyclass]` type

Once the version checks are complete and the API struct loading is established, this technique can be expanded to share #[pyclass] types across the package boundary.

For a type named SharedType, the steps to achieve this are as follows:

  1. base-package-core defines a #[repr(C)] struct which contains the data to be shared across the boundary.

  2. The BaseApi struct defined in the previous sections is extended to include functions to manipulate this struct (these functions will later be provided by base-package).

    At a minimum, this will probably include:

    • get_shared_type: extern "C" fn() -> Py - a function to get the #[pyclass] Python type object for SharedType.

    • create_shared_type: unsafe extern "C" fn(SharedType) -> Option> - a function to create a new instance of the SharedType struct and return it as a Python object.

      This function is unsafe because the caller must ensure that the thread is attached to the interpreter (Python<'py> is a zero-sized type and not FFI-safe).

      The return type on this function is wrapped in Option to allow for failure - see the error handling section for more details.

    • cast_shared_type: for<'a> extern "C" fn(Borrowed<'a, '_, SharedType>) -> &'a SharedType - a function to extract a reference to the SharedType Rust struct from inside a Python object.

  3. With the API struct extended to include these functions, the base-package-core crate can now implement PyO3 traits for SharedType in terms of those functions.

    The crucial traits are:

    • PyTypeInfo - the get_type function can delegate to the get_shared_type function pointer in the API struct.

    • IntoPyPyObject - the into_pyobject function can delegate to the create_shared_type function pointer in the API struct.

    • FromPyObject<'_> - the extract function can delegate to the cast_shared_type function pointer in the API struct.

  4. base-package implements a #[pyclass] which is a thin wrapper around the SharedType struct, defining its Python functionality.

  5. base-package implements the functions defined in the API struct to manipulate the SharedType, and populates the API struct with pointers to these functions as part of creating the capsule.

  6. derived-package uses the existing import_base_package() function to load the API struct, and then can interact with it as a Python type via PyO3's smart pointers.

The example project demonstrates how to do this with a Series type implementing a "mini DataFrame API", to show how to use these stages to perform real work.

Practical considerations for sharing data across the package boundary

As a reminder, there are two key restrictions to data which is shared across the package boundary:

  • Only types with a stable layout, such as #[repr(C)] types, can be shared.
  • Global variables are not shared across the boundary, and in particular for data sharing, the #[global_allocator] is not shared, so data allocated by one package must be freed by the same package.

This means that all Rust standard library types, such as Vec, String, and Box, cannot be shared across the boundary.
Even Rust tuples do not have a stable layout and cannot be shared.

There is a proposal for a #[repr(crabi)] ABI which would define stable layouts for many Rust types, which would make it easier to share data.
This would still not solve types which contain heap allocations.

For now, a practical solution is to use the abi_stable crate, which provides many equivalents to Rust standard library types with stable internal layouts.
It uses vtables to allow for heap-allocated data to be shared, automatically ensuring the same allocator is used to free data as was used to allocate it.
This is the solution used in the example project to share Vec-like and String-like data across the boundary.

Using vtables introduces overhead (e.g. prevents inlining), however this is a necessary consequence of the limitations of sharing data across the package boundary.

Error handling across the boundary { #error-handling }

Similar to many other types, PyO3's PyErr type is not currently #[repr(C)], so cannot be shared across the package boundary.

The simplest approach to handling errors across the boundary is to use Option return types in the API struct, and to return None on error.
This is the approach taken in the example project.

The downside of this approach is that ? does not trivially work in the implementation of the API functions.
The suggested strategy is:

  • Inside the API function implementations, convert PyResult to Option by using PyErr::restore to write the error to the Python thread state, and returning None on error.
  • The wrappers in base-package-core which delegate to the API functions can then convert the Option back to PyResult by using PyErr::fetch to read the error from the Python thread state.

Wrap-up, limitations & future work

This subchapter has attempted to detail how to use capsule objects to create an API for sharing Rust data between multiple PyO3 extension modules.
While complex, this is a workable technique for projects which need to have this functionality.

PyO3 does not yet offer any particular support for this use case and will likely be unable to provide a fully safe API to achieve this while data exchange is limited to #[repr(C)] types with pitfalls such as duplicate global variables.
Over time PyO3 might accumulate utilities for common pieces of this process.

Some places where this process process easier in the future include:

Better `#[pyclass]` support

At the moment the base-package-core Rust API cannot use PyO3's #[pyclass] macro, because of the global variable backing the #[pyclass] implementation being a problem when duplicated into base-package and derived-package.

A possible solution is that PyO3 could have an option like #[pyclass(shareable)], which could automatically generate the kind of PyTypeInfo / IntoPyObject / FromPyObject implementations which need to be hand-rolled at present.

To make this work, PyO3 would probably need to support module state, and require the base-package which exports the shared type to manually initialize it and store it in module state.

There are many open design questions about how to make that work elegantly.

`#[repr(C)]` error handling

At present, PyO3's PyErr type is a complex internal state machine which allows for lazy creation of Python exceptions.
This was convenient in early implementations of PyO3 but carried internal complexity and overhead.

PyO3's APIs have been trending in recent years towards allowing Rust code to use Rust error types for full control of overhead, e.g. the IntoPyObject and FromPyObject traits have a type Error, and the #[pyfunction] macros accept any Result<T, E> as long as the error type implements a conversion to PyErr.

It is likely that PyO3 will eventually transition the PyErr type to be a thin wrapper around Py, which would allow it to have a stable layout and participate in error handling across the boundary.
This primarily requires consideration about how to nudge existing dependents of PyErr's "lazy" internals towards better practices.

Better support for stable-layout types

The restriction of needing #[repr(C)] types to achieve a stable ABI for data sharing creates a lot of friction at the boundary.
It can also have implications for both efficiency and implementation of the base-package-core types.

The author's experience is that the Rust compiler doesn't yet have a mechanism to comprehensively lint against accidentally sharing types with unstable layouts across the boundary.
There may be value in an upstream effort to implement this so that projects using this capsule mechanism can avoid easy mistakes.

Outside of the Rust project itself, abi_stable crate is the most complete solution currently available to have a convenient way to correctly create a stable ABI.
However, abi_stable also appears to be largely unmaintained, so users wanting to depend on its functionality may need to consider reviving or forking it.

Furthermore, PyO3 currently doesn't implement FromPyObject or IntoPyObject conversions for abi_stable types.
This makes working with these types somewhat awkward / inefficient at the Python boundary.
PyO3 could add an optional feature for this, however given the API surface involved it would be better to resolve the question of maintenance of abi_stable before adding such functionality to PyO3.

File: guide/src/building-and-distribution/multiple-python-versions.md

Supporting multiple Python versions

PyO3 supports all actively-supported Python 3 and PyPy versions.
As much as possible, this is done internally to PyO3 so that your crate's code does not need to adapt to the differences between each version.
However, as Python features grow and change between versions, PyO3 cannot offer a completely identical API for every Python version.
This may require you to add conditional compilation to your crate or runtime checks for the Python version.

This section of the guide first introduces the pyo3-build-config crate, which you can use as a build-dependency to add additional #[cfg] flags which allow you to support multiple Python versions at compile-time.

Second, we'll show how to check the Python version at runtime.
This can be useful when building for multiple versions with the abi3 feature, where the Python API compiled against is not always the same as the one in use.

Conditional compilation for different Python versions

The pyo3-build-config exposes multiple #[cfg] flags which can be used to conditionally compile code for a given Python version.
PyO3 itself depends on this crate, so by using it you can be sure that you are configured correctly for the Python version PyO3 is building against.

This allows us to write code like the following

rust,ignore
#[cfg(Py_3_9)]
fn function_only_supported_on_python_3_9_and_up() {}

#[cfg(not(Py_3_9))]
fn function_only_supported_before_python_3_9() {}

#[cfg(not(Py_LIMITED_API))]
fn function_incompatible_with_stable_abi_feature() {}

The following sections first show how to add these #[cfg] flags to your build process, and then cover some common patterns flags in a little more detail.

To see a full reference of all the #[cfg] flags provided, see the pyo3-build-cfg docs.

Using `pyo3-build-config`

You can use the #[cfg] flags in just two steps:

  1. Add pyo3-build-config to your crate's build dependencies in Cargo.toml:

    toml
    [build-dependencies]
pyo3-build-config = {{#PYO3_CRATE_VERSION}}

  2. Add a build.rs file to your crate with the following contents:

    rust,ignore
    fn main() {
    // If you have an existing build.rs file, just add this line to it.
    pyo3_build_config::use_pyo3_cfgs();
}

After these steps you are ready to annotate your code!

Common usages of `pyo3-build-cfg` flags

The #[cfg] flags added by pyo3-build-cfg can be combined with all of Rust's logic in the #[cfg] attribute to create very precise conditional code generation.
The following are some common patterns implemented using these flags:

text
#[cfg(Py_3_9)]

This #[cfg] marks code that will only be present on Python 3.9 and upwards.
There are similar options Py_3_10, Py_3_11, Py_3_12 and so on for each minor version.

text
#[cfg(not(Py_3_9))]

This #[cfg] marks code that will only be present on Python versions before (but not including) Python 3.9.

text
#[cfg(Py_GIL_DISABLED)]

This #[cfg] marks code that will only be present for free-threaded builds of CPython.
You might use this in situations where the GIL provides thread safety but the free-threaded build needs extra synchronization logic.

text
#[cfg(not(Py_LIMITED_API))]

This #[cfg] marks code that is only available when building for the unlimited Python API (i.e. PyO3's abi3 or abi3t features are not enabled).
This might be useful if you want to ship your extension module as an abi3 or abi3t wheel and also allow users to compile it from source to make use of optimizations only possible with the unlimited API.

text
#[cfg(any(Py_3_9, not(Py_LIMITED_API)))]

This #[cfg] marks code which is available when running Python 3.9 or newer, or when using the unlimited API with an older Python version.
Patterns like this are commonly seen on Python APIs which were added to the limited Python API in a specific minor version.

text
#[cfg(all(Py_GIL_DISABLED, Py_LIMITED_API))]

This marks code which is available only for the free-threaded abi3t stable ABI.
You might use this if you need some sort of custom locking implementation in code that makes strong assumptions about the GIL providing thread safety.

text
#[cfg(PyPy)]

This #[cfg] marks code which is running on PyPy.

Checking the Python version at runtime

When building with PyO3's abi3 or abi3t features, your extension module will be compiled against a specific minimum version of Python, but may be running on newer Python versions.

For example with PyO3's abi3-py39 feature, your extension module will be compiled as if it were for Python 3.9.
If you were using pyo3-build-config, #[cfg(Py_3_9)] would be present.
Your user could freely install and run your abi3 extension module on Python 3.10.

There's no way to detect your user doing that at compile time, so instead you need to fall back to runtime checks.

PyO3 provides the APIs Python::version() and Python::version_info() to query the running Python version.
This allows you to do the following, for example:

rust
use pyo3::Python;

Python::attach(|py| {
    // PyO3 supports Python 3.9 and up.
    assert!(py.version_info() >= (3, 9));
    assert!(py.version_info() >= (3, 9, 0));
});

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#binding Browse binding catalog → #ffi Browse ffi catalog → #python Browse python catalog → #python-c-api Browse python-c-api catalog → #rust Browse rust catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →