{"owner":"PyO3","repo":"pyo3","hasSkills":true,"totalSkillsCount":1,"totalTokensCount":2776,"categories":["project-spec"],"hasMcp":false,"mcpConfig":null,"found":["Architecture.md"],"skills":{"Architecture.md":"<!-- This file contains a rough overview of the PyO3 codebase. -->\n<!-- Please do not make descriptions too specific, so that we can easily -->\n<!-- keep this file in sync with the codebase. -->\n\n# PyO3: Architecture\n\nThis document roughly describes the high-level architecture of PyO3.\nIf you want to become familiar with the codebase you are in the right place!\n\n## Overview\n\nPyO3 provides a bridge between Rust and Python, based on the [Python/C API].\nThus, PyO3 has low-level bindings of these API as its core.\nOn top of that, we have higher-level bindings to operate Python objects safely.\nAlso, to define Python classes and functions in Rust code, we have `trait PyClass` and a set of\nprotocol traits (e.g., `PyIterProtocol`) for supporting object protocols (i.e., `__dunder__` methods).\nSince implementing `PyClass` requires lots of boilerplate, we have a proc-macro `#[pyclass]`.\n\nTo summarize, there are six main parts to the PyO3 codebase.\n\n1. [Low-level bindings of Python/C API.](#1-low-level-bindings-of-python-capi)\n   - [`pyo3-ffi`] and [`src/ffi`]\n2. [Bindings to Python objects.](#2-bindings-to-python-objects)\n   - [`src/instance.rs`] and [`src/types`]\n3. [`PyClass` and related functionalities.](#3-pyclass-and-related-functionalities)\n   - [`src/pycell.rs`], [`src/pyclass.rs`], and more\n4. [Procedural macros to simplify usage for users.](#4-procedural-macros-to-simplify-usage-for-users)\n   - [`src/impl_`], [`pyo3-macros`] and [`pyo3-macros-backend`]\n5. [`build.rs` and `pyo3-build-config`](#5-buildrs-and-pyo3-build-config)\n   - [`build.rs`](https://github.com/PyO3/pyo3/tree/main/build.rs)\n   - [`pyo3-build-config`]\n\n## 1. Low-level bindings of Python/C API\n\n[`pyo3-ffi`] contains wrappers of the [Python/C API]. This is currently done by hand rather than\nautomated tooling because:\n\n- it gives us best control about how to adapt C conventions to Rust, and\n- there are many Python interpreter versions we support in a single set of files.\n\nWe aim to provide straight-forward Rust wrappers resembling the file structure of [`cpython/Include`](https://github.com/python/cpython/tree/main/Include).\n\nWe are continuously updating the module to match the latest CPython version which PyO3 supports (i.e. as of time of writing Python 3.13). The tracking issue is [#1289](https://github.com/PyO3/pyo3/issues/1289), and contribution is welcome.\n\nIn the [`pyo3-ffi`] crate, there is lots of conditional compilation such as `#[cfg(Py_LIMITED_API)]`,\n`#[cfg(Py_3_9)]`, and `#[cfg(PyPy)]`.\n`Py_LIMITED_API` corresponds to `#define Py_LIMITED_API` macro in Python/C API.\nWith `Py_LIMITED_API`, we can build a Python-version-agnostic binary called an\n[abi3 wheel](https://pyo3.rs/latest/building-and-distribution.html#py_limited_apiabi3abi3t).\n`Py_3_9` means that the API is available from Python >= 3.9.\nThere are also `Py_3_10`, `Py_3_11`, and so on.\n`PyPy` means that the API definition is for PyPy.\nThose flags are set in [`build.rs`](#6-buildrs-and-pyo3-build-config).\n\n## 2. Bindings to Python objects\n\n[`src/types`] contains bindings to [built-in types](https://docs.python.org/3/library/stdtypes.html)\nof Python, such as `dict` and `list`.\nFor historical reasons, Python's `object` is called `PyAny` in PyO3 and located in [`src/types/any.rs`].\n\nCurrently, `PyAny` is a straightforward wrapper of `ffi::PyObject`, defined as:\n\n```rust\n#[repr(transparent)]\npub struct PyAny(UnsafeCell<ffi::PyObject>);\n```\n\nConcrete Python objects are implemented by wrapping `PyAny`, e.g.,:\n\n```rust\n#[repr(transparent)]\npub struct PyDict(PyAny);\n```\n\nThese types are not intended to be accessed directly, and instead are used through the `Py<T>` and `Bound<T>` smart pointers.\n\nWe have some macros in [`src/types/mod.rs`] which make it easier to implement APIs for concrete Python types.\n\n## 3. `PyClass` and related functionalities\n\n[`src/pycell.rs`], [`src/pyclass.rs`], and [`src/type_object.rs`] contain types and\ntraits to make `#[pyclass]` work.\nAlso, [`src/pyclass_init.rs`] and [`src/impl_/pyclass.rs`] have related functionalities.\n\nTo realize object-oriented programming in C, all Python objects have `ob_base: PyObject` as their\nfirst field in their structure definition. Thanks to this guarantee, casting `*mut A` to `*mut PyObject`\nis valid if `A` is a Python object.\n\nTo ensure this guarantee, we have a wrapper struct `PyClassObject<T>` in [`src/pycell/impl_.rs`] which is roughly:\n\n```rust\n#[repr(C)]\npub struct PyClassObject<T> {\n    ob_base: crate::ffi::PyObject,\n    inner: T,\n}\n```\n\nThus, when copying a Rust struct to a Python object, we first allocate `PyClassObject` on the Python heap and then\nmove `T` into it.\n\nThe primary way to interact with Python objects implemented in Rust is through the `Bound<'py, T>` smart pointer.\nBy having the `'py` lifetime of the `Python<'py>` token, this ties the lifetime of the `Bound<'py, T>` smart pointer to the lifetime for which the thread is attached to the Python interpreter and allows PyO3 to call Python APIs at maximum efficiency.\n\n`Bound<'py, T>` requires that `T` implements `PyClass`.\nThis trait is somewhat complex and derives many traits, but the most important one is `PyTypeInfo`\nin [`src/type_object.rs`].\n`PyTypeInfo` is also implemented for built-in types.\nIn Python, all objects have their types, and types are also objects of `type`.\nFor example, you can see `type({})` shows `dict` and `type(type({}))` shows `type` in Python REPL.\n`T: PyTypeInfo` implies that `T` has a corresponding type object.\n\n### Protocol methods\n\nPython has some built-in special methods called dunder methods, such as `__iter__`.\nThey are called \"slots\" in the [abstract objects layer](https://docs.python.org/3/c-api/abstract.html) in\nPython/C API.\nWe provide a way to implement those protocols similarly, by recognizing special\nnames in `#[pymethods]`, with a few new ones for slots that can not be\nimplemented in Python, such as GC support.\n\n## 4. Procedural macros to simplify usage for users.\n\n[`pyo3-macros`] provides five proc-macro APIs: `pymodule`, `pyfunction`, `pyclass`,\n`pymethods`, and `#[derive(FromPyObject)]`.\n[`pyo3-macros-backend`] has the actual implementations of these APIs.\n[`src/impl_`] contains `#[doc(hidden)]` functionality used in code generated by these proc-macros,\nsuch as parsing function arguments.\n\n## 5. `build.rs` and `pyo3-build-config`\n\nPyO3 supports a wide range of OSes, interpreters and use cases. The correct environment must be\ndetected at build time in order to set up relevant conditional compilation correctly. This logic\nis captured in the [`pyo3-build-config`] crate, which is a `build-dependency` of `pyo3` and\n`pyo3-macros`, and can also be used by downstream users in the same way.\n\nIn [`pyo3-build-config`]'s `build.rs` the build environment is detected and inlined into the crate\nas a \"config file\". This works in all cases except for cross-compiling, where it is necessary to\ncapture this from the `pyo3` `build.rs` to get some extra environment variables that Cargo doesn't\nset for build dependencies.\n\nThe `pyo3` `build.rs` also runs some safety checks such as ensuring the Python version detected is\nactually supported.\n\nSome of the functionality of `pyo3-build-config`:\n\n- Find the interpreter for build and detect the Python version.\n  - We have to set some version flags like `#[cfg(Py_3_9)]`.\n  - If the interpreter is PyPy, we set `#[cfg(PyPy)`.\n  - If the `PYO3_CONFIG_FILE` environment variable is set then that file's contents will be used\n    instead of any detected configuration.\n  - If the `PYO3_NO_PYTHON` environment variable is set then the interpreter detection is bypassed\n    entirely and only abi3 extensions can be built.\n- Check if we are building a Python extension.\n  - If we are building an extension (e.g., Python library installable by `pip`),\n    we don't link `libpython` on most platforms (to allow for statically-linked Python interpreters).\n    The `PYO3_BUILD_EXTENSION_MODULE` environment variable suppresses linking.\n- Cross-compiling configuration\n  - If `TARGET` architecture and `HOST` architecture differ, we can find cross compile information\n    from environment variables (`PYO3_CROSS_LIB_DIR`, `PYO3_CROSS_PYTHON_VERSION` and\n    `PYO3_CROSS_PYTHON_IMPLEMENTATION`) or system files.\n    When cross compiling extension modules it is often possible to make it work without any\n    additional user input.\n  - On Windows, `pyo3-ffi` uses Rust's `raw-dylib` linking feature to link against the Python DLL\n    directly without needing import libraries (`.lib` files). The build script emits a `pyo3_dll`\n    cfg with the target DLL name, and the `extern_libpython!` macro expands to the appropriate\n    `#[link(name = \"...\", kind = \"raw-dylib\")]` attribute. This enables cross compiling Python\n    extensions for Windows without having to install any Windows Python libraries.\n    This can be opted out by setting `PYO3_USE_RAW_DYLIB` to anything other than \"1\".\n\n<!-- External Links -->\n\n[python/c api]: https://docs.python.org/3/c-api/\n\n<!-- Crates -->\n\n[`pyo3-macros`]: https://github.com/PyO3/pyo3/tree/main/pyo3-macros\n[`pyo3-macros-backend`]: https://github.com/PyO3/pyo3/tree/main/pyo3-macros-backend\n[`pyo3-build-config`]: https://github.com/PyO3/pyo3/tree/main/pyo3-build-config\n[`pyo3-ffi`]: https://github.com/PyO3/pyo3/tree/main/pyo3-ffi\n\n<!-- Directories -->\n\n[`src/class`]: https://github.com/PyO3/pyo3/tree/main/src/class\n[`src/ffi`]: https://github.com/PyO3/pyo3/tree/main/src/ffi\n[`src/types`]: https://github.com/PyO3/pyo3/tree/main/src/types\n\n<!-- Files -->\n\n[`src/impl_`]: https://github.com/PyO3/pyo3/blob/main/src/impl_\n[`src/instance.rs`]: https://github.com/PyO3/pyo3/tree/main/src/instance.rs\n[`src/pycell.rs`]: https://github.com/PyO3/pyo3/tree/main/src/pycell.rs\n[`src/pyclass.rs`]: https://github.com/PyO3/pyo3/tree/main/src/pyclass.rs\n[`src/pyclass_init.rs`]: https://github.com/PyO3/pyo3/tree/main/src/pyclass_init.rs\n[`src/pyclass_slot.rs`]: https://github.com/PyO3/pyo3/tree/main/src/pyclass_slot.rs\n[`src/type_object.rs`]: https://github.com/PyO3/pyo3/tree/main/src/type_object.rs\n[`src/class/methods.rs`]: https://github.com/PyO3/pyo3/tree/main/src/class/methods.rs\n[`src/class/impl_.rs`]: https://github.com/PyO3/pyo3/tree/main/src/class/impl_.rs\n[`src/types/any.rs`]: https://github.com/PyO3/pyo3/tree/main/src/types/any.rs\n[`src/types/mod.rs`]: https://github.com/PyO3/pyo3/tree/main/src/types/mod.rs\n"},"files":{"Architecture.md":"<!-- This file contains a rough overview of the PyO3 codebase. -->\n<!-- Please do not make descriptions too specific, so that we can easily -->\n<!-- keep this file in sync with the codebase. -->\n\n# PyO3: Architecture\n\nThis document roughly describes the high-level architecture of PyO3.\nIf you want to become familiar with the codebase you are in the right place!\n\n## Overview\n\nPyO3 provides a bridge between Rust and Python, based on the [Python/C API].\nThus, PyO3 has low-level bindings of these API as its core.\nOn top of that, we have higher-level bindings to operate Python objects safely.\nAlso, to define Python classes and functions in Rust code, we have `trait PyClass` and a set of\nprotocol traits (e.g., `PyIterProtocol`) for supporting object protocols (i.e., `__dunder__` methods).\nSince implementing `PyClass` requires lots of boilerplate, we have a proc-macro `#[pyclass]`.\n\nTo summarize, there are six main parts to the PyO3 codebase.\n\n1. [Low-level bindings of Python/C API.](#1-low-level-bindings-of-python-capi)\n   - [`pyo3-ffi`] and [`src/ffi`]\n2. [Bindings to Python objects.](#2-bindings-to-python-objects)\n   - [`src/instance.rs`] and [`src/types`]\n3. [`PyClass` and related functionalities.](#3-pyclass-and-related-functionalities)\n   - [`src/pycell.rs`], [`src/pyclass.rs`], and more\n4. [Procedural macros to simplify usage for users.](#4-procedural-macros-to-simplify-usage-for-users)\n   - [`src/impl_`], [`pyo3-macros`] and [`pyo3-macros-backend`]\n5. [`build.rs` and `pyo3-build-config`](#5-buildrs-and-pyo3-build-config)\n   - [`build.rs`](https://github.com/PyO3/pyo3/tree/main/build.rs)\n   - [`pyo3-build-config`]\n\n## 1. Low-level bindings of Python/C API\n\n[`pyo3-ffi`] contains wrappers of the [Python/C API]. This is currently done by hand rather than\nautomated tooling because:\n\n- it gives us best control about how to adapt C conventions to Rust, and\n- there are many Python interpreter versions we support in a single set of files.\n\nWe aim to provide straight-forward Rust wrappers resembling the file structure of [`cpython/Include`](https://github.com/python/cpython/tree/main/Include).\n\nWe are continuously updating the module to match the latest CPython version which PyO3 supports (i.e. as of time of writing Python 3.13). The tracking issue is [#1289](https://github.com/PyO3/pyo3/issues/1289), and contribution is welcome.\n\nIn the [`pyo3-ffi`] crate, there is lots of conditional compilation such as `#[cfg(Py_LIMITED_API)]`,\n`#[cfg(Py_3_9)]`, and `#[cfg(PyPy)]`.\n`Py_LIMITED_API` corresponds to `#define Py_LIMITED_API` macro in Python/C API.\nWith `Py_LIMITED_API`, we can build a Python-version-agnostic binary called an\n[abi3 wheel](https://pyo3.rs/latest/building-and-distribution.html#py_limited_apiabi3abi3t).\n`Py_3_9` means that the API is available from Python >= 3.9.\nThere are also `Py_3_10`, `Py_3_11`, and so on.\n`PyPy` means that the API definition is for PyPy.\nThose flags are set in [`build.rs`](#6-buildrs-and-pyo3-build-config).\n\n## 2. Bindings to Python objects\n\n[`src/types`] contains bindings to [built-in types](https://docs.python.org/3/library/stdtypes.html)\nof Python, such as `dict` and `list`.\nFor historical reasons, Python's `object` is called `PyAny` in PyO3 and located in [`src/types/any.rs`].\n\nCurrently, `PyAny` is a straightforward wrapper of `ffi::PyObject`, defined as:\n\n```rust\n#[repr(transparent)]\npub struct PyAny(UnsafeCell<ffi::PyObject>);\n```\n\nConcrete Python objects are implemented by wrapping `PyAny`, e.g.,:\n\n```rust\n#[repr(transparent)]\npub struct PyDict(PyAny);\n```\n\nThese types are not intended to be accessed directly, and instead are used through the `Py<T>` and `Bound<T>` smart pointers.\n\nWe have some macros in [`src/types/mod.rs`] which make it easier to implement APIs for concrete Python types.\n\n## 3. `PyClass` and related functionalities\n\n[`src/pycell.rs`], [`src/pyclass.rs`], and [`src/type_object.rs`] contain types and\ntraits to make `#[pyclass]` work.\nAlso, [`src/pyclass_init.rs`] and [`src/impl_/pyclass.rs`] have related functionalities.\n\nTo realize object-oriented programming in C, all Python objects have `ob_base: PyObject` as their\nfirst field in their structure definition. Thanks to this guarantee, casting `*mut A` to `*mut PyObject`\nis valid if `A` is a Python object.\n\nTo ensure this guarantee, we have a wrapper struct `PyClassObject<T>` in [`src/pycell/impl_.rs`] which is roughly:\n\n```rust\n#[repr(C)]\npub struct PyClassObject<T> {\n    ob_base: crate::ffi::PyObject,\n    inner: T,\n}\n```\n\nThus, when copying a Rust struct to a Python object, we first allocate `PyClassObject` on the Python heap and then\nmove `T` into it.\n\nThe primary way to interact with Python objects implemented in Rust is through the `Bound<'py, T>` smart pointer.\nBy having the `'py` lifetime of the `Python<'py>` token, this ties the lifetime of the `Bound<'py, T>` smart pointer to the lifetime for which the thread is attached to the Python interpreter and allows PyO3 to call Python APIs at maximum efficiency.\n\n`Bound<'py, T>` requires that `T` implements `PyClass`.\nThis trait is somewhat complex and derives many traits, but the most important one is `PyTypeInfo`\nin [`src/type_object.rs`].\n`PyTypeInfo` is also implemented for built-in types.\nIn Python, all objects have their types, and types are also objects of `type`.\nFor example, you can see `type({})` shows `dict` and `type(type({}))` shows `type` in Python REPL.\n`T: PyTypeInfo` implies that `T` has a corresponding type object.\n\n### Protocol methods\n\nPython has some built-in special methods called dunder methods, such as `__iter__`.\nThey are called \"slots\" in the [abstract objects layer](https://docs.python.org/3/c-api/abstract.html) in\nPython/C API.\nWe provide a way to implement those protocols similarly, by recognizing special\nnames in `#[pymethods]`, with a few new ones for slots that can not be\nimplemented in Python, such as GC support.\n\n## 4. Procedural macros to simplify usage for users.\n\n[`pyo3-macros`] provides five proc-macro APIs: `pymodule`, `pyfunction`, `pyclass`,\n`pymethods`, and `#[derive(FromPyObject)]`.\n[`pyo3-macros-backend`] has the actual implementations of these APIs.\n[`src/impl_`] contains `#[doc(hidden)]` functionality used in code generated by these proc-macros,\nsuch as parsing function arguments.\n\n## 5. `build.rs` and `pyo3-build-config`\n\nPyO3 supports a wide range of OSes, interpreters and use cases. The correct environment must be\ndetected at build time in order to set up relevant conditional compilation correctly. This logic\nis captured in the [`pyo3-build-config`] crate, which is a `build-dependency` of `pyo3` and\n`pyo3-macros`, and can also be used by downstream users in the same way.\n\nIn [`pyo3-build-config`]'s `build.rs` the build environment is detected and inlined into the crate\nas a \"config file\". This works in all cases except for cross-compiling, where it is necessary to\ncapture this from the `pyo3` `build.rs` to get some extra environment variables that Cargo doesn't\nset for build dependencies.\n\nThe `pyo3` `build.rs` also runs some safety checks such as ensuring the Python version detected is\nactually supported.\n\nSome of the functionality of `pyo3-build-config`:\n\n- Find the interpreter for build and detect the Python version.\n  - We have to set some version flags like `#[cfg(Py_3_9)]`.\n  - If the interpreter is PyPy, we set `#[cfg(PyPy)`.\n  - If the `PYO3_CONFIG_FILE` environment variable is set then that file's contents will be used\n    instead of any detected configuration.\n  - If the `PYO3_NO_PYTHON` environment variable is set then the interpreter detection is bypassed\n    entirely and only abi3 extensions can be built.\n- Check if we are building a Python extension.\n  - If we are building an extension (e.g., Python library installable by `pip`),\n    we don't link `libpython` on most platforms (to allow for statically-linked Python interpreters).\n    The `PYO3_BUILD_EXTENSION_MODULE` environment variable suppresses linking.\n- Cross-compiling configuration\n  - If `TARGET` architecture and `HOST` architecture differ, we can find cross compile information\n    from environment variables (`PYO3_CROSS_LIB_DIR`, `PYO3_CROSS_PYTHON_VERSION` and\n    `PYO3_CROSS_PYTHON_IMPLEMENTATION`) or system files.\n    When cross compiling extension modules it is often possible to make it work without any\n    additional user input.\n  - On Windows, `pyo3-ffi` uses Rust's `raw-dylib` linking feature to link against the Python DLL\n    directly without needing import libraries (`.lib` files). The build script emits a `pyo3_dll`\n    cfg with the target DLL name, and the `extern_libpython!` macro expands to the appropriate\n    `#[link(name = \"...\", kind = \"raw-dylib\")]` attribute. This enables cross compiling Python\n    extensions for Windows without having to install any Windows Python libraries.\n    This can be opted out by setting `PYO3_USE_RAW_DYLIB` to anything other than \"1\".\n\n<!-- External Links -->\n\n[python/c api]: https://docs.python.org/3/c-api/\n\n<!-- Crates -->\n\n[`pyo3-macros`]: https://github.com/PyO3/pyo3/tree/main/pyo3-macros\n[`pyo3-macros-backend`]: https://github.com/PyO3/pyo3/tree/main/pyo3-macros-backend\n[`pyo3-build-config`]: https://github.com/PyO3/pyo3/tree/main/pyo3-build-config\n[`pyo3-ffi`]: https://github.com/PyO3/pyo3/tree/main/pyo3-ffi\n\n<!-- Directories -->\n\n[`src/class`]: https://github.com/PyO3/pyo3/tree/main/src/class\n[`src/ffi`]: https://github.com/PyO3/pyo3/tree/main/src/ffi\n[`src/types`]: https://github.com/PyO3/pyo3/tree/main/src/types\n\n<!-- Files -->\n\n[`src/impl_`]: https://github.com/PyO3/pyo3/blob/main/src/impl_\n[`src/instance.rs`]: https://github.com/PyO3/pyo3/tree/main/src/instance.rs\n[`src/pycell.rs`]: https://github.com/PyO3/pyo3/tree/main/src/pycell.rs\n[`src/pyclass.rs`]: https://github.com/PyO3/pyo3/tree/main/src/pyclass.rs\n[`src/pyclass_init.rs`]: https://github.com/PyO3/pyo3/tree/main/src/pyclass_init.rs\n[`src/pyclass_slot.rs`]: https://github.com/PyO3/pyo3/tree/main/src/pyclass_slot.rs\n[`src/type_object.rs`]: https://github.com/PyO3/pyo3/tree/main/src/type_object.rs\n[`src/class/methods.rs`]: https://github.com/PyO3/pyo3/tree/main/src/class/methods.rs\n[`src/class/impl_.rs`]: https://github.com/PyO3/pyo3/tree/main/src/class/impl_.rs\n[`src/types/any.rs`]: https://github.com/PyO3/pyo3/tree/main/src/types/any.rs\n[`src/types/mod.rs`]: https://github.com/PyO3/pyo3/tree/main/src/types/mod.rs\n"},"items":[{"name":"Architecture.md","path":"Architecture.md","rawUrl":"https://raw.githubusercontent.com/PyO3/pyo3/HEAD/Architecture.md","title":"System Architecture & Module Boundaries","category":"project-spec","format":"markdown","content":"<!-- This file contains a rough overview of the PyO3 codebase. -->\n<!-- Please do not make descriptions too specific, so that we can easily -->\n<!-- keep this file in sync with the codebase. -->\n\n# PyO3: Architecture\n\nThis document roughly describes the high-level architecture of PyO3.\nIf you want to become familiar with the codebase you are in the right place!\n\n## Overview\n\nPyO3 provides a bridge between Rust and Python, based on the [Python/C API].\nThus, PyO3 has low-level bindings of these API as its core.\nOn top of that, we have higher-level bindings to operate Python objects safely.\nAlso, to define Python classes and functions in Rust code, we have `trait PyClass` and a set of\nprotocol traits (e.g., `PyIterProtocol`) for supporting object protocols (i.e., `__dunder__` methods).\nSince implementing `PyClass` requires lots of boilerplate, we have a proc-macro `#[pyclass]`.\n\nTo summarize, there are six main parts to the PyO3 codebase.\n\n1. [Low-level bindings of Python/C API.](#1-low-level-bindings-of-python-capi)\n   - [`pyo3-ffi`] and [`src/ffi`]\n2. [Bindings to Python objects.](#2-bindings-to-python-objects)\n   - [`src/instance.rs`] and [`src/types`]\n3. [`PyClass` and related functionalities.](#3-pyclass-and-related-functionalities)\n   - [`src/pycell.rs`], [`src/pyclass.rs`], and more\n4. [Procedural macros to simplify usage for users.](#4-procedural-macros-to-simplify-usage-for-users)\n   - [`src/impl_`], [`pyo3-macros`] and [`pyo3-macros-backend`]\n5. [`build.rs` and `pyo3-build-config`](#5-buildrs-and-pyo3-build-config)\n   - [`build.rs`](https://github.com/PyO3/pyo3/tree/main/build.rs)\n   - [`pyo3-build-config`]\n\n## 1. Low-level bindings of Python/C API\n\n[`pyo3-ffi`] contains wrappers of the [Python/C API]. This is currently done by hand rather than\nautomated tooling because:\n\n- it gives us best control about how to adapt C conventions to Rust, and\n- there are many Python interpreter versions we support in a single set of files.\n\nWe aim to provide straight-forward Rust wrappers resembling the file structure of [`cpython/Include`](https://github.com/python/cpython/tree/main/Include).\n\nWe are continuously updating the module to match the latest CPython version which PyO3 supports (i.e. as of time of writing Python 3.13). The tracking issue is [#1289](https://github.com/PyO3/pyo3/issues/1289), and contribution is welcome.\n\nIn the [`pyo3-ffi`] crate, there is lots of conditional compilation such as `#[cfg(Py_LIMITED_API)]`,\n`#[cfg(Py_3_9)]`, and `#[cfg(PyPy)]`.\n`Py_LIMITED_API` corresponds to `#define Py_LIMITED_API` macro in Python/C API.\nWith `Py_LIMITED_API`, we can build a Python-version-agnostic binary called an\n[abi3 wheel](https://pyo3.rs/latest/building-and-distribution.html#py_limited_apiabi3abi3t).\n`Py_3_9` means that the API is available from Python >= 3.9.\nThere are also `Py_3_10`, `Py_3_11`, and so on.\n`PyPy` means that the API definition is for PyPy.\nThose flags are set in [`build.rs`](#6-buildrs-and-pyo3-build-config).\n\n## 2. Bindings to Python objects\n\n[`src/types`] contains bindings to [built-in types](https://docs.python.org/3/library/stdtypes.html)\nof Python, such as `dict` and `list`.\nFor historical reasons, Python's `object` is called `PyAny` in PyO3 and located in [`src/types/any.rs`].\n\nCurrently, `PyAny` is a straightforward wrapper of `ffi::PyObject`, defined as:\n\n```rust\n#[repr(transparent)]\npub struct PyAny(UnsafeCell<ffi::PyObject>);\n```\n\nConcrete Python objects are implemented by wrapping `PyAny`, e.g.,:\n\n```rust\n#[repr(transparent)]\npub struct PyDict(PyAny);\n```\n\nThese types are not intended to be accessed directly, and instead are used through the `Py<T>` and `Bound<T>` smart pointers.\n\nWe have some macros in [`src/types/mod.rs`] which make it easier to implement APIs for concrete Python types.\n\n## 3. `PyClass` and related functionalities\n\n[`src/pycell.rs`], [`src/pyclass.rs`], and [`src/type_object.rs`] contain types and\ntraits to make `#[pyclass]` work.\nAlso, [`src/pyclass_init.rs`] and [`src/impl_/pyclass.rs`] have related functionalities.\n\nTo realize object-oriented programming in C, all Python objects have `ob_base: PyObject` as their\nfirst field in their structure definition. Thanks to this guarantee, casting `*mut A` to `*mut PyObject`\nis valid if `A` is a Python object.\n\nTo ensure this guarantee, we have a wrapper struct `PyClassObject<T>` in [`src/pycell/impl_.rs`] which is roughly:\n\n```rust\n#[repr(C)]\npub struct PyClassObject<T> {\n    ob_base: crate::ffi::PyObject,\n    inner: T,\n}\n```\n\nThus, when copying a Rust struct to a Python object, we first allocate `PyClassObject` on the Python heap and then\nmove `T` into it.\n\nThe primary way to interact with Python objects implemented in Rust is through the `Bound<'py, T>` smart pointer.\nBy having the `'py` lifetime of the `Python<'py>` token, this ties the lifetime of the `Bound<'py, T>` smart pointer to the lifetime for which the thread is attached to the Python interpreter and allows PyO3 to call Python APIs at maximum efficiency.\n\n`Bound<'py, T>` requires that `T` implements `PyClass`.\nThis trait is somewhat complex and derives many traits, but the most important one is `PyTypeInfo`\nin [`src/type_object.rs`].\n`PyTypeInfo` is also implemented for built-in types.\nIn Python, all objects have their types, and types are also objects of `type`.\nFor example, you can see `type({})` shows `dict` and `type(type({}))` shows `type` in Python REPL.\n`T: PyTypeInfo` implies that `T` has a corresponding type object.\n\n### Protocol methods\n\nPython has some built-in special methods called dunder methods, such as `__iter__`.\nThey are called \"slots\" in the [abstract objects layer](https://docs.python.org/3/c-api/abstract.html) in\nPython/C API.\nWe provide a way to implement those protocols similarly, by recognizing special\nnames in `#[pymethods]`, with a few new ones for slots that can not be\nimplemented in Python, such as GC support.\n\n## 4. Procedural macros to simplify usage for users.\n\n[`pyo3-macros`] provides five proc-macro APIs: `pymodule`, `pyfunction`, `pyclass`,\n`pymethods`, and `#[derive(FromPyObject)]`.\n[`pyo3-macros-backend`] has the actual implementations of these APIs.\n[`src/impl_`] contains `#[doc(hidden)]` functionality used in code generated by these proc-macros,\nsuch as parsing function arguments.\n\n## 5. `build.rs` and `pyo3-build-config`\n\nPyO3 supports a wide range of OSes, interpreters and use cases. The correct environment must be\ndetected at build time in order to set up relevant conditional compilation correctly. This logic\nis captured in the [`pyo3-build-config`] crate, which is a `build-dependency` of `pyo3` and\n`pyo3-macros`, and can also be used by downstream users in the same way.\n\nIn [`pyo3-build-config`]'s `build.rs` the build environment is detected and inlined into the crate\nas a \"config file\". This works in all cases except for cross-compiling, where it is necessary to\ncapture this from the `pyo3` `build.rs` to get some extra environment variables that Cargo doesn't\nset for build dependencies.\n\nThe `pyo3` `build.rs` also runs some safety checks such as ensuring the Python version detected is\nactually supported.\n\nSome of the functionality of `pyo3-build-config`:\n\n- Find the interpreter for build and detect the Python version.\n  - We have to set some version flags like `#[cfg(Py_3_9)]`.\n  - If the interpreter is PyPy, we set `#[cfg(PyPy)`.\n  - If the `PYO3_CONFIG_FILE` environment variable is set then that file's contents will be used\n    instead of any detected configuration.\n  - If the `PYO3_NO_PYTHON` environment variable is set then the interpreter detection is bypassed\n    entirely and only abi3 extensions can be built.\n- Check if we are building a Python extension.\n  - If we are building an extension (e.g., Python library installable by `pip`),\n    we don't link `libpython` on most platforms (to allow for statically-linked Python interpreters).\n    The `PYO3_BUILD_EXTENSION_MODULE` environment variable suppresses linking.\n- Cross-compiling configuration\n  - If `TARGET` architecture and `HOST` architecture differ, we can find cross compile information\n    from environment variables (`PYO3_CROSS_LIB_DIR`, `PYO3_CROSS_PYTHON_VERSION` and\n    `PYO3_CROSS_PYTHON_IMPLEMENTATION`) or system files.\n    When cross compiling extension modules it is often possible to make it work without any\n    additional user input.\n  - On Windows, `pyo3-ffi` uses Rust's `raw-dylib` linking feature to link against the Python DLL\n    directly without needing import libraries (`.lib` files). The build script emits a `pyo3_dll`\n    cfg with the target DLL name, and the `extern_libpython!` macro expands to the appropriate\n    `#[link(name = \"...\", kind = \"raw-dylib\")]` attribute. This enables cross compiling Python\n    extensions for Windows without having to install any Windows Python libraries.\n    This can be opted out by setting `PYO3_USE_RAW_DYLIB` to anything other than \"1\".\n\n<!-- External Links -->\n\n[python/c api]: https://docs.python.org/3/c-api/\n\n<!-- Crates -->\n\n[`pyo3-macros`]: https://github.com/PyO3/pyo3/tree/main/pyo3-macros\n[`pyo3-macros-backend`]: https://github.com/PyO3/pyo3/tree/main/pyo3-macros-backend\n[`pyo3-build-config`]: https://github.com/PyO3/pyo3/tree/main/pyo3-build-config\n[`pyo3-ffi`]: https://github.com/PyO3/pyo3/tree/main/pyo3-ffi\n\n<!-- Directories -->\n\n[`src/class`]: https://github.com/PyO3/pyo3/tree/main/src/class\n[`src/ffi`]: https://github.com/PyO3/pyo3/tree/main/src/ffi\n[`src/types`]: https://github.com/PyO3/pyo3/tree/main/src/types\n\n<!-- Files -->\n\n[`src/impl_`]: https://github.com/PyO3/pyo3/blob/main/src/impl_\n[`src/instance.rs`]: https://github.com/PyO3/pyo3/tree/main/src/instance.rs\n[`src/pycell.rs`]: https://github.com/PyO3/pyo3/tree/main/src/pycell.rs\n[`src/pyclass.rs`]: https://github.com/PyO3/pyo3/tree/main/src/pyclass.rs\n[`src/pyclass_init.rs`]: https://github.com/PyO3/pyo3/tree/main/src/pyclass_init.rs\n[`src/pyclass_slot.rs`]: https://github.com/PyO3/pyo3/tree/main/src/pyclass_slot.rs\n[`src/type_object.rs`]: https://github.com/PyO3/pyo3/tree/main/src/type_object.rs\n[`src/class/methods.rs`]: https://github.com/PyO3/pyo3/tree/main/src/class/methods.rs\n[`src/class/impl_.rs`]: https://github.com/PyO3/pyo3/tree/main/src/class/impl_.rs\n[`src/types/any.rs`]: https://github.com/PyO3/pyo3/tree/main/src/types/any.rs\n[`src/types/mod.rs`]: https://github.com/PyO3/pyo3/tree/main/src/types/mod.rs\n","isInternal":false,"tokens":2776,"sizeBytes":10394}],"systemPromptSnippet":"<yakaai_skills repo=\"PyO3/pyo3\">\n<!-- File: Architecture.md (Tokens: ~2776 | Category: project-spec) -->\n<!-- This file contains a rough overview of the PyO3 codebase. -->\n<!-- Please do not make descriptions too specific, so that we can easily -->\n<!-- keep this file in sync with the codebase. -->\n\n# PyO3: Architecture\n\nThis document roughly describes the high-level architecture of PyO3.\nIf you want to become familiar with the codebase you are in the right place!\n\n## Overview\n\nPyO3 provides a bridge between Rust and Python, based on the [Python/C API].\nThus, PyO3 has low-level bindings of these API as its core.\nOn top of that, we have higher-level bindings to operate Python objects safely.\nAlso, to define Python classes and functions in Rust code, we have `trait PyClass` and a set of\nprotocol traits (e.g., `PyIterProtocol`) for supporting object protocols (i.e., `__dunder__` methods).\nSince implementing `PyClass` requires lots of boilerplate, we have a proc-macro `#[pyclass]`.\n\nTo summarize, there are six main parts to the PyO3 codebase.\n\n1. [Low-level bindings of Python/C API.](#1-low-level-bindings-of-python-capi)\n   - [`pyo3-ffi`] and [`src/ffi`]\n2. [Bindings to Python objects.](#2-bindings-to-python-objects)\n   - [`src/instance.rs`] and [`src/types`]\n3. [`PyClass` and related functionalities.](#3-pyclass-and-related-functionalities)\n   - [`src/pycell.rs`], [`src/pyclass.rs`], and more\n4. [Procedural macros to simplify usage for users.](#4-procedural-macros-to-simplify-usage-for-users)\n   - [`src/impl_`], [`pyo3-macros`] and [`pyo3-macros-backend`]\n5. [`build.rs` and `pyo3-build-config`](#5-buildrs-and-pyo3-build-config)\n   - [`build.rs`](https://github.com/PyO3/pyo3/tree/main/build.rs)\n   - [`pyo3-build-config`]\n\n## 1. Low-level bindings of Python/C API\n\n[`pyo3-ffi`] contains wrappers of the [Python/C API]. This is currently done by hand rather than\nautomated tooling because:\n\n- it gives us best control about how to adapt C conventions to Rust, and\n- there are many Python interpreter versions we support in a single set of files.\n\nWe aim to provide straight-forward Rust wrappers resembling the file structure of [`cpython/Include`](https://github.com/python/cpython/tree/main/Include).\n\nWe are continuously updating the module to match the latest CPython version which PyO3 supports (i.e. as of time of writing Python 3.13). The tracking issue is [#1289](https://github.com/PyO3/pyo3/issues/1289), and contribution is welcome.\n\nIn the [`pyo3-ffi`] crate, there is lots of conditional compilation such as `#[cfg(Py_LIMITED_API)]`,\n`#[cfg(Py_3_9)]`, and `#[cfg(PyPy)]`.\n`Py_LIMITED_API` corresponds to `#define Py_LIMITED_API` macro in Python/C API.\nWith `Py_LIMITED_API`, we can build a Python-version-agnostic binary called an\n[abi3 wheel](https://pyo3.rs/latest/building-and-distribution.html#py_limited_apiabi3abi3t).\n`Py_3_9` means that the API is available from Python >= 3.9.\nThere are also `Py_3_10`, `Py_3_11`, and so on.\n`PyPy` means that the API definition is for PyPy.\nThose flags are set in [`build.rs`](#6-buildrs-and-pyo3-build-config).\n\n## 2. Bindings to Python objects\n\n[`src/types`] contains bindings to [built-in types](https://docs.python.org/3/library/stdtypes.html)\nof Python, such as `dict` and `list`.\nFor historical reasons, Python's `object` is called `PyAny` in PyO3 and located in [`src/types/any.rs`].\n\nCurrently, `PyAny` is a straightforward wrapper of `ffi::PyObject`, defined as:\n\n```rust\n#[repr(transparent)]\npub struct PyAny(UnsafeCell<ffi::PyObject>);\n```\n\nConcrete Python objects are implemented by wrapping `PyAny`, e.g.,:\n\n```rust\n#[repr(transparent)]\npub struct PyDict(PyAny);\n```\n\nThese types are not intended to be accessed directly, and instead are used through the `Py<T>` and `Bound<T>` smart pointers.\n\nWe have some macros in [`src/types/mod.rs`] which make it easier to implement APIs for concrete Python types.\n\n## 3. `PyClass` and related functionalities\n\n[`src/pycell.rs`], [`src/pyclass.rs`], and [`src/type_object.rs`] contain types and\ntraits to make `#[pyclass]` work.\nAlso, [`src/pyclass_init.rs`] and [`src/impl_/pyclass.rs`] have related functionalities.\n\nTo realize object-oriented programming in C, all Python objects have `ob_base: PyObject` as their\nfirst field in their structure definition. Thanks to this guarantee, casting `*mut A` to `*mut PyObject`\nis valid if `A` is a Python object.\n\nTo ensure this guarantee, we have a wrapper struct `PyClassObject<T>` in [`src/pycell/impl_.rs`] which is roughly:\n\n```rust\n#[repr(C)]\npub struct PyClassObject<T> {\n    ob_base: crate::ffi::PyObject,\n    inner: T,\n}\n```\n\nThus, when copying a Rust struct to a Python object, we first allocate `PyClassObject` on the Python heap and then\nmove `T` into it.\n\nThe primary way to interact with Python objects implemented in Rust is through the `Bound<'py, T>` smart pointer.\nBy having the `'py` lifetime of the `Python<'py>` token, this ties the lifetime of the `Bound<'py, T>` smart pointer to the lifetime for which the thread is attached to the Python interpreter and allows PyO3 to call Python APIs at maximum efficiency.\n\n`Bound<'py, T>` requires that `T` implements `PyClass`.\nThis trait is somewhat complex and derives many traits, but the most important one is `PyTypeInfo`\nin [`src/type_object.rs`].\n`PyTypeInfo` is also implemented for built-in types.\nIn Python, all objects have their types, and types are also objects of `type`.\nFor example, you can see `type({})` shows `dict` and `type(type({}))` shows `type` in Python REPL.\n`T: PyTypeInfo` implies that `T` has a corresponding type object.\n\n### Protocol methods\n\nPython has some built-in special methods called dunder methods, such as `__iter__`.\nThey are called \"slots\" in the [abstract objects layer](https://docs.python.org/3/c-api/abstract.html) in\nPython/C API.\nWe provide a way to implement those protocols similarly, by recognizing special\nnames in `#[pymethods]`, with a few new ones for slots that can not be\nimplemented in Python, such as GC support.\n\n## 4. Procedural macros to simplify usage for users.\n\n[`pyo3-macros`] provides five proc-macro APIs: `pymodule`, `pyfunction`, `pyclass`,\n`pymethods`, and `#[derive(FromPyObject)]`.\n[`pyo3-macros-backend`] has the actual implementations of these APIs.\n[`src/impl_`] contains `#[doc(hidden)]` functionality used in code generated by these proc-macros,\nsuch as parsing function arguments.\n\n## 5. `build.rs` and `pyo3-build-config`\n\nPyO3 supports a wide range of OSes, interpreters and use cases. The correct environment must be\ndetected at build time in order to set up relevant conditional compilation correctly. This logic\nis captured in the [`pyo3-build-config`] crate, which is a `build-dependency` of `pyo3` and\n`pyo3-macros`, and can also be used by downstream users in the same way.\n\nIn [`pyo3-build-config`]'s `build.rs` the build environment is detected and inlined into the crate\nas a \"config file\". This works in all cases except for cross-compiling, where it is necessary to\ncapture this from the `pyo3` `build.rs` to get some extra environment variables that Cargo doesn't\nset for build dependencies.\n\nThe `pyo3` `build.rs` also runs some safety checks such as ensuring the Python version detected is\nactually supported.\n\nSome of the functionality of `pyo3-build-config`:\n\n- Find the interpreter for build and detect the Python version.\n  - We have to set some version flags like `#[cfg(Py_3_9)]`.\n  - If the interpreter is PyPy, we set `#[cfg(PyPy)`.\n  - If the `PYO3_CONFIG_FILE` environment variable is set then that file's contents will be used\n    instead of any detected configuration.\n  - If the `PYO3_NO_PYTHON` environment variable is set then the interpreter detection is bypassed\n    entirely and only abi3 extensions can be built.\n- Check if we are building a Python extension.\n  - If we are building an extension (e.g., Python library installable by `pip`),\n    we don't link `libpython` on most platforms (to allow for statically-linked Python interpreters).\n    The `PYO3_BUILD_EXTENSION_MODULE` environment variable suppresses linking.\n- Cross-compiling configuration\n  - If `TARGET` architecture and `HOST` architecture differ, we can find cross compile information\n    from environment variables (`PYO3_CROSS_LIB_DIR`, `PYO3_CROSS_PYTHON_VERSION` and\n    `PYO3_CROSS_PYTHON_IMPLEMENTATION`) or system files.\n    When cross compiling extension modules it is often possible to make it work without any\n    additional user input.\n  - On Windows, `pyo3-ffi` uses Rust's `raw-dylib` linking feature to link against the Python DLL\n    directly without needing import libraries (`.lib` files). The build script emits a `pyo3_dll`\n    cfg with the target DLL name, and the `extern_libpython!` macro expands to the appropriate\n    `#[link(name = \"...\", kind = \"raw-dylib\")]` attribute. This enables cross compiling Python\n    extensions for Windows without having to install any Windows Python libraries.\n    This can be opted out by setting `PYO3_USE_RAW_DYLIB` to anything other than \"1\".\n\n<!-- External Links -->\n\n[python/c api]: https://docs.python.org/3/c-api/\n\n<!-- Crates -->\n\n[`pyo3-macros`]: https://github.com/PyO3/pyo3/tree/main/pyo3-macros\n[`pyo3-macros-backend`]: https://github.com/PyO3/pyo3/tree/main/pyo3-macros-backend\n[`pyo3-build-config`]: https://github.com/PyO3/pyo3/tree/main/pyo3-build-config\n[`pyo3-ffi`]: https://github.com/PyO3/pyo3/tree/main/pyo3-ffi\n\n<!-- Directories -->\n\n[`src/class`]: https://github.com/PyO3/pyo3/tree/main/src/class\n[`src/ffi`]: https://github.com/PyO3/pyo3/tree/main/src/ffi\n[`src/types`]: https://github.com/PyO3/pyo3/tree/main/src/types\n\n<!-- Files -->\n\n[`src/impl_`]: https://github.com/PyO3/pyo3/blob/main/src/impl_\n[`src/instance.rs`]: https://github.com/PyO3/pyo3/tree/main/src/instance.rs\n[`src/pycell.rs`]: https://github.com/PyO3/pyo3/tree/main/src/pycell.rs\n[`src/pyclass.rs`]: https://github.com/PyO3/pyo3/tree/main/src/pyclass.rs\n[`src/pyclass_init.rs`]: https://github.com/PyO3/pyo3/tree/main/src/pyclass_init.rs\n[`src/pyclass_slot.rs`]: https://github.com/PyO3/pyo3/tree/main/src/pyclass_slot.rs\n[`src/type_object.rs`]: https://github.com/PyO3/pyo3/tree/main/src/type_object.rs\n[`src/class/methods.rs`]: https://github.com/PyO3/pyo3/tree/main/src/class/methods.rs\n[`src/class/impl_.rs`]: https://github.com/PyO3/pyo3/tree/main/src/class/impl_.rs\n[`src/types/any.rs`]: https://github.com/PyO3/pyo3/tree/main/src/types/any.rs\n[`src/types/mod.rs`]: https://github.com/PyO3/pyo3/tree/main/src/types/mod.rs\n</yakaai_skills>"}