pyodide Documentation

Pyodide is a Python distribution for the browser and Node.js based on WebAssembly

14,771 stars Python Python 1 file Β· ~17,705 tokens #emscripten#pyodide#python#wasm#webassembly
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy pyodide on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name pyodide --restart always pyodide/pyodide:latest

File: README.md




Pyodide is a Python distribution for the browser and Node.js based on WebAssembly.

What is Pyodide?

Pyodide is a port of CPython to WebAssembly/Emscripten.

Pyodide makes it possible to install and run Python packages in the browser with
micropip. Any pure Python package with a wheel
available on PyPi is supported. Many packages with C, C++, and Rust extensions
have also been ported for use with Pyodide. These include many general-purpose
packages such as regex, PyYAML, and cryptography, and scientific Python packages
including NumPy, pandas, SciPy, Matplotlib, and scikit-learn.

Pyodide comes with a robust Javascript ⟺ Python foreign function interface so
that you can freely mix these two languages in your code with minimal friction.
This includes full support for error handling, async/await, and much more.

When used inside a browser, Python has full access to the Web APIs.

Try Pyodide (no installation needed)

Try Pyodide in a
REPL directly in
your browser. For further information, see the
documentation.

Getting Started

The Components of the Pyodide Project

The Pyodide project consists of the following components:

  1. A build of CPython with a few patches (https://github.com/pyodide/pyodide/tree/main/cpython/)
  2. A JS/Python foreign function interface
    (https://github.com/pyodide/pyodide/tree/main/src/core and
    https://github.com/pyodide/pyodide/tree/main/src/py)
  3. JavaScript code for creating and managing Pyodide interpreters
    (https://github.com/pyodide/pyodide/tree/main/src/js)
  4. A choice of Emscripten platform, which consists of a version + ABI-sensitive
    flags + static libraries to link. This is described
    here and implemented in
    (https://github.com/pyodide/pyodide/tree/main/Makefile.envs)
  5. A toolchain for cross compiling
    testing and
    installing packages for Pyodide.

History

Pyodide was created in 2018 by Michael Droettboom
at Mozilla as part of the Iodide
project. Iodide is an experimental
web-based notebook environment for literate scientific computing and
communication.

Iodide is no longer maintained. If you want to use Pyodide in an interactive
client-side notebook, see Pyodide notebook
environments.

Contributing

Please view the contributing
guide for tips
on filing issues, making changes, and submitting pull requests. Pyodide is an
independent and community-driven open-source project. The decision-making
process is outlined in the Project
governance.

Communication

Sponsors

For a full list of current and historical sponsors, please see the Funding section of our About page.

Pyodide also has a large number of small donors. If you’re interested in supporting Pyodide, check out our OpenCollective and GitHub Sponsors pages.

Special thanks

License

Pyodide uses the Mozilla Public License Version
2.0.

File: cpython/README.md

Credits

Based on the hard work of @dgym on
cpython-emscripten.

File: docs/development/abi.md

The PyEmscripten Platform

(pyodide-platform-abi)=

What is the PyEmscripten Platform?

The PyEmscripten platform defines a binary interface for an Emscripten
application. If an Emscripten application and an Emscripten shared library are
both built to target this platform, then the application will be able to load
and run the shared library.

To build a shared library that is compatible with a given version of Pyodide, it
is necessary to identify which PyEmscripten platform that version of Pyodide
uses and follow the corresponding shared library build instructions. This allows
wheels to be built for Pyodide that will load and run correctly.

The Emscripten compiler makes no ABI stability guarantees between versions, and
several linker flags can adjust the ABI. Therefore, Python packages built for
Emscripten must match the ABI-sensitive compiler and linker flags used to build
the interpreter to avoid load-time or run-time errors.

To balance ABI stability needs of package maintainers with flexibility to adopt
new platform features and bug fixes, Pyodide adopts a new PyEmscripten platform
for each feature release of Python. The platform tags take the form
pyemscripten_${YEAR}_${PATCH}_wasm32 (e.g., pyemscripten_2026_0_wasm32 for
Python 3.14).

Each PyEmscripten platform specifies:

  • the version of the Emscripten compiler to be used,
  • what libraries are statically linked to the application,
  • the stack unwinding ABI to be used,
  • how the loader handles dependency lookup, and
  • various additional compile and linker flags

The PyEmscripten platform definition does not include anything about Python and
in particular it is agnostic to the Python version it is intended to be used
with. However, for clarity we indicate in the platform documentation which
Python version we plan to use each platform with.

The PyEmscripten platform is specified by {pep}783, and wheels using this
tag are supported on PyPI. Prior to {pep}783, it was called the Pyodide
platform instead of the PyEmscripten platform (and used the form
pyodide_${YEAR}_${PATCH}_wasm32).

Platform Versions

For background on why specific flags were chosen, see ABI-sensitive flags.

{eval-rst}
.. toctree::
   :hidden:

   abi/315.md
   abi/314.md
   abi/313.md
   abi/312.md
   abi/flags.md

General

Building for the Emscripten target

To build C/C++ projects, use the Emscripten compiler toolchain emcc.

To build Rust projects, use rustc --target wasm32-unknown-emscripten or
cargo build --target wasm32-unknown-emscripten. When building, emcc must be
on the path or linking will fail.

Making a shared library

Emscripten shared libraries use the the WebAssembly binary
format and have a
dynamic linking section.

emcc will ignore the -shared flag. To make a shared library with emcc, you
must pass -sSIDE_MODULE=1 or -sSIDE_MODULE=2.

With Rust version 1.95.0 and later, Rust will automatically detect that the
crate is a cdylib and link an Emscripten shared library. For older Rust
versions, to make a shared library with rustc, pass
-C link-arg=-sSIDE_MODULE=2. To build a shared library with cargo, put
-C link-arg=-sSIDE_MODULE=2 in the RUSTFLAGS environment variable.

Static Libraries Linked to the Main Binary

Many libraries cannot be dynamically linked on the Emscripten platform, most
frequently because they contain JavaScript code. If they are to be made
available for packages to use, they must be statically linked to the
interpreter. There are also a few libraries that are needed for a Python builtin
module. As a part of the platform definition, we document which versions of
which libraries are linked to the main binary.

Other Library Dependencies

Other library dependencies should either be statically linked to the extension
module or built as a shared library and vendored into the wheel in a .libs
directory. Like on native platforms, the RPATH of the dependent extension module
should also be set as appropriate to ensure that the dependency is resolved
correctly.

No pthreads support

-pthread must not be used at compile or link time. If -pthread is used, the
resulting libraries will not load.

Controlling the Set of Exported Symbols

All symbols that form part of a binary module's interface must be exported. It
is desirable to produce the minimal list of exported symbols to keep download
size and runtime to a minimum. Not exporting symbols also reduces chances of
symbol collisions.

Linking a shared libraries with -sSIDE_MODULE=1 will pass -whole-archive to
wasm-ld and so force inclusion of all object files and all symbols. Linking
with -sSIDE_MODULE=2 will only include symbols that are explicitly listed with
-sEXPORTED_FUNCTIONS=<export list>. The name of each symbol in the list must
be prefixed with an underscore. For the smallest result, it is recommended to
link with:

text
-sSIDE_MODULE=2 -sEXPORTED_FUNCTIONS=["_PyInit_MyCModule1", "_PyInit_MyCModule2]

File: docs/development/building-from-sources.md

(building_from_source)=

Building from source

{warning}
If you are building the latest development version of Pyodide from the `main`
branch, please make sure to follow the build instructions from the dev
version of the documentation at
[pyodide.org/en/latest/](https://pyodide.org/en/latest/development/building-from-sources.html)

Pyodide can be built from sources on different platforms,

  • on Linux it is easiest using the Pyodide Docker image. This approach
    works with any native operating system as long as Docker is installed. You
    can also build on your native Linux OS if the correct build prerequisites
    are installed.
  • on MacOS it is recommended to install dependencies via conda-forge or
    using Homebrew, particularly with the M1 ARM CPU. Building with Docker is
    possible but very slow.
  • It is not possible to build on Windows, but you can use Windows Subsystem
    for Linux     to
    create a Linux build environment.

Build instructions

The Pyodide repository has a git submodule called pyodide-build. Make sure to
do a recursive clone:

bash
git clone --recursive https://github.com/pyodide/pyodide

or if you have already cloned the Pyodide repository without the --recursive
flag, you may initialize the submodule with:

bash
git submodule update --init

If you change git branches, make sure you update pyodide-build with
git submodule update.

Using Docker

We provide a Debian-based x86_64 Docker image
(pyodide/pyodide-env) on
Docker Hub with the dependencies already installed to make it easier to build
Pyodide.

{note}
These Docker images are also available from the Github packages at
[`github.com/orgs/pyodide/packages`](https://github.com/orgs/pyodide/packages).

  1. Install Docker

  2. From a git checkout of Pyodide, run ./run_docker

  3. Run make to build.

{note}
You can control the resources allocated to the build by setting the env
vars `EMSDK_NUM_CORE`, `EMCC_CORES` and `PYODIDE_JOBS` (the default for each is
4).

If running make deterministically stops at some point,
increasing the maximum RAM usage available to the docker container might help.
(The RAM available to the container is different from the physical RAM capacity of the machine.)
Ideally,
at least 3 GB of RAM should be available to the docker container to build
Pyodide smoothly. These settings can be changed via Docker preferences (see
here).

You can edit the files in the shared pyodide source folder on your host
machine (outside of Docker), and then repeatedly run make inside the Docker
environment to test your changes.

Using the "Docker" dev container

We provide a dev container configuration that is equivalent to the use of
./run_docker script. It can be used in Visual Studio Code and
on GitHub Codespaces.
When prompted, select "Docker".

Using the "Conda" dev container

We provide another dev container configuration that corresponds to
the "Linux with conda" method described below. When Visual Studio Code or
GitHub Codespaces
prompts for the dev container configuration, select "Conda".

Using `make`

Make sure the prerequisites for
emsdk are installed. Pyodide will
build a custom, patched version of emsdk, so there is no need to build it
yourself prior.

You need Python 3.14.2 to run the build scripts. To make sure that the correct
Python is used during the build it is recommended to use a virtual
environment
or a conda environment.

{tab-set}
```{tab-item} Linux

To build on Linux, you need:

- A working native compiler toolchain, enough to build
  [CPython](https://devguide.python.org/getting-started/setup-building/index.html#linux).
- CMake (required to install Emscripten)

```

```{tab-item} Linux with conda


You would need a working native compiler toolchain, enough to build
  [CPython](https://devguide.python.org/getting-started/setup-building/index.html#linux), for example,
- `apt install build-essential` on Debian based systems.
- Conda which can be installed from [MiniForge](https://github.com/conda-forge/miniforge)

Then install the required Python version and other build dependencies in a separate conda environment,

    conda env create -f environment.yml
    conda activate pyodide-env

```
```{tab-item} macOS with conda

You would need,
- System libraries in the root directory:
  `xcode-select --install`
- Conda which can be installed using [Miniforge](https://github.com/conda-forge/miniforge) (both for Intel and M1 CPU)


Then install the required Python version and other build dependencies in a separate conda environment,

    conda env create -f environment.yml
    conda activate pyodide-env

```

```{tab-item} macOS with Homebrew

To build on macOS with Homebrew, you need:

- System command line tools:
  `xcode-select --install`
- [Homebrew](https://brew.sh/) for installing dependencies
- Pyodide requires CMake3 but Homebrew installs CMake4.
  Use pip for cmake `pip install cmake==3.31`
- `brew install coreutils cmake autoconf automake libtool libffi ccache wget rustup-init`
- Initialize Rust (needed for building test-rust-* packages): `rustup-init`
- Install the GNU patch and
  GNU sed (`brew install gpatch gnu-sed`)
  and re-defining them temporarily as [`patch`](https://formulae.brew.sh/formula/gpatch) and
  [`sed`](https://formulae.brew.sh/formula/gnu-sed).

```
{note}
If you encounter issues with the requirements, it is useful to check the exact
list in the
[Dockerfile](https://github.com/pyodide/pyodide/blob/main/Dockerfile) which is
tested in the CI.

You can install the Python dependencies from the requirement file at the root of Pyodide folder:
pip install -r requirements.txt

After installing the build prerequisites, run from the command line:

bash
make

(building-full-distribution)=

Building a full Pyodide distribution

By running make, you build a minimal Pyodide distribution that includes only
the core packages. Pyodide contains a large number of packages, and building
all of them takes a long time. If you want to build a full distribution, or make a custom
distribution with a subset of packages, there are two options:

  1. Use pre-built packages from the pyodide-recipes repository.

Packages included in the Pyodide distribution are built separately and stored in
the pyodide-recipes repository.

Setting ENABLE_PREBUILT_PACKAGES environment variable to 1 will download
and install pre-built packages from the pyodide-recipes repository.

bash
ENABLE_PREBUILT_PACKAGES=1 make
  1. Build a subset of packages from the pyodide-recipes repository.

If you want to build a subset of packages, you can clone the
pyodide-recipes repository and build the packages you need.

bash
git clone https://github.com/pyodide/pyodide-recipes

# Build the packages you need
pyodide build-recipes "numpy, scipy, toolz, attrs" --recipe-dir pyodide-recipes/packages --install

Dependencies of the listed packages will be built automatically as well. The
package names must match the folder names in pyodide-recipes/packages exactly; in particular
they are case-sensitive.

There are also some meta-packages that can be used to build a subset of packages:

  • "tag:min-scipy-stack": includes the "core" meta-package as well as some
    core packages from the Scientific Python stack and their dependencies:
    "numpy", "scipy", "pandas", "matplotlib", "scikit-learn", "joblib",
    "pytest". This option is non exhaustive and is mainly intended to make build
    faster while testing a diverse set of scientific packages.
  • "tag:library": includes all shared and static libraries.
  • "tag:shared_library": includes all shared libraries. Added in conjunction with
    the "library" tag.
  • "tag:static_library": includes all static libraries. Added in conjunction with
    the "library" tag.
  • "tag:pyodide-test": includes all packages built to test the Pyodide build
    system's behaviour.
  • "tag:rust": includes all packages that require a Rust toolchain to build,
    and their dependencies.
  • "*" builds all packages
  • You can exclude a package by prefixing it with "!".

micropip is always automatically included.

Environment variables

The following environment variables additionally impact the build:

  • PYODIDE_JOBS: the -j option passed to the emmake make command when
    applicable for parallel compilation. Default: 3.
  • PYODIDE_BASE_URL: Base URL where Pyodide packages are deployed. It must end
    with a trailing /. Default: ./ to load Pyodide packages from the same
    base URL path as where pyodide.js is located. Example:
    {{PYODIDE_CDN_URL}}
  • EXTRA_CFLAGS: Add extra compilation flags.
  • EXTRA_LDFLAGS: Add extra linker flags.

Setting EXTRA_CFLAGS="-D DEBUG_F" provides detailed diagnostic information
whenever error branches are taken inside the Pyodide core code. These error
messages are frequently helpful even when the problem is a fatal configuration
problem and Pyodide cannot even be initialized. These error branches occur also
in correctly working code, but they are relatively uncommon so in practice the
amount of noise generated isn't too large. The shorthand make debug
automatically sets this flag.

In certain cases, setting EXTRA_LDFLAGS="-s ASSERTIONS=1 or ASSERTIONS=2 can
also be helpful, but this slows down the linking and the runtime speed of
Pyodide a lot and generates a large amount of noise in the console.

File: docs/development/contributing.md

(how_to_contribute)=

How to Contribute

Thank you for your interest in contributing to Pyodide! There are many ways to
contribute, and we appreciate all of them. Here are some guidelines & pointers
for diving into it.

Development Workflow

To contribute code, see the following steps,

  1. Fork the Pyodide repository https://github.com/pyodide/pyodide on Github.

  2. If you are on Linux, you can skip this step. On Windows and MacOS you have a
    choice. The first option is to manually install Docker:

    • on MacOS follow these instructions
    • on Windows, install WSL
      2      , then Docker.
      Note that Windows filesystem access from WSL2 is very slow and should
      be avoided when building Pyodide.

    The second option is to use a service that provides a Linux
    development environment, such as

  3. Clone your fork of Pyodide

    text
    git clone --recursive https://github.com/<your-username>/pyodide.git

    and add the upstream remote,

    text
    git remote add upstream https://github.com/pyodide/pyodide.git

  4. While the build will happen inside Docker you still need a development
    environment with Python 3.14 and ideally Node.js. These can be installed
    for instance with,

    text
    conda env create -f environment.yml
conda activate pyodide-env

    or via your system package manager.

  5. Install requirements (it's recommended to use a virtualenv or a conda env),

    text
    pip install -r requirements.txt

  6. Enable prek for code style,

    text
    prek install

    This will run a set of linters for each commit.

  7. Follow {ref}building_from_source instructions.

  8. See {ref}testing documentation.

Code of Conduct

Pyodide has adopted a {ref}code-of-conduct that we expect all contributors and
core members to adhere to.

Development

Work on Pyodide happens on GitHub. Core members and contributors can make Pull
Requests to fix issues and add features, which all go through the same review
process. We’ll detail how you can start making PRs below.

We’ll do our best to keep main in a non-breaking state, ideally with tests
always passing. The unfortunate reality of software development is sometimes
things break. As such, main cannot be expected to remain reliable at all
times. We recommend using the latest stable version of Pyodide.

Pyodide follows semantic versioning - major versions for
breaking changes (x.0.0), minor versions for new features (0.x.0), and patches
for bug fixes (0.0.x).

We keep a file, {ref}docs/project/changelog.md <changelog>, outlining changes
to Pyodide in each release. We like to think of the audience for changelogs
as non-developers who primarily run the latest stable. So the change log will
primarily outline user-visible changes such as new features and deprecations,
and will exclude things that might otherwise be inconsequential to the end user
experience, such as infrastructure or refactoring.

Bugs & Issues

We use Github Issues for
announcing and discussing bugs and features. Use
this link to report a
bug or issue. We provide a template to give you a guide for how to file
optimally. If you have the chance, please search the existing issues before
reporting a bug. It's possible that someone else has already reported your
error. This doesn't always work, and sometimes it's hard to know what to search
for, so consider this extra credit. We won't mind if you accidentally file a
duplicate report.

Core contributors are monitoring new issues & comments all the time, and will
label & organize issues to align with development priorities.

How to Contribute

Pull requests are the primary mechanism we use to change Pyodide. GitHub itself
has some
great documentation
on using the Pull Request feature. We use the "fork and pull" model
described here,
where contributors push changes to their personal fork and create pull requests
to bring those changes into the source repository.

Please make pull requests against the main branch.

If you’re looking for a way to jump in and contribute, our list of
good first issues
is a great place to start.

If you’d like to fix a currently-filed issue, please take a look at the comment
thread on the issue to ensure no one is already working on it. If no one has
claimed the issue, make a comment stating you’d like to tackle it in a PR. If
someone has claimed the issue but has not worked on it in a few weeks, make a
comment asking if you can take over, and we’ll figure it out from there.

We use pytest, driving
Selenium as our testing framework. Every PR will
automatically run through our tests, and our test framework will alert you on
GitHub if your PR doesn’t pass all of them. If your PR fails a test, try to
figure out whether or not you can update your code to make the test pass again,
or ask for help. As a policy we will not accept a PR that fails any of our
tests, and will likely ask you to add tests if your PR adds new functionality.
Writing tests can be scary, but they make open-source contributions easier for
everyone to assess. Take a moment and look through how we’ve written our tests,
and try to make your tests match. If you are having trouble, we can help you get
started on our test-writing journey.

All code submissions should pass make lint. Python is checked with ruff (for
linting and formatting) and mypy. JavaScript and TypeScript are checked with
biome. C is checked with clang-format.

Contributing to the β€œcore” C Code

See {ref}contributing-core.

Documentation

Documentation is a critical part of any open source project, and we are very
welcome to any documentation improvements. Pyodide has a documentation written
in Markdown in the docs/ folder. We use the
MyST
for parsing Markdown in sphinx. You may want to have a look at the
MyST syntax guide
when contributing, in particular regarding
cross-referencing sections.

Building the docs

From the directory docs, first install the Python dependencies with
pip install -r requirements-doc.txt. You also need to install JsDoc, which is a
node dependency. Install it with sudo npm install jsdoc. Then to
build the docs run make html. The built documentation will be in the
subdirectory docs/_build/html. To view them, cd into _build/html and
start a file server, for instance http-server.

Migrating patches

It often happens that patches need to be migrated between different versions of
upstream packages.

If patches fail to apply automatically, one solution can be to

  1. Checkout the initial version of the upstream package in a separate repo, and
    create a branch from it.
  2. Add existing patches with git apply
  3. Checkout the new version of the upstream package and create a branch from it.
  4. Cherry-pick patches to the new version,
    text
    git cherry-pick <commit-hash>
    and resolve conflicts.
  5. Re-export last N commits as patches e.g.
    text
    git format-patch -<N> -N --no-stat HEAD -o <out_dir>

Maintainer information

For information about making releases see {ref}maintainer-information.

License

All contributions to Pyodide will be licensed under the
Mozilla Public License 2.0 (MPL 2.0).
This is considered a "weak copyleft" license. Check out the tl;drLegal entry for more
information, as well as Mozilla's
MPL 2.0 FAQ if you need further
clarification on what is and isn't permitted.

{eval-rst}
.. toctree::
   :hidden:

   core.md
   maintainers.md

File: docs/development/core.md

(contributing-core)=

Contributing to the "core" C Code

This file is intended as guidelines to help contributors trying to modify the C source files in src/core.

What the files do

The primary purpose of core is to implement {ref}type translations <type-translations> between Python and JavaScript. Here is a breakdown of the purposes of the files.

  • main -- responsible for configuring and initializing the Python interpreter, initializing the other source files, and creating the _pyodide_core module which is used to expose Python objects to pyodide_py. main.c also tries to generate fatal initialization error messages to help with debugging when there is a mistake in the initialization code.
  • keyboard_interrupt -- This sets up the keyboard interrupts system for using Pyodide with a webworker.

Backend utilities

  • hiwire -- A helper framework. It is impossible for wasm to directly hold owning references to JavaScript objects. The primary purpose of hiwire is to act as a surrogate owner for JavaScript references by holding the references in a JavaScript Map. hiwire also defines a wide variety of EM_JS helper functions to do JavaScript operations on the held objects. The primary type that hiwire exports is JsRef. References are created with Hiwire.new_value (only can be done from JavaScript) and must be destroyed from C with hiwire_decref or hiwire_CLEAR, or from JavaScript with Hiwire.decref.
  • error_handling -- defines macros useful for error propagation and for adapting JavaScript functions to the CPython calling convention. See more in the {ref}error_handling_macros section.

Type conversion from JavaScript to Python

  • js2python -- Translates basic types from JavaScript to Python, leaves more complicated stuff to jsproxy.
  • jsproxy -- Defines Python classes to proxy complex JavaScript types into Python. A complex file responsible for many of the core behaviors of Pyodide.

Type conversion from Python to JavaScript

  • python2js -- Translates types from Python to JavaScript, implicitly converting basic types and creating pyproxies for others. It also implements explicit conversion from Python to JavaScript (the toJs method).
  • python2js_buffer -- Attempts to convert Python objects that implement the Python Buffer Protocol. This includes bytes objects, memoryviews, array.array and a wide variety of types exposed by extension modules like numpy. If the data is a 1d array in a contiguous block it can be sliced directly out of the wasm heap to produce a JavaScript TypedArray, but JavaScript does not have native support for pointers, so higher dimensional arrays are more complicated.
  • pyproxy -- Defines a JavaScript Proxy object that passes calls through to a Python object. Another important core file, PyProxy.apply is the primary entrypoint into Python code. pyproxy.c is much simpler than jsproxy.c though.

CPython APIs

Conventions for indicating errors

The two main ways to indicate errors:

  1. If the function returns a pointer, (most often PyObject*, char*, or const char*) then to indicate an error set an exception and return NULL.
  2. If the function returns int or float and a correct output must be nonnegative, to indicate an error set an exception and return -1.

Certain functions have "successful errors" like PyIter_Next (successful error is StopIteration) and PyDict_GetItemWithError (successful error is KeyError). These functions will return NULL without setting an exception to indicate the "successful error" occurred. Check what happened with PyErr_Occurred. Also, functions that return int for which -1 is a valid return value will return -1 with no error set to indicate that the result is -1 and -1 with an error set if an error did occur. The simplest way to handle this is to always check PyErr_Occurred.

Lastly, the argument parsing functions PyArg_ParseTuple, PyArg_Parse, etc are edge cases. These return true on success and return false and set an error on failure.

Python APIs to avoid:

  • PyDict_GetItem, PyDict_GetItemString, and _PyDict_GetItemId
    These APIs do not do correct error reporting and there is talk in the Python community of deprecating them going forward. Instead, use PyDict_GetItemWithError and _PyDict_GetItemIdWithError (there is no PyDict_GetItemStringWithError API because use of GetXString APIs is also discouraged).

  • PyObject_HasAttrString, PyObject_GetAttrString, PyDict_GetItemString, PyDict_SetItemString, PyMapping_HasKeyString etc, etc.
    These APIs cause wasteful repeated string conversion.
    If the string you are using is a constant, e.g., PyDict_GetItemString(dict, "identifier"), then make an id with Py_Identifier(identifier) and then use _PyDict_GetItemId(&PyId_identifier). If the string is not constant, convert it to a Python object with PyUnicode_FromString() and then use e.g., PyDict_GetItem.

  • PyModule_AddObject. This steals a reference on success but not on failure and requires unique cleanup code. Instead, use PyObject_SetAttr.

(error_handling_macros)=

Error Handling Macros

The file error_handling.h defines several macros to help make error handling as simple and uniform as possible.

Error Propagation Macros

In a language with exception handling as a feature, error propagation requires no explicit code, it is only if you want to prevent an error from propagating that you use a try/catch block. On the other hand, in C all error propagation must be done explicitly.

We define macros to help make error propagation look as simple and uniform as possible.
They can only be used in a function that begins with a FAIL_RETURN_VALUE(sentinel) statement, which declares the error sentinel fail_return and a success flag. On failure the macros return fail_return directly; resource cleanup is handled by _Defer blocks that run on every return (see structure of functions section below). When compiled with DEBUG_F, these commands will write a message to console.error reporting the line, function, and file where the error occurred.

  • FAIL_RETURN_VALUE(sentinel) -- declare fail_return = sentinel (e.g. NULL, -1, or JS_ERROR) and bool success = true. Must be the first statement of the function body, before any DECLARE_PY_OBJECT / _Defer.
  • FAIL() -- unconditionally set success = false and return fail_return;.
  • FAIL_IF_NULL(ptr) -- FAIL() if ptr == NULL. This should be used with any function that returns a pointer and follows the standard Python calling convention.
  • FAIL_IF_MINUS_ONE(num) -- FAIL() if num == -1. This should be used with any function that returns a number and follows the standard Python calling convention.
  • FAIL_IF_NONZERO(num) -- FAIL() if num != 0. Can be used with functions that return any nonzero error code on failure.
  • FAIL_IF_ERR_OCCURRED() -- FAIL() if the Python error indicator is set (in other words if PyErr_Occurred()).
  • FAIL_IF_JS_ERROR(val) -- FAIL() if the JsVal is a JavaScript error sentinel.

JavaScript to CPython calling convention adaptors

If we call a JavaScript function from C and that JavaScript function throws an error, it is impossible to catch it in C. We define two EM_JS adaptors to convert from the JavaScript calling convention to the CPython calling convention. The point of this is to ensure that errors that occur in EM_JS functions can be handled in C code using the FAIL_*`` macros. When compiled with DEBUG_F, when a JavaScript error is thrown a message will also be written to console.error`. The wrappers do roughly the following:

javascript
try {
  // body of function here
} catch (e) {
  // wrap e in a Python exception and set the Python error indicator
  // return error code
}

There are two variants: EM_JS_NUM returns -1 as the error code, EM_JS_REF returns NULL == 0 as the error code. A couple of simple examples:
Use EM_JS_REF when return value is a JsRef:

javascript
EM_JS_REF(JsRef, hiwire_call, (JsRef idfunc, JsRef idargs), {
  let jsfunc = Hiwire.get_value(idfunc);
  let jsargs = Hiwire.get_value(idargs);
  return Hiwire.new_value(jsfunc(... jsargs));
});

Use EM_JS_REF when return value is a PyObject:

javascript
EM_JS_REF(PyObject*, __js2python, (JsRef id), {
  // body here
});

If the function returns void, use EM_JS_NUM with return type errcode. errcode is a typedef for int. EM_JS_NUM will automatically return -1 if an error occurs and 0 if not:

javascript
EM_JS_NUM(errcode, hiwire_set_member_int, (JsRef idobj, int idx, JsRef idval), {
  Hiwire.get_value(idobj)[idx] = Hiwire.get_value(idval);
});

If the function returns int or bool use EM_JS_NUM:

javascript
EM_JS_NUM(int, hiwire_get_length, (JsRef idobj), {
  return Hiwire.get_value(idobj).length;
});

These wrappers enable the following sort of code:

python
try:
  jsfunc()
except JsException:
  print("Caught an exception thrown in JavaScript!")

Structure of functions

In C it takes special care to correctly and cleanly handle both reference counting and exception propagation. In Python (or other higher level languages), all references are released in an implicit finally block at the end of the function. Implicitly, it is as if you wrote:

python
def f():
  try: # implicit
    a = do_something()
    b = do_something_else()
    c = a + b
    return some_func(c)
  finally:
    # implicit, free references both on successful exit and on exception
    decref(a)
    decref(b)
    decref(c)

Freeing all references at the end of the function allows us to separate reference counting boilerplate from the "actual logic" of the function definition. When a function does correct error propagation, there will be many different execution paths, roughly linearly many in the length of the function. For example, the above pseudocode could exit in five different ways: do_something could raise an exception, do_something_else could raise an exception, a + b could raise an exception, some_func could raise an exception, or the function could return successfully. (Even a Python function like def f(a,b,c,d): return (a + b) * c - d has four execution paths.) The point of the try/finally block is that we know the resources are freed correctly without checking once for each execution path.

To do this, we divide any function that produces more than a couple of owned PyObject*s or JsRefs into several "segments".
The more owned references there are in a function and the longer it is, the more important it becomes to follow this style carefully.
By being as consistent as possible, we reduce the burden on people reading the code to double-check that you are not leaking memory or errors. In short functions it is fine to do something ad hoc.

  1. The guard block. The first block of a function does sanity checks on the inputs and argument parsing, but only to the extent possible without creating any owned references. If you check more complicated invariants on the inputs in a way that requires creating owned references, this logic belongs in the body block.

Here's an example of a METH_VARARGS function:

c
PyObject*
JsImport_CreateModule(PyObject* self, PyObject* args)
{
  // Guard
  PyObject* name;
  PyObject* jsproxy;
  // PyArg_UnpackTuple uses an unusual calling convention:
  // It returns `false` on failure...
  if (!PyArg_UnpackTuple(args, "create_module", 2, 2, &spec, &jsproxy)) {
    return NULL;
  }
  if (!JsProxy_Check(jsproxy)) {
    PyErr_SetString(PyExc_TypeError, "package is not an instance of jsproxy");
    return NULL;
  }
  1. Forward declaration of owned references. This starts by declaring the error sentinel and success flag with FAIL_RETURN_VALUE(NULL) (use -1 for functions returning an int error code). FAIL() returns this sentinel and sets success = false, which ON_FAIL cleanup blocks use to decide whether to clear the result. Then declare every reference counted variable that we will create during execution of the function, attaching a _Defer block to release it (DECLARE_PY_OBJECT(x) does this in one step). Finally, declare the variable that we are planning to return.
    Typically, this will be called result, but in this case the function is named CreateModule so we name the return variable module.
c
FAIL_RETURN_VALUE(NULL);
  // Note: these are all the objects that we will own. If a function returns
  // a borrow, we XINCREF the result so that the _Defer block can CLEAR it.
  // Reference counting is hard, so it's good to be as explicit and consistent
  // as possible!
  DECLARE_PY_OBJECT(sys_modules);
  DECLARE_PY_OBJECT(importlib_machinery);
  DECLARE_PY_OBJECT(ModuleSpec);
  DECLARE_PY_OBJECT(spec);
  DECLARE_PY_OBJECT(__dir__);
  DECLARE_PY_OBJECT(module_dict);
  // result
  PyObject* module = NULL;
  ON_FAIL({ Py_CLEAR(module); });
  1. The body of the function. The vast majority of API calls can return error codes. You MUST check every fallible API for an error. Also, as you are writing the code, you should look up every Python API you use that returns a reference to determine whether it returns a borrowed reference or a new one. If it returns a borrowed reference, immediately Py_XINCREF() the result to convert it into an owned reference (before FAIL_IF_NULL, to be consistent with the case where you use custom error handling).
c
name = PyUnicode_FromString(name_utf8);
  FAIL_IF_NULL(name);
  sys_modules = PyImport_GetModuleDict(); // returns borrow
  Py_XINCREF(sys_modules);
  FAIL_IF_NULL(sys_modules);
  module = PyDict_GetItemWithError(sys_modules, name); // returns borrow
  Py_XINCREF(module);
  FAIL_IF_NULL(module);
  if(module && !JsImport_Check(module)){
    PyErr_Format(PyExc_KeyError,
      "Cannot mount with name '%s': there is an existing module by this name that was not mounted with 'pyodide.mountPackage'."
      , name
    );
    FAIL();
  }
// ... [SNIP]
  1. The finally block. Here we will clear all the variables we declared at the top in exactly the same order. Do not clear the arguments! They are borrowed. According to the standard Python function calling convention, they are the responsibility of the calling code.
c
success = true;
finally:
  Py_CLEAR(sys_modules);
  Py_CLEAR(importlib_machinery);
  Py_CLEAR(ModuleSpec);
  Py_CLEAR(spec);
  Py_CLEAR(__dir__);
  Py_CLEAR(module_dict);
  if(!success){
    Py_CLEAR(result);
  }
  return result;
}

One case where you do need to Py_CLEAR a variable in the body of a function is if that variable is allocated in a loop:

c
// refcounted variable declarations
  PyObject* pyentry = NULL;
  // ... other stuff
  Py_ssize_t n = PySequence_Length(pylist);
  for (Py_ssize_t i = 0; i < n; i++) {
    pyentry = PySequence_GetItem(pydir, i);
    FAIL_IF_MINUS_ONE(do_something(pyentry));
    Py_CLEAR(pyentry); // important to use Py_CLEAR and not Py_decref.
  }

  success = true
finally:
  // have to clear pyentry at end too in case do_something failed in the loop body
  Py_CLEAR(pyentry);

Testing

Any nonstatic C function called some_name defined not using EM_JS will be exposed as pyodide._module._some_name, and this can be used in tests to good effect. If the arguments / return value are not just numbers and booleans, it may take some effort to set up the function call.

If you want to test an EM_JS function, consider moving the body of the function to an API defined on Module. You should still wrap the function with EM_JS_REF or EM_JS_NUM in order to get a function with the CPython calling convention.

File: docs/development/debugging.md

(debugging)=

Debugging tips

See Emscripten's page about
debugging which has
extensive info about the various debugging options available. The Wasm Binary
Toolkit is super helpful for analyzing
.wasm, .so, .a, and .o files.

Also whenever you can reproduce a bug in chromium make sure to use a
chromium-based browser (e.g., chrome) for debugging. They are better at it.

Run `prettier` on `pyodide.asm.mjs`

Before doing any debugger I strongly recommend running
npx prettier -w pyodide.asm.mjs. This makes everything much easier.

(build-with-symbols)=

Building with symbols or source maps

Building with symbols retains the names of all internal functions, but will not
prevent function calls from being inlined away and does not retain a source map
to the original C code. Symbols do not cause much of a slowdown and only cause a
minor increase in code size. Building with source maps disables many
optimizations such as inlining and slows down execution by quite a lot.

To rebuild with symbols you can use the following:

sh
touch src/core/jsproxy.h
PYODIDE_SYMBOLS=1 make -C cpython rebuild-all
PYODIDE_SYMBOLS=1 make all-but-packages

To use source maps, use PYODIDE_SOURCEMAPS=1 instead of PYODIDE_SYMBOLS=1.
To get the benefits of source maps, you need the
C/C++ devtoools support Chrome extension.

When source maps are present and the devtools extension is enabled, you will not
be able to see the raw disassembly, which makes it hard to inspect the state of
the program in the debugger because there is no support for executing C
expressions. The extension provides no way to see the raw assembly without
disabling the extension and refreshing the page.

(function-signature-mismatch)=

Linker error: function signature mismatch

You may get linker errors as follows:

text
wasm-ld: error: function signature mismatch: some_func
>>> defined as (i32, i32) -> i32 in some_static_lib.a(a.o)
>>> defined as (i32) -> i32 in b.o

This is especially common in Scipy. Oftentimes it isn't too hard to figure out
what is going wrong because it told you the both the symbol name (some_func)
and the object files involved (this is much easier than the runtime version of
this error!). If you can't tell what is going on from looking at the source
files, it's time to pull out wasm-objdump. In this case a.o is part of
some_static_lib.a so you first need to get it out with
ar -x some_static_lib.a a.o.
Now we can check if a.o imports or defines some_func.
To check for imports, use wasm-objdump a.o -j Import -x | grep some_func.
If a.o is importing some_func you should see a line like:
- func[0] sig=1 <env.some_func> <- env.some_func in the output.

If not, you will see nothing or things like some_func2. To check if a.o
defines some_func (this is a bit redundant because you can conclude whether or
not does from whether it imports it) we can use:
wasm-objdump a.o -j Function -x | grep some_func, if
a.o defines some_func you will see something like:
- func[0] sig=0 <some_func>.

Now the question is what these signatures mean (though we already know this from
the linker error). To find out what signature 0 is, you can use
wasm-objdump a.o -j Type -x | grep "type\[0\]".

Using this, we can verify that a.o imports some_func with signature
(i32, i32) -> i32 but b.o exports it with signature (i32) -> i32,
hence the linker error.

This process works in basically the same way for already-linked .so and
.wasm files, which can help if you get the load-time version of this linker
error.

Misencoded Wasm

On a very rare occasion you may run into a misencoded object file. This can
cause different tools to crash, wasm-ld may panic, etc. wasm-objdump will
just generate a useless error message. In this case, I recommend
wasm-objdump -s --debug 2>&1 | grep -i error -C 20 (or pipe to less), which will result in
more diagnostic information. Sometimes the crash happens quite a lot later than the actual error,
look for suspiciously large constants, these are often the first sign of something gone haywire.

After this, you can get out a hex editor and consult the
WebAssembly binary specification
Cross reference against the hex addresses appearing in wasm-objdump --debug.
With enough diligence you can locate the problem.

Debugging RuntimeError: function signature mismatch

First recompile with a source map as suggested in {ref}build-with-symbols, and
disable the C/C++ DevTools extension (if it's enabled).

The browser console will show something like the following. Click on the
innermost stack trace:

Clicking the offset will (hopefully) take you to the corresponding wasm
instruction, which should be a call_indirect. If the offset is too large
(somewhere between 0x0200000 and 0x0300000) you will instead see
;; text is truncated due to size, see {ref}text-truncated-due-to-size. In
this example we see the following:

So we think we are calling a function pointer with signature
(param i32 i32) (result i32)
meaning that it takes two i32 inputs and returns one i32 output. Set a
breakpoint by clicking on the address, then refresh the page and run the
reproduction again. Sometimes these are on really hot code paths (as in the
present example) so you probably only want to set the breakpoint once Pyodide is
finished loading. If your reproduction passes through the breakpoint multiple
times before crashing you can do the usual chore of counting how many times you
have to press "Resume" before the crash. Suppose you've done all this, and we've
got the vm stopped at the bad instruction just before crashing:

The bottom value on the stack is the function pointer. In this case it's the
fourth item on the stack, so you can type the following into the console:

js
> pyodide._module.wasmTable.get(stack[4].value) // stack[4].value === 13109
< Ζ’ $one() { [native code] }

So the bad function pointer's symbol is one! Now clicking on $one brings you
to the source for it:

and we see the function pointer has signature (param $var0 i32) (result i32),
meaning it takes one i32 input and returns one i32 output. Note that if the
function had void return type it might look like (param $var0 i32 $var1 i32)
(with no result). Confusion between i32 and void return type is the single
most common cause of this error.

Now we basically know the cause of the trouble. You can look up cfunction_call
in the CPython source code with the help of ripgrep and locate the line that
generates this call, and look up one in the appropriate source and find the
signature. Another approach to locate the call site would be to recompile with
-g3 and use source maps {ref}build-with-symbols to locate the problematic
source code. With the same process of reproduce crash ==> click innermost stack
frame ==> see source file and line where the error occurs. In this case we see
that the crash is on the line:

c
result = _PyCFunction_TrampolineCall(meth, self, args);

in the file /src/cpython/build/Python-3.11.0dev0/Objects/methodobject.c.
Unfortunately, source maps are useless for the harder problem of finding the
callee because compiling with -g3 increases the number of function pointers so
the function pointer we are calling is in a different spot. I know of no way to
determine the bad function pointer when compiling with -g3.

Sometimes (particularly with Scipy/OpenBLAS/libf2c) the issue will be a
mismatch between
(param i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32) (result i32) and
(param i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32) (result i32)

(14 vs 15 parameters) which might be a little hard to discern. I copy the
signature into the Javascript console and run "i32 ... i32".split(" ").length
in this case.

(text-truncated-due-to-size)=

Dealing with `;; text is truncated due to size`

If you are debugging and run into the dreaded ;; text is truncated due to size
error message, the solution is to compile a modified version of Chrome devtools
with a larger wasm size cap. Surprisingly, this is not actually all that hard.

These instructions are adapted from here:
https://www.diverto.hr/en/blog/2020-08-15-WebAssembly-limit/

In short,

text
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
./fetch devtools-frontend
cd devtools-frontend

Apply the following change:

diff
--- a/front_end/entrypoints/wasmparser_worker/WasmParserWorker.ts
+++ b/front_end/entrypoints/wasmparser_worker/WasmParserWorker.ts
@@ -55,7 +55,7 @@ export function dissambleWASM(
     const lines = [];
     const offsets = [];
     const functionBodyOffsets = [];
-    const MAX_LINES = 1000 * 1000;
+    const MAX_LINES = 12 * 1000 * 1000;
     let chunkSize: number = 128 * 1024;
     let buffer: Uint8Array = new Uint8Array(chunkSize);
     let pendingSize = 0;

Then build with:

text
gn gen out/Default
autoninja -C out/Default

then

text
cd out/Default/resources/inspector_overlay/
python http.server <some_port>

and then you can start a version of chrome using the modified devtools:

text
chrome --custom-devtools-frontend=http://localhost:<some_port>/

File: docs/development/maintainers.md

(maintainer-information)=

Maintainer information

Making a release

For branch organization we use a variation of the GitHub
Flow with
the latest release branch named stable (due to ReadTheDocs constraints).

Making a major release

Assume for concreteness that we are releasing version 0.20.0.

Preparation

  1. Make a tracking issue with a title like "0.20.0 release planning". Add the
    checklist:

    • Update packages
    • Look for open PRs to add to the release milestone
    • Make sure all PRs in the release milestone are merged
    • Write release notes
    • Tidy changelog

  2. Make a release notes blog post at pyodide-blog:
    https://github.com/pyodide/pyodide-blog

  3. Generate the list of contributors for the release at the end of the release
    notes blog post with:

    sh
    git shortlog -s 0.19.0.. | cut -f2- | grep -v '\[bot\]' | sort --ignore-case | tr '\n' ';' | sed 's/;/, /g;s/, $//' | fold -s

    where 0.19.0 is the tag for the last major release.

  4. Read the changelog and tidy it up by adding subsections, organizing, and proof
    reading it. Make a pull request with these changes titled "Rearrange changelog
    for 0.20.0 release" and merge it.

  5. Make sure all the PRs that we want to release are merged and that the release notes are ready.

Releasing

  1. Switch to the main branch

  2. From the root of the repository run:

    text
    ./tools/bump_version.py 0.20.0 --tag

    This makes a release commit and tags it.

  3. Push the release commit and tag to upstream. This triggers the release CI.

    text
    git push upstream main 0.20.0

  4. Wait for CI to pass and release to be created.

  5. Rename the stable branch to a release branch for the previous major
    version. For instance if last release was, 0.20.0, the corresponding
    release branch would be 0.20.X:

    sh
    git fetch upstream stable:stable
git branch 0.20.X stable
git push -u upstream 0.20.X

  6. Create a new stable branch:

    sh
    git switch main
git switch -C stable
git push upstream stable --force

  7. Set the version back to next development version with:

    sh
    git switch main
./tools/bump_version.py 0.21.0 --dev
git push upstream main

Making a minor release

Assume for concreteness that we are releasing version 0.27.2.

Preparation

  1. Go through the commits on the main branch since the last release, find ones
    you want to backport and add the "needs backport" label to the pull requests.
    You can do this manually in the web interface on the GitHub PR or you can use

    sh
    ./tools/backport.py add-pr

  2. List out the needs backport PRs that are missing changelog entries with

    sh
    ./tools/backport.py missing-changelogs

    and double check that every PR that should have a changelog does have one.

  3. Read the changelog and tidy it up by adding subsections, organizing, and
    proof reading it. Make a pull request with these changes titled e.g.,
    "Rearrange changelog for 0.27.2 release" and merge it.

  4. Make the backport branch (on top of stable):

    text
    ./tools/backport.py backport-branch

  5. Make the update-changelog branch (on top of main) with:

    text
    ./tools/backport.py changelog-branch

  6. Open PRs for these two branches with:

    text
    ./tools/backport.py open-release-prs

  7. Use the backport branch PR as the release tracker.

  8. Make sure that the CI passes on the backports branch and it is approved. When
    it does pass, set the date for the release in the changelog with:

    text
    ./tools/backport.py set-date
git switch backports-for-0.27.2
git push -f
git switch changelog-for-0.27.2
git push -f

    Then merge the two PRs.

  9. Run

    text
    ./tools/backport.py clear-prs

    to clear all the "needs backport" labels.

Releasing

  1. Switch to the stable branch and git pull.
  2. From the root of the repository run:
    text
    ./tools/backport.py bump-version --tag
    This makes a release commit and tags it.
  3. Push the release commit and tag to upstream/stable. This triggers the release
    CI.
    text
    git push upstream stable 0.27.2
  4. Wait for CI to pass and the release to be created.

Making an alpha release

Assume for concreteness that we are releasing 0.28.0a1.

Preparation

Any single maintainer can decide on their own to make an alpha release, it is
not required to discuss it with other maintainers.

Name the first alpha release x.x.xa1 and in subsequent alphas increment the
final number. No preparation is necessary. Do not make any changes to the
changelog.

Release instructions

  1. Switch to the main branch and git pull.
  2. From the root of the repository run:
    text
    ./tools/bump_version.py 0.28.0a1 --tag
    This makes a release commit and tags it.
  3. Push the release commit and tag to upstream/main. This triggers the release
    CI.
    text
    git push upstream main 0.28.0a1
  4. Put the version back with:
    text
    git revert 0.28.0a1 -n && git commit -m "Back to development version"
git push upstream main
  5. Wait for CI to pass and the release to be created.

Fixing documentation for a released version

Cherry pick the corresponding documentation commits to the stable branch. Use
git commit --amend to add [skip ci] to the commit message.

Updating the Docker image

Anyone with an account on hub.docker.com can follow the following steps:

  1. Make whatever changes are needed to the Dockerfile.
  2. Build the docker image with docker build . in the Pyodide root directory.
    If the build succeeds, docker will give you a hash for the built image.
  3. Use python ./tools/docker_image_tag.py to find out what the new image tag
    should be. Tag the image with:
    sh
    docker image tag  <your-docker-username>/pyodide-env:
  4. Push the image with:
    sh
    docker image push <your-docker-username>/pyodide-env:
  5. Replace the image in .circleci/config.yml with your newly created image.
    Open a pull request with your changes to Dockerfile and .circleci/config.yml.
  6. When the tests pass and the pull request is approved, a maintainer must copy
    the new image into the pyodide dockerhub account.
  7. Then replace the image tag in .circleci/config.yml,
    .devcontainer/devcontainer.json, and run_docker with the new image under
    the pyodide dockerhub account.

It's also possible to update the docker image by pushing your changes to the
Dockerfile to a branch in the pyodide/pyodide repo (not on a fork) and
clicking Run workflow on
https://github.com/pyodide/pyodide/actions/workflows/docker_image.yml.

(updating-packages)=

Updating packages

Before updating the Python version and before making a major Pyodide release, we
try to update all packages that are not too much trouble. Run

sh
make -C packages update-all

to update all packages and make a pull request with these changes. There will be
build/test failures, revert the packages that fail the build or tests and make a
note to update them independently.

Updating pyodide-build

To change the version of pyodide-build, change the commit of the pyodide-build submodule.

bash
cd pyodide-build
git checkout "<COMMIT HASH>"

To test with a fork of pyodide-build, change the .gitmodules file to point to your fork and update the commit hash

ini
# .gitmodules
[submodule "pyodide-build"]
	path = pyodide-build
	url = https://github.com/<yourfork>/pyodide-build
bash
git submodule sync
cd pyodide-build
git checkout "<COMMIT HASH>"

Upgrading the ABI

Each year, we aim to release a new Pyodide ABI version with a new CPython and
Emscripten version.

Similar to the CPython alpha release, we stabilize the ABI early before
releasing the new Pyodide version so that package maintainers can update their
packages to use the new ABI (#5580).

0. Stabilize and freeze the ABI

Discuss with the maintainers to decide what ABI changes to make before the
release.

Once the ABI changes are decided, create a 20XX_0 branch (targeting year) from
the main branch. All PRs that change the ABI should be merged to the 20XX_0
branch.

1. Disable `USE_PREBUILT_PACKAGES`

Pyodide CI uses prebuilt packages built with the ABI of the last release. To
upgrade the ABI, you need to disable them, otherwise the CI will fail.

2. Upgrade Emscripten and CPython version

Update the Emscripten and CPython versions respectively.

Updating the Emscripten version is not mandatory, but it is recommended to
benefit from the latest features and bug fixes.

Updating Emscripten and CPython separately is often easier than updating them
together.

2.1. Upgrade the Emscripten version

Updating Emscripten requires the following three steps:

  1. Rebase the patches in emsdk/patches onto the new Emscripten version.
  2. Update the Emscripten version in Makefile.envs.
  3. Update the struct_info JSON file in src/js/ to match the version of the
    file in Emscripten.

All three steps are automated by tools/update_emscripten.py. To update, run:

sh
./tools/update_emscripten.py new_version

If there are rebase conflicts, you will have to manually finish the rebase. Once
the rebase is completed, rerun update_emscripten.py. It will start over the
rebase from scratch but reuse your conflict resolutions using the git rerere
feature.

After this is done, commit all the changes and open a PR. There are frequently
complicated CI failures.

2.2. Upgrade CPython version

Prerequisites

The desired version of CPython must be available at:

  1. The "specific release" section of https://www.python.org/downloads
  2. https://hub.docker.com/_/python
  3. https://github.com/actions/python-versions/releases

Steps

  1. Follow the steps in "Updating the Docker image" to create a Docker image for
    the new Python version.

  2. Make sure you are in a Python virtual environment with the new version of
    Python and with requirements.txt installed. (It is also possible to work in
    the Docker image as an alternative.)

  3. Update the Python version in Makefile.envs.

  4. Update the Python version in the following locations:

    • .github/workflows/main.yml
    • docs/conf.py
    • docs/development/contributing.md
    • docs/development/building-and-testing-packages.md
    • environment.yml
    • .pre-commit-config.yaml
    • pyproject.toml

    (TODO: make this list shorter.)

  5. Check python/cpython-source-deps for
    updated versions of bundled C libraries. Pyodide bundles zstd from this repository
    (see ZSTDTARBALL in cpython/Makefile). Update the tag and URL if a newer version
    is available, and verify the build still works.

  6. Rebase the patches:

    • Clone CPython and cd into it. Checkout the Python version you are upgrading
      from. For instance, if the old version is 3.13.2, use git checkout v3.13.2
      (Python tags have a leading v). Run:

      sh
      git am ~/path/to/pyodide/cpython/patches/*

    • Rebase the patches onto the new version of Python. For instance, if updating
      from Python v3.13.2 to Python 3.14.1:

      sh
      git rebase v3.13.2 --onto v3.14.1

    • Resolve conflicts and drop patches that have been upstreamed. If you have
      conflicts, make sure you are using diff3:

      sh
      git config --global merge.conflictstyle diff3

    • Generate the new patches:

      sh
      rm ~/path/to/pyodide/cpython/patches/*
git format-patch v3.14.1 -o ~/path/to/pyodide/cpython/patches/

  7. Try to build Python with make -C cpython. Fix any build errors. If you
    modify the Python source in-tree after a failed build, it may be useful to
    run make rebuild.

  8. Try to finish the build with a top-level make. Fix compile errors in
    src/core and any link errors. It may be useful to apply
    upgrade_pythoncapi.py --no-compat
    to the C extension in src/core.

    The file most tightly coupled to the CPython version is
    src/core/stack_switching/pystate.c. Consult the
    greenlet TPythonState.cpp
    to figure out how to fix it.

  9. Run:

    sh
    python tools/make_test_list.py

    Then run the core tests with pytest src/tests/test_core_python.py and
    either fix the failures or update src/tests/python_tests.yaml to skip or
    xfail them.

  10. Try to build packages with:

    sh
    pyodide build-recipes '*'

  11. Fix failing package tests.

  12. Update standard library stubs in src/templates. We currently have
    webbrowser.py and ssl.py that we implement ourselves. If you are
    updating the Python version, you may need to update these stubs. Review the
    Python docs for the standard library modules and check if the APIs have
    changed. Search for "in version 3.XX" to find the relevant changes.

Old major Python upgrades

version pr
3.13 {pr}5498
3.12 {pr}4435
3.11 {pr}3252
3.10 {pr}2225
3.9 {pr}1637
3.8 {pr}712
3.7 {pr}77

3. Release a new Pyodide alpha version

After all the changes are merged, release a new Pyodide alpha version so that
people can build packages with the new ABI.

4. Update packages and release new package set

TODO: Update the process after PEP 783 is accepted.

Go to pyodide/pyodide-recipes and
follow the guidelines in
MAINTAINERS.md.
You'll need to create a tag for the existing package sets and update
default_cross_build_env_url in the main branch.

If you updated the Python version, also update the python version used in the
CI. When opening the PR, include the full build label to trigger a full build
of all packages; otherwise, only a small set of packages will be built.

Most likely, some packages will fail to build. You can disable them by adding
_disabled: true to the recipes of the failing packages. You don't need to
handle all the build failures in the same PRβ€”you can open multiple PRs to fix
the build failures or let the package maintainers handle them.

Create a new tag for the new package set when the build is successful. This will
create a new release with the new package set.

5. Re-enable `USE_PREBUILT_PACKAGES` and upgrade the ABI in the main branch

Go back to pyodide/pyodide and update the PYODIDE_PREBUILT_PACKAGES_BASE
variable in Makefile.envs to the new package set you created in the previous
step.

Then re-enable USE_PREBUILT_PACKAGES in .circleci/config.yml and open a PR
to merge it into the main branch.

6. Release a new Pyodide version

After enabling most of the packages that were available in the previous ABI
version, we are ready to release a new Pyodide version with the new ABI.

Update {ref}pyodide-platform-abi to match the new ABI.

Merge the 20XX_0 branch into main and open a PR, then release a new Pyodide
version with the new ABI.

Explore Related Technologies & Topics

View All 9,111+ Repositories β†’

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#emscripten Browse emscripten catalog β†’ #pyodide Browse pyodide catalog β†’ #python Browse python catalog β†’ #wasm Browse wasm catalog β†’ #webassembly Browse webassembly catalog β†’ ⚑ MCP Servers Model Context Protocol β†’ 🧠 Agent Skills Prompt & rule files β†’ πŸ”Œ OpenAPI Specs REST & Swagger schemas β†’