### Code Of Conduct Code of Conduct --------------- Our Pledge ~~~~~~~~~~ In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to make participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation. Our Standards ~~~~~~~~~~~~~ Examples of behavior that contributes to creating a positive environment include: * Using welcoming and inclusive language * Being respectful of differing viewpoints and experiences * Gracefully accepting constructive criticism * Focusing on what is best for the community * Showing empathy towards other community members Examples of unacceptable behavior by participants include: * The use of sexualized language or imagery and unwelcome sexual attention or advances * Trolling, insulting/derogatory comments, and personal or political attacks * Public or private harassment * Publishing others' private information, such as a physical or electronic address, without explicit permission * Other conduct which could reasonably be considered inappropriate in a professional setting Our Responsibilities ~~~~~~~~~~~~~~~~~~~~ Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior. Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful. Scope ~~~~~ This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers. Enforcement ~~~~~~~~~~~ Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at snakecharmers@ethereum.org. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately. Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership. Attribution ~~~~~~~~~~~ This Code of Conduct is adapted from the `Contributor Covenant `_, version 1.4, available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html --- ### Constants .. _constants: Constants ========= The web3.constants module contains commonly used values. Strings ******* .. code-block:: python #The Address Zero, which is 20 bytes (40 nibbles) of zero. web3.constants.ADDRESS_ZERO #The hexadecimal version of Max uint256. web3.constants.MAX_INT #The Hash Zero, which is 32 bytes (64 nibbles) of zero. web3.constants.HASH_ZERO Int *** .. code-block:: python #The amount of Wei in one Ether web3.constants.WEI_PER_ETHER --- ### Contributing .. _contributing: Contributing ------------ Thanks for your interest in contributing to web3.py! Read on to learn what would be helpful and how to go about it. If you get stuck along the way, reach for help in the `Python Discord server`_. How to Help ~~~~~~~~~~~ Without code: * Answer user questions within GitHub issues, Stack Overflow, or the `Python Discord server`_. * Write or record tutorial content. * Improve our documentation (including typo fixes). * `Open an issue `_ on GitHub to document a bug. Include as much detail as possible, e.g., how to reproduce the issue and any exception messages. With code: * Fix a bug that has been reported in an issue. * Add a feature that has been documented in an issue. * Add a missing test case. .. warning:: **Before you start:** always ask if a change would be desirable or let us know that you plan to work on something! We don't want to waste your time on changes we can't accept or duplicated effort. Your Development Environment ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. note:: Use of a virtual environment is strongly advised for minimizing dependency issues. See `this article `_ for usage patterns. All pull requests are made from a fork of the repository; use the GitHub UI to create a fork. web3.py depends on `submodules `_, so when you clone your fork to your local machine, include the ``--recursive`` flag: .. code:: sh $ git clone --recursive https://github.com//web3.py.git $ cd web3.py Finally, install all development dependencies: .. code:: sh $ python -m pip install -e ".[dev]" $ pre-commit install Using Docker ^^^^^^^^^^^^ Developing within Docker is not required, but if you prefer that workflow, use the *sandbox* container provided in the **docker-compose.yml** file. To start up the test environment, run: .. code:: sh $ docker compose up -d This will build a Docker container set up with an environment to run the Python test code. To run the core tests from your local machine: .. code:: sh $ docker compose exec sandbox bash -c 'pytest tests/core' The container does not have ``go-ethereum`` installed, so you can exclude those tests by using the ``-k "not goethereum"`` flag. .. code:: sh $ docker compose exec sandbox bash -c 'pytest tests/integration -k "not goethereum"' You can run arbitrary commands inside the Docker container by using the ``bash -c`` prefix. .. code:: sh $ docker compose exec sandbox bash -c 'pwd && ls' Or, if you would like to open a session to the container, run: .. code:: sh $ docker compose exec sandbox bash Running The Tests ~~~~~~~~~~~~~~~~~ A great way to explore the code base is to run the tests. First, install the test dependencies: .. code:: sh $ python -m pip install -e ".[test]" You can run all tests with: .. code:: sh $ pytest However, running the entire test suite takes a very long time and is generally impractical. Typically, you'll just want to run a subset instead, like: .. code:: sh $ pytest tests/core/eth-module/test_accounts.py Linting is also performed by the CI and locally with each commit. You can save yourself some time by checking for linting errors manually: .. code:: sh $ make lint It is important to understand that each pull request must pass the full test suite as part of the CI check. This test suite will run in the CI anytime a pull request is opened or updated. Writing Tests ~~~~~~~~~~~~~ We strongly encourage contributors to write good tests for their code as part of the code review process. This helps ensure that your code is doing what it should be doing. We strongly encourage you to use our existing tests for both guidance and homogeneity / consistency across our tests. We use ``pytest`` for our tests. For more specific pytest guidance, please refer to the `pytest documentation`_. Within the ``pytest`` scope, :file:`conftest.py` files are used for common code shared between modules that exist within the same directory as that particular :file:`conftest.py` file. Unit Testing and eth-tester Tests ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Our unit tests are grouped together with tests against the ``eth-tester`` library, using the ``py-evm`` library as a backend, via the ``EthereumTesterProvider``. These tests live under appropriately named child directories within the ``/tests`` directory. The core of these tests live under ``/tests/core``. Do your best to follow the existing structure when adding a test and make sure that its location makes sense. Integration Testing ^^^^^^^^^^^^^^^^^^^ Our integration test suite setup lives under the ``/tests/integration`` directory. The integration test suite is dependent on what we call "fixtures" (not to be confused with pytest fixtures). These zip file fixtures, which also live in the ``/tests/integration`` directory, are configured to run the specific client we are testing against along with a genesis configuration that gives our tests some pre-determined useful objects (like unlocked, pre-loaded accounts) to be able to interact with the client when we run our tests. The parent ``/integration`` directory houses some common configuration shared across all client tests, whereas the ``/go_ethereum`` directory houses common code to be shared across geth-specific provider tests. Though the setup and run configurations exist across the different files within ``/tests/integration``, our integration module tests are written across different files within ``/web3/_utils/module_testing``. * :file:`common.py` files within the client directories contain code that is shared across all provider tests (http, ipc, and ws). This is mostly used to override tests that span across all providers. * :file:`conftest.py` files within each of these directories contain mostly code that can be *used* by all test files that exist within the same directory or subdirectories of the :file:`conftest.py` file. This is mostly used to house pytest fixtures to be shared among our tests. Refer to the `pytest documentation on fixtures`_ for more information. * ``test_{client}_{provider}.py`` files (e.g. :file:`test_goethereum_http.py`) are where client-and-provider-specific test configurations exist. This is mostly used to override tests specific to the provider type for the respective client. The integration tests are each run in insolation to prevent muddied contexts. Because they are run in isolation, they can be parallelized with ``pytest-xdist`` in order to speed up the test suite. To run the tests in parallel, you can use the ``-n`` flag with ``pytest``. For example, to run the tests in parallel with 4 workers, you can use the following command: .. code:: sh $ pytest tests/integration/go_ethereum/path/to/module/or/test -n 4 Working With Test Contracts ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Contracts used for testing exist under ``web3/_utils/contract_sources``. These contracts get compiled via the ``compile_contracts.py`` script in the same directory. To use this script, simply pass the Solidity version to be used to compile the contracts as an argument at the command line. Arguments for the script are: -v or --version Solidity version to be used to compile the contracts. If blank, the script uses the latest available version from solcx. -f or --filename If left blank, all .sol files will be compiled and the respective contract data will be generated. Pass in a specific ``.sol`` filename here to compile just one file. To run the script, you will need the ``py-solc-x`` library for compiling the files as well as ``black`` for code formatting. You can install those with: .. code:: sh $ python -m pip install py-solc-x black The following example compiles all the contracts and generates their respective contract data that is used across our test files for the test suites. This data gets generated within the ``contract_data`` subdirectory within the ``contract_sources`` folder. .. code-block:: bash $ cd ../web3.py/web3/_utils/contract_sources $ python compile_contracts.py -v 0.8.17 Compiling OffchainLookup ... ... reformatted ... To compile and generate contract data for only one ``.sol`` file, specify using the filename with the ``-f`` (or ``--filename``) argument flag. .. code-block:: bash $ cd ../web3.py/web3/_utils/contract_sources $ python compile_contracts.py -v 0.8.17 -f OffchainLookup.sol Compiling OffchainLookup.sol reformatted ... If there is any contract data that is not generated via the script but is important to pass on to the integration tests, the ``_custom_contract_data.py`` file within the ``contract_data`` subdirectory can be used to store that information when appropriate. Be sure to re-generate the integration test fixture after running the script to update the contract bytecodes for the integration test suite - see the :ref:`generating_fixtures` section below. Manual Testing ~~~~~~~~~~~~~~ To import and test an unreleased version of web3.py in another context, you can install it from your development directory: .. code:: sh $ python -m pip install -e ../path/to/web3py Code Style ~~~~~~~~~~ We use `pre-commit `_ to enforce a consistent code style across the library. This tool runs automatically with every commit, but you can also run it manually with: .. code:: sh $ make lint If you need to make a commit that skips the ``pre-commit`` checks, you can do so with ``git commit --no-verify``. We use Black as part of our linting. To ignore the commits that introduced Black in git history, you can configure your git environment like so: .. code:: sh $ git config blame.ignoreRevsFile .git-blame-ignore-revs This library uses `type hints`_, which are enforced by the ``mypy`` tool (part of the ``pre-commit`` checks). All new code is required to land with type hints, with the exception of code within the ``tests`` directory. Documentation ~~~~~~~~~~~~~ Good documentation will lead to quicker adoption and happier users. Please check out our guide on `how to create documentation`_ for the Python Ethereum ecosystem. Pull requests generate their own preview of the latest documentation at ``https://web3py--.org.readthedocs.build/en//``. Pull Requests ~~~~~~~~~~~~~ It's a good idea to make pull requests early on. A pull request represents the start of a discussion, and doesn't necessarily need to be the final, finished submission. See GitHub's documentation for `working on pull requests`_. Once you've made a pull request take a look at the Circle CI build status in the GitHub interface and make sure all tests are passing. In general, pull requests that do not pass the CI build yet won't get reviewed unless explicitly requested. If the pull request introduces changes that should be reflected in the release notes, please add a **newsfragment** file as explained `here `_. If possible, the change to the release notes file should be included in the commit that introduces the feature or bugfix. .. _generating_fixtures: Generating New Fixtures ~~~~~~~~~~~~~~~~~~~~~~~ Our integration tests make use of Geth private networks. When new versions of the client software are introduced, new fixtures should be generated. Before generating new fixtures, make sure you have the test dependencies installed: .. code:: sh $ python -m pip install -e ".[test]" .. note:: A "fixture" is a pre-synced network. It's the result of configuring and running a client, deploying the test contracts, and saving the resulting state for testing web3.py functionality against. Geth Fixtures ^^^^^^^^^^^^^ 1. Install the desired Geth version on your machine locally. We recommend `py-geth`_ for this purpose, because it enables you to easily manage multiple versions of Geth. Note that ``py-geth`` will need updating to support each new Geth version as well. Adding newer Geth versions to py-geth is straightforward; see past commits for a template. If py-geth has the Geth version you need, install that version locally. For example: .. code:: sh $ python -m geth.install v1.16.7 2. Specify the Geth binary and run the fixture creation script (from within the web3.py directory): .. code:: sh $ GETH_BINARY=~/.py-geth/geth-v1.16.7/bin/geth python ./tests/integration/generate_fixtures/go_ethereum.py 3. The output of this script is your fixture, a zip file, which is now stored in ``/tests/integration/``. The ``/tests/integration/go_ethereum/conftest.py`` and ``/web3/tools/benchmark/node.py`` files should be updated automatically to point to this new fixture. Delete the old fixture. 4. Run the tests. To ensure that the tests run with the correct Geth version locally, you may again include the ``GETH_BINARY`` environment variable. 5. The ``geth_version`` and ``pygeth_version`` parameter defaults in ``/.circleci/config.yml`` should be automatically updated to match the ``go-ethereum`` version used to generate the test fixture and the ``py-geth`` version that supports installing it. CI Testing With a Nightly Geth Build ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Occasionally you'll want to have CI run the test suite against an unreleased version of Geth - e.g. to test upcoming hard fork changes. The workflow described below is for testing only, as updates will only be merged into main once the Geth release is published and the test runs are updated to use the new stable version. 1. Configure ``tests/integration/generate_fixtures/go_ethereum/common.py`` as needed. 2. Geth automagically compiles new builds for every commit that gets merged into the codebase. Download the desired build from the `develop builds `_. 3. Build your test fixture, passing in the binary you just downloaded via ``GETH_BINARY``. Don't forget to update the ``/tests/integration/go_ethereum/conftest.py`` file to point to your new fixture. 4. Our CI runs on Ubuntu, so download the corresponding 64-bit Linux `develop build `_, then add it to the root of your web3.py directory. Rename the binary ``custom_geth``. 5. In ``.circleci/config.yml``, update the ``geth_version`` pipeline parameter to "custom". This will trigger the custom Geth build to be used in the CI test suite. 6. Create a PR and let CI do its thing. Releasing ~~~~~~~~~ Releases are typically done from the ``main`` branch, except when releasing a beta (in which case the beta is released from ``main``, and the previous stable branch is released from said branch). Final test before each release ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Before releasing a new version, build and test the package that will be released: .. code:: sh $ git checkout main && git pull $ make package-test This will build the package and install it in a temporary virtual environment. Follow the instructions to activate the venv and test whatever you think is important. Review the documentation that will get published: .. code:: sh $ make docs Validate and preview the release notes: .. code:: sh $ make validate-newsfragments Build the release notes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Before bumping the version number, build the release notes. You must include the part of the version to bump (see below), which changes how the version number will show in the release notes. .. code:: sh $ make notes bump=$$VERSION_PART_TO_BUMP$$ If there are any errors, be sure to re-run make notes until it works. Push the release to github & pypi ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ After confirming that the release package looks okay, release a new version: .. code:: sh $ make release bump=$$VERSION_PART_TO_BUMP$$ This command will: - Bump the version number as specified in ``.pyproject.toml`` and ``setup.py``. - Create a git commit and tag for the new version. - Build the package. - Push the commit and tag to github. - Push the new package files to pypi. Which version part to bump ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``$$VERSION_PART_TO_BUMP$$`` must be one of: ``major``, ``minor``, ``patch``, ``stage``, or ``devnum``. The version format for this repo is ``{major}.{minor}.{patch}`` for stable, and ``{major}.{minor}.{patch}-{stage}.{devnum}`` for unstable (``stage`` can be alpha or beta). If you are in a beta version, ``make release bump=stage`` will switch to a stable. To issue an unstable version when the current version is stable, specify the new version explicitly, like ``make release bump="--new-version 4.0.0-alpha.1"``. You can see what the result of bumping any particular version part would be with ``bump-my-version show-bump``. .. _Python Discord server: https://discord.gg/GHryRvPB84 .. _style guide: https://github.com/ethereum/snake-charmers-tactical-manual/blob/main/style-guide.md .. _type hints: https://www.python.org/dev/peps/pep-0484/ .. _how to create documentation: https://github.com/ethereum/snake-charmers-tactical-manual/blob/main/documentation.md .. _working on pull requests: https://help.github.com/articles/about-pull-requests/ .. _py-geth: https://github.com/ethereum/py-geth .. _pytest documentation: https://docs.pytest.org/en/latest .. _pytest documentation on fixtures: https://docs.pytest.org/en/latest/how-to/fixtures.html --- ### Ens .. py:module:: ens ENS API =========== :doc:`ens_overview` has a friendly overview. Continue below for the detailed specs on each method and class in the ens module. ens\.ens module ---------------- .. automodule:: ens.ens :members: ens\.async_ens module --------------------- .. automodule:: ens.async_ens :members: ens\.exceptions module ---------------------- .. automodule:: ens.exceptions :members: :show-inheritance: --- ### Ens Overview .. _ens_overview: Ethereum Name Service (ENS) =========================== The Ethereum Name Service (ENS) is analogous to the Domain Name Service. It enables users and developers to use human-friendly names in place of error-prone hexadecimal addresses, content hashes, and more. The :mod:`ens` module is included with web3.py. It provides an interface to look up domains and addresses, add resolver records, or get and set metadata. Setup ----- Create an :class:`~ens.ENS` object (named ``ns`` below) in one of three ways: 1. Automatic detection 2. Specify an instance of a :ref:`provider ` 3. From an existing :class:`web3.Web3` object .. code-block:: python # automatic detection from ens.auto import ns # or, with a provider from web3 import IPCProvider from ens import ENS provider = IPCProvider(...) ns = ENS(provider) # or, with a w3 instance # Note: This inherits the w3 middleware from the w3 instance and adds a stalecheck middleware to the middleware onion. # It also inherits the provider and codec from the w3 instance, as well as the ``strict_bytes_type_checking`` flag value. from ens import ENS w3 = Web3(...) ns = ENS.from_web3(w3) Asynchronous support is available via the ``AsyncENS`` module: .. code-block:: python from ens import AsyncENS ns = AsyncENS(provider) Note that an ``ens`` module instance is also available on the ``w3`` instance. The first time it's used, web3.py will create the ``ens`` instance using ``ENS.from_web3(w3)`` or ``AsyncENS.from_web3(w3)`` as appropriate. .. code-block:: python # instantiate w3 instance from web3 import Web3, IPCProvider w3 = Web3(IPCProvider(...)) # use the module w3.ens.address('ethereum.eth') .. py:attribute:: ens.strict_bytes_type_checking The ``ENS`` instance has a ``strict_bytes_type_checking`` flag that toggles the flag with the same name on the ``Web3`` instance attached to the ``ENS`` instance. You may disable the stricter bytes type checking that is loaded by default using this flag. For more examples, see :ref:`disable-strict-byte-check` If instantiating a standalone ENS instance using ``ENS.from_web3()``, the ENS instance will inherit the value of the flag on the Web3 instance at time of instantiation. .. doctest:: >>> from web3 import Web3, EthereumTesterProvider >>> from ens import ENS >>> w3 = Web3(EthereumTesterProvider()) >>> assert w3.strict_bytes_type_checking # assert strict by default >>> w3.is_encodable('bytes2', b'1') False >>> w3.strict_bytes_type_checking = False >>> w3.is_encodable('bytes2', b'1') # zero-padded, so encoded to: b'1\x00' True >>> ns = ENS.from_web3(w3) >>> # assert inherited from w3 at time of instantiation via ENS.from_web3() >>> assert ns.strict_bytes_type_checking is False >>> ns.w3.is_encodable('bytes2', b'1') True >>> # assert these are now separate instances >>> ns.strict_bytes_type_checking = True >>> ns.w3.is_encodable('bytes2', b'1') False >>> # assert w3 flag value remains >>> assert w3.strict_bytes_type_checking is False >>> w3.is_encodable('bytes2', b'1') True However, if accessing the ``ENS`` class via the ``Web3`` instance as a module (``w3.ens``), since all modules use the same ``Web3`` object reference under the hood (the parent ``w3`` object), changing the ``strict_bytes_type_checking`` flag value on ``w3`` also changes the flag state for ``w3.ens.w3`` and all modules. .. doctest:: >>> from web3 import Web3, EthereumTesterProvider >>> w3 = Web3(EthereumTesterProvider()) >>> assert w3.strict_bytes_type_checking # assert strict by default >>> w3.is_encodable('bytes2', b'1') False >>> w3.strict_bytes_type_checking = False >>> w3.is_encodable('bytes2', b'1') # zero-padded, so encoded to: b'1\x00' True >>> assert w3 == w3.ens.w3 # assert same object >>> assert not w3.ens.w3.strict_bytes_type_checking >>> w3.ens.w3.is_encodable('bytes2', b'1') True >>> # sanity check on eth module as well >>> assert not w3.eth.w3.strict_bytes_type_checking >>> w3.eth.w3.is_encodable('bytes2', b'1') True Usage ----- Name Info ~~~~~~~~~ .. _ens_get_address: Get the Address for an ENS Name ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: python from ens.auto import ns eth_address = ns.address('ens.eth') assert eth_address == '0xFe89cc7aBB2C4183683ab71653C4cdc9B02D44b7' The ``ENS`` module has no opinion as to which **TLD (Top Level Domain)** you can use, but will not infer a TLD if it is not provided with the name. Multichain Address Resolution +++++++++++++++++++++++++++++ `ENSIP-9 `_ introduced multichain address resolution, allowing users to resolve addresses from different chains, specified by the coin type index from `SLIP44 `_. The ``address()`` method on the ``ENS`` class supports multichain address resolution via the ``coin_type`` keyword argument. .. code-block:: python from ens.auto import ns eth_address = ns.address('ens.eth', coin_type=60) # ETH is coin_type 60 assert eth_address == '0xFe89cc7aBB2C4183683ab71653C4cdc9B02D44b7' Get the ENS Name for an Address ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: python domain = ns.name('0xFe89cc7aBB2C4183683ab71653C4cdc9B02D44b7') # name() also accepts the bytes version of the address assert ns.name(b'\xfe\x89\xccz\xbb,A\x83h:\xb7\x16S\xc4\xcd\xc9\xb0-D\xb7') == domain # confirm that the name resolves back to the address that you looked up: assert ns.address(domain) == '0xFe89cc7aBB2C4183683ab71653C4cdc9B02D44b7' .. note:: For accuracy, and as a recommendation from the ENS documentation on `reverse resolution `_, the ``ENS`` module now verifies that the forward resolution matches the address with every call to get the ``name()`` for an address. This is the only sure way to know whether the reverse resolution is correct. Anyone can claim any name, only forward resolution implies that the owner of the name gave their stamp of approval. Get the Owner of a Name ^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: python eth_address = ns.owner('exchange.eth') .... Set Up Your Name and Address ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Link a Name to an Address ^^^^^^^^^^^^^^^^^^^^^^^^^ You can set up your name so that :meth:`~ens.ENS.address` will show the address it points to. In order to do so, you must already be the owner of the domain (or its parent). .. code-block:: python ns.setup_address('ens.eth', '0xFe89cc7aBB2C4183683ab71653C4cdc9B02D44b7') In the common case where you want to point the name to the owning address, you can skip the address. .. code-block:: python ns.setup_address('ens.eth') You can claim arbitrarily deep subdomains. .. code-block:: python ns.setup_address('supreme.executive.power.derives.from.a.mandate.from.the.masses.ens.eth') # wait for the transaction to be mined, then: assert ( ns.address('supreme.executive.power.derives.from.a.mandate.from.the.masses.ens.eth') == '0xFe89cc7aBB2C4183683ab71653C4cdc9B02D44b7' ) .. warning:: Gas costs scale up with the number of subdomains! Multichain Address Support ++++++++++++++++++++++++++ `ENSIP-9 `_ introduced multichain address resolution, allowing users to resolve addresses from different chains, specified by the coin type index from `SLIP44 `_. The ``setup_address()`` method on the ``ENS`` class supports multichain address setup via the ``coin_type`` keyword argument. .. code-block:: python from ens.auto import ns ns.setup_address('ens.eth', coin_type=60) # ETH is coin_type 60 assert ns.address('ens.eth', coin_type=60) == '0xFe89cc7aBB2C4183683ab71653C4cdc9B02D44b7' Link an Address to a Name ^^^^^^^^^^^^^^^^^^^^^^^^^ You can set up your address so that :meth:`~ens.ENS.name` will show the name that points to it. This is like Caller ID. It enables you and others to take an account and determine what name points to it. Sometimes this is referred to as "reverse" resolution. The ENS Reverse Resolver is used for this functionality. .. code-block:: python ns.setup_name('ens.eth', '0xFe89cc7aBB2C4183683ab71653C4cdc9B02D44b7') If you don't supply the address, :meth:`~ens.ENS.setup_name` will assume you want the address returned by :meth:`~ens.ENS.address`. .. code-block:: python ns.setup_name('ens.eth') If the name doesn't already point to an address, :meth:`~ens.ENS.setup_name` will call :meth:`~ens.ENS.setup_address` for you. Wait for the transaction to be mined, then: .. code-block:: python assert ns.name('0xFe89cc7aBB2C4183683ab71653C4cdc9B02D44b7') == 'ens.eth' .... Text Records ~~~~~~~~~~~~ Set Text Metadata for an ENS Record ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ As the owner of an ENS record, you can add text metadata. A list of supported fields can be found in the `ENS documentation `_. You'll need to setup the address first, and then the text can be set: .. code-block:: python ns.setup_address('ens.eth', '0xFe89cc7aBB2C4183683ab71653C4cdc9B02D44b7') ns.set_text('ens.eth', 'url', 'https://example.com') A transaction dictionary can be passed as the last argument if desired: .. code-block:: python transaction_dict = {'from': '0x123...'} ns.set_text('ens.eth', 'url', 'https://example.com', transaction_dict) If the transaction dictionary is not passed, sensible defaults will be used, and if a transaction dictionary is passed but does not have a ``from`` value, the default will be the ``owner``. Read Text Metadata for an ENS Record ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Anyone can read the data from an ENS Record: .. code-block:: python url = ns.get_text('ens.eth', 'url') assert url == 'https://example.com' .... Working With Resolvers ~~~~~~~~~~~~~~~~~~~~~~ Get the Resolver for an ENS Record ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ You can get the resolver for an ENS name via the :meth:`~ens.ENS.resolver` method. .. code-block:: python >>> resolver = ns.resolver('ens.eth') >>> resolver.address '0x5B2063246F2191f18F2675ceDB8b28102e957458' .... Wildcard Resolution Support --------------------------- The ``ENS`` module supports Wildcard Resolution for resolvers that implement the ``ExtendedResolver`` interface as described in `ENSIP-10 `_. Resolvers that implement the extended resolver interface should return ``True`` when calling the ``supportsInterface()`` function with the extended resolver interface id ``"0x9061b923"`` and should resolve subdomains to a unique address. --- ### Filters .. _filtering: Events and Logs =============== If you're on this page, you're likely looking for an answer to this question: **How do I know when a specific contract is used?** You have several options: 1. Query blocks for transactions that include the contract address in the ``"to"`` field. This contrived example is searching the latest block for any transactions sent to the WETH_ contract. .. code-block:: python WETH_ADDRESS = '0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2' block = w3.eth.get_block('latest') for tx_hash in block.transactions: tx = w3.eth.get_transaction(tx_hash) if tx['to'] == WETH_ADDRESS: print(f'Found interaction with WETH contract! {tx}') 2. Query for logs emitted by a contract. After instantiating a web3.py Contract object, you can :ref:`fetch logs ` for any event listed in the ABI. In this example, we query for ``Transfer`` events in the latest block and log out the results. .. code-block:: python WETH_ADDRESS = '0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2' WETH_ABI = '[{"constant":true,"inputs":[],"name":"name","outputs":[{"name":"","type":"string"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"guy","type":"address"},{"name":"wad","type":"uint256"}],"name":"approve","outputs":[{"name":"","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"totalSupply","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"src","type":"address"},{"name":"dst","type":"address"},{"name":"wad","type":"uint256"}],"name":"transferFrom","outputs":[{"name":"","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":false,"inputs":[{"name":"wad","type":"uint256"}],"name":"withdraw","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":true,"inputs":[],"name":"decimals","outputs":[{"name":"","type":"uint8"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[{"name":"","type":"address"}],"name":"balanceOf","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":true,"inputs":[],"name":"symbol","outputs":[{"name":"","type":"string"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"dst","type":"address"},{"name":"wad","type":"uint256"}],"name":"transfer","outputs":[{"name":"","type":"bool"}],"payable":false,"stateMutability":"nonpayable","type":"function"},{"constant":false,"inputs":[],"name":"deposit","outputs":[],"payable":true,"stateMutability":"payable","type":"function"},{"constant":true,"inputs":[{"name":"","type":"address"},{"name":"","type":"address"}],"name":"allowance","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"payable":true,"stateMutability":"payable","type":"fallback"},{"anonymous":false,"inputs":[{"indexed":true,"name":"src","type":"address"},{"indexed":true,"name":"guy","type":"address"},{"indexed":false,"name":"wad","type":"uint256"}],"name":"Approval","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"src","type":"address"},{"indexed":true,"name":"dst","type":"address"},{"indexed":false,"name":"wad","type":"uint256"}],"name":"Transfer","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"dst","type":"address"},{"indexed":false,"name":"wad","type":"uint256"}],"name":"Deposit","type":"event"},{"anonymous":false,"inputs":[{"indexed":true,"name":"src","type":"address"},{"indexed":false,"name":"wad","type":"uint256"}],"name":"Withdrawal","type":"event"}]' weth_contract = w3.eth.contract(address=WETH_ADDRESS, abi=WETH_ABI) # fetch transfer events in the last block logs = weth_contract.events.Transfer().get_logs(from_block=w3.eth.block_number) for log in logs: print(f"Transfer of {w3.from_wei(log.args.wad, 'ether')} WETH from {log.args.src} to {log.args.dst}") See an advanced example of fetching log history :ref:`here `. 3. Subscribe to events for real-time updates. When using a persistent connection provider (:class:`~web3.providers.persistent.WebSocketProvider` or :class:`~web3.providers.persistent.AsyncIPCProvider`), the :meth:`subscribe() ` method can be used to establish a new event subscription. This example subscribes to ``Transfer`` events of the WETH contract. .. code-block:: python import asyncio from web3 import AsyncWeb3, WebSocketProvider from eth_abi.abi import decode WETH_ADDRESS = "0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2" async def subscribe_to_transfer_events(): async with AsyncWeb3(WebSocketProvider("...")) as w3: transfer_event_topic = w3.keccak(text="Transfer(address,address,uint256)") filter_params = { "address": WETH_ADDRESS, "topics": [transfer_event_topic], } subscription_id = await w3.eth.subscribe("logs", filter_params) print(f"Subscribing to transfer events for WETH at {subscription_id}") async for payload in w3.socket.process_subscriptions(): result = payload["result"] from_addr = decode(["address"], result["topics"][1])[0] to_addr = decode(["address"], result["topics"][2])[0] amount = decode(["uint256"], result["data"])[0] print(f"{w3.from_wei(amount, 'ether')} WETH from {from_addr} to {to_addr}") asyncio.run(subscribe_to_transfer_events()) For more usage examples see the docs on :ref:`subscription-examples`. 4. Use a filter. .. warning :: While filters can be a very convenient way to monitor for blocks, transactions, or events, they are notoriously unreliable. Both remote and locally hosted nodes have a reputation for occasionally dropping filters, and some remote node providers don't support filter-related RPC calls at all. .. py:module:: web3.utils.filters The :meth:`web3.eth.Eth.filter` method can be used to set up filters for: * Pending Transactions: ``w3.eth.filter("pending")`` * New Blocks ``w3.eth.filter("latest")`` * Event Logs Through the contract instance api: .. code-block:: python event_filter = my_contract.events.myEvent.create_filter(from_block='latest', argument_filters={'arg1':10}) Or built manually by supplying `valid filter params `_: .. code-block:: python event_filter = w3.eth.filter({"address": contract_address}) * Attaching to an existing filter .. code-block:: python existing_filter = w3.eth.filter(filter_id="0x0") .. note :: Creating event filters requires that your Ethereum node has an API support enabled for filters. Note that Infura support for filters does not offer access to `pending` filters. To get event logs on other stateless nodes please see :class:`web3.contract.ContractEvents`. Filter Class ------------ .. py:class:: Filter(web3, filter_id) .. py:attribute:: Filter.filter_id The ``filter_id`` for this filter as returned by the ``eth_newFilter`` RPC method when this filter was created. .. py:method:: Filter.get_new_entries() Retrieve new entries for this filter. Logs will be retrieved using the :func:`web3.eth.Eth.get_filter_changes` which returns only new entries since the last poll. .. py:method:: Filter.get_all_entries() Retrieve all entries for this filter. Logs will be retrieved using the :func:`web3.eth.Eth.get_filter_logs` which returns all entries that match the given filter. .. py:method:: Filter.format_entry(entry) Hook for subclasses to modify the format of the log entries this filter returns, or passes to its callback functions. By default this returns the ``entry`` parameter unmodified. .. py:method:: Filter.is_valid_entry(entry) Hook for subclasses to add additional programmatic filtering. The default implementation always returns ``True``. Block and Transaction Filter Classes ------------------------------------ .. py:class:: BlockFilter(...) ``BlockFilter`` is a subclass of :class:`Filter`. You can setup a filter for new blocks using ``web3.eth.filter('latest')`` which will return a new :class:`BlockFilter` object. .. code-block:: python new_block_filter = w3.eth.filter('latest') new_block_filter.get_new_entries() .. note:: ``"safe"`` and ``"finalized"`` block identifiers are not yet supported for ``eth_newBlockFilter``. .. py:class:: TransactionFilter(...) ``TransactionFilter`` is a subclass of :class:`Filter`. You can setup a filter for new blocks using ``web3.eth.filter('pending')`` which will return a new :class:`TransactionFilter` object. .. code-block:: python new_transaction_filter = w3.eth.filter('pending') new_transaction_filter.get_new_entries() Event Log Filters ----------------- You can set up a filter for event logs using the web3.py contract api: :meth:`web3.contract.Contract.events.your_event_name.create_filter`, which provides some conveniences for creating event log filters. Refer to the following example: .. code-block:: python event_filter = my_contract.events..create_filter(from_block="latest", argument_filters={'arg1':10}) event_filter.get_new_entries() See :meth:`web3.contract.Contract.events.your_event_name.create_filter()` documentation for more information. You can set up an event log filter like the one above with ``web3.eth.filter`` by supplying a dictionary containing the standard filter parameters. Assuming that ``arg1`` is indexed, the equivalent filter creation would look like: .. code-block:: python event_signature_hash = web3.keccak(text="eventName(uint32)").hex() event_filter = web3.eth.filter({ "address": myContract_address, "topics": [event_signature_hash, "0x000000000000000000000000000000000000000000000000000000000000000a"], }) The ``topics`` argument is order-dependent. For non-anonymous events, the first item in the topic list is always the keccack hash of the event signature. Subsequent topic items are the hex encoded values for indexed event arguments. In the above example, the second item is the ``arg1`` value ``10`` encoded to its hex string representation. In addition to being order-dependent, there are a few more points to recognize when specifying topic filters: Given a transaction log with topics [A, B], the following topic filters will yield a match: - [] "anything" - [A] "A in first position (and anything after)" - [None, B] "anything in first position AND B in second position (and anything after)" - [A, B] "A in first position AND B in second position (and anything after)" - [[A, B], [A, B]] "(A OR B) in first position AND (A OR B) in second position (and anything after)" See the JSON-RPC documentation for `eth_newFilter `_ more information on the standard filter parameters. .. note:: Though ``"finalized"`` and ``"safe"`` block identifiers are not yet part of the specifications for ``eth_newFilter``, they are supported by web3.py and may or may not yield expected results depending on the node being accessed. Creating a log filter by either of the above methods will return a :class:`LogFilter` instance. .. py:class:: LogFilter(web3, filter_id, log_entry_formatter=None, data_filter_set=None) The :py:class:`LogFilter` class is a subclass of :class:`Filter`. See the :class:`Filter` documentation for inherited methods. :class:`LogFilter` provides the following additional methods: .. py:method:: LogFilter.set_data_filters(data_filter_set) Provides a means to filter on the log data, in other words the ability to filter on values from un-indexed event arguments. The parameter ``data_filter_set`` should be a list or set of 32-byte hex encoded values. Examples: Listening For Events ------------------------------ Synchronous ~~~~~~~~~~~ .. code-block:: python from web3 import Web3, IPCProvider import time # instantiate Web3 instance w3 = Web3(IPCProvider(...)) def handle_event(event): print(event) def log_loop(event_filter, poll_interval): while True: for event in event_filter.get_new_entries(): handle_event(event) time.sleep(poll_interval) def main(): block_filter = w3.eth.filter('latest') log_loop(block_filter, 2) if __name__ == '__main__': main() .. _asynchronous_filters: Asynchronous Filter Polling ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Starting with web3 version 4, the ``watch`` method was taken out of the web3 filter objects. There are many decisions to be made when designing a system regarding threading and concurrency. Rather than force a decision, web3 leaves these choices up to the user. Below are some example implementations of asynchronous filter-event handling that can serve as starting points. Single threaded concurrency with ``async`` and ``await`` ```````````````````````````````````````````````````````` Beginning in python 3.5, the ``async`` and ``await`` built-in keywords were added. These provide a shared api for coroutines that can be utilized by modules such as the built-in asyncio_. Below is an example event loop using asyncio_, that polls multiple web3 filter object, and passes new entries to a handler. .. code-block:: python from web3 import Web3, IPCProvider import asyncio # instantiate Web3 instance w3 = Web3(IPCProvider(...)) def handle_event(event): print(event) # and whatever async def log_loop(event_filter, poll_interval): while True: for event in event_filter.get_new_entries(): handle_event(event) await asyncio.sleep(poll_interval) def main(): block_filter = w3.eth.filter('latest') tx_filter = w3.eth.filter('pending') loop = asyncio.get_event_loop() try: loop.run_until_complete( asyncio.gather( log_loop(block_filter, 2), log_loop(tx_filter, 2))) finally: loop.close() if __name__ == '__main__': main() Read the asyncio_ documentation for more information. Running the event loop in a separate thread ``````````````````````````````````````````` Here is an extended version of above example, where the event loop is run in a separate thread, releasing the ``main`` function for other tasks. .. code-block:: python from web3 import Web3, IPCProvider from threading import Thread import time # instantiate Web3 instance w3 = Web3(IPCProvider(...)) def handle_event(event): print(event) # and whatever def log_loop(event_filter, poll_interval): while True: for event in event_filter.get_new_entries(): handle_event(event) time.sleep(poll_interval) def main(): block_filter = w3.eth.filter('latest') worker = Thread(target=log_loop, args=(block_filter, 5), daemon=True) worker.start() # .. do some other stuff if __name__ == '__main__': main() Here are some other libraries that provide frameworks for writing asynchronous python: * gevent_ * twisted_ * celery_ Examples -------- .. _advanced_token_fetch: Advanced example: Fetching all token transfer events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In this example, we show how to fetch all events of a certain event type from the Ethereum blockchain. There are three challenges when working with a large set of events: * How to incrementally update an existing database of fetched events * How to deal with interruptions in long running processes * How to deal with `eth_getLogs` JSON-RPC call query limitations * How to handle Ethereum minor chain reorganisations in (near) real-time data eth_getLogs limitations ``````````````````````` Ethereum JSON-RPC API servers, like Geth, do not provide an easy way to paginate over events, only over blocks. There's no request that can find the first block with an event or how many events occur within a range of blocks. The only feedback the JSON-RPC service will give you is whether the ``eth_getLogs`` call failed. In this example script, we provide two kinds of heuristics to deal with this issue. The script scans events in a chunk of blocks (start block number - end block number). Then it uses two methods to find how many events there are likely to be in a block window: * Dynamically set the block range window size, while never exceeding a threshold (e.g., 10,000 blocks). * In the case of ``eth_getLogs``, the JSON-RPC call gives a timeout error, decrease the end block number and tries again with a smaller block range window. Example code ```````````` The following example code is divided into a reusable ``EventScanner`` class and then a demo script that: * fetches all transfer events for `RCC token `_, * can incrementally run again to check if there are new events, * handles interruptions (e.g., CTRL+C abort) gracefully, * writes all ``Transfer`` events in a single file JSON database, so that other process can consume them, * uses the `tqdm `_ library for progress bar output in a console, * only supports ``HTTPS`` providers, because JSON-RPC retry logic depends on the implementation details of the underlying protocol, * disables the default exception retry configuration because it does not know how to handle the shrinking block range window for ``eth_getLogs``, and * consumes around 20k JSON-RPC API calls. The script can be run with: ``python ./eventscanner.py ``. .. code-block:: python """A stateful event scanner for Ethereum-based blockchains using web3.py. With the stateful mechanism, you can do one batch scan or incremental scans, where events are added wherever the scanner left off. """ import datetime import time import logging from abc import ABC, abstractmethod from typing import Tuple, Optional, Callable, List, Iterable, Dict, Any from web3 import Web3 from web3.contract import Contract from web3.datastructures import AttributeDict from web3.exceptions import BlockNotFound from eth_abi.codec import ABICodec # Currently this method is not exposed over official web3 API, # but we need it to construct eth_getLogs parameters from web3._utils.filters import construct_event_filter_params from web3._utils.events import get_event_data logger = logging.getLogger(__name__) class EventScannerState(ABC): """Application state that remembers what blocks we have scanned in the case of crash. """ @abstractmethod def get_last_scanned_block(self) -> int: """Number of the last block we have scanned on the previous cycle. :return: 0 if no blocks scanned yet """ @abstractmethod def start_chunk(self, block_number: int): """Scanner is about to ask data of multiple blocks over JSON-RPC. Start a database session if needed. """ @abstractmethod def end_chunk(self, block_number: int): """Scanner finished a number of blocks. Persistent any data in your state now. """ @abstractmethod def process_event(self, block_when: datetime.datetime, event: AttributeDict) -> object: """Process incoming events. This function takes raw events from Web3, transforms them to your application internal format, then saves them in a database or some other state. :param block_when: When this block was mined :param event: Symbolic dictionary of the event data :return: Internal state structure that is the result of event transformation. """ @abstractmethod def delete_data(self, since_block: int) -> int: """Delete any data since this block was scanned. Purges any potential minor reorg data. """ class EventScanner: """Scan blockchain for events and try not to abuse JSON-RPC API too much. Can be used for real-time scans, as it detects minor chain reorganisation and rescans. Unlike the easy web3.contract.Contract, this scanner can scan events from multiple contracts at once. For example, you can get all transfers from all tokens in the same scan. You *should* disable the default ``exception_retry_configuration`` on your provider for Web3, because it cannot correctly throttle and decrease the `eth_getLogs` block number range. """ def __init__(self, w3: Web3, contract: Contract, state: EventScannerState, events: List, filters: Dict[str, Any], max_chunk_scan_size: int = 10000, max_request_retries: int = 30, request_retry_seconds: float = 3.0): """ :param contract: Contract :param events: List of web3 Event we scan :param filters: Filters passed to get_logs :param max_chunk_scan_size: JSON-RPC API limit in the number of blocks we query. (Recommendation: 10,000 for mainnet, 500,000 for testnets) :param max_request_retries: How many times we try to reattempt a failed JSON-RPC call :param request_retry_seconds: Delay between failed requests to let JSON-RPC server to recover """ self.logger = logger self.contract = contract self.w3 = w3 self.state = state self.events = events self.filters = filters # Our JSON-RPC throttling parameters self.min_scan_chunk_size = 10 # 12 s/block = 120 seconds period self.max_scan_chunk_size = max_chunk_scan_size self.max_request_retries = max_request_retries self.request_retry_seconds = request_retry_seconds # Factor how fast we increase the chunk size if results are found # # (slow down scan after starting to get hits) self.chunk_size_decrease = 0.5 # Factor how fast we increase chunk size if no results are found self.chunk_size_increase = 2.0 @property def address(self): return self.token_address def get_block_timestamp(self, block_num) -> datetime.datetime: """Get Ethereum block timestamp""" try: block_info = self.w3.eth.get_block(block_num) except BlockNotFound: # Block was not mined yet, # minor chain reorganisation? return None last_time = block_info["timestamp"] return datetime.datetime.utcfromtimestamp(last_time) def get_suggested_scan_start_block(self): """Get where we should start to scan for new token events. If there are no prior scans, start from block 1. Otherwise, start from the last end block minus ten blocks. We rescan the last ten scanned blocks in the case there were forks to avoid misaccounting due to minor single block works (happens once in an hour in Ethereum). These heuristics could be made more robust, but this is for the sake of simple reference implementation. """ end_block = self.get_last_scanned_block() if end_block: return max(1, end_block - self.NUM_BLOCKS_RESCAN_FOR_FORKS) return 1 def get_suggested_scan_end_block(self): """Get the last mined block on Ethereum chain we are following.""" # Do not scan all the way to the final block, as this # block might not be mined yet return self.w3.eth.block_number - 1 def get_last_scanned_block(self) -> int: return self.state.get_last_scanned_block() def delete_potentially_forked_block_data(self, after_block: int): """Purge old data in the case of blockchain reorganisation.""" self.state.delete_data(after_block) def scan_chunk(self, start_block, end_block) -> Tuple[int, datetime.datetime, list]: """Read and process events between to block numbers. Dynamically decrease the size of the chunk if the case JSON-RPC server pukes out. :return: tuple(actual end block number, when this block was mined, processed events) """ block_timestamps = {} get_block_timestamp = self.get_block_timestamp # Cache block timestamps to reduce some RPC overhead # Real solution might include smarter models around block def get_block_when(block_num): if block_num not in block_timestamps: block_timestamps[block_num] = get_block_timestamp(block_num) return block_timestamps[block_num] all_processed = [] for event_type in self.events: # Callable that takes care of the underlying web3 call def _fetch_events(_start_block, _end_block): return _fetch_events_for_all_contracts(self.w3, event_type, self.filters, from_block=_start_block, to_block=_end_block) # Do `n` retries on `eth_getLogs`, # throttle down block range if needed end_block, events = _retry_web3_call( _fetch_events, start_block=start_block, end_block=end_block, retries=self.max_request_retries, delay=self.request_retry_seconds) for evt in events: idx = evt["logIndex"] # Integer of the log index position in the block, null when its pending # We cannot avoid minor chain reorganisations, but # at least we must avoid blocks that are not mined yet assert idx is not None, "Somehow tried to scan a pending block" block_number = evt["blockNumber"] # Get UTC time when this event happened (block mined timestamp) # from our in-memory cache block_when = get_block_when(block_number) logger.debug("Processing event %s, block: %s count: %s", evt['event'], block_number, block_number) processed = self.state.process_event(block_when, evt) all_processed.append(processed) end_block_timestamp = get_block_when(end_block) return end_block, end_block_timestamp, all_processed def estimate_next_chunk_size(self, current_chuck_size: int, event_found_count: int): """Try to figure out optimal chunk size Our scanner might need to scan the whole blockchain for all events * We want to minimize API calls over empty blocks * We want to make sure that one scan chunk does not try to process too many entries once, as we try to control commit buffer size and potentially asynchronous busy loop * Do not overload node serving JSON-RPC API by asking data for too many events at a time Currently Ethereum JSON-API does not have an API to tell when a first event occurred in a blockchain and our heuristics try to accelerate block fetching (chunk size) until we see the first event. These heuristics exponentially increase the scan chunk size depending on if we are seeing events or not. When any transfers are encountered, we are back to scanning only a few blocks at a time. It does not make sense to do a full chain scan starting from block 1, doing one JSON-RPC call per 20 blocks. """ if event_found_count > 0: # When we encounter first events, reset the chunk size window current_chuck_size = self.min_scan_chunk_size else: current_chuck_size *= self.chunk_size_increase current_chuck_size = max(self.min_scan_chunk_size, current_chuck_size) current_chuck_size = min(self.max_scan_chunk_size, current_chuck_size) return int(current_chuck_size) def scan(self, start_block, end_block, start_chunk_size=20, progress_callback=Optional[Callable]) -> Tuple[ list, int]: """Perform a token balances scan. Assumes all balances in the database are valid before start_block (no forks sneaked in). :param start_block: The first block included in the scan :param end_block: The last block included in the scan :param start_chunk_size: How many blocks we try to fetch over JSON-RPC on the first attempt :param progress_callback: If this is an UI application, update the progress of the scan :return: [All processed events, number of chunks used] """ assert start_block <= end_block current_block = start_block # Scan in chunks, commit between chunk_size = start_chunk_size last_scan_duration = last_logs_found = 0 total_chunks_scanned = 0 # All processed entries we got on this scan cycle all_processed = [] while current_block <= end_block: self.state.start_chunk(current_block, chunk_size) # Print some diagnostics to logs to try to fiddle with real world JSON-RPC API performance estimated_end_block = min(current_block + chunk_size, end_block) logger.debug( "Scanning token transfers for blocks: %s - %s, chunk size %s, last chunk scan took %s, last logs found %s", current_block, estimated_end_block, chunk_size, last_scan_duration, last_logs_found, ) start = time.time() actual_end_block, end_block_timestamp, new_entries = self.scan_chunk(current_block, estimated_end_block) # Where does our current chunk scan ends - are we out of chain yet? current_end = actual_end_block last_scan_duration = time.time() - start all_processed += new_entries # Print progress bar if progress_callback: progress_callback(start_block, end_block, current_block, end_block_timestamp, chunk_size, len(new_entries)) # Try to guess how many blocks to fetch over `eth_getLogs` API next time chunk_size = self.estimate_next_chunk_size(chunk_size, len(new_entries)) # Set where the next chunk starts current_block = current_end + 1 total_chunks_scanned += 1 self.state.end_chunk(current_end) return all_processed, total_chunks_scanned def _retry_web3_call(func, start_block, end_block, retries, delay) -> Tuple[int, list]: """A custom retry loop to throttle down block range. If our JSON-RPC server cannot serve all incoming `eth_getLogs` in a single request, we retry and throttle down block range for every retry. For example, Go Ethereum does not indicate what is an acceptable response size. It just fails on the server-side with a "context was cancelled" warning. :param func: A callable that triggers Ethereum JSON-RPC, as func(start_block, end_block) :param start_block: The initial start block of the block range :param end_block: The initial start block of the block range :param retries: How many times we retry :param delay: Time to sleep between retries """ for i in range(retries): try: return end_block, func(start_block, end_block) except Exception as e: # Assume this is HTTPConnectionPool(host='localhost', port=8545): Read timed out. (read timeout=10) # from Go Ethereum. This translates to the error "context was cancelled" on the server side: # https://github.com/ethereum/go-ethereum/issues/20426 if i < retries - 1: # Give some more verbose info than the default middleware logger.warning( "Retrying events for block range %s - %s (%s) failed with %s , retrying in %s seconds", start_block, end_block, end_block-start_block, e, delay, ) # Decrease the `eth_getBlocks` range end_block = start_block + ((end_block - start_block) // 2) # Let the JSON-RPC to recover e.g. from restart time.sleep(delay) continue else: logger.warning("Out of retries") raise def _fetch_events_for_all_contracts( w3, event, argument_filters: Dict[str, Any], from_block: int, to_block: int) -> Iterable: """Get events using eth_getLogs API. This method is detached from any contract instance. This is a stateless method, as opposed to create_filter. It can be safely called against nodes which do not provide `eth_newFilter` API, like Infura. """ if from_block is None: raise Web3TypeError("Missing mandatory keyword argument to get_logs: from_block") # Currently no way to poke this using a public web3.py API. # This will return raw underlying ABI JSON object for the event abi = event._get_event_abi() # Depending on the Solidity version used to compile # the contract that uses the ABI, # it might have Solidity ABI encoding v1 or v2. # We just assume the default that you set on Web3 object here. # More information here https://eth-abi.readthedocs.io/en/latest/index.html codec: ABICodec = w3.codec # Here we need to poke a bit into Web3 internals, as this # functionality is not exposed by default. # Construct JSON-RPC raw filter presentation based on human readable Python descriptions # Namely, convert event names to their keccak signatures # More information here: # https://github.com/ethereum/web3.py/blob/e176ce0793dafdd0573acc8d4b76425b6eb604ca/web3/_utils/filters.py#L71 data_filter_set, event_filter_params = construct_event_filter_params( abi, codec, address=argument_filters.get("address"), argument_filters=argument_filters, from_block=from_block, to_block=to_block ) logger.debug("Querying eth_getLogs with the following parameters: %s", event_filter_params) # Call JSON-RPC API on your Ethereum node. # get_logs() returns raw AttributedDict entries logs = w3.eth.get_logs(event_filter_params) # Convert raw binary data to Python proxy objects as described by ABI all_events = [] for log in logs: # Convert raw JSON-RPC log result to human readable event by using ABI data # More information how process_log works here # https://github.com/ethereum/web3.py/blob/fbaf1ad11b0c7fac09ba34baff2c256cffe0a148/web3/_utils/events.py#L200 evt = get_event_data(codec, abi, log) # Note: This was originally yield, # but deferring the timeout exception caused the throttle logic not to work all_events.append(evt) return all_events if __name__ == "__main__": # Simple demo that scans all the token transfers of RCC token (11k). # The demo supports persistent state by using a JSON file. # You will need an Ethereum node for this. # Running this script will consume around 20k JSON-RPC calls. # With locally running Geth, the script takes 10 minutes. # The resulting JSON state file is 2.9 MB. import sys import json from web3.providers.rpc import HTTPProvider # We use tqdm library to render a nice progress bar in the console # https://pypi.org/project/tqdm/ from tqdm import tqdm # RCC has around 11k Transfer events # https://etherscan.io/token/0x9b6443b0fb9c241a7fdac375595cea13e6b7807a RCC_ADDRESS = "0x9b6443b0fb9c241a7fdac375595cea13e6b7807a" # Reduced ERC-20 ABI, only Transfer event ABI = """[ { "anonymous": false, "inputs": [ { "indexed": true, "name": "from", "type": "address" }, { "indexed": true, "name": "to", "type": "address" }, { "indexed": false, "name": "value", "type": "uint256" } ], "name": "Transfer", "type": "event" } ] """ class JSONifiedState(EventScannerState): """Store the state of scanned blocks and all events. All state is an in-memory dict. Simple load/store massive JSON on start up. """ def __init__(self): self.state = None self.fname = "test-state.json" # How many second ago we saved the JSON file self.last_save = 0 def reset(self): """Create initial state of nothing scanned.""" self.state = { "last_scanned_block": 0, "blocks": {}, } def restore(self): """Restore the last scan state from a file.""" try: self.state = json.load(open(self.fname, "rt")) print(f"Restored the state, previously {self.state['last_scanned_block']} blocks have been scanned") except (IOError, json.decoder.JSONDecodeError): print("State starting from scratch") self.reset() def save(self): """Save everything we have scanned so far in a file.""" with open(self.fname, "wt") as f: json.dump(self.state, f) self.last_save = time.time() # # EventScannerState methods implemented below # def get_last_scanned_block(self): """The number of the last block we have stored.""" return self.state["last_scanned_block"] def delete_data(self, since_block): """Remove potentially reorganised blocks from the scan data.""" for block_num in range(since_block, self.get_last_scanned_block()): if block_num in self.state["blocks"]: del self.state["blocks"][block_num] def start_chunk(self, block_number, chunk_size): pass def end_chunk(self, block_number): """Save at the end of each block, so we can resume in the case of a crash or CTRL+C""" # Next time the scanner is started we will resume from this block self.state["last_scanned_block"] = block_number # Save the database file for every minute if time.time() - self.last_save > 60: self.save() def process_event(self, block_when: datetime.datetime, event: AttributeDict) -> str: """Record a ERC-20 transfer in our database.""" # Events are keyed by their transaction hash and log index # One transaction may contain multiple events # and each one of those gets their own log index # event_name = event.event # "Transfer" log_index = event.logIndex # Log index within the block # transaction_index = event.transactionIndex # Transaction index within the block txhash = event.transactionHash.hex() # Transaction hash block_number = event.blockNumber # Convert ERC-20 Transfer event to our internal format args = event["args"] transfer = { "from": args["from"], "to": args.to, "value": args.value, "timestamp": block_when.isoformat(), } # Create empty dict as the block that contains all transactions by txhash if block_number not in self.state["blocks"]: self.state["blocks"][block_number] = {} block = self.state["blocks"][block_number] if txhash not in block: # We have not yet recorded any transfers in this transaction # (One transaction may contain multiple events if executed by a smart contract). # Create a tx entry that contains all events by a log index self.state["blocks"][block_number][txhash] = {} # Record ERC-20 transfer in our database self.state["blocks"][block_number][txhash][log_index] = transfer # Return a pointer that allows us to look up this event later if needed return f"{block_number}-{txhash}-{log_index}" def run(): if len(sys.argv) < 2: print("Usage: eventscanner.py http://your-node-url") sys.exit(1) api_url = sys.argv[1] # Enable logs to the stdout. # DEBUG is very verbose level logging.basicConfig(level=logging.INFO) provider = HTTPProvider(api_url) # Disable the default JSON-RPC retry configuration # as it correctly cannot handle eth_getLogs block range provider.exception_retry_configuration = None w3 = Web3(provider) # Prepare stub ERC-20 contract object abi = json.loads(ABI) ERC20 = w3.eth.contract(abi=abi) # Restore/create our persistent state state = JSONifiedState() state.restore() # chain_id: int, w3: Web3, abi: Dict, state: EventScannerState, events: List, filters: Dict, max_chunk_scan_size: int=10000 scanner = EventScanner( w3=w3, contract=ERC20, state=state, events=[ERC20.events.Transfer], filters={"address": RCC_ADDRESS}, # How many maximum blocks at the time we request from JSON-RPC # and we are unlikely to exceed the response size limit of the JSON-RPC server max_chunk_scan_size=10000 ) # Assume we might have scanned the blocks all the way to the last Ethereum block # that mined a few seconds before the previous scan run ended. # Because there might have been a minor Ethereum chain reorganisations # since the last scan ended, we need to discard # the last few blocks from the previous scan results. chain_reorg_safety_blocks = 10 scanner.delete_potentially_forked_block_data(state.get_last_scanned_block() - chain_reorg_safety_blocks) # Scan from [last block scanned] - [latest ethereum block] # Note that our chain reorg safety blocks cannot go negative start_block = max(state.get_last_scanned_block() - chain_reorg_safety_blocks, 0) end_block = scanner.get_suggested_scan_end_block() blocks_to_scan = end_block - start_block print(f"Scanning events from blocks {start_block} - {end_block}") # Render a progress bar in the console start = time.time() with tqdm(total=blocks_to_scan) as progress_bar: def _update_progress(start, end, current, current_block_timestamp, chunk_size, events_count): if current_block_timestamp: formatted_time = current_block_timestamp.strftime("%d-%m-%Y") else: formatted_time = "no block time available" progress_bar.set_description(f"Current block: {current} ({formatted_time}), blocks in a scan batch: {chunk_size}, events processed in a batch {events_count}") progress_bar.update(chunk_size) # Run the scan result, total_chunks_scanned = scanner.scan(start_block, end_block, progress_callback=_update_progress) state.save() duration = time.time() - start print(f"Scanned total {len(result)} Transfer events, in {duration} seconds, total {total_chunks_scanned} chunk scans performed") run() .. _WETH: https://etherscan.io/token/0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2#code .. _asyncio: https://docs.python.org/3/library/asyncio.html .. _gevent: https://www.gevent.org/ .. _twisted: https://twistedmatrix.com/ .. _celery: https://www.celeryproject.org/ --- ### Formatters .. _formatters: Formatters ========== Formatters are a core part of web3.py's data transformation pipeline. They convert data between Python-friendly formats and the hexadecimal formats required by the Ethereum JSON-RPC specification. This page explains how formatters work, what the default formatters do, and how to customize them. .. note:: For a deep dive into how requests flow through web3.py, including formatters, see the excellent blog post: `Web3.py Internals: JSON-RPC Round Trips `_ How Formatters Work ------------------- When you make a call like ``w3.eth.get_balance("0x123...")``, your request goes through several transformation steps before reaching the Ethereum node, and the response goes through similar transformations before being returned to you. The Request/Response Flow ~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: none Your Python Code | v +-----------------+ | Input Mungers | <- Set default values (e.g., block_identifier="latest") +-----------------+ | v +---------------------+ | Request Formatters | <- Convert Python types to JSON-RPC format +---------------------+ | v +-----------------+ | Middleware | <- Additional processing (e.g., ENS resolution) +-----------------+ | v +-----------------+ | Provider | <- Send JSON-RPC request to Ethereum node +-----------------+ | v Ethereum Node | v +-----------------+ | Provider | <- Receive JSON-RPC response +-----------------+ | v +-----------------+ | Middleware | <- Process response +-----------------+ | v +--------------------+ | Result Formatters | <- Convert JSON-RPC format to Python types +--------------------+ | v Your Python Code Types of Formatters ~~~~~~~~~~~~~~~~~~~ web3.py uses several types of formatters: 1. **Request Formatters**: Transform outgoing request parameters from Python types to the format expected by Ethereum nodes (e.g., integers to hex strings). 2. **Result Formatters**: Transform incoming response data from JSON-RPC format to Python-friendly types (e.g., hex strings to integers or ``HexBytes``). 3. **Error Formatters**: Process error responses to raise appropriate exceptions with meaningful messages. 4. **Null Result Formatters**: Handle cases where the response is ``None`` or empty. Default Formatters ------------------ web3.py includes a comprehensive set of default formatters that handle common data transformations. These are defined in ``web3._utils.method_formatters``. Pythonic Request Formatters ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``PYTHONIC_REQUEST_FORMATTERS`` convert Python types to JSON-RPC compatible formats before sending requests: .. list-table:: :header-rows: 1 :widths: 40 60 * - Transformation - Example * - Integers to hex strings - ``500000`` → ``"0x7a120"`` * - Block identifiers - ``"latest"`` (unchanged) or ``12345`` → ``"0x3039"`` * - Addresses to checksummed format - Validates and formats addresses * - Transaction parameters - Formats ``gas``, ``gasPrice``, ``value``, ``nonce`` to hex Example transformations by method: - **eth_getBalance**: Block number parameter converted to hex - **eth_call**: Transaction object fields formatted, block identifier converted - **eth_sendTransaction**: All numeric fields (gas, value, nonce) converted to hex - **eth_getBlockByNumber**: Block identifier converted to hex - **eth_getLogs**: ``fromBlock`` and ``toBlock`` converted to hex Pythonic Result Formatters ~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``PYTHONIC_RESULT_FORMATTERS`` convert JSON-RPC responses to Python types: .. list-table:: :header-rows: 1 :widths: 30 35 35 * - Method - Transformation - Example * - eth_getBalance - Hex to integer - ``"0x83a3c396d1a7b40"`` → ``583760663573639744`` * - eth_blockNumber - Hex to integer - ``"0xf4240"`` → ``1000000`` * - eth_getBlock - Full block formatting - Formats all block fields * - eth_getTransaction - Full transaction formatting - Formats all transaction fields * - eth_getTransactionReceipt - Receipt formatting - Formats logs, status, gas used * - eth_chainId - Hex to integer - ``"0x1"`` → ``1`` * - eth_gasPrice - Hex to integer - ``"0x3b9aca00"`` → ``1000000000`` Block Result Formatters ~~~~~~~~~~~~~~~~~~~~~~~ Block responses are formatted with ``BLOCK_RESULT_FORMATTERS``: .. code-block:: python { "baseFeePerGas": to_integer_if_hex, "difficulty": to_integer_if_hex, "gasLimit": to_integer_if_hex, "gasUsed": to_integer_if_hex, "number": to_integer_if_hex, "size": to_integer_if_hex, "timestamp": to_integer_if_hex, "totalDifficulty": to_integer_if_hex, "hash": to_hexbytes(32), "parentHash": to_hexbytes(32), "miner": to_checksum_address, "transactions": [list of tx hashes or full tx objects], # ... and more } Transaction Result Formatters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Transaction responses are formatted with ``TRANSACTION_RESULT_FORMATTERS``: .. code-block:: python { "blockNumber": to_integer_if_hex, "gas": to_integer_if_hex, "gasPrice": to_integer_if_hex, "nonce": to_integer_if_hex, "transactionIndex": to_integer_if_hex, "value": to_integer_if_hex, "v": to_integer_if_hex, "from": to_checksum_address, "to": to_checksum_address, "hash": to_hexbytes(32), "input": HexBytes, "r": to_hexbytes(32), "s": to_hexbytes(32), # ... and more } Log Entry Formatters ~~~~~~~~~~~~~~~~~~~~ Log entries (events) are formatted with ``LOG_ENTRY_FORMATTERS``: .. code-block:: python { "blockNumber": to_integer_if_hex, "logIndex": to_integer_if_hex, "transactionIndex": to_integer_if_hex, "address": to_checksum_address, "blockHash": to_hexbytes(32), "transactionHash": to_hexbytes(32), "data": HexBytes, "topics": [list of HexBytes(32)], } Using FormattingMiddlewareBuilder --------------------------------- For custom formatting needs, you can use the ``FormattingMiddlewareBuilder`` class to create middleware that applies your own formatters. Basic Usage ~~~~~~~~~~~ .. code-block:: python from web3 import Web3 from web3.middleware import FormattingMiddlewareBuilder w3 = Web3(Web3.HTTPProvider("http://localhost:8545")) # Create middleware with custom formatters custom_middleware = FormattingMiddlewareBuilder.build( request_formatters={ "eth_myCustomMethod": lambda params: [p.upper() for p in params] }, result_formatters={ "eth_myCustomMethod": lambda result: result.lower() }, ) # Add to middleware stack w3.middleware_onion.add(custom_middleware) Request Formatters Example ~~~~~~~~~~~~~~~~~~~~~~~~~~ Request formatters receive the parameters list and should return the transformed parameters: .. code-block:: python from web3.middleware import FormattingMiddlewareBuilder def my_request_formatter(params): """Transform request parameters before sending.""" # params is a list of parameters for the RPC method address, block_id = params # Ensure address is lowercase return [address.lower(), block_id] custom_middleware = FormattingMiddlewareBuilder.build( request_formatters={ "eth_getBalance": my_request_formatter, } ) Result Formatters Example ~~~~~~~~~~~~~~~~~~~~~~~~~ Result formatters receive the result value and should return the transformed result: .. code-block:: python from web3.middleware import FormattingMiddlewareBuilder from decimal import Decimal def balance_to_ether(result): """Convert balance from wei to ether.""" wei_balance = int(result, 16) if isinstance(result, str) else result return Decimal(wei_balance) / Decimal(10**18) custom_middleware = FormattingMiddlewareBuilder.build( result_formatters={ "eth_getBalance": balance_to_ether, } ) w3.middleware_onion.add(custom_middleware) # Now get_balance returns Decimal in ether instead of int in wei balance = w3.eth.get_balance("0x...") # Returns Decimal("1.5") instead of 1500000000000000000 Error Formatters Example ~~~~~~~~~~~~~~~~~~~~~~~~ Error formatters process error responses: .. code-block:: python from web3.middleware import FormattingMiddlewareBuilder def custom_error_formatter(error): """Add custom error handling.""" if "revert" in str(error.get("message", "")): error["message"] = f"Contract reverted: {error['message']}" return error custom_middleware = FormattingMiddlewareBuilder.build( error_formatters={ "eth_call": custom_error_formatter, } ) Dynamic Formatters with Builders ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For formatters that need access to the ``Web3`` instance or need to be built dynamically per-request, use the ``sync_formatters_builder`` and ``async_formatters_builder`` options: .. code-block:: python from web3.middleware import FormattingMiddlewareBuilder def build_formatters(w3, method): """Build formatters dynamically based on method and w3 instance.""" request_formatters = {} result_formatters = {} error_formatters = {} if method == "eth_getBalance": # Access w3 instance for dynamic behavior chain_id = w3.eth.chain_id if chain_id == 1: # Mainnet result_formatters["eth_getBalance"] = lambda x: int(x, 16) return { "request_formatters": request_formatters, "result_formatters": result_formatters, "error_formatters": error_formatters, } # For async usage async def async_build_formatters(async_w3, method): """Async version of formatter builder.""" chain_id = await async_w3.eth.chain_id # ... build formatters return { "request_formatters": {}, "result_formatters": {}, "error_formatters": {}, } custom_middleware = FormattingMiddlewareBuilder.build( sync_formatters_builder=build_formatters, async_formatters_builder=async_build_formatters, ) Common Formatter Utilities -------------------------- web3.py provides several utility functions for building formatters in ``web3._utils.formatters`` and ``eth_utils``: .. code-block:: python from eth_utils import ( to_checksum_address, to_hex, to_int, ) from eth_utils.curried import ( apply_formatter_at_index, apply_formatter_if, apply_formatter_to_array, apply_formatters_to_dict, apply_formatters_to_sequence, ) from web3._utils.formatters import ( apply_key_map, hex_to_integer, integer_to_hex, ) # Apply formatter to specific index in params list formatter = apply_formatter_at_index(to_hex, 1) # Format 2nd parameter # Apply formatter conditionally formatter = apply_formatter_if( lambda x: isinstance(x, int), # condition to_hex # formatter to apply if condition is True ) # Apply formatters to each element of an array formatter = apply_formatter_to_array(to_checksum_address) # Apply different formatters to dict keys formatter = apply_formatters_to_dict({ "gasPrice": to_hex, "value": to_hex, "from": to_checksum_address, }) Built-in Middleware Using Formatters ------------------------------------ Several built-in middleware use the formatting system: PythonicMiddleware ~~~~~~~~~~~~~~~~~~ The ``PythonicMiddleware`` applies the default Pythonic formatters to convert between Python types and JSON-RPC formats. This is included in the default middleware stack. .. code-block:: python from web3.middleware import PythonicMiddleware # This is equivalent to: PythonicMiddleware = FormattingMiddlewareBuilder.build( request_formatters=PYTHONIC_REQUEST_FORMATTERS, result_formatters=PYTHONIC_RESULT_FORMATTERS, ) ValidationMiddleware ~~~~~~~~~~~~~~~~~~~~ The ``ValidationMiddleware`` uses formatters to validate transaction parameters like ``chainId`` before sending: .. code-block:: python from web3.middleware import ValidationMiddleware # Included by default, validates transactions have correct chainId EthereumTesterMiddleware ~~~~~~~~~~~~~~~~~~~~~~~~ The ``ethereum_tester_middleware`` uses formatters to transform data between web3.py and the eth-tester backend: .. code-block:: python from web3.providers.eth_tester.middleware import ethereum_tester_middleware Best Practices -------------- 1. **Order Matters**: Request formatters are applied in order, and result formatters are applied in reverse order. Be mindful of the order when adding multiple formatting middleware. 2. **Don't Duplicate**: The default ``PythonicMiddleware`` handles most common transformations. Only add custom formatters for specific needs. 3. **Test Your Formatters**: Formatters can silently transform data incorrectly. Always test with real RPC calls. 4. **Use Type Hints**: When writing custom formatters, use type hints to catch errors early: .. code-block:: python from typing import Any def my_formatter(value: str) -> int: return int(value, 16) 5. **Handle Edge Cases**: Always handle ``None`` values and unexpected types: .. code-block:: python def safe_formatter(value): if value is None: return None return int(value, 16) if isinstance(value, str) else value See Also -------- - :ref:`middleware_internals` - How middleware works in web3.py - :ref:`Modifying_Middleware` - How to add, remove, and configure middleware - `JSON-RPC Specification `_ - The underlying protocol - `Ethereum JSON-RPC API `_ - Ethereum-specific methods --- ### Gas Price .. _Gas_Price: Gas Price API =============== .. warning:: Gas price strategy is only supported for legacy transactions. The London fork introduced ``maxFeePerGas`` and ``maxPriorityFeePerGas`` transaction parameters which should be used over ``gasPrice`` whenever possible. For Ethereum (legacy) transactions, gas price is a delicate property. For this reason, Web3 includes an API for configuring it. The Gas Price API allows you to define Web3's behaviour for populating the gas price. This is done using a "Gas Price Strategy" - a method which takes the Web3 object and a transaction dictionary and returns a gas price (denominated in wei). Retrieving gas price -------------------- To retrieve the gas price using the selected strategy simply call :meth:`~web3.eth.Eth.generate_gas_price` .. code-block:: python >>> web3.eth.generate_gas_price() 20000000000 Creating a gas price strategy ------------------------------- A gas price strategy is implemented as a python method with the following signature: .. code-block:: python def gas_price_strategy(web3, transaction_params=None): ... The method must return a positive integer representing the gas price in wei. To demonstrate, here is a rudimentary example of a gas price strategy that returns a higher gas price when the value of the transaction is higher than 1 Ether. .. code-block:: python from web3 import Web3 def value_based_gas_price_strategy(web3, transaction_params): if transaction_params['value'] > Web3.to_wei(1, 'ether'): return Web3.to_wei(20, 'gwei') else: return Web3.to_wei(5, 'gwei') Selecting the gas price strategy -------------------------------- The gas price strategy can be set by calling :meth:`~web3.eth.Eth.set_gas_price_strategy`. .. code-block:: python from web3 import Web3 def value_based_gas_price_strategy(web3, transaction_params): ... w3 = Web3(...) w3.eth.set_gas_price_strategy(value_based_gas_price_strategy) Available gas price strategies ------------------------------ .. py:module:: web3.gas_strategies.rpc .. py:method:: rpc_gas_price_strategy(web3, transaction_params=None) Makes a call to the `JSON-RPC eth_gasPrice method `_ which returns the gas price configured by the connected Ethereum node. .. py:module:: web3.gas_strategies.time_based .. py:method:: construct_time_based_gas_price_strategy(max_wait_seconds, sample_size=120, probability=98, weighted=False) Constructs a strategy which will compute a gas price such that the transaction will be mined within a number of seconds defined by ``max_wait_seconds`` with a probability defined by ``probability``. The gas price is computed by sampling ``sample_size`` of the most recently mined blocks. If ``weighted=True``, the block time will be weighted towards more recently mined blocks. * ``max_wait_seconds`` The desired maximum number of seconds the transaction should take to mine. * ``sample_size`` The number of recent blocks to sample * ``probability`` An integer representation of the desired probability that the transaction will be mined within ``max_wait_seconds``. 0 means 0% and 100 means 100%. The following ready to use versions of this strategy are available. * ``web3.gas_strategies.time_based.fast_gas_price_strategy``: Transaction mined within 60 seconds. * ``web3.gas_strategies.time_based.medium_gas_price_strategy``: Transaction mined within 5 minutes. * ``web3.gas_strategies.time_based.slow_gas_price_strategy``: Transaction mined within 1 hour. * ``web3.gas_strategies.time_based.glacial_gas_price_strategy``: Transaction mined within 24 hours. .. warning:: Due to the overhead of sampling the recent blocks it is recommended that a caching solution be used to reduce the amount of chain data that needs to be re-fetched for each request. .. code-block:: python from web3 import Web3 from web3.gas_strategies.time_based import medium_gas_price_strategy w3 = Web3(...) w3.eth.set_gas_price_strategy(medium_gas_price_strategy) w3.provider.cache_allowed_requests = True --- ### Index .. meta:: :description: Python Web3 SDK for Ethereum and EVM blockchains gm == .. image:: _static/banner/banner-snek.jpg :alt: Banner Image **web3.py** is a Python library for interacting with Ethereum. It's commonly found in `decentralized apps (dapps)`_ to help with sending transactions, interacting with smart contracts, reading block data, and a variety of other use cases. For project updates, follow `@EthereumPython`_ and sign up for new post notifications on the `blog`_. Getting Started --------------- .. NOTE:: 👋 Brand new to Ethereum? 0. Don't travel alone! Join the Ethereum Python Community `Discord`_. 1. Read this `blog post series`_ for a gentle introduction to Ethereum blockchain concepts. 2. The :ref:`Overview` page will give you a quick idea of what else web3.py can do. 3. Try building a little something! - Ready to code? → :ref:`quickstart` - Quick tour? → :ref:`overview` - Synchronous help? → `Discord`_ - Asynchronous help? → `StackExchange`_ - Report a bug? → `Github`_ - Want to help us? → :ref:`Contribute ` - Looking for inspiration? → :ref:`resources` .. include:: toc.rst .. _decentralized apps (dapps): https://ethereum.org/en/dapps/ .. _@EthereumPython: https://twitter.com/EthereumPython .. _blog: https://snakecharmers.ethereum.org/ .. _blog post series: https://snakecharmers.ethereum.org/a-developers-guide-to-ethereum-pt-1 .. _StackExchange: https://ethereum.stackexchange.com/questions/tagged/web3.py .. _Discord: https://discord.gg/GHryRvPB84 .. _Github: https://github.com/ethereum/web3.py/issues --- ### Internals Web3 Internals ============== .. warning:: This section of the documentation is for advanced users. You should probably stay away from these APIs if you don't know what you are doing. The Web3 library has multiple layers of abstraction between the public api exposed by the web3 object and the backend or node that web3 is connecting to. * **Providers** are responsible for the actual communication with the blockchain such as sending JSON-RPC requests over HTTP or an IPC socket. * **Middleware** provide hooks for monitoring and modifying requests and responses to and from the provider. * **Managers** provide thread safety and primitives to allow for asynchronous usage of web3. Here are some common things you might want to do with these APIs. * Redirect certain RPC requests to different providers such as sending all *read* operations to a provider backed by a remote node and all *write* operations to a local node that you control. * Transparently intercept transactions sent over ``eth_sendTransaction``, sign them locally, and then send them through ``eth_sendRawTransaction``. * Modify the response from an RPC request so that it is returned in different format such as converting all integer values to their hexadecimal representation. * Validate the inputs to RPC requests Request Lifecycle ----------------- Each web3 RPC call passes through these layers in the following manner. .. code-block:: none *********** ************ | Request | | Response | *********** ************ | ^ v | +-----------------------------+ | Manager | +-----------------------------+ | ^ v | +-----------------------------+ | Middleware | +-----------------------------+ | ^ v | +-----------------------------+ | Provider | +-----------------------------+ You can visualize this relationship like an onion, with the Provider at the center. The request originates from the ``Manager``, outside of the onion, passing down through each layer of the onion until it reaches the ``Provider`` at the center. The ``Provider`` then handles the request, producing a response which will then pass back out from the center of the onion, through each layer until it is finally returned by the ``Manager``. Providers --------- A provider is responsible for all direct blockchain interactions. In most cases this means interacting with the JSON-RPC server for an ethereum node over HTTP or an IPC socket. There is however nothing which requires providers to be RPC based, allowing for providers designed for testing purposes which use an in-memory EVM to fulfill requests. Writing your own Provider ~~~~~~~~~~~~~~~~~~~~~~~~~ Writing your own provider requires implementing two required methods as well as setting the middleware the provider should use. .. py:method:: BaseProvider.make_request(method, params) Each provider class **must** implement this method. This method **should** return a JSON object with either a ``'result'`` key in the case of success, or an ``'error'`` key in the case of failure. * ``method`` This will be a string representing the JSON-RPC method that is being called such as ``'eth_sendTransaction'``. * ``params`` This will be a list or other iterable of the parameters for the JSON-RPC method being called. .. py:method:: BaseProvider.is_connected(show_traceback=False) This function should return ``True`` or ``False`` depending on whether the provider should be considered *connected*. For example, an IPC socket based provider should return ``True`` if the socket is open and ``False`` if the socket is closed. If set to ``True``, the optional ``show_traceback`` boolean will raise a ``ProviderConnectionError`` and provide information on why the provider should not be considered *connected*. .. py:attribute:: BaseProvider.middleware This should be an iterable of middleware. You can set a new list of middleware by assigning to ``provider.middleware``, with the first middleware that processes the request at the beginning of the list. Provider Configurations ~~~~~~~~~~~~~~~~~~~~~~~ .. _request_caching: Request Caching ``````````````` /* Detailed source-code truncated for AI context efficiency. */ ````````````````````````````````` ``HTTPProvider`` and ``AsyncHTTPProvider`` instances retry certain requests by default on exceptions. This can be configured via the ``exception_retry_configuration`` property on the provider instance, which takes a :class:`~web3.providers.rpc.utils.ExceptionRetryConfiguration` class as its value. The retry mechanism employs an exponential backoff strategy, starting from the initial value determined by the ``backoff_factor``, and doubling the delay with each attempt, up to the ``retries`` value. Below is an example showing the default options for the retry configuration and how to override them. .. py:class:: web3.providers.rpc.utils.ExceptionRetryConfiguration .. py:attribute:: errors A tuple of exceptions that the provider should retry on. The default is ``HTTPProvider``: ``(ConnectionError, requests.HTTPError, requests.Timeout)`` and ``AsyncHTTPProvider``: ``(aiohttp.ClientError, asyncio.TimeoutError)``. .. py:attribute:: retries The number of retries to attempt. The default is 5. .. py:attribute:: backoff_factor The initial delay multiplier, which doubles with each retry attempt. The default is 0.125. .. py:attribute:: method_allowlist A list of retryable methods. The default is an in-house list of deemed-safe-to- retry methods. .. code-block:: python from web3 import Web3, HTTPProvider from web3.providers.rpc.utils import ( REQUEST_RETRY_ALLOWLIST, ExceptionRetryConfiguration, ) w3 = Web3(HTTPProvider( endpoint_uri="...", exception_retry_configuration=ExceptionRetryConfiguration( errors=DEFAULT_EXCEPTIONS, # number of retries to attempt retries=5, # initial delay multiplier, doubles with each retry attempt backoff_factor=0.125, # an in-house default list of retryable methods method_allowlist=REQUEST_RETRY_ALLOWLIST, ), )) For the different http providers, ``DEFAULT_EXCEPTIONS`` is defined as: - ``HTTPProvider``: ``(ConnectionError, requests.HTTPError, requests.Timeout)`` - ``AsyncHTTPProvider``: ``(ConnectionError, aiohttp.ClientError, asyncio.TimeoutError)`` Setting ``retry_configuration`` to ``None`` will disable retries on exceptions for the provider instance. .. code-block:: python from web3 import Web3, HTTPProvider w3 = Web3(HTTPProvider(endpoint_uri="...", retry_configuration=None) Managers -------- The Manager acts as a gatekeeper for the request/response lifecycle. It is unlikely that you will need to change the Manager as most functionality can be implemented in the Middleware layer. .. _internals__persistent_connection_providers: Request Processing for Persistent Connection Providers ------------------------------------------------------ .. py:class:: web3.providers.persistent.request_processor.RequestProcessor The ``RequestProcessor`` class is responsible for the storing and syncing up of asynchronous requests to responses for a ``PersistentConnectionProvider``. The :class:`~web3.providers.persistent.WebSocketProvider` and the :class:`~web3.providers.persistent.AsyncIPCProvider` are two persistent connection providers. In order to send a request and receive a response to that same request, ``PersistentConnectionProvider`` instances have to match request *id* values to response *id* values coming back from the socket connection. Any provider that does not adhere to the `JSON-RPC 2.0 specification `_ in this way will not work with ``PersistentConnectionProvider`` instances. The specifics of how the request processor handles this are outlined below. Listening for Responses ~~~~~~~~~~~~~~~~~~~~~~~ Implementations of the ``PersistentConnectionProvider`` class have a message listener background task that is called when the socket connection is established. This task is responsible for listening for any and all messages coming in over the socket connection and storing them in the ``RequestProcessor`` instance internal to the ``PersistentConnectionProvider`` instance. The ``RequestProcessor`` instance is responsible for storing the messages in the correct cache, either the one-to-one cache or the one-to-many (subscriptions) queue, depending on whether the message has a JSON-RPC *id* value or not. One-To-One Requests ~~~~~~~~~~~~~~~~~~~ One-to-one requests can be summarized as any request that expects only one response back. An example is using the ``eth`` module API to request the latest block number. .. code-block:: python >>> async def ws_one_to_one_example(): ... async with AsyncWeb3(WebSocketProvider(f"ws://127.0.0.1:8546")) as w3: ... # make a request and expect a single response returned on the same line ... latest_block_num = await w3.eth.block_number >>> asyncio.run(ws_one_to_one_example()) With persistent socket connections, we have to call ``send()`` and asynchronously receive responses via another means, generally by calling ``recv()`` or by iterating on the socket connection for messages. As outlined above, the ``PersistentConnectionProvider`` class has a message listener background task that handles the receiving of messages. Due to this asynchronous nature of sending and receiving, in order to make one-to-one request-to-response calls work, we have to save the request information somewhere so that, when the response is received, we can match it to the original request that was made (i.e. the request with a matching *id* to the response that was received). The stored request information is then used to process the response when it is received, piping it through the response formatters and middleware internal to the *web3.py* library. In order to store the request information, the ``RequestProcessor`` class has an internal ``RequestInformation`` cache. The ``RequestInformation`` class saves important information about a request. .. py:class:: web3._utils.caching.RequestInformation .. py:attribute:: method The name of the method - e.g. "eth_subscribe". .. py:attribute:: params The params used when the call was made - e.g. ("newPendingTransactions", True). .. py:attribute:: response_formatters The formatters that will be used to process the response. .. py:attribute:: middleware_response_processors Any middleware that processes responses that is present on the instance at the time of the request is appended here, in order, so the response may be piped through that logic when it comes in. .. py:attribute:: subscription_id If the request is an ``eth_subscribe`` request, rather than popping this information from the cache when the response to the subscription call comes in (i.e. the subscription *id*), we save the subscription id with the request information so that we can correctly process all subscription messages that come in with that subscription *id*. For one-to-one request-to-response calls, this value is always ``None``. One-to-one responses, those that include a JSON-RPC *id* in the response object, are stored in an internal ``SimpleCache`` class, isolated from any one-to-many responses. When the ``PersistentConnectionProvider`` is looking for a response internally, it will expect the message listener task to store the response in this cache. Since the request *id* is used in the cache key generation, it will then look for a cache key that matches the response *id* with that of the request *id*. If the cache key is found, the response is processed and returned to the user. If the cache key is not found, the operation will time out and raise a ``TimeExhausted`` exception. This timeout can be configured by the user when instantiating the ``PersistentConnectionProvider`` instance via the ``response_timeout`` keyword argument. One-To-Many Requests ~~~~~~~~~~~~~~~~~~~~ One-to-many requests can be summarized by any request that expects many responses as a result of the initial request. The only current example is the ``eth_subscribe`` request. The initial ``eth_subscribe`` request expects only one response, the subscription *id* value, but it also expects to receive many ``eth_subscription`` messages if and when the request is successful. For this reason, the original request is considered a one-to-one request so that a subscription *id* can be returned to the user on the same line. The many responses this call will produce can be handled in one of a few ways. The recommended way to handle one-to-many responses is to use the subscription manager API. The subscription manager API is a public API on the ``AsyncWeb3`` class, when connected to a ``PersistentConnectionProvider`` instance, that allows the user to subscribe to a subscription and handle the many responses asynchronously. The ``subscription_manager`` instance is responsible for handling the many responses that come in over the socket connection, as long as handlers are passed to each subscription call. The subscription manager can also be used to unsubscribe from a subscription when the user is done with it. .. code-block:: python >>> async def new_heads_handler( ... handler_context: NewHeadsSubscriptionContext, ... ) -> None: ... result = handler_context.result ... print(f"New block header: {result}\n") ... if result["number"] > 1234567: ... await handler_context.subscription.unsubscribe() >>> async def ws_subscription_example(): ... async with AsyncWeb3(WebSocketProvider(f"ws://127.0.0.1:8546")) as w3: ... # Subscribe to new block headers and receive the subscription_id. ... # A one-to-one call with a trigger for many responses ... subscription_id = await w3.eth.subscribe("newHeads", handler=new_heads_handler) ... ... # Handle the subscription messages asynchronously using the subscription ... # manager. This will continue until no more subscriptions are present in ... # the subscription manager, or indefinitely if the `run_forever` flag ... # is set to `True`. ... await w3.subscription_manager.handle_subscriptions(run_forever=False) >>> asyncio.run(ws_subscription_example()) The manager can also subscribe to many subscriptions at one time. The ``EthSubscription`` classes, available via ``web3.utils.subscriptions``, provide a friendly API for managing subscriptions. Since each connection and provider instance has its own message listener task and subscription manager instance, you can subscribe to many subscriptions at once and handle the many responses that come in over the socket connections via handlers. The handlers contain: - ``async_w3``: The ``AsyncWeb3`` instance that the subscription was made on. - ``subscription``: The subscription instance that the handler is attached to. - ``result``: The response that came in over the socket connection for the subscription. Subscriptions also accept a ``handler_context`` argument that can be used to pass additional information to the handler when subscribing to a subscription. This can be used to pass in an event object, for example, that can be used to parse a log event when it comes in. .. code-block:: python >>> from web3 import ( ... AsyncWeb3, ... WebSocketProvider, ... AsyncIPCProvider, ... ) >>> from web3.utils.subscriptions import ( ... EthSubscription, ... NewHeadsSubscription, ... NewHeadsSubscriptionContext, ... PendingTxSubscription, ... PendingTxSubscriptionContext, ... LogsSubscription, ... LogsSubscriptionContext, ... ) >>> async def new_heads_handler( ... handler_context: NewHeadsSubscriptionContext, ... ) -> None: ... header = handler_context.result ... print(f"New block header: {header}\n") ... if header["number"] > 1234567: ... await handler_context.subscription.unsubscribe() >>> async def pending_txs_handler( ... handler_context: PendingTxSubscriptionContext, ... ) -> None: ... ... >>> async def log_handler( ... handler_context: LogsSubscriptionContext, ... ) -> None: ... log_receipt = handler_context.result ... # event is now available in the handler context, because we pass it to in the ... # ``handler_context`` when subscribing to the log ... event_data = handler_context.transfer_event.process_log(log_receipt) ... print(f"Log event data: {event_data}\n") >>> async def sub_manager(): ... local_w3 = await AsyncWeb3(AsyncIPCProvider(LOCAL_IPC)) ... ... # subscribe to many subscriptions via the subscription manager with handlers ... weth_contract = local_w3.eth.contract( ... address=local_w3.to_checksum_address("0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2"), ... abi=WETH_ABI, ... ) ... transfer_event = weth_contract.events.Transfer() ... await local_w3.subscription_manager.subscribe( ... [ ... NewHeadsSubscription(label="new-heads-mainnet", handler=new_heads_handler), ... PendingTxSubscription( ... label="pending-tx-mainnet", # optional label ... full_transactions=True, ... handler=pending_tx_handler, ... ), ... LogsSubscription( ... label="WETH transfers", # optional label ... address=weth_contract.address, ... topics=[transfer_event.topic], ... handler=log_handler, ... # optional ``handler_context`` args to help parse a response ... handler_context={"transfer_event": transfer_event}, ... ), ... ] ... ) ... ... public_w3 = await AsyncWeb3(WebSocketProvider(PUBLIC_PROVIDER_WS)) ... # subscribe via eth_subscribe, with handler and label (optional) ... await public_w3.eth.subscribe("public_newHeads", handler=pending_tx_handler, label="new-heads-public-ws") >>> # This will handle all subscriptions until no more subscriptions are present ... # in either subscription manager instance. If the `run_forever` flag is set ... # to `True` on any manager instance, this will run indefinitely. >>> await asyncio.gather( ... public_w3.subscription_manager.handle_subscriptions(), ... local_w3.subscription_manager.handle_subscriptions(), ... ) ... ... # close the connections ... await local_w3.provider.disconnect() ... await public_w3.provider.disconnect() >>> asyncio.run(sub_manager()) The ``process_subscriptions()`` method on the :class:`~web3.providers.persistent.PersistentConnection` class, the public API for interacting with the active persistent socket connection, is also set up to receive ``eth_subscription`` responses over an asynchronous iterator pattern. You can use this method to listen for raw messages and process them as they come in. .. code-block:: python >>> async def ws_subscription_example(): ... async with AsyncWeb3(WebSocketProvider(f"ws://127.0.0.1:8546")) as w3: ... # Subscribe to new block headers and receive the subscription_id. ... # A one-to-one call with a trigger for many responses ... subscription_id = await w3.eth.subscribe("newHeads") ... ... # Listen to the socket for the many responses utilizing the ... # ``w3.socket`` ``PersistentConnection`` public API method ... # ``process_subscriptions()`` ... async for response in w3.socket.process_subscriptions(): ... # Receive only one-to-many responses here so that we don't ... # accidentally return the response for a one-to-one request in this ... # block ... ... print(f"{response}\n") ... ... if some_condition: ... # unsubscribe from new block headers, another one-to-one request ... is_unsubscribed = await w3.eth.unsubscribe(subscription_id) ... if is_unsubscribed: ... break >>> asyncio.run(ws_subscription_example()) One-to-many responses, those that do not include a JSON-RPC *id* in the response object, are stored in an internal ``asyncio.Queue`` instance, isolated from any one-to-one responses. When the ``PersistentConnectionProvider`` is looking for one-to-many responses internally, it will expect the message listener task to store these messages in this queue. Since the order of the messages is important, the queue is a FIFO queue. The ``process_subscriptions()`` method on the ``PersistentConnection`` class is set up to pop messages from this queue as FIFO over an asynchronous iterator pattern. If the stream of messages from the socket is not being interrupted by any other tasks, the queue will generally be in sync with the messages coming in over the socket. That is, the message listener will put a message in the queue and the ``process_subscriptions()`` method will pop that message from the queue and yield control of the loop back to the listener. This will continue until the socket connection is closed or the user unsubscribes from the subscription. If the stream of messages lags a bit, or the provider is not consuming messages but has subscribed to a subscription, this internal queue may fill up with messages until it reaches its max size and then trigger a waiting ``asyncio.Event`` until the provider begins consuming messages from the queue again. For this reason, it's important to begin consuming messages from the queue, via the ``process_subscriptions()`` method, as soon as a subscription is made. --- ### Middleware .. _middleware_internals: Middleware ========== ``Web3`` is instantiated with layers of middleware by default. They sit between the public ``Web3`` methods and the :doc:`providers`, and are used to perform sanity checks, convert data types, enable ENS support, and more. Each layer can modify the request and/or response. While several middleware are enabled by default, others are available for optional use, and you're free to create your own! Each middleware layer gets invoked before the request reaches the provider, and then processes the result after the provider returns, in reverse order. However, it is possible for a middleware to return early from a call without the request ever getting to the provider (or even reaching the middleware that are in deeper layers). .. _Modifying_Middleware: Configuring Middleware ----------------------- Middleware can be added, removed, replaced, and cleared at runtime. To make that easier, you can name the middleware for later reference. Middleware Order ~~~~~~~~~~~~~~~~ Think of the middleware as being layered in an onion, where you initiate a web3.py request at the outermost layer of the onion, and the Ethereum node (like geth) receives and responds to the request inside the innermost layer of the onion. Here is a (simplified) diagram: .. code-block:: none New request from web3.py | | v `````Layer 2`````` ``````` ``````` ````` | ```` ```` v ```` ``` ``` `. ````````Layer 1``````` `.` `` ```` ````` .` `. ``` | ``` `.` .` ``` v ``` `. `. `.` ``` .` `` .` `Layer 0` `` .` `` `. ````` `````` . .` `. `` ``` | ``` .` . . `` `.` | `` . . . `. `` JSON-RPC call .` . .` . . `` | . `` . `` . . v . . . . .` . . . `` . . . Ethereum node .` . . . . . . . . . `` `. | . . . . . .` | .` . . `. .` .` Response .` .` . . . `.` | `.` `. . `. . ``` | ```` `. . . `. ````` v ```` `. `` . .` ```Layer 0`` `` `. . `. `.` `. . `. | `.` `. .` ``` | ``` .` `. ``` v ```` `.` `` `````` ````` .` `` `````Layer 1````` `.` ``` ``` ```` | ``` ````` v ```` `````` ````` `````````Layer 2`````````` | v Returned value in web3.py The middleware are maintained in ``Web3.middleware_onion``. See below for the API. When specifying middleware in a list, or retrieving the list of middleware, they will be returned in the order of outermost layer first and innermost layer last. In the above example, that means that ``w3.middleware_onion.middleware`` would return the middleware in the order of: ``[2, 1, 0]``. .. _middleware_stack_api: Middleware Stack API ~~~~~~~~~~~~~~~~~~~~ To add or remove items in different layers, use the following API: .. py:method:: Web3.middleware_onion.add(middleware, name=None) Middleware will be added to the outermost layer. That means the new middleware will modify the request first, and the response last. You can optionally name it with any hashable object, typically a string. .. code-block:: python >>> w3 = Web3(...) >>> w3.middleware_onion.add(web3.middleware.GasPriceStrategyMiddleware) # or >>> w3.middleware_onion.add(web3.middleware.GasPriceStrategyMiddleware, 'gas_price_strategy') .. py:method:: Web3.middleware_onion.inject(middleware, name=None, layer=None) Inject a named middleware to an arbitrary layer. The current implementation only supports injection at the innermost or outermost layers. Note that injecting to the outermost layer is equivalent to calling :meth:`Web3.middleware_onion.add` . .. code-block:: python # Either of these will put the gas_price_strategy middleware at the innermost layer >>> w3 = Web3(...) >>> w3.middleware_onion.inject(web3.middleware.GasPriceStrategyMiddleware, layer=0) # or >>> w3.middleware_onion.inject(web3.middleware.GasPriceStrategyMiddleware, 'gas_price_strategy', layer=0) .. py:method:: Web3.middleware_onion.remove(middleware) Middleware will be removed from whatever layer it was in. If you added the middleware with a name, use the name to remove it. If you added the middleware as an object, use the object again later to remove it: .. code-block:: python >>> w3 = Web3(...) >>> w3.middleware_onion.remove(web3.middleware.GasPriceStrategyMiddleware) # or >>> w3.middleware_onion.remove('gas_price_strategy') .. py:method:: Web3.middleware_onion.replace(old_middleware, new_middleware) Middleware will be replaced from whatever layer it was in. If the middleware was named, it will continue to have the same name. If it was un-named, then you will now reference it with the new middleware object. .. code-block:: python >>> from web3.middleware import GasPriceStrategyMiddleware, AttributeDictMiddleware >>> w3 = Web3(provider, middleware=[GasPriceStrategyMiddleware, AttributeDictMiddleware]) >>> w3.middleware_onion.replace(GasPriceStrategyMiddleware, AttributeDictMiddleware) # this is now referenced by the new middleware object, so to remove it: >>> w3.middleware_onion.remove(AttributeDictMiddleware) # or, if it was named >>> w3.middleware_onion.replace('gas_price_strategy', AttributeDictMiddleware) # this is still referenced by the original name, so to remove it: >>> w3.middleware_onion.remove('gas_price_strategy') .. py:method:: Web3.middleware_onion.clear() Empty all the middleware, including the default ones. .. code-block:: python >>> w3 = Web3(...) >>> w3.middleware_onion.clear() >>> assert len(w3.middleware_onion) == 0 .. py:attribute:: Web3.middleware_onion.middleware Return all the current middleware for the ``Web3`` instance in the appropriate order for importing into a new ``Web3`` instance. .. code-block:: python >>> w3_1 = Web3(...) # add uniquely named middleware: >>> w3_1.middleware_onion.add(web3.middleware.GasPriceStrategyMiddleware, 'test_middleware') # export middleware from first w3 instance >>> middleware = w3_1.middleware_onion.middleware # import into second instance >>> w3_2 = Web3(..., middleware=middleware) >>> assert w3_1.middleware_onion.middleware == w3_2.middleware_onion.middleware >>> assert w3_2.middleware_onion.get('test_middleware') Instantiate with Custom Middleware ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Instead of working from the default list, you can specify a custom list of middleware when initializing Web3: .. code-block:: python Web3(middleware=[my_middleware1, my_middleware2]) .. warning:: This will *replace* the default middleware. To keep the default functionality, either use ``middleware_onion.add()`` from above, or add the default middleware to your list of new middleware. .. _default_middleware: Default Middleware ------------------ The following middleware are included by default: * ``gas_price_strategy`` * ``ens_name_to_address`` * ``attrdict`` * ``validation`` * ``gas_estimate`` The defaults are defined in the ``get_default_middleware()`` method in ``web3/manager.py``. AttributeDict ~~~~~~~~~~~~~ .. py:class:: web3.middleware.AttributeDictMiddleware This middleware recursively converts any dictionary type in the result of a call to an ``AttributeDict``. This enables dot-syntax access, like ``eth.get_block('latest').number`` in addition to ``eth.get_block('latest')['number']``. .. note:: Accessing a property via attribute breaks type hinting. For this reason, this feature is available as a middleware, which may be removed if desired. ENS Name to Address Resolution ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. py:class:: web3.middleware.ENSNameToAddressMiddleware This middleware converts Ethereum Name Service (ENS) names into the address that the name points to. For example :meth:`w3.eth.send_transaction ` will accept .eth names in the 'from' and 'to' fields. .. note:: This middleware only converts ENS names on chains where the proper ENS contracts are deployed to support this functionality. All other cases will result in a ``NameNotFound`` error. Gas Price Strategy ~~~~~~~~~~~~~~~~~~ .. py:class:: web3.middleware.GasPriceStrategyMiddleware .. warning:: Gas price strategy is only supported for legacy transactions. The London fork introduced ``maxFeePerGas`` and ``maxPriorityFeePerGas`` transaction parameters which should be used over ``gasPrice`` whenever possible. This adds a ``gasPrice`` to transactions if applicable and when a gas price strategy has been set. See :ref:`Gas_Price` for information about how gas price is derived. Buffered Gas Estimate ~~~~~~~~~~~~~~~~~~~~~ .. py:class:: web3.middleware.BufferedGasEstimateMiddleware This adds a gas estimate to transactions if ``gas`` is not present in the transaction parameters. Sets gas to: ``min(w3.eth.estimate_gas + gas_buffer, gas_limit)`` where the gas_buffer default is 100,000 Validation ~~~~~~~~~~ .. py:class:: web3.middleware.ValidationMiddleware This middleware includes block and transaction validators which perform validations for transaction parameters. Optional Middleware ------------------- ``Web3`` includes optional middleware for common use cases. Below is a list of available middleware which are not enabled by default. Stalecheck ~~~~~~~~~~~~ .. py:method:: web3.middleware.StalecheckMiddlewareBuilder This middleware checks how stale the blockchain is, and interrupts calls with a failure if the blockchain is too old. * ``allowable_delay`` is the length in seconds that the blockchain is allowed to be behind of ``time.time()`` Because this middleware takes an argument, you must create the middleware with a method call. .. code-block:: python two_day_stalecheck = StalecheckMiddlewareBuilder.build(60 * 60 * 24 * 2) web3.middleware_onion.add(two_day_stalecheck) If the latest block in the blockchain is older than 2 days in this example, then the middleware will raise a ``StaleBlockchain`` exception on every call except ``web3.eth.get_block()``. .. _geth-poa: Proof of Authority ~~~~~~~~~~~~~~~~~~ .. py:class:: web3.middleware.ExtraDataToPOAMiddleware .. important:: It is **crucial** that this middleware is injected at the 0th layer of the middleware onion, using ``w3.middleware_onion.inject(ExtraDataToPOAMiddleware, layer=0)``, to guarantee it is the *first* middleware to process the response and modify the ``extraData`` field. This ensures it processes the field before any other middleware attempts to validate it. ``ExtraDataToPOAMiddleware`` is required to connect to ``geth --dev`` and may also be needed for other EVM compatible blockchains like Polygon or BNB Chain (Binance Smart Chain). If the middleware is not injected at the 0th layer of the middleware onion, you may get errors like the example below when interacting with your EVM node. .. code-block:: shell web3.exceptions.ExtraDataLengthError: The field extraData is 97 bytes, but should be 32. It is quite likely that you are connected to a POA chain. Refer to http://web3py.readthedocs.io/en/stable/middleware.html#proof-of-authority for more details. The full extraData is: HexBytes('...') The easiest way to connect to a default ``geth --dev`` instance which loads the middleware is: .. code-block:: python >>> from web3.auto.gethdev import w3 # confirm that the connection succeeded >>> w3.client_version 'Geth/v1.14.12-stable-4bb3c89d/linux-amd64/go1.22.4' This example connects to a local ``geth --dev`` instance on Linux with a unique IPC location and loads the middleware: .. code-block:: python >>> from web3 import Web3, IPCProvider # connect to the IPC location started with 'geth --dev --datadir ~/mynode' >>> w3 = Web3(IPCProvider('~/mynode/geth.ipc')) >>> from web3.middleware import ExtraDataToPOAMiddleware # inject the poa compatibility middleware to the innermost layer (0th layer) >>> w3.middleware_onion.inject(ExtraDataToPOAMiddleware, layer=0) # confirm that the connection succeeded >>> w3.client_version 'Geth/v1.14.12-stable-4bb3c89d/linux-amd64/go1.22.4' Why is ``ExtraDataToPOAMiddleware`` necessary? '''''''''''''''''''''''''''''''''''''''''''''' There is no strong community consensus on a single Proof-of-Authority (PoA) standard yet. Some nodes have successful experiments running though. One is go-ethereum (geth), which uses a prototype PoA for its development mode and the Goerli test network. Unfortunately, it does deviate from the yellow paper specification, which constrains the ``extraData`` field in each block to a maximum of 32-bytes. Geth is one such example where PoA uses more than 32 bytes, so this middleware modifies the block data a bit before returning it. .. _local-filter: Locally Managed Log and Block Filters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. py:method:: web3.middleware.LocalFilterMiddleware This middleware provides an alternative to ethereum node managed filters. When used, Log and Block filter logic are handled locally while using the same web3 filter api. Filter results are retrieved using JSON-RPC endpoints that don't rely on server state. .. doctest:: >>> from web3 import Web3, EthereumTesterProvider >>> w3 = Web3(EthereumTesterProvider()) >>> from web3.middleware import LocalFilterMiddleware >>> w3.middleware_onion.add(LocalFilterMiddleware) .. code-block:: python # Normal block and log filter apis behave as before. >>> block_filter = w3.eth.filter("latest") >>> log_filter = myContract.events.myEvent.build_filter().deploy() Signing ~~~~~~~ .. py:method:: web3.middleware.SignAndSendRawMiddlewareBuilder This middleware automatically captures transactions, signs them, and sends them as raw transactions. The ``from`` field on the transaction, or ``w3.eth.default_account`` must be set to the address of the private key for this middleware to have any effect. The ``build`` method for this middleware builder takes a single argument: * ``private_key_or_account`` A single private key or a tuple, list or set of private keys. Keys can be in any of the following formats: * An ``eth_account.LocalAccount`` object * An ``eth_keys.PrivateKey`` object * A raw private key as a hex string or byte string .. important:: Since this middleware signs transactions, it must always run after any middleware that modifies the transaction. Therefore, it is recommended to inject the signing middleware at the 0th layer of the middleware onion using ``w3.middleware_onion.inject(SignAndSendRawMiddlewareBuilder.build(...), layer=0)``. Ensure that any transaction-modifying middleware exists in a higher layer within the onion so that it runs before the signing middleware. .. note:: If used with ``ExtraDataToPOAMiddleware``, the injection order doesn't matter, as the ``extraData`` field isn't involved in transaction signing. The key is ensuring ``SignAndSendRawMiddlewareBuilder`` runs after any middleware that modifies the transaction. .. code-block:: python >>> from web3 import Web3, EthereumTesterProvider >>> w3 = Web3(EthereumTesterProvider) >>> from web3.middleware import SignAndSendRawMiddlewareBuilder >>> from eth_account import Account >>> acct = Account.create('KEYSMASH FJAFJKLDSKF7JKFDJ 1530') >>> w3.middleware_onion.inject(SignAndSendRawMiddlewareBuilder.build(acct), layer=0) >>> w3.eth.default_account = acct.address :ref:`Hosted nodes` (like Infura or Alchemy) only support signed transactions. This often results in ``send_raw_transaction`` being used repeatedly. Instead, we can automate this process with ``SignAndSendRawMiddlewareBuilder.build(private_key_or_account)``. .. code-block:: python >>> from web3 import Web3 >>> w3 = Web3(Web3.HTTPProvider('HTTP_ENDPOINT')) >>> from web3.middleware import SignAndSendRawMiddlewareBuilder >>> from eth_account import Account >>> import os >>> acct = w3.eth.account.from_key(os.environ.get('PRIVATE_KEY')) >>> w3.middleware_onion.inject(SignAndSendRawMiddlewareBuilder.build(acct), layer=0) >>> w3.eth.default_account = acct.address >>> # use `eth_sendTransaction` to automatically sign and send the raw transaction >>> w3.eth.send_transaction(tx_dict) HexBytes('0x09511acf75918fd03de58141d2fd409af4fd6d3dce48eb3aa1656c8f3c2c5c21') Similarly, with AsyncWeb3: .. code-block:: python >>> from web3 import AsyncWeb3 >>> async_w3 = AsyncWeb3(AsyncHTTPProvider('HTTP_ENDPOINT')) >>> from web3.middleware import SignAndSendRawMiddlewareBuilder >>> from eth_account import Account >>> import os >>> acct = async_w3.eth.account.from_key(os.environ.get('PRIVATE_KEY')) >>> async_w3.middleware_onion.inject(SignAndSendRawMiddlewareBuilder.build(acct), layer=0) >>> async_w3.eth.default_account = acct.address >>> # use `eth_sendTransaction` to automatically sign and send the raw transaction >>> await async_w3.eth.send_transaction(tx_dict) HexBytes('0x09511acf75918fd03de58141d2fd409af4fd6d3dce48eb3aa1656c8f3c2c5c21') Now you can send a transaction from acct.address without having to build and sign each raw transaction. When making use of this signing middleware, when sending dynamic fee transactions (recommended over legacy transactions), the transaction ``type`` of ``2`` (or ``'0x2'``) is necessary. This is because transaction signing is validated based on the transaction ``type`` parameter. This value defaults to ``'0x2'`` when ``maxFeePerGas`` and / or ``maxPriorityFeePerGas`` are present as parameters in the transaction as these params imply a dynamic fee transaction. Since these values effectively replace the legacy ``gasPrice`` value, do not set a ``gasPrice`` for dynamic fee transactions. Doing so will lead to validation issues. .. code-block:: python # dynamic fee transaction, introduced by EIP-1559: >>> dynamic_fee_transaction = { ... 'type': '0x2', # optional - defaults to '0x2' when dynamic fee transaction params are present ... 'from': acct.address, # optional if w3.eth.default_account was set with acct.address ... 'to': receiving_account_address, ... 'value': 22, ... 'maxFeePerGas': 2000000000, # required for dynamic fee transactions ... 'maxPriorityFeePerGas': 1000000000, # required for dynamic fee transactions ... } >>> w3.eth.send_transaction(dynamic_fee_transaction) A legacy transaction still works in the same way as it did before EIP-1559 was introduced: .. code-block:: python >>> legacy_transaction = { ... 'to': receiving_account_address, ... 'value': 22, ... 'gasPrice': 123456, # optional - if not provided, gas_price_strategy (if exists) or eth_gasPrice is used ... } >>> w3.eth.send_transaction(legacy_transaction) Creating Custom Middleware -------------------------- To write your own middleware, create a class and extend from the base ``Web3Middleware`` class, then override only the parts of the middleware that make sense for your use case. .. note:: The Middleware API borrows from the Django middleware API introduced in version 1.10.0. If all you need is to modify the params before a request is made, you can override the ``request_processor`` method, make the necessary tweaks to the params, and pass the arguments to the next element in the middleware stack. Need to do some processing on the response? Override the ``response_processor`` method and return the modified response. The pattern: .. code-block:: python from web3.middleware import Web3Middleware class CustomMiddleware(Web3Middleware): def request_processor(self, method, params): # Pre-request processing goes here before passing to the next middleware. return (method, params) def response_processor(self, method, response): # Response processing goes here before passing to the next middleware. return response # If your provider is asynchronous, override the async methods instead: async def async_request_processor(self, method, params): # Pre-request processing goes here before passing to the next middleware. return (method, params) async def async_response_processor(self, method, response): # Response processing goes here before passing to the next middleware. return response If you wish to prevent making a call under certain conditions, you can override the ``wrap_make_request`` method. This allows for defining pre-request processing, skipping or making the request under certain conditions, as well as response processing before passing it to the next middleware. .. code-block:: python from web3.middleware import Web3Middleware class CustomMiddleware(Web3Middleware): def wrap_make_request(self, make_request): def middleware(method, params): # pre-request processing goes here response = make_request(method, params) # make the request # response processing goes here return response return middleware # If your provider is asynchronous, override the async method instead: async def async_wrap_make_request(self, make_request): async def middleware(method, params): # pre-request processing goes here response = await make_request(method, params) # response processing goes here return response return middleware Custom middleware can be added to the stack via the class itself, using the :ref:`middleware_stack_api`. The ``name`` kwarg is optional. For example: .. code-block:: python from web3 import Web3 from my_module import ( CustomMiddleware, ) w3 = Web3(HTTPProvider(endpoint_uri="...")) # add the middleware to the stack as the class w3.middleware_onion.add(CustomMiddleware, name="custom_middleware") --- ### Migration Migration Guide =============== .. _migrating_v6_to_v7: Migrating from v6 to v7 ----------------------- web3.py follows `Semantic Versioning `_, which means that version 7 introduced backwards-incompatible changes. If you're upgrading from web3.py ``v6`` or earlier, you can expect to need to make some changes. Refer to this guide for a summary of breaking changes when updating from ``v6`` to ``v7``. If you are more than one major version behind, you should also review the migration guides for the versions in between. Provider Updates ~~~~~~~~~~~~~~~~ WebSocketProvider ````````````````` ``WebsocketProviderV2``, introduced in web3.py ``v6``, has taken priority over the legacy ``WebsocketProvider``. The ``LegacyWebSocketProvider`` has been deprecated in ``v7`` and is slated for removal in the next major version of the library. In summary: - ``WebsocketProvider`` -> ``LegacyWebSocketProvider`` (and deprecated) - ``WebsocketProviderV2`` -> ``WebSocketProvider`` If migrating from ``WebSocketProviderV2`` to ``WebSocketProvider``, you can expect the following changes: - Instantiation no longer requires the ``persistent_websocket`` method: .. code-block:: python # WebsocketsProviderV2: AsyncWeb3.persistent_websocket(WebsocketProviderV2('...')) # WebSocketProvider: AsyncWeb3(WebSocketProvider('...')) - Handling incoming subscription messages now occurs under a more flexible namespace: ``socket``. The ``AsyncIPCProvider`` uses the same API to listen for messages via an IPC socket. .. code-block:: python # WebsocketsProviderV2: async for message in w3.ws.process_subscriptions(): ... # WebSocketProvider: async for message in w3.socket.process_subscriptions(): ... AsyncIPCProvider (non-breaking feature) ``````````````````````````````````````` An asynchronous IPC provider, ``AsyncIPCProvider``, is newly available in ``v7``. This provider makes use of some of the same internals that the new ``WebSocketProvider`` introduced, allowing it to also support ``eth_subscription``. EthereumTesterProvider ````````````````````` /* Detailed source-code truncated for AI context efficiency. */ `````````````` ``abi_middleware`` is no longer necessary and has been removed. All of the functionality of the ``abi_middleware`` was already handled by web3.py's ABI formatters. For additional context: a bug in the ENS name-to-address middleware would override the formatters. Fixing this bug has removed the need for the ``abi_middleware``. Caching Middleware `````````````````` The following middleware have been removed: - ``simple_cache_middleware`` - ``latest_block_based_cache_middleware`` - ``time_based_cache_middleware`` All caching middleware has been removed in favor of a decorator/wrapper around the ``make_request`` methods of providers with configuration options on the provider class. The configuration options are outlined in the documentation in the :ref:`request_caching` section. If desired, the previous caching middleware can be re-created using the new class-based middleware model overriding the ``wrap_make_request`` (or ``async_wrap_make_request``) method in the middleware class. Result Generating Middleware ```````````````````````````` The following middleware have been removed: - ``fixture_middleware`` - ``result_generator_middleware`` The ``fixture_middleware`` and ``result_generator_middleware`` which were used for testing/mocking purposes have been removed. These have been replaced internally by the ``RequestMocker`` class, utilized for testing via a ``request_mocker`` pytest fixture. HTTP Retry Request Middleware ````````````````````````````` The ``http_retry_request_middleware`` has been removed in favor of a configuration option on the ``HTTPProvider`` and ``AsyncHTTPProvider`` classes. The configuration options are outlined in the documentation in the :ref:`http_retry_requests` section. Normalize Request Parameters Middleware ``````````````````````````````````````` The ``normalize_request_parameters`` middleware was not used anywhere internally and has been removed. Remaining camelCase -> snake_case Updates ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The following arguments have been renamed across the library from camelCase to snake_case in all methods where they are passed in as a kwarg. - ``fromBlock`` -> ``from_block`` - ``toBlock`` -> ``to_block`` - ``blockHash`` -> ``block_hash`` Note that if a dictionary is used instead, say to a call such as `eth_getLogs`, the keys in the dictionary should be camelCase. This is because the dictionary is passed directly to the JSON-RPC request, where the keys are expected to be in camelCase. Changes to Exception Handling ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All Python standard library exceptions that were raised from within web3.py have been replaced with custom ``Web3Exception`` classes. This change allows for better control over exception handling, being able to distinguish between exceptions raised by web3.py and those raised from elsewhere in a codebase. The following exceptions have been replaced: - ``AssertionError`` -> ``Web3AssertionError`` - ``ValueError`` -> ``Web3ValueError`` - ``TypeError`` -> ``Web3TypeError`` - ``AttributeError`` -> ``Web3AttributeError`` A new ``MethodNotSupported`` exception is now raised when a method is not supported by web3.py. This allows a user to distinguish between when a method is not available on the current provider, ``MethodUnavailable``, and when a method is not supported by web3.py under certain conditions, ``MethodNotSupported``. A ``MismatchedABI`` exception is now raised instead of a ``Web3ValidationError`` in cases where an ABI is not compatible with the data being passed to it. This change allows for more specific error handling when using certain ABI types. JSON-RPC Error Handling ``````````````````````` Rather than a ``ValueError`` being replaced with a ``Web3ValueError`` when a JSON-RPC response comes back with an ``error`` object, a new ``Web3RPCError`` exception is now raised to provide more distinction for JSON-RPC error responses. Some previously existing exceptions now extend from this class since they too are related to JSON-RPC errors: - ``MethodUnavailable`` - ``BlockNotFound`` - ``TransactionNotFound`` - ``TransactionIndexingInProgress`` End of Support and Feature Removals ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Python 3.7 Support Dropped `````````````````````````` Python 3.7 support has been dropped in favor of Python 3.8+. Python 3.7 is no longer supported by the Python core team, and we want to focus our efforts on supporting the latest versions of Python. EthPM Module Removed ```````````````````` The EthPM module has been removed from the library. It was not widely used and has not been functional since around October 2022. It was deprecated in ``v6`` and has been completely removed in ``v7``. Types in the `eth_typing.ethpm `_ module have been deprecated and will be removed from ``eth-typing`` in the next major release. Geth Miner Namespace Removed ```````````````````````````` The ``geth.miner`` namespace, deprecated in ``v6``, has been removed in ``v7``. The ``miner`` namespace was used for managing the concept of a miner in geth. This is no longer a feature in geth and is planned for complete removal in the future, with Ethereum having transitioned to proof-of-stake. Geth Personal Namespace Removed ``````````````````````````````` The ``geth.personal`` namespace, deprecated in ``v6``, has been removed in ``v7``. The ``personal`` namespace was used for managing accounts and keys and was deprecated in geth in ``v1.11.0``. Geth has moved to using ``clef`` for account and key management. ABI Types Removed ``````````````` /* Detailed source-code truncated for AI context efficiency. */ ```````````````````````````````````` In v5, some web3.py exceptions inherited from ``AttributeError``, namely: - ``NoABIFunctionsFound`` - ``NoABIFound`` - ``NoABIEventsFound`` Others inherited from ``ValueError``, namely: - ``InvalidAddress`` - ``NameNotFound`` - ``LogTopicError`` - ``InvalidEventABI`` Now web3.py exceptions inherit from the same base ``Web3Exception``. As such, any code that was expecting a ``ValueError`` or an ``AttributeError`` from web3.py must update to expecting one of the exceptions listed above, or ``Web3Exception``. Similarly, exceptions raised in the EthPM and ENS modules inherit from the base ``EthPMException`` and ``ENSException``, respectively. ValidationError ``````````````` The Python dev tooling ecosystem is moving towards standardizing ``ValidationError``, so users know that they're catching the correct ``ValidationError``. The base ``ValidationError`` is imported from ``eth_utils``. However, we also wanted to empower users to catch all errors emitted by a particular module. So we now have a ``Web3ValidationError``, ``EthPMValidationError``, and an ``ENSValidationError`` that all inherit from the generic ``eth_utils.exceptions.ValidationError``. Web3 class split into Web3 and AsyncWeb3 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The `Web3` class previously contained both sync and async methods. We've separated `Web3` and `AsyncWeb3` functionality to tighten up typing. For example: .. code-block:: python from web3 import Web3, AsyncWeb3 w3 = Web3(Web3.HTTPProvider()) async_w3 = AsyncWeb3(AsyncWeb3.AsyncHTTPProvider()) `dict` to `AttributeDict` conversion moved to middleware ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `Eth` module data returned as key-value pairs was previously automatically converted to an `AttributeDict` by result formatters, which could cause problems with typing. This conversion has been moved to a default `attrdict_middleware` where it can be easily removed if necessary. See the `Eth module `_ docs for more detail. Other Misc Changes ~~~~~~~~~~~~~~~~~~ - ``InfuraKeyNotFound`` exception has been changed to ``InfuraProjectIdNotFound`` - ``SolidityError`` has been removed in favor of ``ContractLogicError`` - When a method is unavailable from a node provider (i.e. a response error code of -32601 is returned), a ``MethodUnavailable`` error is now raised instead of ``ValueError`` - Logs' `data` field was previously formatted with `to_ascii_if_bytes`, now formatted to `HexBytes` - Receipts' `type` field was previously not formatted, now formatted with `to_integer_if_hex` Removals ~~~~~~~~ - Removed unused IBAN module - Removed ``WEB3_INFURA_API_KEY`` environment variable in favor of ``WEB3_INFURA_PROJECT_ID`` - Removed Kovan auto provider - Removed deprecated ``sha3`` and ``soliditySha3`` methods in favor of ``keccak`` and ``solidityKeccak`` - Remove Parity Module and References Other notable changes ~~~~~~~~~~~~~~~~~~~~~ - The ``ipfshttpclient`` library is now opt-in via a web3 install extra. This only affects the ethpm ipfs backends, which rely on the library. .. _migrating_v4_to_v5: Migrating from v4 to v5 ----------------------- Web3.py follows `Semantic Versioning `_, which means that version 5 introduced backwards-incompatible changes. If your project depends on Web3.py v4, then you'll probably need to make some changes. Here are the most common required updates: Python 3.5 no longer supported ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You will need to upgrade to either Python 3.6 or 3.7 ``eth-abi`` v1 no longer supported ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You will need to upgrade the ``eth-abi`` dependency to v2 Changes to base API ~~~~~~~~~~~~~~~~~~~ JSON-RPC Updates ```````````````` In v4, JSON-RPC calls that looked up transactions or blocks and didn't find them, returned ``None``. Now if a transaction or block is not found, a ``BlockNotFound`` or a ``TransactionNotFound`` error will be thrown as appropriate. This applies to the following web3 methods: - :meth:`~web3.eth.Eth.getTransaction` will throw a ``TransactionNotFound`` error - :meth:`~web3.eth.Eth.getTransactionReceipt` will throw a ``TransactionNotFound`` error - :meth:`~web3.eth.Eth.getTransactionByBlock` will throw a ``TransactionNotFound`` error - :meth:`~web3.eth.Eth.getTransactionCount` will throw a ``BlockNotFound`` error - :meth:`~web3.eth.Eth.getBlock` will throw a ``BlockNotFound`` error - :meth:`~web3.eth.Eth.getUncleCount` will throw a ``BlockNotFound`` error - :meth:`~web3.eth.Eth.getUncleByBlock` will throw a ``BlockNotFound`` error Removed Methods ``````````````` - ``contract.buildTransaction`` was removed for ``contract.functions.buildTransaction.`` - ``contract.deploy`` was removed for ``contract.constructor.transact`` - ``contract.estimateGas`` was removed for ``contract.functions..estimateGas`` - ``contract.call`` was removed for ``contract...call`` - ``contract.transact`` was removed for ``contract...transact`` - ``contract.eventFilter`` was removed for ``contract.events..createFilter`` - ``middleware_stack`` was renamed to :meth:`~Web3.middleware_onion` - ``web3.miner.hashrate`` was a duplicate of :meth:`~web3.eth.Eth.hashrate` and was removed. - ``web3.version.network`` was a duplicate of :meth:`~web3.net.Net.version` and was removed. - ``web3.providers.tester.EthereumTesterProvider`` and ``web3.providers.tester.TestRPCProvider`` have been removed for :meth:`~web3.providers.eth_tester.EthereumTesterProvider` - ``web3.eth.enableUnauditedFeatures`` was removed - ``web3.txpool`` was moved to :meth:`~web3.geth.txpool` - ``web3.version.node`` was removed for ``web3.clientVersion`` - ``web3.version.ethereum`` was removed for :meth:`~web3.eth.Eth.protocolVersion` - Relocated personal RPC endpoints to reflect Parity and Geth implementations: - ``web3.personal.listAccounts`` was removed for :meth:`~web3.geth.personal.listAccounts` or :meth:`~web3.parity.personal.listAccounts` - ``web3.personal.importRawKey`` was removed for :meth:`~web3.geth.personal.importRawKey` or :meth:`~web3.parity.personal.importRawKey` - ``web3.personal.newAccount`` was removed for :meth:`~web3.geth.personal.newAccount` or :meth:`~web3.parity.personal.newAccount` - ``web3.personal.lockAccount`` was removed for :meth:`~web3.geth.personal.lockAccount` - ``web3.personal.unlockAccount`` was removed for :meth:`~web3.geth.personal.unlockAccount` or :meth:`~web3.parity.personal.unlockAccount` - ``web3.personal.sendTransaction`` was removed for :meth:`~web3.geth.personal.sendTransaction` or :meth:`~web3.parity.personal.sendTransaction` - Relocated ``web3.admin`` module to ``web3.geth`` namespace - Relocated ``web3.miner`` module to ``web3.geth`` namespace Deprecated Methods `````````````````` Expect the following methods to be removed in v6: - ``web3.sha3`` was deprecated for :meth:`~Web3.keccak` - ``web3.soliditySha3`` was deprecated for :meth:`~Web3.solidityKeccak` - :meth:`~web3.net.Net.chainId` was deprecated for :meth:`~web3.eth.Eth.chainId`. Follow issue `#1293 `_ for details - ``web3.eth.getCompilers()`` was deprecated and will not be replaced - :meth:`~web3.eth.getTransactionFromBlock()` was deprecated for :meth:`~Web3.getTransactionByBlock` Deprecated ConciseContract and ImplicitContract ````````````````````````````````````````````` /* Detailed source-code truncated for AI context efficiency. */ ```````````````` In v4 ``Web3.toDecimal()`` is renamed: :meth:`~Web3.toInt` for improved clarity. It does not return a :class:`decimal.Decimal`, it returns an :class:`int`. Removed Methods ``````````````` - ``Web3.toUtf8`` was removed for :meth:`~Web3.toText`. - ``Web3.fromUtf8`` was removed for :meth:`~Web3.toHex`. - ``Web3.toAscii`` was removed for :meth:`~Web3.toBytes`. - ``Web3.fromAscii`` was removed for :meth:`~Web3.toHex`. - ``Web3.fromDecimal`` was removed for :meth:`~Web3.toHex`. Provider Access ~~~~~~~~~~~~~~~~~ In v4, ``w3.currentProvider`` was removed, in favor of ``w3.providers``. Disambiguating String Inputs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ There are a number of places where an arbitrary string input might be either a byte-string that has been hex-encoded, or unicode characters in text. These are named ``hexstr`` and ``text`` in Web3.py. You specify which kind of :class:`str` you have by using the appropriate keyword argument. See examples in :ref:`overview_type_conversions`. In v3, some methods accepted a :class:`str` as the first positional argument. In v4, you must pass strings as one of ``hexstr`` or ``text`` keyword arguments. Notable methods that no longer accept ambiguous strings: - :meth:`~Web3.sha3` - :meth:`~Web3.toBytes` Contracts ~~~~~~~~~ - When a contract returns the ABI type ``string``, Web3.py v4 now returns a :class:`str` value by decoding the underlying bytes using UTF-8. - When a contract returns the ABI type ``bytes`` (of any length), Web3.py v4 now returns a :class:`bytes` value Personal API ~~~~~~~~~~~~ ``w3.personal.signAndSendTransaction`` is no longer available. Use :meth:`w3.personal.sendTransaction() ` instead. --- ### Overview .. _overview: Overview ======== The purpose of this page is to give you a sense of everything web3.py can do and to serve as a quick reference guide. You'll find a summary of each feature with links to learn more. Configuration ------------- After installing web3.py (via ``pip install web3``), you'll need to configure a provider endpoint and any middleware you want to use beyond the defaults. Providers ~~~~~~~~~ :doc:`providers` are how web3.py connects to a blockchain. The library comes with the following built-in providers: - :class:`~web3.providers.rpc.HTTPProvider` for connecting to http and https based JSON-RPC servers. - :class:`~web3.providers.ipc.IPCProvider` for connecting to ipc socket based JSON-RPC servers. - :class:`~web3.providers.async_rpc.AsyncHTTPProvider` for connecting to http and https based JSON-RPC servers asynchronously. - :class:`~web3.providers.persistent.AsyncIPCProvider` for connecting to ipc socket based JSON-RPC servers asynchronously via a persistent connection. - :class:`~web3.providers.persistent.WebSocketProvider` for connecting to websocket based JSON-RPC servers asynchronously via a persistent connection. Examples ```````` .. code-block:: python >>> from web3 import Web3, AsyncWeb3 # IPCProvider: >>> w3 = Web3(Web3.IPCProvider('./path/to/filename.ipc')) >>> w3.is_connected() True # HTTPProvider: >>> w3 = Web3(Web3.HTTPProvider('http://127.0.0.1:8545')) >>> w3.is_connected() True # AsyncHTTPProvider: >>> w3 = AsyncWeb3(AsyncWeb3.AsyncHTTPProvider('http://127.0.0.1:8545')) >>> await w3.is_connected() True # -- Persistent Connection Providers -- # # WebSocketProvider: >>> w3 = await AsyncWeb3(AsyncWeb3.WebSocketProvider('ws://127.0.0.1:8546')) >>> await w3.is_connected() True # AsyncIPCProvider >>> w3 = await AsyncWeb3(AsyncWeb3.AsyncIPCProvider('./path/to/filename.ipc')) >>> await w3.is_connected() True For more context, see the :doc:`providers` documentation. Middleware ~~~~~~~~~~ Your web3.py instance may be further configured via :doc:`middleware`. web3.py middleware is described using an onion metaphor, where each layer of middleware may affect both the incoming request and outgoing response from your provider. The documentation includes a :ref:`visualization ` of this idea. Several middleware are :ref:`included by default `. You may add to (:meth:`add `, :meth:`inject `, :meth:`replace `) or disable (:meth:`remove `, :meth:`clear `) any of these middleware. Accounts and Private Keys ------------------------- Private keys are required to approve any transaction made on your behalf. The manner in which your key is secured will determine how you create and send transactions in web3.py. A local node, like `Geth `_, may manage your keys for you. You can reference those keys using the :attr:`web3.eth.accounts ` property. A hosted node, like `Infura `_, will have no knowledge of your keys. In this case, you'll need to have your private key available locally for signing transactions. Full documentation on the distinction between keys can be found :ref:`here `. The separate guide to :doc:`transactions` may also help clarify how to manage keys. Base API -------- The :ref:`Web3 ` class includes a number of convenient utility functions: Encoding and Decoding Helpers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - :meth:`Web3.is_encodable() ` - :meth:`Web3.to_bytes() ` - :meth:`Web3.to_hex() ` - :meth:`Web3.to_int() ` - :meth:`Web3.to_json() ` - :meth:`Web3.to_text() ` Address Helpers ~~~~~~~~~~~~~~~ - :meth:`Web3.is_address() ` - :meth:`Web3.is_checksum_address() ` - :meth:`Web3.to_checksum_address() ` Currency Conversions ~~~~~~~~~~~~~~~~~~~~ - :meth:`Web3.from_wei() ` - :meth:`Web3.to_wei() ` Cryptographic Hashing ~~~~~~~~~~~~~~~~~~~~~ - :meth:`Web3.keccak() ` - :meth:`Web3.solidity_keccak() ` web3.eth API ------------ The most commonly used APIs for interacting with Ethereum can be found under the :ref:`web3-eth` namespace. Fetching Data ~~~~~~~~~~~~~ Viewing account balances (:meth:`get_balance `), transactions (:meth:`get_transaction `), and block data (:meth:`get_block `) are some of the most common starting points in web3.py. API ``` - :meth:`web3.eth.get_balance() ` - :meth:`web3.eth.get_block() ` - :meth:`web3.eth.get_block_transaction_count() ` - :meth:`web3.eth.get_code() ` - :meth:`web3.eth.get_proof() ` - :meth:`web3.eth.get_storage_at() ` - :meth:`web3.eth.get_transaction() ` - :meth:`web3.eth.get_transaction_by_block() ` - :meth:`web3.eth.get_transaction_count() ` - :meth:`web3.eth.get_uncle_by_block() ` - :meth:`web3.eth.get_uncle_count() ` Sending Transactions ~~~~~~~~~~~~~~~~~~~~ The most common use cases will be satisfied with :meth:`send_transaction ` or the combination of :meth:`sign_transaction ` and :meth:`send_raw_transaction `. For more context, see the full guide to :doc:`transactions`. .. note:: If interacting with a smart contract, a dedicated API exists. See the next section, :ref:`Contracts `. API ``` - :meth:`web3.eth.send_transaction() ` - :meth:`web3.eth.sign_transaction() ` - :meth:`web3.eth.send_raw_transaction() ` - :meth:`web3.eth.replace_transaction() ` - :meth:`web3.eth.modify_transaction() ` - :meth:`web3.eth.wait_for_transaction_receipt() ` - :meth:`web3.eth.get_transaction_receipt() ` - :meth:`web3.eth.sign() ` - :meth:`web3.eth.sign_typed_data() ` - :meth:`web3.eth.estimate_gas() ` - :meth:`web3.eth.generate_gas_price() ` - :meth:`web3.eth.set_gas_price_strategy() ` .. _overview_contracts: Contracts --------- web3.py can help you deploy, read from, or execute functions on a deployed contract. Deployment requires that the contract already be compiled, with its bytecode and ABI available. This compilation step can be done within `Remix `_ or one of the many contract development frameworks, such as `Ape `_. Once the contract object is instantiated, calling ``transact`` on the :meth:`constructor ` method will deploy an instance of the contract: .. code-block:: python >>> ExampleContract = w3.eth.contract(abi=abi, bytecode=bytecode) >>> tx_hash = ExampleContract.constructor().transact() >>> tx_receipt = w3.eth.wait_for_transaction_receipt(tx_hash) >>> tx_receipt.contractAddress '0x8a22225eD7eD460D7ee3842bce2402B9deaD23D3' Once a deployed contract is loaded into a Contract object, the functions of that contract are available on the ``functions`` namespace: .. code-block:: python >>> deployed_contract = w3.eth.contract(address=tx_receipt.contractAddress, abi=abi) >>> deployed_contract.functions.myFunction(42).transact() If you want to read data from a contract (or see the result of transaction locally, without executing it on the network), you can use the :meth:`ContractFunction.call ` method, or the more concise :attr:`ContractCaller ` syntax: .. code-block:: python # Using ContractFunction.call >>> deployed_contract.functions.getMyValue().call() 42 # Using ContractCaller >>> deployed_contract.caller().getMyValue() 42 For more, see the full :ref:`Contracts` documentation. API ~~~ - :meth:`web3.eth.contract() ` - :attr:`Contract.address ` - :attr:`Contract.abi ` - :attr:`Contract.bytecode ` - :attr:`Contract.bytecode_runtime ` - :attr:`Contract.functions ` - :attr:`Contract.events ` - :attr:`Contract.fallback ` - :meth:`Contract.constructor() ` - :meth:`Contract.encode_abi() ` - :attr:`web3.contract.ContractFunction ` - :attr:`web3.contract.ContractEvents ` Events, Logs, and Filters ------------------------- If you want to react to new blocks being mined or specific events being emitted by a contract, you can leverage ``get_logs``, subscriptions, or filters. See the :doc:`filters` guide for more information. API ~~~ - :meth:`web3.eth.subscribe() ` - :meth:`web3.eth.filter() ` - :meth:`web3.eth.get_filter_changes() ` - :meth:`web3.eth.get_filter_logs() ` - :meth:`web3.eth.uninstall_filter() ` - :meth:`web3.eth.get_logs() ` - :meth:`Contract.events.your_event_name.create_filter() ` - :meth:`Contract.events.your_event_name.build_filter() ` - :meth:`Filter.get_new_entries() ` - :meth:`Filter.get_all_entries() ` - :meth:`Filter.format_entry() ` - :meth:`Filter.is_valid_entry() ` Net API ------- Some basic network properties are available on the ``web3.net`` object: - :attr:`web3.net.listening` - :attr:`web3.net.peer_count` - :attr:`web3.net.version` ENS --- `Ethereum Name Service (ENS) `_ provides the infrastructure for human-readable addresses. If an address is registered with the ENS registry, the domain name can be used in place of the address itself. For example, the registered domain name ``ethereum.eth`` will resolve to the address ``0xde0B295669a9FD93d5F28D9Ec85E40f4cb697BAe``. web3.py has support for ENS, documented :ref:`here `. --- ### Providers .. _providers: Providers ========= Using Ethereum requires access to an Ethereum node. If you have the means, you're encouraged to `run your own node`_. (Note that you do not need to stake ether to run a node.) If you're unable to run your own node, you can use a `remote node`_. Once you have access to a node, you can connect to it using a **provider**. Providers generate `JSON-RPC`_ requests and return the response. This is done by submitting the request to an HTTP, WebSocket, or IPC socket-based server. .. note:: web3.py supports one provider per instance. If you have an advanced use case that requires multiple providers, create and configure a new web3 instance per connection. If you are already happily connected to your Ethereum node, then you can skip the rest of this providers section. .. _run your own node: https://ethereum.org/en/developers/docs/nodes-and-clients/run-a-node/ .. _remote node: https://ethereum.org/en/developers/docs/nodes-and-clients/nodes-as-a-service/ .. _JSON-RPC: https://ethereum.org/en/developers/docs/apis/json-rpc/ .. _choosing_provider: Choosing a Provider ------------------- Most nodes have a variety of ways to connect to them. Most commonly: 1. IPC (uses local filesystem: fastest and most secure) 2. WebSocket (works remotely, faster than HTTP) 3. HTTP (more nodes support it) If you're not sure how to decide, choose this way: - If you have the option of running web3.py on the same machine as the node, choose IPC. - If you must connect to a node on a different computer, use WebSocket. - If your node does not support WebSocket, use HTTP. Once you have decided how to connect, you'll select and configure the appropriate provider class: - :class:`~web3.providers.rpc.HTTPProvider` - :class:`~web3.providers.ipc.IPCProvider` - :class:`~web3.providers.async_rpc.AsyncHTTPProvider` - :class:`~web3.providers.persistent.AsyncIPCProvider` (Persistent Connection Provider) - :class:`~web3.providers.persistent.WebSocketProvider` (Persistent Connection Provider) Each provider above links to the documentation on how to properly initialize that provider. Once you have reviewed the relevant documentation for the provider of your choice, you are ready to :ref:`get started with web3.py`. Provider via Environment Variable ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Alternatively, you can set the environment variable ``WEB3_PROVIDER_URI`` before starting your script, and web3 will look for that provider first. Valid formats for this environment variable are: - ``file:///path/to/node/rpc-json/file.ipc`` - ``http://192.168.1.2:8545`` - ``https://node.ontheweb.com`` - ``ws://127.0.0.1:8546`` Auto-initialization Provider Shortcuts -------------------------------------- Geth dev Proof of Authority ~~~~~~~~~~~~~~~~~~~~~~~~~~~ To connect to a ``geth --dev`` Proof of Authority instance with the POA middleware loaded by default: .. code-block:: python >>> from web3.auto.gethdev import w3 # confirm that the connection succeeded >>> w3.is_connected() True Or, connect to an async web3 instance: .. code-block:: python >>> from web3.auto.gethdev import async_w3 >>> await async_w3.provider.connect() # confirm that the connection succeeded >>> await async_w3.is_connected() True Built In Providers ------------------ Web3 ships with the following providers which are appropriate for connecting to local and remote JSON-RPC servers. HTTPProvider ~~~~~~~~~~~~ .. py:class:: web3.providers.rpc.HTTPProvider(endpoint_uri, request_kwargs={}, session=None, exception_retry_configuration=ExceptionRetryConfiguration()) This provider handles interactions with an HTTP or HTTPS based JSON-RPC server. * ``endpoint_uri`` should be the full URI to the RPC endpoint such as ``'https://localhost:8545'``. For RPC servers behind HTTP connections running on port 80 and HTTPS connections running on port 443 the port can be omitted from the URI. * ``request_kwargs`` should be a dictionary of keyword arguments which will be passed onto each http/https POST request made to your node. * ``session`` allows you to pass a ``requests.Session`` object initialized as desired. * ``exception_retry_configuration`` is an instance of the :class:`~web3.providers.rpc.utils.ExceptionRetryConfiguration` class which allows you to configure how the provider should handle exceptions when making certain requests. Setting this to ``None`` will disable exception retries. .. code-block:: python >>> from web3 import Web3 >>> w3 = Web3(Web3.HTTPProvider("http://127.0.0.1:8545")) Note that you should create only one HTTPProvider with the same provider URL per python process, as the HTTPProvider recycles underlying TCP/IP network connections, for better performance. Multiple HTTPProviders with different URLs will work as expected. Under the hood, the ``HTTPProvider`` uses the python requests library for making requests. If you would like to modify how requests are made, you can use the ``request_kwargs`` to do so. A common use case for this is increasing the timeout for each request. .. code-block:: python >>> from web3 import Web3 >>> w3 = Web3(Web3.HTTPProvider("http://127.0.0.1:8545", request_kwargs={'timeout': 60})) To tune the connection pool size, you can pass your own ``requests.Session``. .. code-block:: python >>> from web3 import Web3 >>> adapter = requests.adapters.HTTPAdapter(pool_connections=20, pool_maxsize=20) >>> session = requests.Session() >>> session.mount('http://', adapter) >>> session.mount('https://', adapter) >>> w3 = Web3(Web3.HTTPProvider("http://127.0.0.1:8545", session=session)) IPCProvider ~~~~~~~~~~~ .. py:class:: web3.providers.ipc.IPCProvider(ipc_path=None, timeout=10) This provider handles interaction with an IPC Socket based JSON-RPC server. * ``ipc_path`` is the filesystem path to the IPC socket: .. code-block:: python >>> from web3 import Web3 >>> w3 = Web3(Web3.IPCProvider("~/Library/Ethereum/geth.ipc")) If no ``ipc_path`` is specified, it will use a default depending on your operating system. - On Linux and FreeBSD: ``~/.ethereum/geth.ipc`` - On Mac OS: ``~/Library/Ethereum/geth.ipc`` - On Windows: ``\\.\pipe\geth.ipc`` AsyncHTTPProvider ~~~~~~~~~~~~~~~~~ .. py:class:: web3.providers.rpc.AsyncHTTPProvider(endpoint_uri, request_kwargs={}, exception_retry_configuration=ExceptionRetryConfiguration()) This provider handles interactions with an HTTP or HTTPS based JSON-RPC server asynchronously. * ``endpoint_uri`` should be the full URI to the RPC endpoint such as ``'https://localhost:8545'``. For RPC servers behind HTTP connections running on port 80 and HTTPS connections running on port 443 the port can be omitted from the URI. * ``request_kwargs`` should be a dictionary of keyword arguments which will be passed onto each http/https POST request made to your node. * ``exception_retry_configuration`` is an instance of the :class:`~web3.providers.rpc.utils.ExceptionRetryConfiguration` class which allows you to configure how the provider should handle exceptions when making certain requests. Setting this to ``None`` will disable exception retries. The ``cache_async_session()`` method allows you to use your own ``aiohttp.ClientSession`` object. .. code-block:: python >>> from aiohttp import ClientSession >>> from web3 import AsyncWeb3, AsyncHTTPProvider >>> w3 = AsyncWeb3(AsyncHTTPProvider(endpoint_uri)) >>> # If you want to pass in your own session: >>> custom_session = ClientSession() >>> await w3.provider.cache_async_session(custom_session) # This method is an async method so it needs to be handled accordingly >>> # when you're finished, disconnect: >>> w3.provider.disconnect() Under the hood, the ``AsyncHTTPProvider`` uses the python `aiohttp `_ library for making requests. Persistent Connection Providers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Persistent Connection Base Class ++++++++++++++++++++++++++++++++ .. note:: This class is not meant to be used directly. If your provider class inherits from this class, look to these docs for additional configuration options. .. py:class:: web3.providers.persistent.PersistentConnectionProvider(\ request_timeout: float = 50.0, \ subscription_response_queue_size: int = 500, \ silence_listener_task_exceptions: bool = False \ max_connection_retries: int = 5, \ request_information_cache_size: int = 500, \ ) This is a base provider class, inherited by the following providers: - :class:`~web3.providers.persistent.WebSocketProvider` - :class:`~web3.providers.persistent.AsyncIPCProvider` It handles interactions with a persistent connection to a JSON-RPC server. Among its configuration, it houses all of the :class:`~web3.providers.persistent.request_processor.RequestProcessor` logic for handling the asynchronous sending and receiving of requests and responses. See the :ref:`internals__persistent_connection_providers` section for more details on the internals of persistent connection providers. * ``request_timeout`` is the timeout in seconds, used when sending data over the connection and waiting for a response to be received from the listener task. Defaults to ``50.0``. * ``subscription_response_queue_size`` is the size of the queue used to store subscription responses, defaults to ``500``. While messages are being consumed, this queue should never fill up as it is a transient queue and meant to handle asynchronous receiving and processing of responses. When in sync with the socket stream, this queue should only ever store 1 to a few messages at a time. * ``silence_listener_task_exceptions`` is a boolean that determines whether exceptions raised by the listener task are silenced. Defaults to ``False``, raising any exceptions that occur in the listener task. * ``max_connection_retries`` is the maximum number of times to retry a connection to the provider when initializing the provider. Defaults to ``5``. * ``request_information_cache_size`` specifies the size of the transient cache for storing request details, enabling the provider to process responses based on the original request information. Defaults to ``500``. AsyncIPCProvider ++++++++++++++++ .. py:class:: web3.providers.persistent.AsyncIPCProvider(ipc_path=None, max_connection_retries=5) This provider handles asynchronous, persistent interaction with an IPC Socket based JSON-RPC server. * ``ipc_path`` is the filesystem path to the IPC socket: * ``read_buffer_limit`` is the maximum size of data, in bytes, that can be read from the socket at one time. Defaults to 20MB (20 * 1024 * 1024). Raises ``ReadBufferLimitReached`` if the limit is reached, suggesting that the buffer limit be increased. This provider inherits from the :class:`~web3.providers.persistent.PersistentConnectionProvider` class. Refer to the :class:`~web3.providers.persistent.PersistentConnectionProvider` documentation for details on additional configuration options available for this provider. If no ``ipc_path`` is specified, it will use a default depending on your operating system. - On Linux and FreeBSD: ``~/.ethereum/geth.ipc`` - On Mac OS: ``~/Library/Ethereum/geth.ipc`` - On Windows: ``\\.\pipe\geth.ipc`` WebSocketProvider +++++++++++++++++ .. py:class:: web3.providers.persistent.WebSocketProvider(endpoint_uri: str, websocket_kwargs: Dict[str, Any] = {}, use_text_frames: bool = False) This provider handles interactions with an WS or WSS based JSON-RPC server. * ``endpoint_uri`` should be the full URI to the RPC endpoint such as ``'ws://localhost:8546'``. * ``websocket_kwargs`` this should be a dictionary of keyword arguments which will be passed onto the ws/wss websocket connection. * ``use_text_frames`` will ensure websocket data is sent as text frames for servers that do not support binary communication. This provider inherits from the :class:`~web3.providers.persistent.PersistentConnectionProvider` class. Refer to the :class:`~web3.providers.persistent.PersistentConnectionProvider` documentation for details on additional configuration options available for this provider. Under the hood, the ``WebSocketProvider`` uses the python websockets library for making requests. If you would like to modify how requests are made, you can use the ``websocket_kwargs`` to do so. See the `websockets connection`_ docs for available arguments. .. _subscription-examples: Using Persistent Connection Providers +++++++++++++++++++++++++++++++++++++ The ``AsyncWeb3`` class may be used as a context manager, utilizing the ``async with`` syntax, when instantiating with a :class:`~web3.providers.persistent.PersistentConnectionProvider`. This will automatically close the connection when the context manager exits and is the recommended way to initiate a persistent connection to the provider. A similar example using a ``websockets`` connection as an asynchronous context manager can be found in the `websockets connection`_ docs. .. code-block:: python >>> import asyncio >>> from web3 import AsyncWeb3 >>> from web3.providers.persistent import ( ... AsyncIPCProvider, ... WebSocketProvider, ... ) >>> LOG = True # toggle debug logging >>> if LOG: ... import logging ... # logger = logging.getLogger("web3.providers.AsyncIPCProvider") # for the AsyncIPCProvider ... logger = logging.getLogger("web3.providers.WebSocketProvider") # for the WebSocketProvider ... logger.setLevel(logging.DEBUG) ... logger.addHandler(logging.StreamHandler()) >>> async def context_manager_subscription_example(): ... # async with AsyncWeb3(AsyncIPCProvider("./path/to.filename.ipc") as w3: # for the AsyncIPCProvider ... async with AsyncWeb3(WebSocketProvider(f"ws://127.0.0.1:8546")) as w3: # for the WebSocketProvider ... # subscribe to new block headers ... subscription_id = await w3.eth.subscribe("newHeads") ... ... async for response in w3.socket.process_subscriptions(): ... print(f"{response}\n") ... # handle responses here ... ... if some_condition: ... # unsubscribe from new block headers and break out of ... # iterator ... await w3.eth.unsubscribe(subscription_id) ... break ... ... # still an open connection, make any other requests and get ... # responses via send / receive ... latest_block = await w3.eth.get_block("latest") ... print(f"Latest block: {latest_block}") ... ... # the connection closes automatically when exiting the context ... # manager (the `async with` block) >>> asyncio.run(context_manager_subscription_example()) The ``AsyncWeb3`` class may also be used as an asynchronous iterator, utilizing the ``async for`` syntax, when instantiating with a :class:`~web3.providers.persistent.PersistentConnectionProvider`. This may be used to set up an indefinite websocket connection and reconnect automatically if the connection is lost. A similar example using a ``websockets`` connection as an asynchronous iterator can be found in the `websockets connection`_ docs. .. _`websockets connection`: https://websockets.readthedocs.io/en/stable/reference/asyncio/client.html#websockets.client.connect .. code-block:: python >>> import asyncio >>> import websockets >>> from web3 import AsyncWeb3 >>> from web3.providers.persistent import ( ... AsyncIPCProvider, ... WebSocketProvider, ... ) >>> async def subscription_iterator_example(): ... # async for w3 in AsyncWeb3(AsyncIPCProvider("./path/to/filename.ipc")): # for the AsyncIPCProvider ... async for w3 in AsyncWeb3(WebSocketProvider(f"ws://127.0.0.1:8546")): # for the WebSocketProvider ... try: ... ... ... except websockets.ConnectionClosed: ... continue # run the example >>> asyncio.run(subscription_iterator_example()) Awaiting the instantiation with a :class:`~web3.providers.persistent.PersistentConnectionProvider`, or instantiating and awaiting the ``connect()`` method is also possible. Both of these examples are shown below. .. code-block:: python >>> async def await_instantiation_example(): ... # w3 = await AsyncWeb3(AsyncIPCProvider("./path/to/filename.ipc")) # for the AsyncIPCProvider ... w3 = await AsyncWeb3(WebSocketProvider(f"ws://127.0.0.1:8546")) # for the WebSocketProvider ... ... # some code here ... ... # manual cleanup ... await w3.provider.disconnect() # run the example >>> asyncio.run(await_instantiation_example()) .. code-block:: python >>> async def await_provider_connect_example(): ... # w3 = AsyncWeb3(AsyncIPCProvider("./path/to/filename.ipc")) # for the AsyncIPCProvider ... w3 = AsyncWeb3(WebSocketProvider(f"ws://127.0.0.1:8546")) # for the WebSocketProvider ... await w3.provider.connect() ... ... # some code here ... ... # manual cleanup ... await w3.provider.disconnect() # run the example >>> asyncio.run(await_provider_connect_example()) :class:`~web3.providers.persistent.PersistentConnectionProvider` classes use the :class:`~web3.providers.persistent.request_processor.RequestProcessor` class under the hood to sync up the receiving of responses and response processing for one-to-one and one-to-many request-to-response requests. Refer to the :class:`~web3.providers.persistent.request_processor.RequestProcessor` documentation for details. AsyncWeb3 with Persistent Connection Providers ++++++++++++++++++++++++++++++++++++++++++++++ When an ``AsyncWeb3`` class is connected to a :class:`~web3.providers.persistent.PersistentConnectionProvider`, some attributes and methods become available. .. py:attribute:: socket The public API for interacting with the websocket connection is available via the ``socket`` attribute of the ``Asyncweb3`` class. This attribute is an instance of the :class:`~web3.providers.persistent.persistent_connection.PersistentConnection` class and is the main interface for interacting with the socket connection. Interacting with the Persistent Connection ++++++++++++++++++++++++++++++++++++++++++ .. py:class:: web3.providers.persistent.persistent_connection.PersistentConnection This class handles interactions with a persistent socket connection. It is available via the ``socket`` attribute on the ``AsyncWeb3`` class. The ``PersistentConnection`` class has the following methods and attributes: .. py:attribute:: subscriptions This attribute returns the current active subscriptions as a dict mapping the subscription ``id`` to a dict of metadata about the subscription request. .. py:method:: process_subscriptions() This method is available for listening to websocket subscriptions indefinitely. It is an asynchronous iterator that yields strictly one-to-many (e.g. ``eth_subscription`` responses) request-to-response messages from the websocket connection. To receive responses for one-to-one request-to-response calls, use the standard API for making requests via the appropriate module (e.g. ``block_num = await w3.eth.block_number``) The responses from this method are formatted by *web3.py* formatters and run through the middleware that were present at the time of subscription. Examples on how to use this method can be seen above in the `Using Persistent Connection Providers`_ section. .. py:method:: send(method: RPCEndpoint, params: Sequence[Any]) This method is available strictly for sending raw requests to the socket, if desired. It is not recommended to use this method directly, as the responses will not be formatted by *web3.py* formatters or run through the middleware. Instead, use the methods available on the respective web3 module. For example, use ``w3.eth.get_block("latest")`` instead of ``w3.socket.send("eth_getBlockByNumber", ["latest", True])``. .. py:method:: recv() The ``recv()`` method can be used to receive the next response for a request from the socket. The response from this method is the raw response. This is not the recommended way to receive a response for a request, as it is not formatted by *web3.py* formatters or run through the middleware. Instead, use the methods available on the respective web3 module (e.g. ``block_num = await w3.eth.block_number``) for retrieving responses for one-to-one request-to-response calls. .. py:method:: make_request(method: RPCEndpoint, params: Sequence[Any]) This method is available for making requests to the socket and retrieving the response. It is not recommended to use this method directly, as the responses will not be properly formatted by *web3.py* formatters or run through the middleware. Instead, use the methods available on the respective web3 module. For example, use ``w3.eth.get_block("latest")`` instead of ``w3.socket.make_request("eth_getBlockByNumber", ["latest", True])``. AutoProvider ~~~~~~~~~~~~ :class:`~web3.providers.auto.AutoProvider` is the default used when initializing :class:`web3.Web3` without any providers. There's rarely a reason to use it explicitly. .. py:currentmodule:: web3.providers.eth_tester EthereumTesterProvider ~~~~~~~~~~~~~~~~~~~~~~ .. warning:: Experimental: This provider is experimental. There are still significant gaps in functionality. However it is being actively developed and supported. .. py:class:: EthereumTesterProvider(ethereum_tester=None, api_endpoints=None) .. py:class:: AsyncEthereumTesterProvider(ethereum_tester=None, api_endpoints=None) This provider integrates with the ``eth-tester`` library. The ``ethereum_tester`` constructor argument should be an instance of the :class:`~eth_tester.EthereumTester` or a subclass of :class:`~eth_tester.backends.base.BaseChainBackend` class provided by the ``eth-tester`` library. The ``api_endpoints`` argument should be a ``dict`` of RPC endpoints. You can see the structure and defaults `here `_. If you would like a custom ``eth-tester`` instance to test with, see the ``eth-tester`` library `documentation `_ for details. .. code-block:: python >>> from web3 import Web3, EthereumTesterProvider >>> w3 = Web3(EthereumTesterProvider()) .. NOTE:: To install the needed dependencies to use EthereumTesterProvider, you can install the pip extras package that has the correct interoperable versions of the ``eth-tester`` and ``py-evm`` dependencies needed: e.g. ``pip install "web3[tester]"`` ---