## File: README.md

# curl_cffi

[](https://pepy.tech/projects/curl-cffi)

[](https://badge.fury.io/py/curl-cffi)
[](https://t.me/impersonate_pro)
[](https://discord.gg/kJqMHHgdn2)

[Documentation](https://curl-cffi.readthedocs.io)

Python binding for [curl-impersonate fork](https://github.com/lexiforest/curl-impersonate)
via [cffi](https://cffi.readthedocs.io/en/latest/). For commercial support, visit [impersonate.pro](https://impersonate.pro).

`curl_cffi` is the most popular Python binding for `curl`. Unlike other pure
python http clients like `httpx` or `requests`, `curl_cffi` can impersonate
browsers' TLS/JA3 and HTTP/2 fingerprints. If you are blocked by some
website for no obvious reason, you can give `curl_cffi` a try.

Python 3.10 is the minimum supported version since v0.14.

## Recent highlights

- 💨 http/3 fingerprints and UDP socks5 proxy support was added in `v0.15.0`!
- 🦞 Added `curl-cffi` CLI and skills for debugging and for claws/agents.

## Recall.ai - API for meeting recordings

[](https://www.recall.ai/?utm_source=github&utm_medium=sponsorship&utm_campaign=lexiforest-curl_cffi)

If you’re looking for a meeting recording API, consider checking out [Recall.ai](https://www.recall.ai/?utm_source=github&utm_medium=sponsorship&utm_campaign=lexiforest-curl_cffi), an API that records Zoom, Google Meet, Microsoft Teams, in-person meetings, and more.

## Sponsors

Maintenance of this project is made possible by all the [contributors](https://github.com/lexiforest/curl_cffi/graphs/contributors) and [sponsors](https://github.com/sponsors/lexiforest). If you'd like to sponsor this project and have your avatar or company logo appear below [click here](https://github.com/sponsors/lexiforest). 💖

------

### Bypass Cloudflare with API

[](https://yescaptcha.com/i/stfnIO)

Yescaptcha is a proxy service that bypasses Cloudflare and uses the API interface to
obtain verified cookies (e.g. `cf_clearance`). Click [here](https://yescaptcha.com/i/stfnIO)
to register: <https://yescaptcha.com/i/stfnIO>

------

[](https://hypersolutions.co/?utm_source=github&utm_medium=readme&utm_campaign=curl_cffi)

TLS fingerprinting alone isn't enough for modern bot protection. [Hyper Solutions](https://hypersolutions.co?utm_source=github&utm_medium=readme&utm_campaign=curl_cffi) provides the missing piece - API endpoints that generate valid antibot tokens for:

Akamai • DataDome • Kasada • Incapsula

No browser automation. Just simple API calls that return the exact cookies and headers these systems require.

🚀 [Get Your API Key](https://hypersolutions.co?utm_source=github&utm_medium=readme&utm_campaign=curl_cffi) | 📖 [Docs](https://docs.justhyped.dev) | 💬 [Discord](https://discord.gg/akamai)

------

## Impersonate Suite

`curl-cffi` is part of the impersonate suite.

- [curl-impersonate](https://github.com/lexiforest/curl-impersonate). A curl distribution that impersonates browsers.
- [curl_cffi](https://github.com/lexiforest/curl_cffi). Python binding to curl-impersonate.
- [impers](https://github.com/lexiforest/impers). Node.js binding to curl-impersonate.
- [impersonate.pro](https://impersonate.pro). Commercial support, more fingerprints and integrated solutions.

## Features

- Supports JA3/TLS and http2 fingerprints impersonation, including recent browsers and custom fingerprints.
- Much faster than requests/httpx, on par with aiohttp/pycurl, see [benchmarks](https://github.com/lexiforest/curl_cffi/tree/main/benchmark).
- Mimics the requests API, no need to learn another one.
- Pre-compiled, so you don't have to compile on your machine.
- Supports `asyncio` with proxy rotation on each request.
- Supports http 2.0, which requests does not.
- Supports http 3.0, with fingerprints and udp proxy.
- Supports websocket.
- MIT licensed.

||requests|aiohttp|httpx|pycurl|curl_cffi|
|---|---|---|---|---|---|
|http/2|❌|❌|✅|✅|✅|
|http/3|❌|❌|❌|☑️<sup>1</sup>|✅<sup>2</sup>|
|sync|✅|❌|✅|✅|✅|
|async|❌|✅|✅|❌|✅|
|websocket|❌|✅|❌|❌|✅|
|native retry|❌|❌|❌|❌|✅|
|fingerprints|❌|❌|❌|❌|✅|
|speed|🐇|🐇🐇|🐇|🐇🐇|🐇🐇|

Notes:

1. For pycurl, http/3 is usually disabled at compile time by default.
2. http/3 support since v0.11.4, http/3 proxy and fingerprints since v0.15.0.

### curl-cffi CLI(new)

Since v0.15, `curl_cffi` comes with a CLI called `curl-cffi`, you can use it for debugging
a certain url with the `--impersonate` option. It can also serve as a `web_fetch`
replacement for "claws" and "agents".

||curl|httpie|curl-cffi|
|---|---|---|---|
|http/2|✅|❌|✅|
|http/3|☑️<sup>1</sup>|❌|✅|
|human-friendly|☑️<sup>2</sup>|✅|✅|
|colorful|❌|✅|✅<sup>3</sup>|
|fingerprints|❌|❌|✅|

Notes:

1. You need an http/3 enabled curl build, it's not enabled by default, at leat on my machine.
2. As a long time command line user, I personally feel very comfortable using `curl -X POST httpbin.org`, but some users may prefer `http GET httpbin.org` syntax. If you prefer the curl syntax, you can keep using `curl-impersonate`.
3. Install `curl_cffi[cli]` for colorful CLI output. Without `rich`, the CLI uses plain text output.

## Install

    pip install curl_cffi --upgrade

This should work on Linux, macOS and Windows out of the box.

On macOS, you can also install via Homebrew:

    brew install lexiforest/tap/curl-cffi

<small>Android support, including Termux, is currently in beta, you can install the beta release for testing.
For BSD systems, we need to get libcurl-impersonate compile first, and then add support in curl_cffi.
If you are using these OSes, please lend an hand.</small>

To install beta releases:

    pip install curl_cffi --upgrade --pre

To install unstable version from GitHub:

    git clone https://github.com/lexiforest/curl_cffi/
    cd curl_cffi
    make preprocess
    pip install .

## Usage

`curl_cffi` comes with a low-level `curl` API and a high-level `requests`-like API.
`curl_cffi` also bundles with a CLI called `curl-cffi`.

### CLI

```sh
curl-cffi get tls.browserleaks.com/json

# curl-cffi can be hard to type, use an alias if you want
alias imp=curl-cffi
imp get tls.browserleaks.com/json --impersonate chrome
```

For a complete CLI guide, see [docs](https://curl-cffi.readthedocs.io).

### requests-like

```python
import curl_cffi

# Notice the impersonate parameter
r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome")

print(r.json())
# output: {..., "ja3n_hash": "aa56c057ad164ec4fdcb7a5a283be9fc", ...}
# the js3n fingerprint should be the same as target browser

# To keep using the latest browser version as `curl_cffi` updates,
# simply set impersonate="chrome" without specifying a version.
# Other similar values are: "safari" and "safari_ios"
r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome")

# Use http/3 with impersonation
r = curl_cffi.get(
    "https://fp.impersonate.pro/api/http3",
    http_version="v3",
    impersonate="chrome"
)

# To pin a specific version, use version numbers together.
r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome124")

# To impersonate other than browsers, bring your own ja3/akamai strings
# See examples directory for details.
r = curl_cffi.get("https://tls.browserleaks.com/json", ja3=..., akamai=...)

# http/socks proxies are supported
proxies = {"https": "http://localhost:3128"}
r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome", proxies=proxies)

proxies = {"https": "socks://localhost:3128"}
r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome", proxies=proxies)
```

### Sessions

```python
s = curl_cffi.Session()

# httpbin is a http test website, this endpoint makes the server set cookies
s.get("https://httpbin.org/cookies/set/foo/bar")
print(s.cookies)
# <Cookies[<Cookie foo=bar for httpbin.org />]>

# retrieve cookies again to verify
r = s.get("https://httpbin.org/cookies")
print(r.json())
# {'cookies': {'foo': 'bar'}}
```

### Supported impersonate browsers

`curl_cffi` supports the same browser versions preset as supported by our [fork](https://github.com/lexiforest/curl-impersonate) of [curl-impersonate](https://github.com/lwthiker/curl-impersonate):

The open source version of `curl_cffi` includes versions when we are adding new capabilities for impersonating.
If you see a version, e.g. `chrome135`, was skipped, it's simply because there's nothing new or we were busy at that time. 
You can simply impersonate it with your own headers and the previous browser target.

For a full list of preset fingerprints, see the [curl-impersonate docs](https://curl-impersonate.readthedocs.io/en/latest/fingerprints.html). 
We will no longer put duplicated and outdated info here.

If you don't want to look up the headers/etc by yourself, consider buying commercial support from [impersonate.pro](https://impersonate.pro).
We have comprehensive browser tls, http and JavaScript fingerprints database for almost all the browser versions on various platforms.

Since v0.15.1, you can use `curl-cffi update` to retrieve the latest fingerprints, without updating to a new version.
We offer the Safari, Chrome, Firefox updates for free and others as part of the [commercial plan](https://impersonate.pro).

The current number of fingerprints:


 


To see the current list of fingerprints on your device, use the command line:

```sh
curl-cffi list
```

To update fingerprints from impersonate.pro, use the command line:

```sh
curl-cffi update
```

If you are trying to impersonate a target other than a browser, use `ja3=...`, `akamai=...`, `extra_fp=...`, and `perk=...`
to specify your own customized fingerprints. See the [docs on impersonation](https://curl-cffi.readthedocs.io/en/latest/impersonate/_index.html) for details.


### Asyncio

```python
from curl_cffi import AsyncSession

async with AsyncSession() as s:
    r = await s.get("https://example.com")
```

More concurrency:

```python
import asyncio
from curl_cffi import AsyncSession

urls = [
    "https://google.com/",
    "https://facebook.com/",
    "https://twitter.com/",
]

async with AsyncSession() as s:
    tasks = []
    for url in urls:
        task = s.get(url)
        tasks.append(task)
    results = await asyncio.gather(*tasks)
```

For low-level APIs, Scrapy integration and other advanced topics, see the
[docs](https://curl-cffi.readthedocs.io) for more details.

### WebSockets

```python
from curl_cffi import WebSocket

def on_message(ws: WebSocket, message: str | bytes):
    print(message)

ws = WebSocket(on_message=on_message)
ws.run_forever("wss://api.gemini.com/v1/marketdata/BTCUSD")
```

### Asyncio WebSockets

```python
import asyncio
from curl_cffi import AsyncSession

async with AsyncSession() as session:
    async with session.ws_connect("wss://echo.websocket.org") as ws:
        await asyncio.gather(*[ws.send_str("Hello, World!") for _ in range(10)])
        async for message in ws:
            print(message)
```

See the WebSocket [docs](https://curl-cffi.readthedocs.io/en/latest/websockets.html) for full details and advanced options.

## Ecosystem

- Integrating with Scrapy: [divtiply/scrapy-curl-cffi](https://github.com/divtiply/scrapy-curl-cffi), [jxlil/scrapy-impersonate](https://github.com/jxlil/scrapy-impersonate) and [tieyongjie/scrapy-fingerprint](https://github.com/tieyongjie/scrapy-fingerprint).
- Integrating with [requests](https://github.com/el1s7/curl-adapter), [httpx](https://github.com/vgavro/httpx-curl-cffi) as adapter.
- Integrating with captcha resolvers: [YesCaptcha](https://yescaptcha.com/i/stfnIO).

## Acknowledgement

- Originally forked from [multippt/python_curl_cffi](https://github.com/multippt/python_curl_cffi), which is under the MIT license.
- Headers/Cookies files are copied from [httpx](https://github.com/encode/httpx/blob/master/httpx/_models.py), which is under the BSD license.
- Asyncio support is inspired by Tornado's curl http client.
- The synchronous WebSocket API is inspired by [websocket_client](https://github.com/websocket-client/websocket-client).
- The asynchronous WebSocket API is inspired by [aiohttp](https://github.com/aio-libs/aiohttp).

## Contributing

When submitting an PR, please use a different branch other than `main` and check the
"Allow edits by maintainers" box, so I can update your PR with lint or style fixes. Thanks!

### AI Policy

- Using AI is neither encouraged nor discouraged, use it by your own choice.
- The bottom line here is that every line of code should be **reviewed by human**, and should be [proven to work](https://simonwillison.net/2025/Dec/18/code-proven-to-work/).
- It's not guaranteed that AI will come up with the cleanest solution, you are responsible to guide it to the right way you know.
- Fix any lint errors, make sure your code follows the established convention in this project.
- LLM tends to generate extensive or none comments, revise the comments and make sure they are concise and helpful.
- It's absolutely **not acceptable** to generate the entire PR summary by LLM. To communicate with other human, use words from a human.
- The only acceptable exception is to fix grammar issues if you are not a native English speaker.
- The essence here is to keep [Human in the loop](https://discourse.llvm.org/t/rfc-llvm-ai-tool-policy-human-in-the-loop/89159)

You can even feed the policy above to your "copilot" to let it adjust the style for you. :P

---

## File: benchmark/README.md

Benchmark
======

benchmark between curl_cffi and other python http clients

Sync clients
------

- curl_cffi
- requests
- pycurl
- [python-tls-client](https://github.com/FlorianREGAZ/Python-Tls-Client.git)
- httpx

Async clients
------

- curl_cffi
- httpx
- aiohttp

Target
------

All the clients run with session/client enabled.

Async WebSocket
------

Two distinct benchmarks are provided to evaluate the performance of the `AsyncWebSocket` implementation under different conditions.

1. Simple Throughput Test ([`client`](ws_bench_1_client.py), [`server`](ws_bench_1_server.py))

    This is a lightweight, in-memory benchmark designed to measure the raw throughput and overhead of the WebSocket client. The server sends a repeating chunk of random bytes from memory, and the client receives it. This test is useful for quick sanity checks and detecting performance regressions under ideal, CPU-cached conditions.

2. Verified Streaming Test ([`code`](ws_bench_2.py))

    This is a rigorous, end-to-end test. It first generates a multi-gigabyte file of random data and its SHA256 hash. The benchmark then streams this file from disk over the WebSocket connection. The receiving end calculates the hash of the incoming stream and verifies it against the original, ensuring complete data integrity.

    It measures the performance of the entire system pipeline, including Disk I/O speed, CPU hashing speed, and network transfer. On many systems, it is likely to be bottlenecked by the CPU's hashing performance or the disk speed. The result of the first run should be discarded as this reflects disk read speed rather than code performance, when the file is not cached in RAM.

Prerequisites
------

- Python 3.10+
- Pip packages

```bash
pip install aiohttp curl_cffi
```

> `uvloop` is highly recommended for performance on Linux and macOS. The benchmarks will automatically fall back to the standard asyncio event loop if it is not installed or on Windows.

Setup
------

1. TLS certificate (optional)

    These benchmarks are configured to use WSS (secure WebSockets) by default on Linux and macOS. To generate a self-signed certificate:

    ```bash
    openssl req -x509 -newkey rsa:2048 -nodes -keyout localhost.key -out localhost.crt -days 365 -subj "/CN=localhost"
    ```

    > **Note**: If you are on any platform and skip certificate generation, the benchmarks will use the insecure `ws://` instead.

2. Configuration

    The benchmark parameters (total data size, chunk size) can be modified by editing the [`TestConfig`](ws_bench_utils.py#L76) class. By default, both benchmarks are configured for `10 GiB` of data transfer.

WebSocket Benchmarks
------

It is recommended to run the server and client in different terminal windows.

Benchmark 1: Simple Throughput Test
------

1. Start the Server:

    ```bash
    python ws_bench_1_server.py
    ```

2. Run the Client:

    ```bash
    python ws_bench_1_client.py
    ```

Benchmark 2: Verified Streaming Test
------

1. Generate Test File (Initial Setup):

    This command will create a large (`10 GiB`) file named `testdata.bin` and its hash:

    ```bash
    python ws_bench_2.py generate
    ```

    > Ensure you have sufficient disk space available

2. Start the Server:

    ```bash
    python ws_bench_2.py server
    ```

3. Run the Client (Choose one):

    > This benchmark requires enough available RAM **equal to the size of the random data** (e.g. `10 GiB`).

    - To test download speed (server sends, client receives):

    ```bash
    python ws_bench_2.py client --test download
    ```

    - To test upload speed (client sends, server receives):

    ```bash
    python ws_bench_2.py client --test upload
    ```

Performance Considerations
------

Benchmark results can vary significantly based on system-level factors. The following should be kept in mind:

- **Loopback Interface**: These tests run on the loopback interface (`127.0.0.1`), which does not represent real-world internet conditions (latency, packet loss, etc.).

- **CPU Affinity**: For maximum consistency, especially on multi-core or multi-CPU (NUMA) systems, you can pin the server and client processes to specific CPU cores. This avoids performance penalties from processes migrating between cores or crossing CPU socket boundaries.

    **On Linux:**
    Use `taskset` to specify a CPU core (e.g., core 0 for the server, core 1 for the client).

    ```bash
    # Terminal 1
    taskset -c 0 python ws_bench_1_server.py

    # Terminal 2
    taskset -c 1 python ws_bench_1_client.py
    ```

    **On Windows:**
    Use the `start /affinity` command. The affinity mask is a hexadecimal number (`1` for CPU 0, `2` for CPU 1, `4` for CPU 2, etc.).

    ```powershell
    # PowerShell/CMD 1
    start /affinity 1 python ws_bench_1_server.py

    # PowerShell/CMD 2
    start /affinity 2 python ws_bench_1_client.py
    ```

- **Concurrent Tests**: The [`ws_bench_1_client.py`](ws_bench_1_client.py) benchmark mode can be changed to download/upload/concurrent by changing the [`BenchmarkDirection`](ws_bench_utils.py#L100) enum. A concurrent test completes when both directions finish.

- **Queue Sizes**: Adjust the [`send_queue`](ws_bench_utils.py#L90) and [`recv_queue`](ws_bench_utils.py#L89) sizes within the [`TestConfig`](ws_bench_utils.py#L76) class to observe the impact on performance and backpressure.

---

## File: docs/advanced.rst

Advanced Topics
**************

Proxies
=======

You can use the ``proxy`` parameter:

.. code-block:: python

    import curl_cffi

    curl_cffi.get(url, proxy="http://user:pass@example.com:3128")

You can also use the ``http_proxy``, ``https_proxy``, and ``ws_proxy``, ``wss_proxy``
environment variables, respectively.

.. warning::

   For beginners, a very common mistake is to add ``https://`` prefix to the ``https`` proxy.

   For explanation of differences between ``http_proxy`` and ``https_proxy``, please see
   `#6 <https://github.com/lexiforest/curl_cffi/issues/6>`_.

For compatibility with ``requests``, we also support using dicts.

.. code-block:: python

    import curl_cffi

    proxies = {
        "http": "http://localhost:3128",
        "https": "http://localhost:3128"
    }
    curl_cffi.get(url, proxies=proxies)


.. note::

   Prefer the single `proxy` parameter, unless you do have different proxies for http and https


Response cache
==============

``Session`` can cache successful responses on disk. This is especially useful for
tests and local mocks: the first request records a real response, and later
requests with the same method, URL and request body are served from the cache.

The simplest form is to pass a number of seconds or a ``timedelta``:

.. code-block:: python

    from datetime import timedelta
    from curl_cffi import Session

    with Session(cache=timedelta(minutes=10)) as s:
        first = s.get("https://example.com/api/users")
        second = s.get("https://example.com/api/users")  # served from cache

By default, cached files are stored in a ``curl_cffi_cache`` directory under the
system temporary directory. Use ``FileCacheBackend`` when you want a stable cache
directory, for example inside a test fixture:

.. code-block:: python

    from datetime import timedelta
    from curl_cffi import FileCacheBackend, Session

    cache = FileCacheBackend(
        path="tests/.cache/http",
        expires=timedelta(hours=1),
    )

    with Session(cache=cache) as s:
        response = s.get("https://api.example.test/items")

Only successful responses are stored by default. Cached responses preserve the
response body, status code, headers, cookies and common curl metadata. On a cache
hit, response cookies are also merged into the session cookie jar, matching normal
``Session`` behavior.

Ignoring query parameters
-------------------------

You can ignore volatile query parameters when building cache keys. This is useful
for analytics, cache-busting or nonce parameters that should not create separate
cache entries.

.. code-block:: python

    from datetime import timedelta
    from curl_cffi import FileCacheBackend, Session

    cache = FileCacheBackend(
        expires=timedelta(minutes=10),
        ignored=["utm_source", "cache_buster"],
    )

    with Session(cache=cache) as s:
        s.get("https://example.com/items?item=1&utm_source=a")
        s.get("https://example.com/items?item=1&utm_source=b")  # same cache entry


Clearing cache files
--------------------

Call ``clear`` to remove cache entries managed by that backend:

.. code-block:: python

    cache.clear()

``FileCacheBackend.clear`` only removes cache-looking files from its own cache
directory.


Custom backends
---------------

To store responses somewhere other than the filesystem, subclass
``CacheBackend`` and implement the storage methods:

.. code-block:: python

    from datetime import timedelta
    from curl_cffi import CacheBackend, Session

    class DictCache(CacheBackend):
        def __init__(self):
            super().__init__(expires=timedelta(minutes=10))
            self.data = {}

        def _read_payload(self, key):
            return self.data.get(key)

        def _write_payload(self, key, payload):
            self.data[key] = payload

        def _delete_payload(self, key):
            self.data.pop(key, None)

        def clear(self):
            self.data.clear()

    with Session(cache=DictCache()) as s:
        response = s.get("https://example.com/api")


Limitations
-----------

- The built-in cache currently works with synchronous ``Session`` only.
  ``AsyncSession`` rejects the ``cache`` option because the backend API is
  blocking.
- Streamed responses and requests using ``content_callback`` are not cached.
- The default key uses request method, normalized URL and request body. Be careful
  with authenticated or cookie-specific responses; use a custom backend or a
  separate cache directory when different users may receive different content for
  the same URL.


Low-level curl API
=========

Although we provide an easy to use ``requests``-like API, sometimes, you may prefer to use the ``curl``-like API.

The curl API is very much like what you may have used -- ``pycurl``, with extra impersonation support.


.. code-block:: python

    from curl_cffi import Curl, CurlOpt
    from io import BytesIO

    buffer = BytesIO()
    c = Curl()
    c.setopt(CurlOpt.URL, b'https://tls.browserleaks.com/json')
    c.setopt(CurlOpt.WRITEDATA, buffer)

    c.impersonate("chrome124")

    c.perform()
    c.close()
    body = buffer.getvalue()
    print(body.decode())

For a complete list of options, see :doc:`api`


Using ``CURLOPT_*`` in requests API
===================================

Sometimes, you know an option from libcurl, but we haven't exposed it in the requests API.
You can simply add the ``curl_options`` dict to apply the option.

.. code-block:: python

    from curl_cffi import CurlHttpVersion, CurlOpt, requests

    response = requests.get(
        "https://example.com",
        curl_options={CurlOpt.HTTP_VERSION: CurlHttpVersion.V1_1},
    )


.. note::

   Using curl_options is preferred over using ``session.curl.setopt``, the latter may get
   overriden internally, while the former is executed after all options have been set.


Selecting http version
======================

The recommended and default http version is http/2, the present and most widely used http version
as of 2025.

According to `Wikipedia <https://en.wikipedia.org/wiki/HTTP>`_, the marketshare is:

- HTTP/1.1, 33.8%
- HTTP/2, 35.3%
- HTTP/3, 30.9%

To change http versions, use the ``http_version`` parameter.

.. code-block:: python

   import curl_cffi
   curl_cffi.get("https://cloudflare-quic.com", http_version="v3")

Common values are: ``v1``, ``v2``, ``v3`` and ``v3only``.

To get the actual used http version, you need to compare the response field with const from libcurl:

.. code-block:: python

    >>> from curl_cffi import CurlHttpVersion
    >>> r = curl_cffi.get("https://example.com", http_version="v2")
    >>> r.http_version == CurlHttpVersion.V2_0
    True


Keeping session alive in http/2
======

With http/2, you can optionally send a ping frame to keep the connection alive when not actively using it.


.. code-block:: python

   from curl_cffi import Session

   s = Session()
   s.get("https://example.com")
   s.upkeep()

``AsyncSession`` also exposes ``upkeep`` for idle connections in its pool:

.. code-block:: python

   from curl_cffi import AsyncSession

   async with AsyncSession() as s:
       await s.get("https://example.com")
       await s.upkeep()

---

## File: docs/api.rst

API References
======

curl low levl APIs
------------------

Curl
~~~~~~

.. autoclass:: curl_cffi.Curl

   .. automethod:: __init__
   .. automethod:: debug
   .. automethod:: setopt
   .. automethod:: getinfo
   .. automethod:: version
   .. automethod:: impersonate
   .. automethod:: perform
   .. automethod:: duphandle
   .. automethod:: reset
   .. automethod:: parse_cookie_headers
   .. automethod:: get_reason_phrase
   .. automethod:: parse_status_line
   .. automethod:: close
   .. automethod:: ws_recv
   .. automethod:: ws_send
   .. automethod:: ws_close

AsyncCurl
~~~~~~

.. autoclass:: curl_cffi.AsyncCurl

   .. automethod:: __init__
   .. automethod:: add_handle
   .. automethod:: remove_handle
   .. automethod:: set_result
   .. automethod:: set_exception
   .. automethod:: setopt
   .. automethod:: socket_action
   .. automethod:: process_data
   .. automethod:: close

CurlMime
~~~~~~

.. autoclass:: curl_cffi.CurlMime

   .. automethod:: __init__
   .. automethod:: addpart
   .. automethod:: from_list
   .. automethod:: attach
   .. automethod:: close

Constants
~~~~~~~~~

Enum values used by ``setopt`` and ``getinfo`` can be accessed from ``CurlOpt`` and
``CurlInfo``.

.. autoclass:: curl_cffi.CurlOpt
.. autoclass:: curl_cffi.CurlInfo
.. autoclass:: curl_cffi.CurlMOpt
.. autoclass:: curl_cffi.CurlECode
.. autoclass:: curl_cffi.CurlHttpVersion
.. autoclass:: curl_cffi.CurlWsFlag
.. autoclass:: curl_cffi.CurlSslVersion

requests-like API
-----------------

request method
~~~~~~~~~~~~~~

``requests.get``, ``requests.post``, etc are just aliases of ``.request(METHOD, ...)``

.. autofunction:: curl_cffi.requests.request


Sessions
~~~~~~~

.. autoclass:: curl_cffi.requests.Session

   .. automethod:: __init__
   .. automethod:: request
   .. automethod:: stream
   .. automethod:: ws_connect


.. autoclass:: curl_cffi.requests.AsyncSession

   .. automethod:: __init__
   .. automethod:: request
   .. automethod:: stream
   .. automethod:: close
   .. automethod:: ws_connect

Cache
~~~~~

.. autoclass:: curl_cffi.requests.CacheBackend

   .. automethod:: __init__
   .. automethod:: should_cache_request
   .. automethod:: should_store_response
   .. automethod:: get
   .. automethod:: set
   .. automethod:: delete
   .. automethod:: _read_payload
   .. automethod:: _write_payload
   .. automethod:: _delete_payload
   .. automethod:: clear

.. autoclass:: curl_cffi.requests.FileCacheBackend

   .. automethod:: __init__
   .. automethod:: clear

Headers
~~~~~~~

.. autoclass:: curl_cffi.requests.Headers

   .. autoproperty:: encoding
   .. automethod:: raw
   .. automethod:: multi_items
   .. automethod:: get
   .. automethod:: get_list
   .. automethod:: update
   .. automethod:: __getitem__
   .. automethod:: __setitem__
   .. automethod:: __delitem__

Cookies
~~~~~~~

.. autoclass:: curl_cffi.requests.Cookies

   .. automethod:: set
   .. automethod:: get
   .. automethod:: delete
   .. automethod:: clear
   .. automethod:: update
   .. automethod:: __getitem__
   .. automethod:: __setitem__
   .. automethod:: __delitem__

Request, Response
~~~~~~

.. autoclass:: curl_cffi.requests.Request

.. autoclass:: curl_cffi.requests.Response

   .. automethod:: raise_for_status
   .. automethod:: iter_lines
   .. automethod:: iter_content
   .. automethod:: json
   .. automethod:: close
   .. automethod:: aiter_lines
   .. automethod:: aiter_content
   .. automethod:: atext
   .. automethod:: acontent
   .. automethod:: aclose

Asyncio
-------

WebSocket
---------

.. autoclass:: curl_cffi.requests.WebSocket

   .. automethod:: __init__
   .. automethod:: connect
   .. automethod:: recv
   .. automethod:: recv_str
   .. automethod:: recv_json
   .. automethod:: send
   .. automethod:: send_binary
   .. automethod:: send_bytes
   .. automethod:: send_str
   .. automethod:: send_json
   .. automethod:: ping
   .. automethod:: run_forever
   .. automethod:: close

.. autoclass:: curl_cffi.requests.AsyncWebSocket

   .. automethod:: __init__
   .. automethod:: recv
   .. automethod:: recv_str
   .. automethod:: recv_json
   .. automethod:: send
   .. automethod:: send_binary
   .. automethod:: send_bytes
   .. automethod:: send_str
   .. automethod:: send_json
   .. automethod:: ping
   .. automethod:: close

Exceptions and Warnings
-----------------------

Exceptions
~~~~~~~~~~~~~~

We try to follow the `requests` exception hirearchy, however, some are missing, while
some are added.

If an exception is marked as "not used", please catch the base exception.


.. autoclass:: curl_cffi.requests.exceptions.RequestException
.. autoclass:: curl_cffi.requests.exceptions.CookieConflict
.. autoclass:: curl_cffi.requests.exceptions.SessionClosed
.. autoclass:: curl_cffi.requests.exceptions.ImpersonateError
.. autoclass:: curl_cffi.requests.exceptions.InvalidJSONError
.. autoclass:: curl_cffi.requests.exceptions.HTTPError
.. autoclass:: curl_cffi.requests.exceptions.IncompleteRead
.. autoclass:: curl_cffi.requests.exceptions.ConnectionError
.. autoclass:: curl_cffi.requests.exceptions.DNSError
.. autoclass:: curl_cffi.requests.exceptions.ProxyError
.. autoclass:: curl_cffi.requests.exceptions.SSLError
.. autoclass:: curl_cffi.requests.exceptions.CertificateVerifyError
.. autoclass:: curl_cffi.requests.exceptions.Timeout
.. autoclass:: curl_cffi.requests.exceptions.ConnectTimeout
.. autoclass:: curl_cffi.requests.exceptions.ReadTimeout
.. autoclass:: curl_cffi.requests.exceptions.URLRequired
.. autoclass:: curl_cffi.requests.exceptions.TooManyRedirects
.. autoclass:: curl_cffi.requests.exceptions.MissingSchema
.. autoclass:: curl_cffi.requests.exceptions.InvalidSchema
.. autoclass:: curl_cffi.requests.exceptions.InvalidURL
.. autoclass:: curl_cffi.requests.exceptions.InvalidHeader
.. autoclass:: curl_cffi.requests.exceptions.InvalidProxyURL
.. autoclass:: curl_cffi.requests.exceptions.ChunkedEncodingError
.. autoclass:: curl_cffi.requests.exceptions.ContentDecodingError
.. autoclass:: curl_cffi.requests.exceptions.StreamConsumedError
.. autoclass:: curl_cffi.requests.exceptions.RetryError
.. autoclass:: curl_cffi.requests.exceptions.UnrewindableBodyError
.. autoclass:: curl_cffi.requests.exceptions.InterfaceError

Warnings
~~~~~~~~~~~~~~

.. autoclass:: curl_cffi.requests.exceptions.RequestsWarning
.. autoclass:: curl_cffi.requests.exceptions.FileModeWarning
.. autoclass:: curl_cffi.requests.exceptions.RequestsDependencyWarning

---

## File: docs/cookies.rst

Cookies
=======

Save and load cookies
------

Do not use ``get_dict`` to dump and load cookies. Cookies are more than just plain
key-value pairs.

Using pickle:

.. code-block:: python

    # example from: https://github.com/encode/httpx/issues/895
    import pickle
    # import httpx
    import curl_cffi

    def save_cookies(client):
        with open("cookies.pk", "wb") as f:
            pickle.dump(client.cookies.jar._cookies, f)

    def load_cookies():
        if not os.path.isfile("cookies.pk"):
            return None
        with open("cookies.pk", "rb") as f:
            return pickle.load(f)

    # client = httpx.Client(cookies=load_cookies())
    client = curl_cffi.Session()
    client.get("https://httpbin.org/cookies/set/foo/bar")
    save_cookies(client)

    client = curl_cffi.Session()
    client.cookies.jar._cookies.update(load_cookies())
    print(client.cookies.get("foo"))


Using mozilla cookie jar:

See: https://github.com/lexiforest/curl_cffi/issues/381

TODO: expose libcurl's native cookies.txt support.


Discard cookies when using Session
----------------------------------

You may need to discard cookies when using sessions, especially when using ``AsyncSession``.

Use the ``discard_cookies`` option.


.. code-block:: python

    s = requests.Session()

    set_url = "https://httpbin.org/cookies"

    r = s.get(set_url)
    assert r.cookies["foo"] == s.cookies["foo"]
    old_cookie = r.cookies["foo"]

    # Let's start discarding cookies
    s.discard_cookies = True
    r = s.get(set_url)
    assert r.cookies["foo"] != s.cookies["foo"]
    assert old_cookie == s.cookies["foo"]

    # The behavior can be reverted
    s.discard_cookies = False
    r = s.get(set_url)
    assert r.cookies["foo"] == s.cookies["foo"]
    old_cookie = r.cookies["foo"]

    # Also works as request parameter
    r = s.get(set_url, discard_cookies=True)
    assert r.cookies["foo"] != s.cookies["foo"]
    assert old_cookie == s.cookies["foo"]

---

## File: docs/dev.rst

Development and contributing
==============

This page documents how to compile curl-impersonate and curl-cffi from source. If binary
package is not available on your platform, you may refer to this page for some inspirations.

First, you need to check if there are libcurl-impersonate binaries for you platform. If
so, you can simply download and install them.

For now, a pre-compiled `libcurl-impersonate` is downloaded from github and built
into a bdist wheel, which is a binary package format used by PyPI. However, the
right way is to download curl and curl-impersonate sources on our side and compile
them all together.

By default, source builds download the platform-specific library into a temporary
directory. Set ``IMPERSONATE_BUILD_DIR`` to reuse an existing download or choose
where it is downloaded and unpacked. The directory must contain the library filename
configured for the current platform, such as ``libcurl-impersonate.so`` or
``libcurl-impersonate.a``::

    IMPERSONATE_BUILD_DIR=/path/to/lib pip install .

The default link type is configured per platform. Set ``IMPERSONATE_LINK_TYPE``
to ``dynamic`` or ``static`` to override it. A dynamic local build can be used
without merging its static dependency archives::

    IMPERSONATE_BUILD_DIR=/path/to/lib \
        IMPERSONATE_LINK_TYPE=dynamic pip install -e .

On macOS, the directory must contain ``libcurl-impersonate.dylib`` and any
versioned symlinks required by its install name. The directory is recorded as a
runtime search path in the extension module.

macOS

To install the local editable build:

.. code-block:: shell

    # This is for using the libcurl-impersonate built by GitHub actions

    sudo mkdir /Users/runner
    sudo chmod 777 /Users/runner

    # Dependencies

    brew install libidn2 zstd

    # Then install

    pip install -e .[test]
    pip install -e .[dev]

Contributing
------------

When opening PR, please do not use the ``main`` branch in your fork, otherwise I cannot
add my modification, such as unittests.

---

## File: docs/faq.rst

FAQ
==========================

What does the pro version offer? Is the open source project still maintained?
----------------------------------------------------------------------------

Yes, the open source project is maintained as before.

In the `pro version <https://impersonate.pro>`_, we provide:

- weekly update of targets
- profiles for mobile browsers and apps
- some private detection fields
- http/3 fingerprints and proxy support

And, a better financial situation will help the open source version better maintained.

Why does the JA3 fingerprints change for Chrome 110+ impersonation?
------

This is intended.

Chrome introduces ``ClientHello`` permutation in version 110, which means the order of
extensions will be random, thus JA3 fingerprints will be random. So, when comparing
JA3 fingerprints of ``curl_cffi`` and a browser, they may differ. However, this does not
mean that TLS fingerprints will not be a problem, ``ClientHello`` extension order is just
one factor of how servers can tell automated requests from browsers.

Roughly, this can be mitigated like:

.. code-block::

    ja3 = md5(list(extensions), ...other arguments)
    ja3n = md5(set(extensions), ...other arguments)

See more from `this article <https://www.fastly.com/blog/a-first-look-at-chromes-tls-clienthello-permutation-in-the-wild>`_
and `curl-impersonate notes <https://github.com/lwthiker/curl-impersonate/pull/148>`_.

Can I bypass Cloudflare with this project? or any other specific site.
------

Short answer is: it depends.

TLS and http2 fingerprints are just one of the many factors Cloudflare considers. Other
factors include but are not limited to: IP quality, request rate, JS fingerprints, etc.

There are different protection levels for website owners to choose. For the most basic
ones, TLS fingerprints alone maybe enough, but for higher levels, you may need to find
a better proxy IP provider and use browser automation tools like playwright.

If you are in a hurry or just want the professionals to take care of the hard parts,
you can consider the commercial solutions from our sponsors:

- `Yescaptcha <https://yescaptcha.com/i/stfnIO>`_, captcha resolver and proxy service for bypassing Cloudflare.
- `ScrapeNinja <https://scrapeninja.net/?utm_source=github&utm_medium=banner&utm_campaign=cffi>`_, Managed web scraping API.

For details, see the `Sponsor` section on front page.


I'm getting certs errors
------

The simplest way is to turn off cert verification by ``verify=False``:

.. code-block:: python

    r = curl_cffi.get("https://example.com", verify=False)


ErrCode: 77, Reason: error setting certificate verify locations
------

On Windows, if your Python environment or CA bundle path contains non-ASCII characters
(e.g. accents), libcurl may fail to open the CA file when passed as a narrow ``char*``.
``curl_cffi`` now encodes file-path options (e.g. ``CAINFO``, ``PROXY_CAINFO``,
``SSLCERT``) using the system's preferred ANSI code page on Windows to ensure correct
file access. This fixes most occurrences of error 77.

How to use with fiddler/charles to intercept content
------

Fiddler and Charles uses man-in-the-middle self-signed certs to intercept TLS traffic,
to use with them, simply set ``verify=False``.


ErrCode: 92, Reason: 'HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)'
------

This error(http/2 stream 0) has been reported many times ever since `curl_cffi` was
published, but I still can not find a reproducible way to trigger it. Given that the
majority users are behind proxies, the situation is even more difficult to deal with.

I'm even not sure it's a bug introduced in libcurl, curl-impersonate or curl_cffi, or
it's just a server error. Depending on your context, here are some general suggestions
for you:

- First, try removing the ``Content-Length`` header from you request.
- Try to see if this error was caused by proxies, if so, use better proxies.
- If it stops working after a while, maybe you're just being blocked by, such as, Akamai.
- Force http/1.1 mode. Some websites' h2 implementation is simply broken.
- See if the url works in your real browser.
- Find a stable way to reproduce it, so we can finally fix, or at least bypass it.

To force curl to use http 1.1 only.

.. code-block:: python

    import curl_cffi

    r = curl_cffi.get("https://postman-echo.com", http_version=curl_cffi.CurlHttpVersion.V1_1)

Related issues:

- `#19 <https://github.com/lexiforest/curl_cffi/issues/19>`_, 
- `#42 <https://github.com/lexiforest/curl_cffi/issues/42>`_, 
- `#79 <https://github.com/lexiforest/curl_cffi/issues/79>`_, 
- `#165 <https://github.com/lexiforest/curl_cffi/issues/165>`_, 


Packaging with PyInstaller
------

If you encountered any issue with PyInstaller, here are a list of options provided by the
community:

Add the ``--hidden-import`` option.

.. code-block::

   pyinstaller -F .\example.py --hidden-import=_cffi_backend --collect-all curl_cffi

Add other paths:

.. code-block::

   pyinstaller --noconfirm --onefile --console \
       --paths "C:/Users/Administrator/AppData/Local/Programs/Python/Python39" \
       --add-data "C:/Users/Administrator/AppData/Local/Programs/Python/Python39/Lib/site-packages/curl_cffi.libs/libcurl-cbb416caa1dd01638554eab3f38d682d.dll;." \
       --collect-data "curl_cffi" \
       "C:/Users/Administrator/Desktop/test_script.py"


See also: 

- `#5 <https://github.com/lexiforest/curl_cffi/issues/5>`_
- `#48 <https://github.com/lexiforest/curl_cffi/issues/48>`_


How to change the order of headers?
------

By default, setting ``impersonate`` parameter will bring the corresponding headers. If
you want to change the order or use your own headers, you need to turn off that and bring
your own headers. For stored fingerprint targets, you can also load and edit a
``Fingerprint`` object directly.

.. code-block::

   fingerprint = curl_cffi.get_fingerprint("edge_146_macos_26")
   fingerprint.headers["User-Agent"] = "..."
   requests.get(url, impersonate=fingerprint)


How to deal with encoding/decoding errors?
------------------------------------------

Use ``chardet`` or ``cchardet``

.. code-block::

    >>> import curl_cffi
    >>> r = curl_cffi.get("https://example.com/messy_codec.html")
    >>> import chardet
    >>> chardet.detect(r.content)
    {'encoding': 'GB2312', 'confidence': 0.99, 'language': 'Chinese'}

Or use regex or lxml to parse the meta header:

.. code-block::

    <meta http-equiv="Content-Type" content="text/html; charset=gbk" />