httpx Documentation

File: docs/advanced/authentication.md Copy Chapter

Authentication can either be included on a per-request basis...

>>> auth = httpx.BasicAuth(username="username", password="secret")
>>> client = httpx.Client()
>>> response = client.get("https://www.example.com/", auth=auth)

Or configured on the client instance, ensuring that all outgoing requests will include authentication credentials...

>>> auth = httpx.BasicAuth(username="username", password="secret")
>>> client = httpx.Client(auth=auth)
>>> response = client.get("https://www.example.com/")

Basic authentication Copy Chapter

HTTP basic authentication is an unencrypted authentication scheme that uses a simple encoding of the username and password in the request Authorization header. Since it is unencrypted it should typically only be used over https, although this is not strictly enforced.

>>> auth = httpx.BasicAuth(username="finley", password="secret")
>>> client = httpx.Client(auth=auth)
>>> response = client.get("https://httpbin.org/basic-auth/finley/secret")
>>> response
<Response [200 OK]>

Digest authentication Copy Chapter

HTTP digest authentication is a challenge-response authentication scheme. Unlike basic authentication it provides encryption, and can be used over unencrypted http connections. It requires an additional round-trip in order to negotiate the authentication.

>>> auth = httpx.DigestAuth(username="olivia", password="secret")
>>> client = httpx.Client(auth=auth)
>>> response = client.get("https://httpbin.org/digest-auth/auth/olivia/secret")
>>> response
<Response [200 OK]>
>>> response.history
[<Response [401 UNAUTHORIZED]>]

NetRC authentication Copy Chapter

HTTPX can be configured to use a .netrc config file for authentication.

The .netrc config file allows authentication credentials to be associated with specified hosts. When a request is made to a host that is found in the netrc file, the username and password will be included using HTTP basic authentication.

Example .netrc file:

machine example.org
login example-username
password example-password

machine python-httpx.org
login other-username
password other-password

Some examples of configuring .netrc authentication with httpx.

Use the default .netrc file in the users home directory:

>>> auth = httpx.NetRCAuth()
>>> client = httpx.Client(auth=auth)

Use an explicit path to a .netrc file:

>>> auth = httpx.NetRCAuth(file="/path/to/.netrc")
>>> client = httpx.Client(auth=auth)

Use the NETRC environment variable to configure a path to the .netrc file,
or fallback to the default.

>>> auth = httpx.NetRCAuth(file=os.environ.get("NETRC"))
>>> client = httpx.Client(auth=auth)

The NetRCAuth() class uses the netrc.netrc() function from the Python standard library. See the documentation there for more details on exceptions that may be raised if the .netrc file is not found, or cannot be parsed.

Custom authentication schemes Copy Chapter

When issuing requests or instantiating a client, the auth argument can be used to pass an authentication scheme to use. The auth argument may be one of the following...

• A two-tuple of username/password, to be used with basic authentication.
• An instance of httpx.BasicAuth(), httpx.DigestAuth(), or httpx.NetRCAuth().
• A callable, accepting a request and returning an authenticated request instance.
• An instance of subclasses of httpx.Auth.

The most involved of these is the last, which allows you to create authentication flows involving one or more requests. A subclass of httpx.Auth should implement def auth_flow(request), and yield any requests that need to be made...

class MyCustomAuth(httpx.Auth):
    def __init__(self, token):
        self.token = token

    def auth_flow(self, request):
        # Send the request, with a custom `X-Authentication` header.
        request.headers['X-Authentication'] = self.token
        yield request

If the auth flow requires more than one request, you can issue multiple yields, and obtain the response in each case...

class MyCustomAuth(httpx.Auth):
    def __init__(self, token):
        self.token = token

    def auth_flow(self, request):
      response = yield request
      if response.status_code == 401:
          # If the server issues a 401 response then resend the request,
          # with a custom `X-Authentication` header.
          request.headers['X-Authentication'] = self.token
          yield request

Custom authentication classes are designed to not perform any I/O, so that they may be used with both sync and async client instances. If you are implementing an authentication scheme that requires the request body, then you need to indicate this on the class using a requires_request_body property.

You will then be able to access request.content inside the .auth_flow() method.

class MyCustomAuth(httpx.Auth):
    requires_request_body = True

    def __init__(self, token):
        self.token = token

    def auth_flow(self, request):
      response = yield request
      if response.status_code == 401:
          # If the server issues a 401 response then resend the request,
          # with a custom `X-Authentication` header.
          request.headers['X-Authentication'] = self.sign_request(...)
          yield request

    def sign_request(self, request):
        # Create a request signature, based on `request.method`, `request.url`,
        # `request.headers`, and `request.content`.
        ...

Similarly, if you are implementing a scheme that requires access to the response body, then use the requires_response_body property. You will then be able to access response body properties and methods such as response.content, response.text, response.json(), etc.

class MyCustomAuth(httpx.Auth):
    requires_response_body = True

    def __init__(self, access_token, refresh_token, refresh_url):
        self.access_token = access_token
        self.refresh_token = refresh_token
        self.refresh_url = refresh_url

    def auth_flow(self, request):
        request.headers["X-Authentication"] = self.access_token
        response = yield request

        if response.status_code == 401:
            # If the server issues a 401 response, then issue a request to
            # refresh tokens, and resend the request.
            refresh_response = yield self.build_refresh_request()
            self.update_tokens(refresh_response)

            request.headers["X-Authentication"] = self.access_token
            yield request

    def build_refresh_request(self):
        # Return an `httpx.Request` for refreshing tokens.
        ...

    def update_tokens(self, response):
        # Update the `.access_token` and `.refresh_token` tokens
        # based on a refresh response.
        data = response.json()
        ...

If you do need to perform I/O other than HTTP requests, such as accessing a disk-based cache, or you need to use concurrency primitives, such as locks, then you should override .sync_auth_flow() and .async_auth_flow() (instead of .auth_flow()). The former will be used by httpx.Client, while the latter will be used by httpx.AsyncClient.

import asyncio
import threading
import httpx


class MyCustomAuth(httpx.Auth):
    def __init__(self):
        self._sync_lock = threading.RLock()
        self._async_lock = asyncio.Lock()

    def sync_get_token(self):
        with self._sync_lock:
            ...

    def sync_auth_flow(self, request):
        token = self.sync_get_token()
        request.headers["Authorization"] = f"Token {token}"
        yield request

    async def async_get_token(self):
        async with self._async_lock:
            ...

    async def async_auth_flow(self, request):
        token = await self.async_get_token()
        request.headers["Authorization"] = f"Token {token}"
        yield request

If you only want to support one of the two methods, then you should still override it, but raise an explicit RuntimeError.

import httpx
import sync_only_library


class MyCustomAuth(httpx.Auth):
    def sync_auth_flow(self, request):
        token = sync_only_library.get_token(...)
        request.headers["Authorization"] = f"Token {token}"
        yield request

    async def async_auth_flow(self, request):
        raise RuntimeError("Cannot use a sync authentication class with httpx.AsyncClient")

File: docs/advanced/clients.md Copy Chapter

!!! hint
If you are coming from Requests, httpx.Client() is what you can use instead of requests.Session().

Why use a Client? Copy Chapter

!!! note "TL;DR"
If you do anything more than experimentation, one-off scripts, or prototypes, then you should use a Client instance.

More efficient usage of network resources

When you make requests using the top-level API as documented in the Quickstart guide, HTTPX has to establish a new connection for every single request (connections are not reused). As the number of requests to a host increases, this quickly becomes inefficient.

On the other hand, a Client instance uses HTTP connection pooling. This means that when you make several requests to the same host, the Client will reuse the underlying TCP connection, instead of recreating one for every single request.

This can bring significant performance improvements compared to using the top-level API, including:

• Reduced latency across requests (no handshaking).
• Reduced CPU usage and round-trips.
• Reduced network congestion.

Extra features

Client instances also support features that aren't available at the top-level API, such as:

• Cookie persistence across requests.
• Applying configuration across all outgoing requests.
• Sending requests through HTTP proxies.
• Using HTTP/2.

The other sections on this page go into further detail about what you can do with a Client instance.

Usage Copy Chapter

The recommended way to use a Client is as a context manager. This will ensure that connections are properly cleaned up when leaving the with block:

with httpx.Client() as client:
    ...

Alternatively, you can explicitly close the connection pool without block-usage using .close():

client = httpx.Client()
try:
    ...
finally:
    client.close()

Making requests Copy Chapter

Once you have a Client, you can send requests using .get(), .post(), etc. For example:

>>> with httpx.Client() as client:
...     r = client.get('https://example.com')
...
>>> r
<Response [200 OK]>

These methods accept the same arguments as httpx.get(), httpx.post(), etc. This means that all features documented in the Quickstart guide are also available at the client level.

For example, to send a request with custom headers:

>>> with httpx.Client() as client:
...     headers = {'X-Custom': 'value'}
...     r = client.get('https://example.com', headers=headers)
...
>>> r.request.headers['X-Custom']
'value'

Sharing configuration across requests Copy Chapter

Clients allow you to apply configuration to all outgoing requests by passing parameters to the Client constructor.

For example, to apply a set of custom headers on every request:

>>> url = 'http://httpbin.org/headers'
>>> headers = {'user-agent': 'my-app/0.0.1'}
>>> with httpx.Client(headers=headers) as client:
...     r = client.get(url)
...
>>> r.json()['headers']['User-Agent']
'my-app/0.0.1'

Merging of configuration Copy Chapter

When a configuration option is provided at both the client-level and request-level, one of two things can happen:

• For headers, query parameters and cookies, the values are combined together. For example:

>>> headers = {'X-Auth': 'from-client'}
>>> params = {'client_id': 'client1'}
>>> with httpx.Client(headers=headers, params=params) as client:
...     headers = {'X-Custom': 'from-request'}
...     params = {'request_id': 'request1'}
...     r = client.get('https://example.com', headers=headers, params=params)
...
>>> r.request.url
URL('https://example.com?client_id=client1&request_id=request1')
>>> r.request.headers['X-Auth']
'from-client'
>>> r.request.headers['X-Custom']
'from-request'

• For all other parameters, the request-level value takes priority. For example:

>>> with httpx.Client(auth=('tom', 'mot123')) as client:
...     r = client.get('https://example.com', auth=('alice', 'ecila123'))
...
>>> _, _, auth = r.request.headers['Authorization'].partition(' ')
>>> import base64
>>> base64.b64decode(auth)
b'alice:ecila123'

If you need finer-grained control on the merging of client-level and request-level parameters, see Request instances.

Other Client-only configuration options Copy Chapter

Additionally, Client accepts some configuration options that aren't available at the request level.

For example, base_url allows you to prepend an URL to all outgoing requests:

>>> with httpx.Client(base_url='http://httpbin.org') as client:
...     r = client.get('/headers')
...
>>> r.request.url
URL('http://httpbin.org/headers')

For a list of all available client parameters, see the Client API reference.

Request instances Copy Chapter

For maximum control on what gets sent over the wire, HTTPX supports building explicit Request instances:

request = httpx.Request("GET", "https://example.com")

To dispatch a Request instance across to the network, create a Client instance and use .send():

with httpx.Client() as client:
    response = client.send(request)
    ...

If you need to mix client-level and request-level options in a way that is not supported by the default Merging of parameters, you can use .build_request() and then make arbitrary modifications to the Request instance. For example:

headers = {"X-Api-Key": "...", "X-Client-ID": "ABC123"}

with httpx.Client(headers=headers) as client:
    request = client.build_request("GET", "https://api.example.com")

    print(request.headers["X-Client-ID"])  # "ABC123"

    # Don't send the API key for this particular request.
    del request.headers["X-Api-Key"]

    response = client.send(request)
    ...

Monitoring download progress Copy Chapter

If you need to monitor download progress of large responses, you can use response streaming and inspect the response.num_bytes_downloaded property.

This interface is required for properly determining download progress, because the total number of bytes returned by response.content or response.iter_content() will not always correspond with the raw content length of the response if HTTP response compression is being used.

For example, showing a progress bar using the tqdm library while a response is being downloaded could be done like this…

import tempfile

import httpx
from tqdm import tqdm

with tempfile.NamedTemporaryFile() as download_file:
    url = "https://speed.hetzner.de/100MB.bin"
    with httpx.stream("GET", url) as response:
        total = int(response.headers["Content-Length"])

        with tqdm(total=total, unit_scale=True, unit_divisor=1024, unit="B") as progress:
            num_bytes_downloaded = response.num_bytes_downloaded
            for chunk in response.iter_bytes():
                download_file.write(chunk)
                progress.update(response.num_bytes_downloaded - num_bytes_downloaded)
                num_bytes_downloaded = response.num_bytes_downloaded

Or an alternate example, this time using the rich library…

import tempfile
import httpx
import rich.progress

with tempfile.NamedTemporaryFile() as download_file:
    url = "https://speed.hetzner.de/100MB.bin"
    with httpx.stream("GET", url) as response:
        total = int(response.headers["Content-Length"])

        with rich.progress.Progress(
            "[progress.percentage]{task.percentage:>3.0f}%",
            rich.progress.BarColumn(bar_width=None),
            rich.progress.DownloadColumn(),
            rich.progress.TransferSpeedColumn(),
        ) as progress:
            download_task = progress.add_task("Download", total=total)
            for chunk in response.iter_bytes():
                download_file.write(chunk)
                progress.update(download_task, completed=response.num_bytes_downloaded)

Monitoring upload progress Copy Chapter

If you need to monitor upload progress of large responses, you can use request content generator streaming.

For example, showing a progress bar using the tqdm library.

import io
import random

import httpx
from tqdm import tqdm


def gen():
    """
    this is a complete example with generated random bytes.
    you can replace `io.BytesIO` with real file object.
    """
    total = 32 * 1024 * 1024  # 32m
    with tqdm(ascii=True, unit_scale=True, unit='B', unit_divisor=1024, total=total) as bar:
        with io.BytesIO(random.randbytes(total)) as f:
            while data := f.read(1024):
                yield data
                bar.update(len(data))


httpx.post("https://httpbin.org/post", content=gen())

Multipart file encoding Copy Chapter

As mentioned in the quickstart
multipart file encoding is available by passing a dictionary with the
name of the payloads as keys and either tuple of elements or a file-like object or a string as values.

>>> with open('report.xls', 'rb') as report_file:
...     files = {'upload-file': ('report.xls', report_file, 'application/vnd.ms-excel')}
...     r = httpx.post("https://httpbin.org/post", files=files)
>>> print(r.text)
{
  ...
  "files": {
    "upload-file": "<... binary content ...>"
  },
  ...
}

More specifically, if a tuple is used as a value, it must have between 2 and 3 elements:

• The first element is an optional file name which can be set to None.
• The second element may be a file-like object or a string which will be automatically
  encoded in UTF-8.
• An optional third element can be used to specify the
  MIME type
  of the file being uploaded. If not specified HTTPX will attempt to guess the MIME type based
  on the file name, with unknown file extensions defaulting to "application/octet-stream".
  If the file name is explicitly set to None then HTTPX will not include a content-type
  MIME header field.

>>> files = {'upload-file': (None, 'text content', 'text/plain')}
>>> r = httpx.post("https://httpbin.org/post", files=files)
>>> print(r.text)
{
  ...
  "files": {},
  "form": {
    "upload-file": "text-content"
  },
  ...
}

!!! tip
It is safe to upload large files this way. File uploads are streaming by default, meaning that only one chunk will be loaded into memory at a time.

Non-file data fields can be included in the multipart form using by passing them to data=....

You can also send multiple files in one go with a multiple file field form.
To do that, pass a list of (field, <file>) items instead of a dictionary, allowing you to pass multiple items with the same field.
For instance this request sends 2 files, foo.png and bar.png in one request on the images form field:

>>> with open('foo.png', 'rb') as foo_file, open('bar.png', 'rb') as bar_file:
...     files = [
...         ('images', ('foo.png', foo_file, 'image/png')),
...         ('images', ('bar.png', bar_file, 'image/png')),
...     ]
...     r = httpx.post("https://httpbin.org/post", files=files)

File: docs/advanced/event-hooks.md Copy Chapter

HTTPX allows you to register "event hooks" with the client, that are called
every time a particular type of event takes place.

There are currently two event hooks:

• request - Called after a request is fully prepared, but before it is sent to the network. Passed the request instance.
• response - Called after the response has been fetched from the network, but before it is returned to the caller. Passed the response instance.

These allow you to install client-wide functionality such as logging, monitoring or tracing.

def log_request(request):
    print(f"Request event hook: {request.method} {request.url} - Waiting for response")

def log_response(response):
    request = response.request
    print(f"Response event hook: {request.method} {request.url} - Status {response.status_code}")

client = httpx.Client(event_hooks={'request': [log_request], 'response': [log_response]})

You can also use these hooks to install response processing code, such as this
example, which creates a client instance that always raises httpx.HTTPStatusError
on 4xx and 5xx responses.

def raise_on_4xx_5xx(response):
    response.raise_for_status()

client = httpx.Client(event_hooks={'response': [raise_on_4xx_5xx]})

!!! note
Response event hooks are called before determining if the response body
should be read or not.

If you need access to the response body inside an event hook, you'll
need to call `response.read()`, or for AsyncClients, `response.aread()`.

The hooks are also allowed to modify request and response objects.

def add_timestamp(request):
    request.headers['x-request-timestamp'] = datetime.now(tz=datetime.utc).isoformat()

client = httpx.Client(event_hooks={'request': [add_timestamp]})

Event hooks must always be set as a list of callables, and you may register
multiple event hooks for each type of event.

As well as being able to set event hooks on instantiating the client, there
is also an .event_hooks property, that allows you to inspect and modify
the installed hooks.

client = httpx.Client()
client.event_hooks['request'] = [log_request]
client.event_hooks['response'] = [log_response, raise_on_4xx_5xx]

!!! note
If you are using HTTPX's async support, then you need to be aware that
hooks registered with httpx.AsyncClient MUST be async functions,
rather than plain functions.

File: docs/advanced/extensions.md Copy Chapter

Extensions

Request and response extensions provide a untyped space where additional information may be added.

Extensions should be used for features that may not be available on all transports, and that do not fit neatly into the simplified request/response model that the underlying httpcore package uses as its API.

Several extensions are supported on the request:

# Request timeouts actually implemented as an extension on
# the request, ensuring that they are passed throughout the
# entire call stack.
client = httpx.Client()
response = client.get(
    "https://www.example.com",
    extensions={"timeout": {"connect": 5.0}}
)
response.request.extensions["timeout"]
{"connect": 5.0}

And on the response:

client = httpx.Client()
response = client.get("https://www.example.com")
print(response.extensions["http_version"])  # b"HTTP/1.1"
# Other server responses could have been
# b"HTTP/0.9", b"HTTP/1.0", or b"HTTP/1.1"

Request Extensions Copy Chapter

`"trace"` Copy Section

The trace extension allows a callback handler to be installed to monitor the internal
flow of events within the underlying httpcore transport.

The simplest way to explain this is with an example:

import httpx

def log(event_name, info):
    print(event_name, info)

client = httpx.Client()
response = client.get("https://www.example.com/", extensions={"trace": log})
# connection.connect_tcp.started {'host': 'www.example.com', 'port': 443, 'local_address': None, 'timeout': None}
# connection.connect_tcp.complete {'return_value': <httpcore.backends.sync.SyncStream object at 0x1093f94d0>}
# connection.start_tls.started {'ssl_context': <ssl.SSLContext object at 0x1093ee750>, 'server_hostname': b'www.example.com', 'timeout': None}
# connection.start_tls.complete {'return_value': <httpcore.backends.sync.SyncStream object at 0x1093f9450>}
# http11.send_request_headers.started {'request': <Request [b'GET']>}
# http11.send_request_headers.complete {'return_value': None}
# http11.send_request_body.started {'request': <Request [b'GET']>}
# http11.send_request_body.complete {'return_value': None}
# http11.receive_response_headers.started {'request': <Request [b'GET']>}
# http11.receive_response_headers.complete {'return_value': (b'HTTP/1.1', 200, b'OK', [(b'Age', b'553715'), (b'Cache-Control', b'max-age=604800'), (b'Content-Type', b'text/html; charset=UTF-8'), (b'Date', b'Thu, 21 Oct 2021 17:08:42 GMT'), (b'Etag', b'"3147526947+ident"'), (b'Expires', b'Thu, 28 Oct 2021 17:08:42 GMT'), (b'Last-Modified', b'Thu, 17 Oct 2019 07:18:26 GMT'), (b'Server', b'ECS (nyb/1DCD)'), (b'Vary', b'Accept-Encoding'), (b'X-Cache', b'HIT'), (b'Content-Length', b'1256')])}
# http11.receive_response_body.started {'request': <Request [b'GET']>}
# http11.receive_response_body.complete {'return_value': None}
# http11.response_closed.started {}
# http11.response_closed.complete {'return_value': None}

The event_name and info arguments here will be one of the following:

• {event_type}.{event_name}.started, <dictionary of keyword arguments>
• {event_type}.{event_name}.complete, {"return_value": <...>}
• {event_type}.{event_name}.failed, {"exception": <...>}

Note that when using async code the handler function passed to "trace" must be an async def ... function.

The following event types are currently exposed...

Establishing the connection

• "connection.connect_tcp"
• "connection.connect_unix_socket"
• "connection.start_tls"

HTTP/1.1 events

• "http11.send_request_headers"
• "http11.send_request_body"
• "http11.receive_response"
• "http11.receive_response_body"
• "http11.response_closed"

HTTP/2 events

• "http2.send_connection_init"
• "http2.send_request_headers"
• "http2.send_request_body"
• "http2.receive_response_headers"
• "http2.receive_response_body"
• "http2.response_closed"

The exact set of trace events may be subject to change across different versions of httpcore. If you need to rely on a particular set of events it is recommended that you pin installation of the package to a fixed version.

`"sni_hostname"` Copy Section

The server's hostname, which is used to confirm the hostname supplied by the SSL certificate.

If you want to connect to an explicit IP address rather than using the standard DNS hostname lookup, then you'll need to use this request extension.

For example:

# Connect to '185.199.108.153' but use 'www.encode.io' in the Host header,
# and use 'www.encode.io' when SSL verifying the server hostname.
client = httpx.Client()
headers = {"Host": "www.encode.io"}
extensions = {"sni_hostname": "www.encode.io"}
response = client.get(
    "https://185.199.108.153/path",
    headers=headers,
    extensions=extensions
)

`"timeout"` Copy Section

A dictionary of str: Optional[float] timeout values.

May include values for 'connect', 'read', 'write', or 'pool'.

For example:

# Timeout if a connection takes more than 5 seconds to established, or if
# we are blocked waiting on the connection pool for more than 10 seconds.
client = httpx.Client()
response = client.get(
    "https://www.example.com",
    extensions={"timeout": {"connect": 5.0, "pool": 10.0}}
)

This extension is how the httpx timeouts are implemented, ensuring that the timeout values are associated with the request instance and passed throughout the stack. You shouldn't typically be working with this extension directly, but use the higher level timeout API instead.

`"target"` Copy Section

The target that is used as the HTTP target instead of the URL path.

This enables support constructing requests that would otherwise be unsupported.

• URL paths with non-standard escaping applied.
• Forward proxy requests using an absolute URI.
• Tunneling proxy requests using CONNECT with hostname as the target.
• Server-wide OPTIONS * requests.

Some examples:

Using the 'target' extension to send requests without the standard path escaping rules...

# Typically a request to "https://www.example.com/test^path" would
# connect to "www.example.com" and send an HTTP/1.1 request like...
#
# GET /test%5Epath HTTP/1.1
#
# Using the target extension we can include the literal '^'...
#
# GET /test^path HTTP/1.1
#
# Note that requests must still be valid HTTP requests.
# For example including whitespace in the target will raise a `LocalProtocolError`.
extensions = {"target": b"/test^path"}
response = httpx.get("https://www.example.com", extensions=extensions)

The target extension also allows server-wide OPTIONS * requests to be constructed...

# This will send the following request...
#
# CONNECT * HTTP/1.1
extensions = {"target": b"*"}
response = httpx.request("CONNECT", "https://www.example.com", extensions=extensions)

Response Extensions Copy Chapter

`"http_version"` Copy Section

The HTTP version, as bytes. Eg. b"HTTP/1.1".

When using HTTP/1.1 the response line includes an explicit version, and the value of this key could feasibly be one of b"HTTP/0.9", b"HTTP/1.0", or b"HTTP/1.1".

When using HTTP/2 there is no further response versioning included in the protocol, and the value of this key will always be b"HTTP/2".

`"reason_phrase"` Copy Section

The reason-phrase of the HTTP response, as bytes. For example b"OK". Some servers may include a custom reason phrase, although this is not recommended.

HTTP/2 onwards does not include a reason phrase on the wire.

When no key is included, a default based on the status code may be used.

`"stream_id"` Copy Section

When HTTP/2 is being used the "stream_id" response extension can be accessed to determine the ID of the data stream that the response was sent on.

`"network_stream"` Copy Section

The "network_stream" extension allows developers to handle HTTP CONNECT and Upgrade requests, by providing an API that steps outside the standard request/response model, and can directly read or write to the network.

The interface provided by the network stream:

• read(max_bytes, timeout = None) -> bytes
• write(buffer, timeout = None)
• close()
• start_tls(ssl_context, server_hostname = None, timeout = None) -> NetworkStream
• get_extra_info(info) -> Any

This API can be used as the foundation for working with HTTP proxies, WebSocket upgrades, and other advanced use-cases.

See the network backends documentation for more information on working directly with network streams.

Extra network information

The network stream abstraction also allows access to various low-level information that may be exposed by the underlying socket:

response = httpx.get("https://www.example.com")
network_stream = response.extensions["network_stream"]

client_addr = network_stream.get_extra_info("client_addr")
server_addr = network_stream.get_extra_info("server_addr")
print("Client address", client_addr)
print("Server address", server_addr)

The socket SSL information is also available through this interface, although you need to ensure that the underlying connection is still open, in order to access it...

with httpx.stream("GET", "https://www.example.com") as response:
    network_stream = response.extensions["network_stream"]

    ssl_object = network_stream.get_extra_info("ssl_object")
    print("TLS version", ssl_object.version())

File: docs/advanced/proxies.md Copy Chapter

HTTPX supports setting up HTTP proxies via the proxy parameter to be passed on client initialization or top-level API functions like httpx.get(..., proxy=...).

HTTP Proxies Copy Chapter

To route all traffic (HTTP and HTTPS) to a proxy located at http://localhost:8030, pass the proxy URL to the client...

with httpx.Client(proxy="http://localhost:8030") as client:
    ...

For more advanced use cases, pass a mounts dict. For example, to route HTTP and HTTPS requests to 2 different proxies, respectively located at http://localhost:8030, and http://localhost:8031, pass a dict of proxy URLs:

proxy_mounts = {
    "http://": httpx.HTTPTransport(proxy="http://localhost:8030"),
    "https://": httpx.HTTPTransport(proxy="http://localhost:8031"),
}

with httpx.Client(mounts=proxy_mounts) as client:
    ...

For detailed information about proxy routing, see the Routing section.

!!! tip "Gotcha"
In most cases, the proxy URL for the https:// key should use the http:// scheme (that's not a typo!).

This is because HTTP proxying requires initiating a connection with the proxy server. While it's possible that your proxy supports doing it via HTTPS, most proxies only support doing it via HTTP.

For more information, see [FORWARD vs TUNNEL](#forward-vs-tunnel).

Authentication Copy Chapter

Proxy credentials can be passed as the userinfo section of the proxy URL. For example:

with httpx.Client(proxy="http://username:password@localhost:8030") as client:
    ...

Proxy mechanisms Copy Chapter

!!! note
This section describes advanced proxy concepts and functionality.

FORWARD vs TUNNEL Copy Section

In general, the flow for making an HTTP request through a proxy is as follows:

1. The client connects to the proxy (initial connection request).
2. The proxy transfers data to the server on your behalf.

How exactly step 2/ is performed depends on which of two proxying mechanisms is used:

• Forwarding: the proxy makes the request for you, and sends back the response it obtained from the server.
• Tunnelling: the proxy establishes a TCP connection to the server on your behalf, and the client reuses this connection to send the request and receive the response. This is known as an HTTP Tunnel. This mechanism is how you can access websites that use HTTPS from an HTTP proxy (the client "upgrades" the connection to HTTPS by performing the TLS handshake with the server over the TCP connection provided by the proxy).

Troubleshooting proxies Copy Section

If you encounter issues when setting up proxies, please refer to our Troubleshooting guide.

SOCKS Copy Chapter

In addition to HTTP proxies, httpcore also supports proxies using the SOCKS protocol.
This is an optional feature that requires an additional third-party library be installed before use.

You can install SOCKS support using pip:

$ pip install httpx[socks]

You can now configure a client to make requests via a proxy using the SOCKS protocol:

httpx.Client(proxy='socks5://user:pass@host:port')

File: docs/advanced/resource-limits.md Copy Chapter

You can control the connection pool size using the limits keyword
argument on the client. It takes instances of httpx.Limits which define:

• max_keepalive_connections, number of allowable keep-alive connections, or None to always
  allow. (Defaults 20)
• max_connections, maximum number of allowable connections, or None for no limits.
  (Default 100)
• keepalive_expiry, time limit on idle keep-alive connections in seconds, or None for no limits. (Default 5)

limits = httpx.Limits(max_keepalive_connections=5, max_connections=10)
client = httpx.Client(limits=limits)

File: docs/advanced/ssl.md Copy Chapter

When making a request over HTTPS, HTTPX needs to verify the identity of the requested host. To do this, it uses a bundle of SSL certificates (a.k.a. CA bundle) delivered by a trusted certificate authority (CA).

Enabling and disabling verification Copy Section

By default httpx will verify HTTPS connections, and raise an error for invalid SSL cases...

>>> httpx.get("https://expired.badssl.com/")
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)

You can disable SSL verification completely and allow insecure requests...

>>> httpx.get("https://expired.badssl.com/", verify=False)
<Response [200 OK]>

Configuring client instances Copy Section

If you're using a Client() instance you should pass any verify=<...> configuration when instantiating the client.

By default the certifi CA bundle is used for SSL verification.

For more complex configurations you can pass an SSL Context instance...

import certifi
import httpx
import ssl

# This SSL context is equivalent to the default `verify=True`.
ctx = ssl.create_default_context(cafile=certifi.where())
client = httpx.Client(verify=ctx)

Using the truststore package to support system certificate stores...

import ssl
import truststore
import httpx

# Use system certificate stores.
ctx = truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
client = httpx.Client(verify=ctx)

Loding an alternative certificate verification store using the standard SSL context API...

import httpx
import ssl

# Use an explicitly configured certificate store.
ctx = ssl.create_default_context(cafile="path/to/certs.pem")  # Either cafile or capath.
client = httpx.Client(verify=ctx)

Client side certificates Copy Section

Client side certificates allow a remote server to verify the client. They tend to be used within private organizations to authenticate requests to remote servers.

You can specify client-side certificates, using the .load_cert_chain() API...

ctx = ssl.create_default_context()
ctx.load_cert_chain(certfile="path/to/client.pem")  # Optionally also keyfile or password.
client = httpx.Client(verify=ctx)

Working with `SSL_CERT_FILE` and `SSL_CERT_DIR` Copy Section

httpx does respect the SSL_CERT_FILE and SSL_CERT_DIR environment variables by default. For details, refer to the section on the environment variables page.

Making HTTPS requests to a local server Copy Section

When making requests to local servers, such as a development server running on localhost, you will typically be using unencrypted HTTP connections.

If you do need to make HTTPS connections to a local server, for example to test an HTTPS-only service, you will need to create and use your own certificates. Here's one way to do it...

1. Use trustme to generate a pair of server key/cert files, and a client cert file.
2. Pass the server key/cert files when starting your local server. (This depends on the particular web server you're using. For example, Uvicorn provides the --ssl-keyfile and --ssl-certfile options.)
3. Configure httpx to use the certificates stored in client.pem.

ctx = ssl.create_default_context(cafile="client.pem")
client = httpx.Client(verify=ctx)

File: docs/advanced/text-encodings.md Copy Chapter

When accessing response.text, we need to decode the response bytes into a unicode text representation.

By default httpx will use "charset" information included in the response Content-Type header to determine how the response bytes should be decoded into text.

In cases where no charset information is included on the response, the default behaviour is to assume "utf-8" encoding, which is by far the most widely used text encoding on the internet.

Using the default encoding Copy Chapter

To understand this better let's start by looking at the default behaviour for text decoding...

import httpx
# Instantiate a client with the default configuration.
client = httpx.Client()
# Using the client...
response = client.get(...)
print(response.encoding)  # This will either print the charset given in
                          # the Content-Type charset, or else "utf-8".
print(response.text)  # The text will either be decoded with the Content-Type
                      # charset, or using "utf-8".

This is normally absolutely fine. Most servers will respond with a properly formatted Content-Type header, including a charset encoding. And in most cases where no charset encoding is included, UTF-8 is very likely to be used, since it is so widely adopted.

Using an explicit encoding Copy Chapter

In some cases we might be making requests to a site where no character set information is being set explicitly by the server, but we know what the encoding is. In this case it's best to set the default encoding explicitly on the client.

import httpx
# Instantiate a client with a Japanese character set as the default encoding.
client = httpx.Client(default_encoding="shift-jis")
# Using the client...
response = client.get(...)
print(response.encoding)  # This will either print the charset given in
                          # the Content-Type charset, or else "shift-jis".
print(response.text)  # The text will either be decoded with the Content-Type
                      # charset, or using "shift-jis".

Using auto-detection Copy Chapter

In cases where the server is not reliably including character set information, and where we don't know what encoding is being used, we can enable auto-detection to make a best-guess attempt when decoding from bytes to text.

To use auto-detection you need to set the default_encoding argument to a callable instead of a string. This callable should be a function which takes the input bytes as an argument and returns the character set to use for decoding those bytes to text.

There are two widely used Python packages which both handle this functionality:

• chardet - This is a well established package, and is a port of the auto-detection code in Mozilla.
• charset-normalizer - A newer package, motivated by chardet, with a different approach.

Let's take a look at installing autodetection using one of these packages...

$ pip install httpx
$ pip install chardet

Once chardet is installed, we can configure a client to use character-set autodetection.

import httpx
import chardet

def autodetect(content):
    return chardet.detect(content).get("encoding")

# Using a client with character-set autodetection enabled.
client = httpx.Client(default_encoding=autodetect)
response = client.get(...)
print(response.encoding)  # This will either print the charset given in
                          # the Content-Type charset, or else the auto-detected
                          # character set.
print(response.text)

File: docs/advanced/timeouts.md Copy Chapter

HTTPX is careful to enforce timeouts everywhere by default.

The default behavior is to raise a TimeoutException after 5 seconds of
network inactivity.

Setting and disabling timeouts Copy Chapter

You can set timeouts for an individual request:

# Using the top-level API:
httpx.get('http://example.com/api/v1/example', timeout=10.0)

# Using a client instance:
with httpx.Client() as client:
    client.get("http://example.com/api/v1/example", timeout=10.0)

Or disable timeouts for an individual request:

# Using the top-level API:
httpx.get('http://example.com/api/v1/example', timeout=None)

# Using a client instance:
with httpx.Client() as client:
    client.get("http://example.com/api/v1/example", timeout=None)

Setting a default timeout on a client Copy Chapter

You can set a timeout on a client instance, which results in the given
timeout being used as the default for requests made with this client:

client = httpx.Client()              # Use a default 5s timeout everywhere.
client = httpx.Client(timeout=10.0)  # Use a default 10s timeout everywhere.
client = httpx.Client(timeout=None)  # Disable all timeouts by default.

Fine tuning the configuration Copy Chapter

HTTPX also allows you to specify the timeout behavior in more fine grained detail.

There are four different types of timeouts that may occur. These are connect,
read, write, and pool timeouts.

• The connect timeout specifies the maximum amount of time to wait until
  a socket connection to the requested host is established. If HTTPX is unable to connect
  within this time frame, a ConnectTimeout exception is raised.
• The read timeout specifies the maximum duration to wait for a chunk of
  data to be received (for example, a chunk of the response body). If HTTPX is
  unable to receive data within this time frame, a ReadTimeout exception is raised.
• The write timeout specifies the maximum duration to wait for a chunk of
  data to be sent (for example, a chunk of the request body). If HTTPX is unable
  to send data within this time frame, a WriteTimeout exception is raised.
• The pool timeout specifies the maximum duration to wait for acquiring
  a connection from the connection pool. If HTTPX is unable to acquire a connection
  within this time frame, a PoolTimeout exception is raised. A related
  configuration here is the maximum number of allowable connections in the
  connection pool, which is configured by the limits argument.

You can configure the timeout behavior for any of these values...

# A client with a 60s timeout for connecting, and a 10s timeout elsewhere.
timeout = httpx.Timeout(10.0, connect=60.0)
client = httpx.Client(timeout=timeout)

response = client.get('http://example.com/')