## File: README.md # Elasticsearch Python Client *The official Python client for Elasticsearch.* ## Features * Translating basic Python data types to and from JSON * Configurable automatic discovery of cluster nodes * Persistent connections * Load balancing (with pluggable selection strategy) across available nodes * Failed connection penalization (time based - failed connections won't be retried until a timeout is reached) * Support for TLS and HTTP authentication * Thread safety across requests * Pluggable architecture * Helper functions for idiomatically using APIs together ## Installation [Download the latest version of Elasticsearch](https://www.elastic.co/downloads/elasticsearch) or [sign-up](https://cloud.elastic.co/registration?elektra=en-ess-sign-up-page) for a free trial of Elastic Cloud. Refer to the [Installation section](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/getting-started-python.html#_installation) of the getting started documentation. ## Connecting Refer to the [Connecting section](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/getting-started-python.html#_connecting) of the getting started documentation. ## Usage ----- * [Creating an index](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/getting-started-python.html#_creating_an_index) * [Indexing a document](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/getting-started-python.html#_indexing_documents) * [Getting documents](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/getting-started-python.html#_getting_documents) * [Searching documents](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/getting-started-python.html#_searching_documents) * [Updating documents](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/getting-started-python.html#_updating_documents) * [Deleting documents](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/getting-started-python.html#_deleting_documents) * [Deleting an index](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/getting-started-python.html#_deleting_an_index) ## Compatibility Language clients are _forward compatible:_ each client version works with equivalent and later minor versions of Elasticsearch without breaking. Compatibility does not imply full feature parity. New Elasticsearch features are supported only in equivalent client versions. For example, an 8.12 client fully supports Elasticsearch 8.12 features and works with 8.13 without breaking; however, it does not support new Elasticsearch 8.13 features. An 8.13 client fully supports Elasticsearch 8.13 features. | Elasticsearch version | elasticsearch-py branch | | --- | --- | | main | main | | 9.x | 9.x | | 9.x | 8.x | | 8.x | 8.x | Elasticsearch language clients are also _backward compatible_ across minor versions — with default distributions and without guarantees. > [!TIP] > To upgrade to a new major version, first upgrade Elasticsearch, then upgrade the Python Elasticsearch client. If you need to work with multiple client versions, note that older versions are also released as `elasticsearch7` and `elasticsearch8`. ## Documentation Documentation for the client is [available on elastic.co] and [Read the Docs]. [available on elastic.co]: https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/index.html [Read the Docs]: https://elasticsearch-py.readthedocs.io ## Try Elasticsearch and Kibana locally If you want to try Elasticsearch and Kibana locally, you can run the following command: ```bash curl -fsSL https://elastic.co/start-local | sh ``` This will run Elasticsearch at [http://localhost:9200](http://localhost:9200) and Kibana at [http://localhost:5601](http://localhost:5601). More information is available [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/run-elasticsearch-locally.html). ## Contributing See [CONTRIBUTING.md](./CONTRIBUTING.md) ## License This software is licensed under the [Apache License 2.0](./LICENSE). See [NOTICE](./NOTICE). --- ## File: docs/reference/async.md --- mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/async.html --- # Using with asyncio [async] The `elasticsearch` package supports async/await with [asyncio](https://docs.python.org/3/library/asyncio.html) and [aiohttp](https://docs.aiohttp.org). You can either install `aiohttp` directly or use the `[async]` extra: ```bash $ python -m pip install elasticsearch aiohttp # - OR - $ python -m pip install elasticsearch[async] ``` ## Getting Started with Async [_getting_started_with_async] After installation all async API endpoints are available via `~elasticsearch.AsyncElasticsearch` and are used in the same way as other APIs, with an extra `await`: ```python import asyncio from elasticsearch import AsyncElasticsearch client = AsyncElasticsearch() async def main(): resp = await client.search( index="documents", body={"query": {"match_all": {}}}, size=20, ) print(resp) asyncio.run(main()) ``` All APIs that are available under the sync client are also available under the async client. [Reference documentation](https://elasticsearch-py.readthedocs.io/en/latest/async.html#api-reference) ## ASGI Applications and Elastic APM [_asgi_applications_and_elastic_apm] [ASGI](https://asgi.readthedocs.io) (Asynchronous Server Gateway Interface) is a way to serve Python web applications making use of async I/O to achieve better performance. Some examples of ASGI frameworks include FastAPI, Django 3.0+, and Starlette. If you’re using one of these frameworks along with Elasticsearch then you should be using `~elasticsearch.AsyncElasticsearch` to avoid blocking the event loop with synchronous network calls for optimal performance. [Elastic APM](apm-agent-python://reference/index.md) also supports tracing of async Elasticsearch queries like synchronous queries. For an example on how to configure `AsyncElasticsearch` with a popular ASGI framework [FastAPI](https://fastapi.tiangolo.com/) and APM tracing there is a [pre-built example](https://github.com/elastic/elasticsearch-py/tree/master/examples/fastapi-apm) in the `examples/fastapi-apm` directory. See also the [Using OpenTelemetry](/reference/opentelemetry.md) page. ## Trio support If you prefer using Trio instead of asyncio to take advantage of its better structured concurrency support, you can use the HTTPX async node which supports Trio out of the box. ```python import trio from elasticsearch import AsyncElasticsearch client = AsyncElasticsearch( "https://...", api_key="...", node_class="httpxasync") async def main(): resp = await client.info() print(resp.body) trio.run(main) ``` The one limitation of Trio support is that it does not currently support node sniffing, which was not implemented with structured concurrency in mind. ## Frequently Asked Questions [_frequently_asked_questions] ### ValueError when initializing `AsyncElasticsearch`? [_valueerror_when_initializing_asyncelasticsearch] If when trying to use `AsyncElasticsearch` you receive `ValueError: You must have 'aiohttp' installed to use AiohttpHttpNode` you should ensure that you have `aiohttp` installed in your environment (check with `$ python -m pip freeze | grep aiohttp`). Otherwise, async support won’t be available. ### What about the `elasticsearch-async` package? [_what_about_the_elasticsearch_async_package] Previously asyncio was supported separately with the [elasticsearch-async](https://github.com/elastic/elasticsearch-py-async) package. The `elasticsearch-async` package has been deprecated in favor of `AsyncElasticsearch` provided by the `elasticsearch` package in v7.8 and onwards. ### Receiving *Unclosed client session / connector* warning? [_receiving_unclosed_client_session_connector_warning] This warning is created by `aiohttp` when an open HTTP connection is garbage collected. You’ll typically run into this when closing your application. To resolve the issue ensure that `~elasticsearch.AsyncElasticsearch.close` is called before the `~elasticsearch.AsyncElasticsearch` instance is garbage collected. For example if using FastAPI that might look like this: ```python import os from contextlib import asynccontextmanager from fastapi import FastAPI from elasticsearch import AsyncElasticsearch ELASTICSEARCH_URL = os.environ["ELASTICSEARCH_URL"] client = None @asynccontextmanager async def lifespan(app: FastAPI): global client client = AsyncElasticsearch(ELASTICSEARCH_URL) yield await client.close() app = FastAPI(lifespan=lifespan) @app.get("/") async def main(): return await client.info() ``` You can run this example by saving it to `main.py` and executing `ELASTICSEARCH_URL=http://localhost:9200 uvicorn main:app`. ## Async Helpers [_async_helpers] Async variants of all helpers are available in `elasticsearch.helpers` and are all prefixed with `async_*`. You’ll notice that these APIs are identical to the ones in the sync [*Client helpers*](/reference/client-helpers.md) documentation. All async helpers that accept an iterator or generator also accept async iterators and async generators. [Reference documentation](https://elasticsearch-py.readthedocs.io/en/latest/async.html#api-reference) --- ## File: docs/reference/client-helpers.md --- mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/client-helpers.html --- # Client helpers [client-helpers] You can find here a collection of simple helper functions that abstract some specifics of the raw API. ## Bulk helpers [bulk-helpers] There are several helpers for the bulk API since its requirement for specific formatting and other considerations can make it cumbersome if used directly. All bulk helpers accept an instance of `Elasticsearch` class and an iterable `action` (any iterable, can also be a generator, which is ideal in most cases since it allows you to index large datasets without the need of loading them into memory). For asynchronous Python, use the bulk helpers with the `async_` prefix and pass an `AsyncElasticsearch` instance as first argument. The items in the iterable `action` should be the documents we wish to index in several formats. The most common one is the same as returned by `search()`, for example: ```yaml { '_index': 'index-name', '_id': 42, '_routing': 5, 'pipeline': 'my-ingest-pipeline', '_source': { "title": "Hello World!", "body": "..." } } ``` Alternatively, if `_source` is not present, it pops all metadata fields from the doc and use the rest as the document data: ```yaml { "_id": 42, "_routing": 5, "title": "Hello World!", "body": "..." } ``` The `bulk()` api accepts `index`, `create`, `delete`, and `update` actions. Use the `_op_type` field to specify an action (`_op_type` defaults to `index`): ```yaml { '_op_type': 'delete', '_index': 'index-name', '_id': 42, } { '_op_type': 'update', '_index': 'index-name', '_id': 42, 'doc': {'question': 'The life, universe and everything.'} } ``` ## Scan [scan] Simple abstraction on top of the `scroll()` API - a simple iterator that yields all hits as returned by underlining scroll requests. By default scan does not return results in any pre-determined order. To have a standard order in the returned documents (either by score or explicit sort definition) when scrolling, use `preserve_order=True`. This may be an expensive operation and will negate the performance benefits of using `scan`. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```py from elasticsearch.helpers import scan es = Elasticsearch(hosts=['https://localhost:9200']) scan( es, query={"query": {"match": {"title": "python"}}}, index="orders-*" ) ``` ::: :::{tab-item} Async Python :sync: async ```py from elasticsearch.helpers import async_scan es = AsyncElasticsearch(hosts=['https://localhost:9200']) async def main(): await async_scan( es, query={"query": {"match": {"title": "python"}}}, index="orders-*" asyncio.run(main()) ``` ::: :::: --- ## File: docs/reference/configuration.md --- mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/config.html navigation_title: Configuration --- # Python client configuration for {{es}} [config] This page contains information about the most important configuration options of the Python {{es}} client. ## TLS/SSL [tls-and-ssl] The options in this section can only be used when the node is configured for HTTPS. An error will be raised if using these options with an HTTP node. ### Verifying server certificates [_verifying_server_certificates] The typical route to verify a cluster certificate is via a "CA bundle" which can be specified via the `ca_certs` parameter. If no options are given and the [certifi package](https://github.com/certifi/python-certifi) is installed then certifi’s CA bundle is used by default. If you have your own CA bundle to use you can configure via the `ca_certs` parameter: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python client = Elasticsearch( "https://...", ca_certs="/path/to/certs.pem" ) ``` ::: :::{tab-item} Async Python :sync: async ```python client = AsyncElasticsearch( "https://...", ca_certs="/path/to/certs.pem" ) ``` ::: :::: If using a generated certificate or certificate with a known fingerprint you can use the `ssl_assert_fingerprint` to specify the fingerprint which tries to match the server’s leaf certificate during the TLS handshake. If there is any matching certificate the connection is verified, otherwise a `TlsError` is raised. In Python 3.9 and earlier only the leaf certificate will be verified but in Python 3.10+ private APIs are used to verify any certificate in the certificate chain. This helps when using certificates that are generated on a multi-node cluster. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python client = Elasticsearch( "https://...", ssl_assert_fingerprint=( "315f5bdb76d078c43b8ac0064e4a0164612b1fce77c869345bfc94c75894edd3" ) ) ``` ::: :::{tab-item} Async Python :sync: async ```python client = AsyncElasticsearch( "https://...", ssl_assert_fingerprint=( "315f5bdb76d078c43b8ac0064e4a0164612b1fce77c869345bfc94c75894edd3" ) ) ``` ::: :::: To disable certificate verification use the `verify_certs=False` parameter. This option should be avoided in production, instead use the other options to verify the clusters' certificate. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python client = Elasticsearch( "https://...", verify_certs=False ) ``` ::: :::{tab-item} Async Python :sync: async ```python client = AsyncElasticsearch( "https://...", verify_certs=False ) ``` ::: :::: ### TLS versions [_tls_versions] Configuring the minimum TLS version to connect to is done via the `ssl_version` parameter. By default this is set to a minimum value of TLSv1.2. Use the `ssl.TLSVersion` enumeration to specify versions. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python import ssl client = Elasticsearch( ..., ssl_version=ssl.TLSVersion.TLSv1_2 ) ``` ::: :::{tab-item} Async Python :sync: async ```python import ssl client = AsyncElasticsearch( ..., ssl_version=ssl.TLSVersion.TLSv1_2 ) ``` ::: :::: ### Client TLS certificate authentication [_client_tls_certificate_authentication] Elasticsearch can be configured to authenticate clients via TLS client certificates. Client certificate and keys can be configured via the `client_cert` and `client_key` parameters: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python client = Elasticsearch( ..., client_cert="/path/to/cert.pem", client_key="/path/to/key.pem", ) ``` ::: :::{tab-item} Async Python :sync: async ```python client = AsyncElasticsearch( ..., client_cert="/path/to/cert.pem", client_key="/path/to/key.pem", ) ``` ::: :::: ### Using an SSLContext [_using_an_sslcontext] For advanced users an `ssl.SSLContext` object can be used for configuring TLS via the `ssl_context` parameter. The `ssl_context` parameter can’t be combined with any other TLS options except for the `ssl_assert_fingerprint` parameter. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python import ssl # Create and configure an SSLContext ctx = ssl.create_default_context() ctx.load_verify_locations(...) client = Elasticsearch( ..., ssl_context=ctx ) ``` ::: :::{tab-item} Async Python :sync: async ```python import ssl # Create and configure an SSLContext ctx = ssl.create_default_context() ctx.load_verify_locations(...) client = AsyncElasticsearch( ..., ssl_context=ctx ) ``` ::: :::: ## HTTP compression [compression] Compression of HTTP request and response bodies can be enabled with the `http_compress` parameter. If enabled then HTTP request bodies will be compressed with `gzip` and HTTP responses will include the `Accept-Encoding: gzip` HTTP header. By default compression is disabled. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python client = Elasticsearch( ..., http_compress=True # Enable compression! ) ``` ::: :::{tab-item} Async Python :sync: async ```python client = AsyncElasticsearch( ..., http_compress=True # Enable compression! ) ``` ::: :::: HTTP compression is recommended to be enabled when requests are traversing the network. Compression is automatically enabled when connecting to Elastic Cloud. ## Request timeouts [timeouts] Requests can be configured to timeout if taking too long to be serviced. The `request_timeout` parameter can be passed via the client constructor or the client `.options()` method. When the request times out the node will raise a `ConnectionTimeout` exception which can trigger retries. Setting `request_timeout` to `None` will disable timeouts. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python client = Elasticsearch( ..., request_timeout=10 # 10 second timeout ) # Search request will timeout in 5 seconds client.options(request_timeout=5).search(...) ``` ::: :::{tab-item} Async Python :sync: async ```python client = AsyncElasticsearch( ..., request_timeout=10 # 10 second timeout ) async def example(): # Search request will timeout in 5 seconds await client.options(request_timeout=5).search(...) ``` ::: :::: ### API and server timeouts [_api_and_server_timeouts] There are API-level timeouts to take into consideration when making requests which can cause the request to timeout on server-side rather than client-side. You may need to configure both a transport and API level timeout for long running operations. In the example below there are three different configurable timeouts for the `cluster.health` API all with different meanings for the request: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python client.options( # Amount of time to wait for an HTTP response to start. request_timeout=30 ).cluster.health( # Amount of time to wait to collect info on all nodes. timeout=30, # Amount of time to wait for info from the master node. master_timeout=10, ) ``` ::: :::{tab-item} Async Python :sync: async ```python async def example(): await client.options( # Amount of time to wait for an HTTP response to start. request_timeout=30 ).cluster.health( # Amount of time to wait to collect info on all nodes. timeout=30, # Amount of time to wait for info from the master node. master_timeout=10, ) ``` ::: :::: ## Retries [retries] Requests can be retried if they don’t return with a successful response. This provides a way for requests to be resilient against transient failures or overloaded nodes. The maximum number of retries per request can be configured via the `max_retries` parameter. Setting this parameter to 0 disables retries. This parameter can be set in the client constructor or per-request via the client `.options()` method: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python client = Elasticsearch( ..., max_retries=5 ) # For this API request we disable retries with 'max_retries=0' client.options(max_retries=0).index( index="blogs", document={ "title": "..." } ) ``` ::: :::{tab-item} Async Python :sync: async ```python client = AsyncElasticsearch( ..., max_retries=5 ) async def example(): # For this API request we disable retries with 'max_retries=0' await client.options(max_retries=0).index( index="blogs", document={ "title": "..." } ) ``` ::: :::: ### Retrying on connection errors and timeouts [_retrying_on_connection_errors_and_timeouts] Connection errors are automatically retried if retries are enabled. Retrying requests on connection timeouts can be enabled or disabled via the `retry_on_timeout` parameter. This parameter can be set on the client constructor or via the client `.options()` method: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python client = Elasticsearch( ..., retry_on_timeout=True ) client.options(retry_on_timeout=False).info() ``` ::: :::{tab-item} Async Python :sync: async ```python client = AsyncElasticsearch( ..., retry_on_timeout=True ) async def example(): await client.options(retry_on_timeout=False).info() ``` ::: :::: ### Retrying status codes [_retrying_status_codes] By default if retries are enabled `retry_on_status` is set to `(429, 502, 503, 504)`. This parameter can be set on the client constructor or via the client `.options()` method. Setting this value to `()` will disable the default behavior. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python client = Elasticsearch( ..., retry_on_status=() ) # Retry this API on '500 Internal Error' statuses client.options(retry_on_status=[500]).index( index="blogs", document={ "title": "..." } ) ``` ::: :::{tab-item} Async Python :sync: async ```python client = AsyncElasticsearch( ..., retry_on_status=() ) async def example(): # Retry this API on '500 Internal Error' statuses await client.options(retry_on_status=[500]).index( index="blogs", document={ "title": "..." } ) ``` ::: :::: ### Exponential backoff [_exponential_backoff] By default, retries are executed in quick succession, without any pause between them. If desired, the client can be configured to use an exponential backoff strategy that spaces out the retries to give the server more time to recover from a transient failure. The exponential backoff algorithm used by the client is called "Equal Jitter." It is described in detail in the [Exponential Backoff And Jitter](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) article, published in the AWS Architecture blog. Exponential backoff delays between retries are configured with the `retry_backoff_base` and `retry_backoff_cap` arguments. As with other configuration options, these can be given in the client's constructor to be used in all calls made with the client, or passed in the client's `.options()` method for use only with specific calls. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python client = Elasticsearch( ..., max_retries=5, retry_backoff_base=1, retry_backoff_cap=20, ) ::: :::{tab-item} Async Python :sync: async ```python client = AsyncElasticsearch( ..., max_retries=5, retry_backoff_base=1, retry_backoff_cap=20, ) ``` /* Detailed source-code truncated for AI context efficiency. */ ```python client = Elasticsearch(...) # API request is robust against the index not existing: resp = client.options(ignore_status=404).indices.delete(index="delete-this") resp.meta.status # Can be either '2XX' or '404' # API request is robust against the index already existing: resp = client.options(ignore_status=[400]).indices.create( index="create-this", mapping={ "properties": {"field": {"type": "integer"}} } ) resp.meta.status # Can be either '2XX' or '400' ``` ::: :::{tab-item} Async Python :sync: async ```python client = AsyncElasticsearch(...) async def example(): # API request is robust against the index not existing: resp = await client.options(ignore_status=404).indices.delete(index="delete-this") resp.meta.status # Can be either '2XX' or '404' # API request is robust against the index already existing: resp = await client.options(ignore_status=[400]).indices.create( index="create-this", mapping={ "properties": {"field": {"type": "integer"}} } ) resp.meta.status # Can be either '2XX' or '400' ``` ::: :::: When using the `ignore_status` parameter the error response will be returned serialized like a non-error response. In these cases it can be useful to inspect the HTTP status of the response. To do this you can inspect the `resp.meta.status`. ## Sniffing for new nodes [sniffing] Additional nodes can be discovered by a process called "sniffing" where the client will query the cluster for more nodes that can handle requests. Sniffing can happen at three different times: on client instantiation, before requests, and on a node failure. These three behaviors can be enabled and disabled with the `sniff_on_start`, `sniff_before_requests`, and `sniff_on_node_failure` parameters. ::::{important} When using an HTTP load balancer or proxy you cannot use sniffing functionality as the cluster would supply the client with IP addresses to directly connect to the cluster, circumventing the load balancer. Depending on your configuration this might be something you don’t want or break completely. :::: ### Waiting between sniffing attempts [_waiting_between_sniffing_attempts] To avoid needlessly sniffing too often there is a delay between attempts to discover new nodes. This value can be controlled via the `min_delay_between_sniffing` parameter. ### Filtering nodes which are sniffed [_filtering_nodes_which_are_sniffed] By default nodes which are marked with only a `master` role will not be used. To change the behavior the parameter `sniffed_node_callback` can be used. To mark a sniffed node not to be added to the node pool return `None` from the `sniffed_node_callback`, otherwise return a `NodeConfig` instance. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from typing import Optional, Dict, Any from elastic_transport import NodeConfig from elasticsearch import Elasticsearch def filter_master_eligible_nodes( node_info: Dict[str, Any], node_config: NodeConfig ) -> Optional[NodeConfig]: # This callback ignores all nodes that are master eligible # instead of master-only nodes (default behavior) if "master" in node_info.get("roles", ()): return None return node_config client = Elasticsearch( "https://localhost:9200", sniffed_node_callback=filter_master_eligible_nodes ) ``` ::: :::{tab-item} Async Python :sync: async ```python from typing import Optional, Dict, Any from elastic_transport import NodeConfig from elasticsearch import AsyncElasticsearch def filter_master_eligible_nodes( node_info: Dict[str, Any], node_config: NodeConfig ) -> Optional[NodeConfig]: # This callback ignores all nodes that are master eligible # instead of master-only nodes (default behavior) if "master" in node_info.get("roles", ()): return None return node_config client = AsyncElasticsearch( "https://localhost:9200", sniffed_node_callback=filter_master_eligible_nodes ) ``` ::: :::: The `node_info` parameter is part of the response from the `nodes.info()` API, below is an example of what that object looks like: ```json { "name": "SRZpKFZ", "transport_address": "127.0.0.1:9300", "host": "127.0.0.1", "ip": "127.0.0.1", "version": "5.0.0", "build_hash": "253032b", "roles": ["master", "data", "ingest"], "http": { "bound_address": ["[fe80::1]:9200", "[::1]:9200", "127.0.0.1:9200"], "publish_address": "1.1.1.1:123", "max_content_length_in_bytes": 104857600 } } ``` ## Node Pool [node-pool] ### Selecting a node from the pool [_selecting_a_node_from_the_pool] You can specify a node selector pattern via the `node_selector_class` parameter. The supported values are `round_robin` and `random`. Default is `round_robin`. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python client = Elasticsearch( ..., node_selector_class="round_robin" ) ``` ::: :::{tab-item} Async Python :sync: async ```python client = AsyncElasticsearch( ..., node_selector_class="round_robin" ) ``` ::: :::: Custom selectors are also supported: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elastic_transport import NodeSelector class CustomSelector(NodeSelector): def select(nodes): ... client = Elasticsearch( ..., node_selector_class=CustomSelector ) ``` ::: :::{tab-item} Async Python :sync: async ```python from elastic_transport import NodeSelector class CustomSelector(NodeSelector): def select(nodes): ... client = AsyncElasticsearch( ..., node_selector_class=CustomSelector ) ``` ::: :::: ### Marking nodes dead and alive [_marking_nodes_dead_and_alive] Individual nodes of Elasticsearch can have transient connectivity or load issues which may make them unable to service requests. To combat this the pool of nodes will detect when a node isn’t able to service requests due to transport or API errors. After a node has been timed out it will be moved back to the set of "alive" nodes but only after the node returns a successful response will the node be marked as "alive" in terms of consecutive errors. The `dead_node_backoff_factor` and `max_dead_node_backoff` parameters can be used to configure how long the node pool will put the node into timeout with each consecutive failure. Both parameters use a unit of seconds. The calculation is equal to `min(dead_node_backoff_factor * (2 ** (consecutive_failures - 1)), max_dead_node_backoff)`. ## Serializers [serializer] Serializers transform bytes on the wire into native Python objects and vice-versa. By default the client ships with serializers for `application/json`, `application/x-ndjson`, `text/*`, `application/vnd.apache.arrow.stream` and `application/mapbox-vector-tile`. You can define custom serializers via the `serializers` parameter: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch import Elasticsearch, JsonSerializer class JsonSetSerializer(JsonSerializer): """Custom JSON serializer that handles Python sets""" def default(self, data: Any) -> Any: if isinstance(data, set): return list(data) return super().default(data) client = Elasticsearch( ..., # Serializers are a mapping of 'mimetype' to Serializer class. serializers={"application/json": JsonSetSerializer()} ) ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch import AsyncElasticsearch, JsonSerializer class JsonSetSerializer(JsonSerializer): """Custom JSON serializer that handles Python sets""" def default(self, data: Any) -> Any: if isinstance(data, set): return list(data) return super().default(data) client = AsyncElasticsearch( ..., # Serializers are a mapping of 'mimetype' to Serializer class. serializers={"application/json": JsonSetSerializer()} ) ``` ::: :::: If the `orjson` package is installed, you can use the faster ``OrjsonSerializer`` for the default mimetype (``application/json``): ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch import Elasticsearch, OrjsonSerializer es = Elasticsearch( ..., serializer=OrjsonSerializer() ) ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch import AsyncElasticsearch, OrjsonSerializer es = AsyncElasticsearch( ..., serializer=OrjsonSerializer() ) ``` ::: :::: orjson is particularly fast when serializing vectors as it has native numpy support. This will be the default in a future release. You can install orjson with the `orjson` extra: ```sh $ python -m pip install elasticsearch[orjson] ``` ## Nodes [nodes] ### Node implementations [_node_implementations] The default node class for synchronous I/O is `urllib3` and the default node class for asynchronous I/O is `aiohttp`. For all of the built-in HTTP node implementations like `urllib3`, `requests`, and `aiohttp` you can specify with a simple string to the `node_class` parameter: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch import Elasticsearch client = Elasticsearch( ..., node_class="requests" ) ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch import AsyncElasticsearch client = AsyncElasticsearch( ..., node_class="httpxasync" ) ``` ::: :::: You can also specify a custom node implementation via the `node_class` parameter: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch import Elasticsearch from elastic_transport import Urllib3HttpNode class CustomHttpNode(Urllib3HttpNode): ... client = Elasticsearch( ... node_class=CustomHttpNode ) ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch import AsyncElasticsearch from elastic_transport import AophttpHttpNode class CustomHttpNode(AiohttpHttpNode): ... client = AsyncElasticsearch( ... node_class=CustomHttpNode ) ``` ::: :::: ### HTTP connections per node [_http_connections_per_node] Each node contains its own pool of HTTP connections to allow for concurrent requests. This value is configurable via the `connections_per_node` parameter: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python client = Elasticsearch( ..., connections_per_node=5 ) ``` ::: ::::{tab-item} Async Python :sync: async ```python client = AsyncElasticsearch( ..., connections_per_node=5 ) ``` ::: ::: --- ## File: docs/reference/connecting.md --- mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/connecting.html --- # Connecting [connecting] This page contains the information you need to connect the Client with {{es}}. ## Connecting to Elastic Cloud [connect-ec] [Elastic Cloud](docs-content://deploy-manage/deploy/elastic-cloud/cloud-hosted.md) is the easiest way to get started with {{es}}. When connecting to Elastic Cloud with the Python {{es}} client you should always use the `cloud_id` parameter to connect. You can find this value within the "Manage Deployment" page after you’ve created a cluster (look in the top-left if you’re in Kibana). We recommend using a Cloud ID whenever possible because your client will be automatically configured for optimal use with Elastic Cloud including HTTPS and HTTP compression. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python import os from elasticsearch import Elasticsearch # Password for the 'elastic' user generated by Elasticsearch ELASTIC_PASSWORD = os.environ['ELASTIC_PASSWORD'] # Found in the 'Manage Deployment' page CLOUD_ID = "deployment-name:dXMtZWFzdDQuZ2Nw..." # Create the client instance client = Elasticsearch( cloud_id=CLOUD_ID, basic_auth=("elastic", ELASTIC_PASSWORD) ) # Successful response! client.info() # {'name': 'instance-0000000000', 'cluster_name': ...} ``` ::: :::{tab-item} Async Python :sync: async ```python import asyncio import os from elasticsearch import AsyncElasticsearch # Password for the 'elastic' user generated by Elasticsearch ELASTIC_PASSWORD = os.environ['ELASTIC_PASSWORD'] # Found in the 'Manage Deployment' page CLOUD_ID = "deployment-name:dXMtZWFzdDQuZ2Nw..." async def main(): # Create the client instance client = AsyncElasticsearch( cloud_id=CLOUD_ID, basic_auth=("elastic", ELASTIC_PASSWORD) ) # Successful response! await client.info() # {'name': 'instance-0000000000', 'cluster_name': ...} asyncio.run(main()) ``` ::: :::: ## Connecting to a self-managed cluster [connect-self-managed-new] By default {{es}} will start with security features like authentication and TLS enabled. To connect to the {{es}} cluster you’ll need to configure the Python {{es}} client to use HTTPS with the generated CA certificate in order to make requests successfully. If you’re getting started with {{es}} we recommend reading the documentation on [configuring](docs-content://deploy-manage/deploy/self-managed/configure-elasticsearch.md) and [starting {{es}}](docs-content://deploy-manage/maintenance/start-stop-services/start-stop-elasticsearch.md) to ensure your cluster is running as expected. When you start {{es}} for the first time you’ll see a distinct block like the one below in the output from {{es}} (you may have to scroll up if it’s been a while): ```sh ---------------------------------------------------------------- -> Elasticsearch security features have been automatically configured! -> Authentication is enabled and cluster connections are encrypted. -> Password for the elastic user (reset with `bin/elasticsearch-reset-password -u elastic`): lhQpLELkjkrawaBoaz0Q -> HTTP CA certificate SHA-256 fingerprint: a52dd93511e8c6045e21f16654b77c9ee0f34aea26d9f40320b531c474676228 ... ---------------------------------------------------------------- ``` Note down the `elastic` user password and HTTP CA fingerprint for the next sections. In the examples below they will be stored in the variables `ELASTIC_PASSWORD` and `CERT_FINGERPRINT` respectively. Depending on the circumstances there are two options for verifying the HTTPS connection, either verifying with the CA certificate itself or via the HTTP CA certificate fingerprint. ### Verifying HTTPS with CA certificates [_verifying_https_with_ca_certificates] Using the `ca_certs` option is the default way the Python {{es}} client verifies an HTTPS connection. The generated root CA certificate can be found in the `certs` directory in your {{es}} config location (`$ES_CONF_PATH/certs/http_ca.crt`). If you’re running {{es}} in Docker there is [additional documentation for retrieving the CA certificate](docs-content://deploy-manage/deploy/self-managed/install-elasticsearch-with-docker.md). Once you have the `http_ca.crt` file somewhere accessible pass the path to the client via `ca_certs`: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python import os from elasticsearch import Elasticsearch # Password for the 'elastic' user generated by Elasticsearch ELASTIC_PASSWORD = os.environ['ELASTIC_PASSWORD'] # Create the client instance client = Elasticsearch( "https://localhost:9200", ca_certs="/path/to/http_ca.crt", basic_auth=("elastic", ELASTIC_PASSWORD) ) # Successful response! client.info() # {'name': 'instance-0000000000', 'cluster_name': ...} ``` ::: :::{tab-item} Async Python :sync: async ```python import asyncio import os from elasticsearch import AsyncElasticsearch # Password for the 'elastic' user generated by Elasticsearch ELASTIC_PASSWORD = os.environ['ELASTIC_PASSWORD'] async def main(): # Create the client instance client = AsyncElasticsearch( "https://localhost:9200", ca_certs="/path/to/http_ca.crt", basic_auth=("elastic", ELASTIC_PASSWORD) ) # Successful response! await client.info() # {'name': 'instance-0000000000', 'cluster_name': ...} asyncio.run(main()) ``` ::: :::: ::::{note} If you don’t specify `ca_certs` or `ssl_assert_fingerprint` then the [certifi package](https://certifiio.readthedocs.io) will be used for `ca_certs` by default if available. :::: ### Verifying HTTPS with certificate fingerprints (Python 3.10 or later) [_verifying_https_with_certificate_fingerprints_python_3_10_or_later] ::::{note} Using this method **requires using Python 3.10 or later** and isn’t available when using the `aiohttp` HTTP client library so can’t be used with `AsyncElasticsearch`. :::: This method of verifying the HTTPS connection takes advantage of the certificate fingerprint value noted down earlier. Take this SHA256 fingerprint value and pass it to the Python {{es}} client via `ssl_assert_fingerprint`: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python import os from elasticsearch import Elasticsearch # Fingerprint either from Elasticsearch startup or above script. # Colons and uppercase/lowercase don't matter when using # the 'ssl_assert_fingerprint' parameter CERT_FINGERPRINT = "A5:2D:D9:35:11:E8:C6:04:5E:21:F1:66:54:B7:7C:9E:E0:F3:4A:EA:26:D9:F4:03:20:B5:31:C4:74:67:62:28" # Password for the 'elastic' user generated by Elasticsearch ELASTIC_PASSWORD = os.environ['ELASTIC_PASSWORD'] client = Elasticsearch( "https://localhost:9200", ssl_assert_fingerprint=CERT_FINGERPRINT, basic_auth=("elastic", ELASTIC_PASSWORD) ) # Successful response! client.info() # {'name': 'instance-0000000000', 'cluster_name': ...} ``` ::: :::{tab-item} Async Python :sync: async ```python import asyncio import os from elasticsearch import AsyncElasticsearch # Fingerprint either from Elasticsearch startup or above script. # Colons and uppercase/lowercase don't matter when using # the 'ssl_assert_fingerprint' parameter CERT_FINGERPRINT = "A5:2D:D9:35:11:E8:C6:04:5E:21:F1:66:54:B7:7C:9E:E0:F3:4A:EA:26:D9:F4:03:20:B5:31:C4:74:67:62:28" # Password for the 'elastic' user generated by Elasticsearch ELASTIC_PASSWORD = os.environ['ELASTIC_PASSWORD'] async def main(): client = AsyncElasticsearch( "https://localhost:9200", ssl_assert_fingerprint=CERT_FINGERPRINT, basic_auth=("elastic", ELASTIC_PASSWORD) ) # Successful response! await client.info() # {'name': 'instance-0000000000', 'cluster_name': ...} asyncio.run(main()) ``` ::: :::: The certificate fingerprint can be calculated using `openssl x509` with the certificate file: ```sh openssl x509 -fingerprint -sha256 -noout -in /path/to/http_ca.crt ``` If you don’t have access to the generated CA file from {{es}} you can use the following script to output the root CA fingerprint of the {{es}} instance with `openssl s_client`: ```sh # Replace the values of 'localhost' and '9200' to the # corresponding host and port values for the cluster. openssl s_client -connect localhost:9200 -servername localhost -showcerts /dev/null \ | openssl x509 -fingerprint -sha256 -noout -in /dev/stdin ``` The output of `openssl x509` will look something like this: ```sh SHA256 Fingerprint=A5:2D:D9:35:11:E8:C6:04:5E:21:F1:66:54:B7:7C:9E:E0:F3:4A:EA:26:D9:F4:03:20:B5:31:C4:74:67:62:28 ``` ## Connecting without security enabled [connect-no-security] ::::{warning} Running {{es}} without security enabled is not recommended. :::: If your cluster is configured with [security explicitly disabled](elasticsearch://reference/elasticsearch/configuration-reference/security-settings.md) then you can connect via HTTP: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch import Elasticsearch # Create the client instance client = Elasticsearch("http://localhost:9200") # Successful response! client.info() # {'name': 'instance-0000000000', 'cluster_name': ...} ``` ::: :::{tab-item} Async Python :sync: async ```python import asyncio from elasticsearch import AsyncElasticsearch async def main(): # Create the client instance client = AsyncElasticsearch("http://localhost:9200") # Successful response! await client.info() # {'name': 'instance-0000000000', 'cluster_name': ...} asyncio.run(main()) ``` ::: :::: ## Connecting to multiple nodes [connect-url] The Python {{es}} client supports sending API requests to multiple nodes in the cluster. This means that work will be more evenly spread across the cluster instead of hammering the same node over and over with requests. To configure the client with multiple nodes you can pass a list of URLs, each URL will be used as a separate node in the pool. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python import os from elasticsearch import Elasticsearch # List of nodes to connect use with different hosts and ports. NODES = [ "https://localhost:9200", "https://localhost:9201", "https://localhost:9202", ] # Password for the 'elastic' user generated by Elasticsearch ELASTIC_PASSWORD = os.environ['ELASTIC_PASSWORD'] client = Elasticsearch( NODES, ca_certs="/path/to/http_ca.crt", basic_auth=("elastic", ELASTIC_PASSWORD) ) ``` ::: :::{tab-item} Async Python :sync: async ```python import os from elasticsearch import AsyncElasticsearch # List of nodes to connect use with different hosts and ports. NODES = [ "https://localhost:9200", "https://localhost:9201", "https://localhost:9202", ] # Password for the 'elastic' user generated by Elasticsearch ELASTIC_PASSWORD = os.environ['ELASTIC_PASSWORD'] client = AsyncElasticsearch( NODES, ca_certs="/path/to/http_ca.crt", basic_auth=("elastic", ELASTIC_PASSWORD) ) ``` ::: :::: By default nodes are selected using round-robin, but alternate node selection strategies can be configured with `node_selector_class` parameter. ::::{note} If your {{es}} cluster is behind a load balancer like when using Elastic Cloud you won’t need to configure multiple nodes. Instead use the load balancer host and port. :::: ## Authentication [authentication] This section contains code snippets to show you how to connect to various {{es}} providers. All authentication methods are supported on the client constructor or via the per-request `.options()` method: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python import os from elasticsearch import Elasticsearch # Authenticate from the constructor client = Elasticsearch( "https://localhost:9200", ca_certs="/path/to/http_ca.crt", basic_auth=("username", os.environ["ELASTIC_PASSWORD"]) ) # Authenticate via the .options() method: client.options( basic_auth=("username", os.environ["ELASTIC_PASSWORD"]) ).indices.get(index="*") # You can persist the authenticated client to use # later or use for multiple API calls: auth_client = client.options(api_key=os.environ["ELASTIC_API_KEY"]) for i in range(10): auth_client.index( index="example-index", document={"field": i} ) ``` ::: :::{tab-item} Async Python :sync: async ```python import asyncio import os from elasticsearch import AsyncElasticsearch async def main(): # Authenticate from the constructor client = AsyncElasticsearch( "https://localhost:9200", ca_certs="/path/to/http_ca.crt", basic_auth=("username", os.environ["ELASTIC_PASSWORD"]) ) # Authenticate via the .options() method: await client.options( basic_auth=("username", os.environ["ELASTIC_PASSWORD"]) ).indices.get(index="*") # You can persist the authenticated client to use # later or use for multiple API calls: auth_client = client.options(api_key=os.environ["ELASTIC_API_KEY"]) for i in range(10): await auth_client.index( index="example-index", document={"field": i} ) asyncio.run(main()) ``` ::: :::: ### HTTP Basic authentication (Username and Password) [auth-basic] HTTP Basic authentication uses the `basic_auth` parameter by passing in a username and password within a tuple: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python import os from elasticsearch import Elasticsearch # Adds the HTTP header 'Authorization: Basic ' client = Elasticsearch( "https://localhost:9200", ca_certs="/path/to/http_ca.crt", basic_auth=("username", os.environ["ELASTIC_PASSWORD"]) ) ``` ::: :::{tab-item} Async Python :sync: async ```python import os from elasticsearch import AsyncElasticsearch # Adds the HTTP header 'Authorization: Basic ' client = AsyncElasticsearch( "https://localhost:9200", ca_certs="/path/to/http_ca.crt", basic_auth=("username", os.environ["ELASTIC_PASSWORD"]) ) ``` ::: :::: ### HTTP Bearer authentication [auth-bearer] HTTP Bearer authentication uses the `bearer_auth` parameter by passing the token as a string. This authentication method is used by [Service Account Tokens](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-service-token) and [Bearer Tokens](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-get-token). ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python import os from elasticsearch import Elasticsearch # Adds the HTTP header 'Authorization: Bearer token-value' client = Elasticsearch( "https://localhost:9200", bearer_auth=os.environ["ELASTIC_TOKEN"] ) ``` ::: :::{tab-item} Async Python :sync: async ```python import os from elasticsearch import AsyncElasticsearch # Adds the HTTP header 'Authorization: Bearer token-value' client = AsyncElasticsearch( "https://localhost:9200", bearer_auth=os.environ["ELASTIC_TOKEN"] ) ``` ::: :::: ### API Key authentication [auth-apikey] You can configure the client to use {{es}}'s API Key for connecting to your cluster. These can be generated through the [Elasticsearch Create API key API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key) or [Kibana Stack Management](docs-content://deploy-manage/api-keys/elasticsearch-api-keys.md#create-api-key). ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch import Elasticsearch # Adds the HTTP header 'Authorization: ApiKey ' client = Elasticsearch( "https://localhost:9200", ca_certs="/path/to/http_ca.crt", api_key=os.environ["ELASTIC_API_KEY"], ) ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch import AsyncElasticsearch # Adds the HTTP header 'Authorization: ApiKey ' client = AsyncElasticsearch( "https://localhost:9200", ca_certs="/path/to/http_ca.crt", api_key=os.environ["ELASTIC_API_KEY"], ) ``` ::: :::: ## Using the Client in a Function-as-a-Service Environment [connecting-faas] This section illustrates the best practices for leveraging the {{es}} client in a Function-as-a-Service (FaaS) environment. The most influential optimization is to initialize the client outside of the function, the global scope. This practice does not only improve performance but also enables background functionality as – for example – [sniffing](https://www.elastic.co/blog/elasticsearch-sniffing-best-practices-what-when-why-how). The following examples provide a skeleton for the best practices. ::::{important} The async client shouldn’t be used within Function-as-a-Service as a new event loop must be started for each invocation. Instead the synchronous `Elasticsearch` client is recommended. :::: ### GCP Cloud Functions [connecting-faas-gcp] ```python import os from elasticsearch import Elasticsearch # Client initialization client = Elasticsearch( cloud_id="deployment-name:ABCD...", api_key=os.environ["ELASTIC_API_KEY"] ) def main(request): # Use the client client.search(index=..., query={"match_all": {}}) ``` ### AWS Lambda [connecting-faas-aws] ```python import os from elasticsearch import Elasticsearch # Client initialization client = Elasticsearch( cloud_id="deployment-name:ABCD...", api_key=os.environ["ELASTIC_API_KEY"] ) def main(event, context): # Use the client client.search(index=..., query={"match_all": {}}) ``` ### Azure Functions [connecting-faas-azure] ```python import azure.functions as func import os from elasticsearch import Elasticsearch # Client initialization client = Elasticsearch( cloud_id="deployment-name:ABCD...", api_key=os.environ["ELASTIC_API_KEY"] ) def main(request: func.HttpRequest) -> func.HttpResponse: # Use the client client.search(index=..., query={"match_all": {}}) ``` Resources used to assess these recommendations: * [GCP Cloud Functions: Tips & Tricks](https://cloud.google.com/functions/docs/bestpractices/tips#use_global_variables_to_reuse_objects_in_future_invocations) * [Best practices for working with AWS Lambda functions](https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html) * [Azure Functions Python developer guide](https://docs.microsoft.com/en-us/azure/azure-functions/functions-reference-python?tabs=azurecli-linux%2Capplication-level#global-variables) * [AWS Lambda: Comparing the effect of global scope](https://docs.aws.amazon.com/lambda/latest/operatorguide/global-scope.html) --- ## File: docs/reference/dsl_configuration.md --- mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/_configuration.html navigation_title: Configuration --- # Python DSL configuration for {{es}} [_configuration] There are several ways to configure connections for the library. The easiest and most useful approach is to define one default connection that can be used every time an API call is made without explicitly passing in other connections. ::::{note} Unless you want to access multiple clusters from your application, it is highly recommended that you use the `create_connection` method and all operations will use that connection automatically. :::: ## Default connection [_default_connection] To define a default connection that can be used globally, use the `connections` module and the `create_connection` method like this: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch.dsl import connections connections.create_connection(hosts=['https://localhost:9200'], request_timeout=20) ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch.dsl import async_connections async_connections.create_connection(hosts=['https://localhost:9200'], request_timeout=20) ``` ::: :::: ### Single connection with an alias [_single_connection_with_an_alias] You can define the `alias` or name of a connection so you can efficiently refer to it later. The default value for `alias` is `default`. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch.dsl import connections connections.create_connection(alias='my_new_connection', hosts=['https://localhost:9200'], request_timeout=60) ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch.dsl import async_connections async_connections.create_connection(alias='my_new_connection', hosts=['https://localhost:9200'], request_timeout=60) ``` ::: :::: Additional keyword arguments (`hosts` and `timeout` in our example) will be passed to the `Elasticsearch` class from `elasticsearch-py`. To see all possible configuration options refer to the [documentation](https://elasticsearch-py.readthedocs.io/en/latest/api/elasticsearch.html). ## Multiple clusters [_multiple_clusters] You can define multiple connections to multiple clusters at the same time using the `configure` method: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch.dsl import connections connections.configure( default={'hosts': 'https://localhost:9200'}, dev={ 'hosts': ['https://esdev1.example.com:9200'], 'sniff_on_start': True } ) ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch.dsl import async_connections async_connections.configure( default={'hosts': 'https://localhost:9200'}, dev={ 'hosts': ['https://esdev1.example.com:9200'], 'sniff_on_start': True } ) ``` ::: :::: Such connections will be constructed lazily when requested for the first time. You can alternatively define multiple connections by adding them one by one as shown in the following example: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python # if you have configuration options to be passed to Elasticsearch.__init__ # this also shows creating a connection with the alias 'qa' connections.create_connection('qa', hosts=['esqa1.example.com'], sniff_on_start=True) # if you already have an Elasticsearch instance ready connections.add_connection('another_qa', my_client) ``` ::: :::{tab-item} Async Python :sync: async ```python # if you have configuration options to be passed to Elasticsearch.__init__ # this also shows creating a connection with the alias 'qa' async_connections.create_connection('qa', hosts=['esqa1.example.com'], sniff_on_start=True) # if you already have an Elasticsearch instance ready async_connections.add_connection('another_qa', my_client) ``` ::: :::: ### Using aliases [_using_aliases] When using multiple connections, you can refer to them using the string alias specified when you created the connection. This example shows how to use an alias to a connection: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = Search(using='qa') ``` ::: :::{tab-item} Async Python :sync: async ```python s = AsyncSearch(using='qa') ``` ::: :::: A `KeyError` will be raised if there is no connection registered with that alias. ## Manual [_manual] If you don’t want to supply a global configuration, you can always pass in your own connection as an instance of `elasticsearch.Elasticsearch` with the parameter `using` wherever it is accepted like this: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = Search(using=Elasticsearch('https://localhost:9200')) ``` ::: :::{tab-item} Async Python :sync: async ```python s = AsyncSearch(using=AsyncElasticsearch('https://localhost:9200')) ``` ::: :::: You can even use this approach to override any connection the object might be already associated with: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = s.using(Elasticsearch('https://otherhost:9200')) ``` ::: :::{tab-item} Async Python :sync: async ```python s = s.using(AsyncElasticsearch('https://otherhost:9200')) ``` ::: :::: ::::{note} When using the `dsl` module, it is highly recommended that you use the built-in serializer (`elasticsearch.dsl.serializer.serializer`) to ensure your objects are correctly serialized into `JSON` every time. The `create_connection` method that is described here (and that the `configure` method uses under the hood) will do that automatically for you, unless you explicitly specify your own serializer. The built-in serializer also allows you to serialize your own objects - define a `to_dict()` method on your objects and that method will be automatically called when serializing your custom objects to `JSON`. :::: --- ## File: docs/reference/dsl_examples.md --- mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/_examples.html navigation_title: Examples --- # {{es}} Python DSL examples [_examples] Refer to the [DSL examples](https://github.com/elastic/elasticsearch-py/tree/master/examples/dsl) directory to see some complex examples using the DSL module. --- ## File: docs/reference/dsl_how_to_guides.md --- mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/_how_to_guides.html --- # How-To Guides [_how_to_guides] ## Search DSL [search_dsl] ### The `Search` object [_the_search_object] The `Search` object represents the entire search request: * queries * filters * aggregations * k-nearest neighbor searches * sort * pagination * highlighting * suggestions * collapsing * additional parameters * associated client The API is designed to be chainable. With the exception of the aggregations functionality this means that the `Search` object is immutable -all changes to the object will result in a shallow copy being created which contains the changes. You can safely pass the `Search` object to foreign code without fear of it modifying your objects as long as it sticks to the `Search` object APIs. You can pass an instance of the [elasticsearch client](https://elasticsearch-py.readthedocs.io/) when instantiating the `Search` object: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch import Elasticsearch from elasticsearch.dsl import Search client = Elasticsearch() s = Search(using=client) ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch import AsyncElasticsearch from elasticsearch.dsl import AsyncSearch client = AsyncElasticsearch() s = AsyncSearch(using=client) ``` ::: :::: You can also define the client at a later time (for more options see the `configuration` chapter): ```python s = s.using(client) ``` ::::{note} All methods return a *copy* of the object, making it safe to pass to outside code. :::: The API is chainable, allowing you to combine multiple method calls in one statement: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = Search().using(client).query(Match("title", "python")) ``` ::: :::{tab-item} Async Python :sync: async ```python s = AsyncSearch().using(client).query(Match("title", "python")) ``` ::: :::: To send the request to Elasticsearch: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python response = s.execute() ``` ::: :::{tab-item} Async Python :sync: async ```python response = await s.execute() ``` ::: :::: If you want to iterate over the hits returned by your search you can iterate over the `Search` or `AsyncSearch` objects: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python for hit in s: print(hit.title) ``` ::: :::{tab-item} Async Python :sync: async ```python async for hit in s: print(hit.title) ``` ::: :::: Search results will be cached. Subsequent calls to `execute` or trying to iterate over an already executed `Search` object will not trigger additional requests being sent to Elasticsearch. To force a new request to be issued specify `ignore_cache=True` when calling `execute`. For debugging purposes you can serialize the `Search` object to a `dict` with the raw Elasticsearch request: ```python print(s.to_dict()) ``` #### Delete By Query [_delete_by_query] You can delete the documents matching a search by calling `delete` on the `Search` object instead of `execute` like this: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = Search(index='i').query(Match("title", "python")) response = s.delete() ``` ::: :::{tab-item} Async Python :sync: async ```python s = AsyncSearch(index='i').query(Match("title", "python")) response = await s.delete() ``` ::: :::: To pass [deletion parameters](https://elasticsearch-py.readthedocs.io/en/latest/api/elasticsearch.html#elasticsearch.Elasticsearch.delete_by_query) in your query, you can add them by calling ``params`` on the ``Search`` object before ``delete`` like this: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = Search(index='i').query("match", title="python") s = s.params(ignore_unavailable=False, wait_for_completion=True) response = s.delete() ``` ::: :::{tab-item} Async Python :sync: async ```python s = AsyncSearch(index='i').query("match", title="python") s = s.params(ignore_unavailable=False, wait_for_completion=True) response = await s.delete() ``` ::: :::: #### Queries [_queries] The `elasticsearch.dsl.query` module provides classes for all Elasticsearch query types. These classes accept keyword arguments in their constructors, which are serialized to the appropriate format to be sent to Elasticsearch. There is a clear one-to-one mapping between the raw query and its equivalent class-based version: ```python >>> from elasticsearch.dsl.query import MultiMatch, Match >>> q = MultiMatch(query='python django', fields=['title', 'body']) >>> q.to_dict() {'multi_match': {'query': 'python django', 'fields': ['title', 'body']}} >>> q = Match("title", {"query": "web framework", "type": "phrase"}) >>> q.to_dict() {'match': {'title': {'query': 'web framework', 'type': 'phrase'}}} ``` An alternative to the class-based queries is to use the `Q` shortcut, passing a query name followed by its parameters, or the raw query as a `dict`: ```python from elasticsearch.dsl import Q Q("multi_match", query='python django', fields=['title', 'body']) Q({"multi_match": {"query": "python django", "fields": ["title", "body"]}}) ``` To add a query to the `Search` object, use the `.query()` method. This works with class-based or `Q` queries: ```python q = Q("multi_match", query='python django', fields=['title', 'body']) s = s.query(q) ``` As a shortcut the `query()` method also accepts all the parameters of the `Q` shortcut directly: ```python s = s.query("multi_match", query='python django', fields=['title', 'body']) ``` If you already have a query object, or a `dict` representing one, you can assign it to the `query` attribute of a `Search` object to add it to it, replacing any previously configured queries: ```python s.query = Q('bool', must=[Q('match', title='python'), Q('match', body='best')]) ``` #### Dotted fields [_dotted_fields] Sometimes you want to refer to a field within another field, either as a multi-field (`title.keyword`) or in a structured `json` document like `address.city`. This is not a problem when using class-based queries, but when working without classes it is often required to pass field names as keyword arguments. To make this easier, you can use `__` (double underscore) in place of a dot in a keyword argument: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = Search() s = s.filter('term', category__keyword='Python') s = s.query('match', address__city='prague') ``` ::: :::{tab-item} Async Python :sync: async ```python s = AsyncSearch() s = s.filter('term', category__keyword='Python') s = s.query('match', address__city='prague') ``` ::: :::: Alternatively you can use Python’s keyword argument unpacking: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = Search() s = s.filter('term', **{'category.keyword': 'Python'}) s = s.query('match', **{'address.city': 'prague'}) ``` ::: :::{tab-item} Async Python :sync: async ```python s = AsyncSearch() s = s.filter('term', **{'category.keyword': 'Python'}) s = s.query('match', **{'address.city': 'prague'}) ``` ::: :::: #### Query combination [_query_combination] Query objects can be combined using logical operators `|`, `&` and `~`: ```python >>> q = Match("title", "python") | Match("title", "django") >>> q.to_dict() {'bool': {'should': [{'match': {'title': 'python'}}, {'match': {'title': 'django'}}]}} >>> q = Match("title", "python") & Match("title", "django") >>> q.to_dict() {'bool': {'must': [{'match': {'title': 'python'}}, {'match': {'title': 'django'}}]}} >>> q = ~Match("title", "python") >>> q.to_dict() {'bool': {'must_not': [{'match': {'title': 'python'}}]}} ``` When you call the `.query()` method multiple times, the `&` operator will be used internally to combine all the queries: ```python s = s.query().query() print(s.to_dict()) # {"query": {"bool": {...}}} ``` If you want to have precise control over the query form, use the `Q` shortcut to directly construct the combined query: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python q = Q('bool', must=[Q('match', title='python')], should=[Q(...), Q(...)], minimum_should_match=1 ) s = Search().query(q) ``` ::: :::{tab-item} Async Python :sync: async ```python q = Q('bool', must=[Q('match', title='python')], should=[Q(...), Q(...)], minimum_should_match=1 ) s = AsyncSearch().query(q) ``` ::: :::: #### Filters [_filters] If you want to add a query in a [filter context](docs-content://explore-analyze/query-filter/languages/querydsl.md) you can use the `filter()` method to make things easier: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch.dsl.query import Terms s = Search() s = s.filter(Terms("tags", ['search', 'python'])) ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch.dsl.query import Terms s = AsyncSearch() s = s.filter(Terms("tags", ['search', 'python'])) ``` ::: :::: Behind the scenes this will produce a `Bool` query and place the specified `terms` query into its `filter` branch, making it equivalent to: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch.dsl.query import Terms, Bool s = Search() s = s.query(Bool(filter=[Terms("tags", ["search", "python"])])) ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch.dsl.query import Terms, Bool s = AsyncSearch() s = s.query(Bool(filter=[Terms("tags", ["search", "python"])])) ``` ::: :::: If you want to use the `post_filter` element for faceted navigation, use the `.post_filter()` method. The `exclude()` method works like `filter()`, but it applies the query as negated: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = Search() s = s.exclude(Terms("tags", ['search', 'python'])) ``` ::: :::{tab-item} Async Python :sync: async ```python s = AsyncSearch() s = s.exclude(Terms("tags", ['search', 'python'])) ``` ::: :::: which is shorthand for: ```python s = s.query(Bool(filter=[~Terms("tags", ["search", "python"])])) ``` #### Aggregations [_aggregations] As with queries, there are classes that represent each aggregation type, all accessible through the `elasticsearch.dsl.aggs` module: ```python from elasticsearch.dsl import aggs a = aggs.Terms(field="tags") # {"terms": {"field": "tags"}} ``` It is also possible to define an aggregation using the `A` shortcut: ```python from elasticsearch.dsl import A A('terms', field='tags') ``` To nest aggregations, you can use the `.bucket()`, `.metric()` and `.pipeline()` methods: ```python a = aggs.Terms(field="category") # {'terms': {'field': 'category'}} a.metric("clicks_per_category", aggs.Sum(field="clicks")) \ .bucket("tags_per_category", aggs.Terms(field="tags")) # { # 'terms': {'field': 'category'}, # 'aggs': { # 'clicks_per_category': {'sum': {'field': 'clicks'}}, # 'tags_per_category': {'terms': {'field': 'tags'}} # } # } ``` To add aggregations to the `Search` object, use the `.aggs` property, which acts as a top-level aggregation: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = Search() a = aggs.Terms(field="category") s.aggs.bucket("category_terms", a) # { # 'aggs': { # 'category_terms': { # 'terms': { # 'field': 'category' # } # } # } # } ``` ::: :::{tab-item} Async Python :sync: async ```python s = AsyncSearch() a = aggs.Terms(field="category") s.aggs.bucket("category_terms", a) # { # 'aggs': { # 'category_terms': { # 'terms': { # 'field': 'category' # } # } # } # } ``` ::: :::: or ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = Search() s.aggs.bucket("articles_per_day", aggs.DateHistogram(field="publish_date", interval="day")) \ .metric("clicks_per_day", aggs.Sum(field="clicks")) \ .pipeline("moving_click_average", aggs.MovingAvg(buckets_path="clicks_per_day")) \ .bucket("tags_per_day", aggs.Terms(field="tags")) s.to_dict() # { # "aggs": { # "articles_per_day": { # "date_histogram": { "interval": "day", "field": "publish_date" }, # "aggs": { # "clicks_per_day": { "sum": { "field": "clicks" } }, # "moving_click_average": { "moving_avg": { "buckets_path": "clicks_per_day" } }, # "tags_per_day": { "terms": { "field": "tags" } } # } # } # } # } ``` ::: :::{tab-item} Async Python :sync: async ```python s = AsyncSearch() s.aggs.bucket("articles_per_day", aggs.DateHistogram(field="publish_date", interval="day")) \ .metric("clicks_per_day", aggs.Sum(field="clicks")) \ .pipeline("moving_click_average", aggs.MovingAvg(buckets_path="clicks_per_day")) \ .bucket("tags_per_day", aggs.Terms(field="tags")) s.to_dict() # { # "aggs": { # "articles_per_day": { # "date_histogram": { "interval": "day", "field": "publish_date" }, # "aggs": { # "clicks_per_day": { "sum": { "field": "clicks" } }, # "moving_click_average": { "moving_avg": { "buckets_path": "clicks_per_day" } }, # "tags_per_day": { "terms": { "field": "tags" } } # } # } # } # } ``` ::: :::: You can access an existing bucket by its name: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = Search() s.aggs.bucket("per_category", aggs.Terms(field="category")) s.aggs["per_category"].metric("clicks_per_category", aggs.Sum(field="clicks")) s.aggs["per_category"].bucket("tags_per_category", aggs.Terms(field="tags")) ``` ::: :::{tab-item} Async Python :sync: async ```python s = AsyncSearch() s.aggs.bucket("per_category", aggs.Terms(field="category")) s.aggs["per_category"].metric("clicks_per_category", aggs.Sum(field="clicks")) s.aggs["per_category"].bucket("tags_per_category", aggs.Terms(field="tags")) ``` ::: :::: ::::{note} When chaining multiple aggregations, there is a difference between what `.bucket()` and `.metric()` methods return - `.bucket()` returns the newly defined bucket while `.metric()` returns its parent bucket to allow further chaining. :::: As opposed to other methods on the `Search` objects, aggregations are defined in-place, without returning a new copy. #### K-Nearest Neighbor Searches [_k_nearest_neighbor_searches] To issue a kNN search, use the `.knn()` method: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = Search() vector = get_embedding("search text") s = s.knn( field="embedding", k=5, num_candidates=10, query_vector=vector ) ``` ::: :::{tab-item} Async Python :sync: async ```python s = AsyncSearch() vector = get_embedding("search text") s = s.knn( field="embedding", k=5, num_candidates=10, query_vector=vector ) ``` ::: :::: The `field`, `k` and `num_candidates` arguments can be given as positional or keyword arguments and are required. In addition to these, `query_vector` or `query_vector_builder` must be given as well. The `.knn()` method can be invoked multiple times to include multiple kNN searches in the request. #### Sorting [_sorting] To specify sorting order, use the `.sort()` method: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = Search().sort( 'category', '-title', {"lines" : {"order" : "asc", "mode" : "avg"}} ) ``` ::: :::{tab-item} Async Python :sync: async ```python s = AsyncSearch().sort( 'category', '-title', {"lines" : {"order" : "asc", "mode" : "avg"}} ) ``` ::: :::: It accepts positional arguments which can be either strings or dictionaries. String value is a field name, optionally prefixed by the `-` sign to specify a descending order. To reset the sorting, call the method with no arguments: ```python s = s.sort() ``` #### Pagination [_pagination] To specify the from/size parameters, apply the standard Python slicing operator on the `Search` instance: ```python s = s[10:20] # {"from": 10, "size": 10} s = s[:20] # {"size": 20} s = s[10:] # {"from": 10} s = s[10:20][2:] # {"from": 12, "size": 8} ``` If you want to access all the documents matched by your query you can use the `scan` method which uses the scan/scroll elasticsearch API: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python for hit in s.scan(): print(hit.title) ``` ::: :::{tab-item} Async Python :sync: async ```python async for hit in s.scan(): print(hit.title) ``` ::: :::: In this case, the results won’t be sorted. #### Highlighting [_highlighting] To set common attributes for highlighting use the `highlight_options` method: ```python s = s.highlight_options(order='score') ``` Enabling highlighting for individual fields is done using the `highlight` method: ```python s = s.highlight('title') # or, including parameters: s = s.highlight('title', fragment_size=50) ``` The fragments in the response will then be available on each `Result` object as `.meta.highlight.FIELD` which will contain the list of fragments: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python response = s.execute() for hit in response: for fragment in hit.meta.highlight.title: print(fragment) ``` ::: :::{tab-item} Async Python :sync: async ```python response = await s.execute() for hit in response: for fragment in hit.meta.highlight.title: print(fragment) ``` ::: :::: #### Suggestions [_suggestions] To specify a suggest request on your `Search` object use the `suggest` method: ```python # check for correct spelling s = s.suggest('my_suggestion', 'pyhton', term={'field': 'title'}) ``` The first argument is the name of the suggestions (name under which it will be returned), second is the actual text you wish the suggester to work on and the keyword arguments will be added to the suggest’s json as-is which means that it should be one of `term`, `phrase` or `completion` to indicate which type of suggester should be used. #### Collapsing [_collapsing] To collapse search results use the `collapse` method on your `Search` object: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = Search().query(Match("message", "GET /search")) # collapse results by user_id s = s.collapse("user_id") ``` ::: :::{tab-item} Async Python :sync: async ```python s = AsyncSearch().query(Match("message", "GET /search")) # collapse results by user_id s = s.collapse("user_id") ``` ::: :::: The top hits will only include one result per `user_id`. You can also expand each collapsed top hit with the `inner_hits` parameter, `max_concurrent_group_searches` being the number of concurrent requests allowed to retrieve the inner hits per group: ```python inner_hits = {"name": "recent_search", "size": 5, "sort": [{"@timestamp": "desc"}]} s = s.collapse("user_id", inner_hits=inner_hits, max_concurrent_group_searches=4) ``` #### More Like This Query [_more_like_this_query] To use Elasticsearch’s `more_like_this` functionality, you can use the MoreLikeThis query type. A simple example is below ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch.dsl.query import MoreLikeThis from elasticsearch.dsl import Search my_text = 'I want to find something similar' s = Search() # We're going to match based only on two fields, in this case text and title s = s.query(MoreLikeThis(like=my_text, fields=['text', 'title'])) # You can also exclude fields from the result to make the response quicker in the normal way s = s.source(exclude=["text"]) response = s.execute() for hit in response: print(hit.title) ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch.dsl.query import MoreLikeThis from elasticsearch.dsl import AsyncSearch async def example(): my_text = 'I want to find something similar' s = AsyncSearch() # We're going to match based only on two fields, in this case text and title s = s.query(MoreLikeThis(like=my_text, fields=['text', 'title'])) # You can also exclude fields from the result to make the response quicker in the normal way s = s.source(exclude=["text"]) response = await s.execute() for hit in response: print(hit.title) ``` ::: :::: #### Extra properties and parameters [_extra_properties_and_parameters] To set extra properties of the search request, use the `.extra()` method. This can be used to define keys in the body that cannot be defined using a specific API method like `explain` or `search_after`: ```python s = s.extra(explain=True) ``` To set query parameters, use the `.params()` method: ```python s = s.params(routing="42") ``` If you need to limit the fields being returned by elasticsearch, use the `source()` method: ```python # only return the selected fields s = s.source(['title', 'body']) # don't return any fields, just the metadata s = s.source(False) # explicitly include/exclude fields s = s.source(includes=["title"], excludes=["user.*"]) # reset the field selection s = s.source(None) ``` #### Serialization and Deserialization [_serialization_and_deserialization] The search object can be serialized into a dictionary by using the `.to_dict()` method. You can also create a `Search` object from a `dict` using the `from_dict` class method. This will create a new `Search` object and populate it using the data from the dict: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = Search.from_dict({"query": {"match": {"title": "python"}}}) ``` ::: :::{tab-item} Async Python :sync: async ```python s = AsyncSearch.from_dict({"query": {"match": {"title": "python"}}}) ``` ::: :::: If you wish to modify an existing `Search` object, overriding it’s properties, instead use the `update_from_dict` method that alters an instance **in-place**: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = Search(index='i') s.update_from_dict({"query": {"match": {"title": "python"}}, "size": 42}) ``` ::: :::{tab-item} Async Python :sync: async ```python s = AsyncSearch(index='i') s.update_from_dict({"query": {"match": {"title": "python"}}, "size": 42}) ``` ::: :::: ### Response [_response] You can execute your search by calling the `.execute()` method that will return a `Response` object. The `Response` object allows you access to any key from the response dictionary using attribute access. It also provides some convenient helpers: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python response = s.execute() print(response.success()) # True print(response.took) # 12 print(response.hits.total.relation) # eq print(response.hits.total.value) # 142 print(response.suggest.my_suggestions) ``` ::: :::{tab-item} Async Python :sync: async ```python await response = s.execute() print(response.success()) # True print(response.took) # 12 print(response.hits.total.relation) # eq print(response.hits.total.value) # 142 print(response.suggest.my_suggestions) ``` ::: :::: If you want to inspect the contents of the `response` objects, use its `to_dict` method to get access to the raw data for pretty printing. #### Hits [_hits] To access the hits returned by the search, use the `hits` property or iterate over the `Response` object: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python response = s.execute() print(f"Total {response.hits.total} hits found.") for h in response: print(h.title, h.body) ``` ::: :::{tab-item} Async Python :sync: async ```python response = await s.execute() print(f"Total {response.hits.total} hits found.") for h in response: print(h.title, h.body) ``` ::: :::: ::::{note} If you are only seeing partial results (for example 10000 or even 10 results), consider using the option `s.extra(track_total_hits=True)` to get a full hit count. :::: #### Result [_result] The individual hits is wrapped in a convenience class that allows attribute access to the keys in the returned dictionary. All the metadata for the results are accessible using `meta` (without the leading `_`): ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python response = s.execute() h = response.hits[0] print(f"/{h.meta.index}/{h.meta.doc_type}/{h.meta.id} returned with score {h.meta.score}") ``` ::: :::{tab-item} Async Python :sync: async ```python response = await s.execute() h = response.hits[0] print(f"/{h.meta.index}/{h.meta.doc_type}/{h.meta.id} returned with score {h.meta.score}") ``` ::: :::: ::::{note} If your document has a field called `meta` you have to access it using the get item syntax: `hit['meta']`. :::: #### Aggregations [_aggregations_2] Aggregations are available through the `aggregations` property: ```python for tag in response.aggregations.per_tag.buckets: print(tag.key, tag.max_lines.value) ``` ### `MultiSearch` [_multisearch] If you need to execute multiple searches at the same time you can use the `MultiSearch` class which will use the `_msearch` API: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch.dsl import MultiSearch, Search from elasticsearch.dsl.query import Term ms = MultiSearch(index='blogs') ms = ms.add(Search().filter(Term("tags", "python"))) ms = ms.add(Search().filter(Term("tags", 'elasticsearch'))) responses = ms.execute() for response in responses: print("Results for query %r." % response._search.query) for hit in response: print(hit.title) ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch.dsl import AsyncMultiSearch, AsyncSearch from elasticsearch.dsl.query import Term async def example(): ms = AsyncMultiSearch(index='blogs') ms = ms.add(AsyncSearch().filter(Term("tags", "python"))) ms = ms.add(AsyncSearch().filter(Term("tags", 'elasticsearch'))) responses = await ms.execute() for response in responses: print("Results for query %r." % response._search.query) for hit in response: print(hit.title) ``` ::: :::: ### `EmptySearch` [_emptysearch] The `EmptySearch` class can be used as a fully compatible version of `Search` that will return no results, regardless of any queries configured. ## Persistence [_persistence_2] You can use the DSL module to define your mappings and a basic persistent layer for your application. For more comprehensive examples have a look at the [DSL examples](https://github.com/elastic/elasticsearch-py/tree/main/examples/dsl) directory in the repository. ### Document [doc_type] If you want to create a model-like wrapper around your documents, use the `Document` class (or the equivalent `AsyncDocument` for asynchronous applications). It can also be used to create all the necessary mappings and settings in Elasticsearch (see [Document life cycle](#life-cycle) below for details). ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from datetime import datetime from elasticsearch.dsl import AsyncDocument, Boolean, InnerDoc, Completion, Keyword, Text, analyzer html_strip = analyzer('html_strip', tokenizer="standard", filter=["standard", "lowercase", "stop", "snowball"], char_filter=["html_strip"] ) class Comment(InnerDoc): author: str = Text(fields={'raw': Keyword()}) content: str = Text(analyzer='snowball') created_at: datetime def age(self): return datetime.now() - self.created_at class Post(Document): title: str title_suggest: str = Completion() created_at: datetime published: bool category: str = Text( analyzer=html_strip, fields={'raw': Keyword()} ) comments: Comment class Index: name = 'blog' def add_comment(self, author, content): self.comments.append( Comment(author=author, content=content, created_at=datetime.now())) def save(self, ** kwargs): self.created_at = datetime.now() return super().save(** kwargs) ``` ::: :::{tab-item} Async Python :sync: async ```python from datetime import datetime from elasticsearch.dsl import AsyncDocument, Boolean, InnerDoc, Completion, Keyword, Text, analyzer html_strip = analyzer('html_strip', tokenizer="standard", filter=["standard", "lowercase", "stop", "snowball"], char_filter=["html_strip"] ) class Comment(InnerDoc): author: str = Text(fields={'raw': Keyword()}) content: str = Text(analyzer='snowball') created_at: datetime def age(self): return datetime.now() - self.created_at class Post(AsyncDocument): title: str title_suggest: str = Completion() created_at: datetime published: bool category: str = Text( analyzer=html_strip, fields={'raw': Keyword()} ) comments: Comment class Index: name = 'blog' def add_comment(self, author, content): self.comments.append( Comment(author=author, content=content, created_at=datetime.now())) async def save(self, ** kwargs): self.created_at = datetime.now() return await super().save(** kwargs) ``` ::: :::: #### Data types [_data_types] The `Document` instances can use native Python types such as `str` and `datetime` for its attributes. In case of `Object` or `Nested` fields an instance of the `InnerDoc` subclass is used, as in the `add_comment` method in the above example, where we are creating an instance of the `Comment` class. There are also specific type classes that were created to make working with some field types easier, for example the `Range` object used in any of the [range fields](elasticsearch://reference/elasticsearch/mapping-reference/range.md): ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch.dsl import Document, DateRange, Keyword, Range class RoomBooking(Document): room = Keyword() dates = DateRange() rb = RoomBooking( room='Conference Room II', dates=Range( gte=datetime(2018, 11, 17, 9, 0, 0), lt=datetime(2018, 11, 17, 10, 0, 0) ) ) # Range supports the in operator correctly: datetime(2018, 11, 17, 9, 30, 0) in rb.dates # True # you can also get the limits and whether they are inclusive or exclusive: rb.dates.lower # datetime(2018, 11, 17, 9, 0, 0), True rb.dates.upper # datetime(2018, 11, 17, 10, 0, 0), False # empty range is unbounded Range().lower # None, False ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch.dsl import AsyncDocument, DateRange, Keyword, Range class RoomBooking(AsyncDocument): room = Keyword() dates = DateRange() rb = RoomBooking( room='Conference Room II', dates=Range( gte=datetime(2018, 11, 17, 9, 0, 0), lt=datetime(2018, 11, 17, 10, 0, 0) ) ) # Range supports the in operator correctly: datetime(2018, 11, 17, 9, 30, 0) in rb.dates # True # you can also get the limits and whether they are inclusive or exclusive: rb.dates.lower # datetime(2018, 11, 17, 9, 0, 0), True rb.dates.upper # datetime(2018, 11, 17, 10, 0, 0), False # empty range is unbounded Range().lower # None, False ``` ::: :::: #### Python Type Hints [_python_type_hints] Document fields can be defined using standard Python type hints if desired. Here are some examples: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from typing import Optional class Post(Document): title: str # same as title = Text(required=True) created_at: Optional[datetime] # same as created_at = Date(required=False) published: bool # same as published = Boolean(required=True) ``` ::: :::{tab-item} Async Python :sync: async ```python from typing import Optional class Post(AsyncDocument): title: str # same as title = Text(required=True) created_at: Optional[datetime] # same as created_at = Date(required=False) published: bool # same as published = Boolean(required=True) ``` ::: :::: :::::{note} When using `Field` subclasses such as `Text`, `Date` and `Boolean` to define attributes, these classes must be given in the right-hand side. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python class Post(Document): title = Text() # correct subtitle: Text # incorrect ``` ::: :::{tab-item} Async Python :sync: async ```python class Post(AsyncDocument): title = Text() # correct subtitle: Text # incorrect ``` ::: :::: Using a `Field` subclass as a Python type hint will result in errors. ::::: Python types are mapped to their corresponding `Field` types according to the following table: | Python type | DSL field | | --- | --- | | `str` | `Text(required=True)` | | `bool` | `Boolean(required=True)` | | `int` | `Integer(required=True)` | | `float` | `Float(required=True)` | | `bytes` | `Binary(required=True)` | | `datetime` | `Date(required=True)` | | `date` | `Date(format="yyyy-MM-dd", required=True)` | To type a field as optional, the standard `Optional` modifier from the Python `typing` package can be used. When using Python 3.10 or newer, "pipe" syntax can also be used, by adding `| None` to a type. The `List` modifier can be added to a field to convert it to an array, similar to using the `multi=True` argument on the `Field` object. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from typing import Optional, List class MyDoc(Document): pub_date: Optional[datetime] # same as pub_date = Date() middle_name: str | None # same as middle_name = Text() authors: List[str] # same as authors = Text(multi=True, required=True) comments: Optional[List[str]] # same as comments = Text(multi=True) ``` ::: :::{tab-item} Async Python :sync: async ```python from typing import Optional, List class MyDoc(AsyncDocument): pub_date: Optional[datetime] # same as pub_date = Date() middle_name: str | None # same as middle_name = Text() authors: List[str] # same as authors = Text(multi=True, required=True) comments: Optional[List[str]] # same as comments = Text(multi=True) ``` ::: :::: A field can also be given a type hint of an `InnerDoc` subclass, in which case it becomes an `Object` field of that class. When the `InnerDoc` subclass is wrapped with `List`, a `Nested` field is created instead. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from typing import List class Address(InnerDoc): ... class Comment(InnerDoc): ... class Post(Document): address: Address # same as address = Object(Address, required=True) comments: List[Comment] # same as comments = Nested(Comment, required=True) ``` ::: :::{tab-item} Async Python :sync: async ```python from typing import List class Address(InnerDoc): ... class Comment(InnerDoc): ... class Post(AsyncDocument): address: Address # same as address = Object(Address, required=True) comments: List[Comment] # same as comments = Nested(Comment, required=True) ``` ::: :::: Unfortunately it is impossible to have Python type hints that uniquely identify every possible Elasticsearch `Field` type. To choose a type that is different than the one that is assigned according to the table above, the desired `Field` instance can be added explicitly as a right-side assignment in the field declaration. The next example creates a field that is typed as `Optional[str]`, but is mapped to `Keyword` instead of `Text`: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python class MyDocument(Document): category: Optional[str] = Keyword() ``` ::: :::{tab-item} Async Python :sync: async ```python class MyDocument(AsyncDocument): category: Optional[str] = Keyword() ``` ::: :::: This form can also be used when additional options need to be given to initialize the field, such as when using custom analyzer settings: ```python class Comment(InnerDoc): content: str = Text(analyzer='snowball') ``` When using type hints as above, subclasses of `Document` and `InnerDoc` inherit some of the behaviors associated with Python dataclasses, as defined by [PEP 681](https://peps.python.org/pep-0681/) and the [dataclass_transform decorator](https://typing.readthedocs.io/en/latest/spec/dataclasses.html#dataclass-transform). To add per-field dataclass options such as `default` or `default_factory`, the `mapped_field()` wrapper can be used on the right side of a typed field declaration: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python class MyDocument(Document): title: str = mapped_field(default="no title") created_at: datetime = mapped_field(default_factory=datetime.now) published: bool = mapped_field(default=False) category: str = mapped_field(Keyword(), default="general") ``` ::: :::{tab-item} Async Python :sync: async ```python class MyDocument(AsyncDocument): title: str = mapped_field(default="no title") created_at: datetime = mapped_field(default_factory=datetime.now) published: bool = mapped_field(default=False) category: str = mapped_field(Keyword(), default="general") ``` ::: :::: The `mapped_field()` wrapper function can optionally be given an explicit field type instance as a first positional argument, as the `category` field does in the example above to be defined as `Keyword` instead of the `Text` default. Static type checkers such as [mypy](https://mypy-lang.org/) and [pyright](https://github.com/microsoft/pyright) can use the type hints and the dataclass-specific options added to the `mapped_field()` function to improve type inference and provide better real-time code completion and suggestions in IDEs. One situation in which type checkers can’t infer the correct type is when using fields as class attributes. Consider the following example: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python class MyDocument(Document): title: str doc = MyDocument() # doc.title is typed as "str" (correct) # MyDocument.title is also typed as "str" (incorrect) ``` ::: :::{tab-item} Async Python :sync: async ```python class MyDocument(AsyncDocument): title: str doc = MyDocument() # doc.title is typed as "str" (correct) # MyDocument.title is also typed as "str" (incorrect) ``` ::: :::: To help type checkers correctly identify class attributes as such, the `M` generic must be used as a wrapper to the type hint, as shown in the next examples: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch.dsl import M class MyDocument(Document): title: M[str] created_at: M[datetime] = mapped_field(default_factory=datetime.now) doc = MyDocument() # doc.title is typed as "str" # doc.created_at is typed as "datetime" # MyDocument.title is typed as "InstrumentedField" # MyDocument.created_at is typed as "InstrumentedField" ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch.dsl import M class MyDocument(AsyncDocument): title: M[str] created_at: M[datetime] = mapped_field(default_factory=datetime.now) doc = MyDocument() # doc.title is typed as "str" # doc.created_at is typed as "datetime" # MyDocument.title is typed as "InstrumentedField" # MyDocument.created_at is typed as "InstrumentedField" ``` ::: :::: The `M` type hint does not provide any runtime behavior and its use is not required, but it can be useful to remove spurious type errors in IDEs or type checking builds. The `InstrumentedField` objects returned when fields are accessed as class attributes are proxies for the field instances that can be used anywhere a field needs to be referenced, such as when specifying sort options in a `Search` object: ```python # sort by creation date descending, and title ascending s = MyDocument.search().sort(-MyDocument.created_at, MyDocument.title) ``` When specifying sorting order, the `+` and `-` unary operators can be used on the class field attributes to indicate ascending and descending order. Finally, it is also possible to define class attributes and request that they are ignored when building the Elasticsearch mapping. One way is to type attributes with the `ClassVar` annotation. Alternatively, the `mapped_field()` wrapper function accepts an `exclude` argument that can be set to `True`: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from typing import ClassVar class MyDoc(Document): title: M[str] created_at: M[datetime] = mapped_field(default_factory=datetime.now) my_var: ClassVar[str] # regular class variable, ignored by Elasticsearch anoter_custom_var: int = mapped_field(exclude=True) # also ignored by Elasticsearch ``` ::: :::{tab-item} Async Python :sync: async ```python from typing import ClassVar class MyDoc(AsyncDocument): title: M[str] created_at: M[datetime] = mapped_field(default_factory=datetime.now) my_var: ClassVar[str] # regular class variable, ignored by Elasticsearch anoter_custom_var: int = mapped_field(exclude=True) # also ignored by Elasticsearch ``` ::: :::: #### Note on dates [_note_on_dates] The DSL module will always respect the timezone information (or lack thereof) on the `datetime` objects passed in or stored in Elasticsearch. Elasticsearch itself interprets all datetimes with no timezone information as `UTC`. If you wish to reflect this in your python code, you can specify `default_timezone` when instantiating a `Date` field: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python class Post(Document): created_at = Date(default_timezone='UTC') ``` ::: :::{tab-item} Async Python :sync: async ```python class Post(AsyncDocument): created_at = Date(default_timezone='UTC') ``` ::: :::: In that case any `datetime` object passed in (or parsed from Elasticsearch) will be treated as if it were in `UTC` timezone. #### Custom field names By default, the `Document` and `AsyncDocument` classes use the names given to the field attributes as the field names in the Elasticsearch index. But sometimes it is necessary for the names of a field in Python and Elasticsearch to be different, such as when the Elasticsearch name is not a valid Python identifier. The following example shows how to define the Elasticsearch field `@timestamp`, used with data streams: :::{tab-item} Standard Python :sync: sync ```python class MyDoc(Document): timestamp: datetime = mapped_field(es_name='@timestamp') ``` ::: :::{tab-item} Async Python :sync: async ```python class MyDoc(AsyncDocument): timestamp: datetime = mapped_field(es_name='@timestamp') ``` ::: :::: If using `Field` subclasses to define attribute types, the `_es_name` argument can be passed with the desired Elasticsearch name: :::{tab-item} Standard Python :sync: sync ```python class MyDoc(Document): timestamp = Date(_es_name='@timestamp') ``` ::: :::{tab-item} Async Python :sync: async ```python class MyDoc(AsyncDocument): timestamp = Date(_es_name='@timestamp') ``` ::: :::: The conversion between the Python and Elasticsearch names happens automatically during serialization and deserialization of `Document` and `AsyncDocument` instances. The Elasticsearch field names must be used when sending requests outside of the document classes. Likewise, responses from Elasticsearch that are not deserialized to a document class will reference the Elasticsearch field names. #### Document life cycle [life-cycle] Before you first use the `Post` document type, you need to create the mappings in Elasticsearch. For that you can either use the `index` object or create the mappings directly by calling the `init` class method: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python # create the mappings in Elasticsearch Post.init() ``` ::: :::{tab-item} Async Python :sync: async ```python async def example(): # create the mappings in Elasticsearch await Post.init() ``` ::: :::: This code will typically be run in the setup for your application during a code deploy, similar to running database migrations. To create a new `Post` document, instantiate the class and pass in any fields you wish to set, you can then use standard attribute setting to change/add more fields. You are not limited to the fields defined explicitly: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python # instantiate the document first = Post(title='My First Blog Post, yay!', published=True) # assign some field values, can be values or lists of values first.category = ['everything', 'nothing'] # every document has an id in meta first.meta.id = 47 # save the document into the cluster first.save() ``` ::: :::{tab-item} Async Python :sync: async ```python async def example(): # instantiate the document first = Post(title='My First Blog Post, yay!', published=True) # assign some field values, can be values or lists of values first.category = ['everything', 'nothing'] # every document has an id in meta first.meta.id = 47 # save the document into the cluster await first.save() ``` ::: :::: All the metadata fields (`id`, `routing`, `index` and so on) can be accessed (and set) using a `meta` attribute or directly using the underscored variant: ```python post = Post(meta={'id': 42}) # prints 42 print(post.meta.id) # override default index post.meta.index = 'my-blog' ``` ::::{note} Having all metadata accessible through `meta` means that this name is reserved and you shouldn’t have a field called `meta` on your document. If you, however, need it you can still access the data using the get item (as opposed to attribute) syntax: `post['meta']`. :::: To retrieve an existing document use the `get` class method: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python # retrieve the document first = Post.get(id=42) # now we can call methods, change fields, ... first.add_comment('me', 'This is nice!') # and save the changes into the cluster again first.save() ``` ::: :::{tab-item} Async Python :sync: async ```python async def example(): # retrieve the document first = Post.get(id=42) # now we can call methods, change fields, ... first.add_comment('me', 'This is nice!') # and save the changes into the cluster again await first.save() ``` ::: :::: The [Update API](https://www.elastic.co/docs/api/doc/elasticsearch/v8/group/endpoint-document) can also be used via the `update` method. By default any keyword arguments, beyond the parameters of the API, will be considered fields with new values. Those fields will be updated on the local copy of the document and then sent over as partial document to be updated: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python # retrieve the document first = Post.get(id=42) # you can update just individual fields which will call the update API # and also update the document in place first.update(published=True, published_by='me') ``` ::: :::{tab-item} Async Python :sync: async ```python async def example(): # retrieve the document first = await Post.get(id=42) # you can update just individual fields which will call the update API # and also update the document in place await first.update(published=True, published_by='me') ``` ::: :::: In case you wish to use a `painless` script to perform the update you can pass in the script string as `script` or the `id` of a [stored script](docs-content://explore-analyze/scripting/modules-scripting-store-and-retrieve.md) via `script_id`. All additional keyword arguments to the `update` method will then be passed in as parameters of the script. The document will not be updated in place. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python # retrieve the document first = Post.get(id=42) # we execute a script in elasticsearch with additional kwargs being passed # as params into the script first.update(script='ctx._source.category.add(params.new_category)', new_category='testing') ``` ::: :::{tab-item} Async Python :sync: async ```python async def example(): # retrieve the document first = await Post.get(id=42) # we execute a script in elasticsearch with additional kwargs being passed # as params into the script await first.update(script='ctx._source.category.add(params.new_category)', new_category='testing') ``` ::: :::: If the document is not found in elasticsearch, an exception (`elasticsearch.NotFoundError`) will be raised. If you wish to return `None` instead just pass in `ignore=404` to suppress the exception: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python p = Post.get(id='not-in-es', ignore=404) if p is None: ... ``` ::: :::{tab-item} Async Python :sync: async ```python async def example(): p = await Post.get(id='not-in-es', ignore=404) if p is None: ... ``` ::: :::: When you wish to retrieve multiple documents at the same time by their `id` you can use the `mget` method: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python posts = Post.mget([42, 47, 256]) ``` ::: :::{tab-item} Async Python :sync: async ```python posts = await Post.mget([42, 47, 256]) ``` ::: :::: `mget` will, by default, raise a `NotFoundError` if any of the documents wasn’t found and `RequestError` if any of the document had resulted in error. You can control this behavior by setting parameters: * `raise_on_error`: If `True` (default) then any error will cause an exception to be raised. Otherwise all documents containing errors will be treated as missing. * `missing`: Can have three possible values: `'none'` (default), `'raise'` and `'skip'`. If a document is missing or errored it will either be replaced with `None`, an exception will be raised or the document will be skipped in the output list entirely. The index associated with the `Document` is accessible via the `_index` class property which gives you access to the `index` class. The `_index` attribute is also home to the `load_mappings` method which will update the mapping on the `Index` from elasticsearch. This is useful if you use dynamic mappings and want the class to be aware of those fields (for example if you wish the `Date` fields to be properly (de)serialized): ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python Post._index.load_mappings() ``` ::: :::{tab-item} Async Python :sync: async ```python await Post._index.load_mappings() ``` ::: :::: To delete a document, call its `delete` method: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python first = Post.get(id=42) first.delete() ``` ::: :::{tab-item} Async Python :sync: async ```python first = await Post.get(id=42) await first.delete() ``` ::: :::: #### Integration with Pydantic models ::::{warning} This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. :::: ::::{note} This feature is available in the Python Elasticsearch client starting with release 9.2.0. :::: Applications that define their data models using [Pydantic](https://docs.pydantic.dev/latest/) can combine these models with Elasticsearch DSL annotations. To take advantage of this option, Pydantic's `BaseModel` base class needs to be replaced with `BaseESModel` (or `AsyncBaseESModel` for asynchronous applications), and then the model can include type annotations for Pydantic and Elasticsearch both, as demonstrated in the following example: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from typing import Annotated from pydantic import Field from elasticsearch import dsl from elasticsearch.dsl.pydantic import BaseESModel class Quote(BaseESModel): quote: str author: Annotated[str, dsl.Keyword()] tags: Annotated[list[str], dsl.Keyword(normalizer="lowercase")] embedding: Annotated[list[float], dsl.DenseVector()] = Field(init=False, default=[]) class Index: name = "quotes" ``` ::: :::{tab-item} Async Python :sync: async ```python from typing import Annotated from pydantic import Field from elasticsearch import dsl from elasticsearch.dsl.pydantic import AsyncBaseESModel class Quote(AsyncBaseESModel): quote: str author: Annotated[str, dsl.Keyword()] tags: Annotated[list[str], dsl.Keyword(normalizer="lowercase")] embedding: Annotated[list[float], dsl.DenseVector()] = Field(init=False, default=[]) class Index: name = "quotes" ``` ::: :::: In this example, the `quote` attribute is annotated with a `str` type hint. Both Pydantic and Elasticsearch use this annotation. The `author` and `tags` attributes have a Python type hint and an Elasticsearch annotation, both wrapped with Python's `typing.Annotated`. When using the `BaseESModel` or `AsyncBaseESModel` classes, the typing information intended for Elasticsearch needs to be defined inside `Annotated`. The `embedding` attribute includes a base Python type and an Elasticsearch annotation in the same format as the other fields, but it adds Pydantic's `Field` definition as a right-hand side assignment. Finally, any other items that need to be defined for the Elasticsearch document class, such as `class Index` and `class Meta` entries (discussed later), can be added as well. The next example demonstrates how to define `Object` and `Nested` fields: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from typing import Annotated from pydantic import BaseModel, Field from elasticsearch import dsl from elasticsearch.dsl.pydantic import BaseESModel class Phone(BaseModel): type: Annotated[str, dsl.Keyword()] = Field(default="Home") number: str class Person(BaseESModel): name: str main_phone: Phone # same as Object(Phone) other_phones: list[Phone] # same as Nested(Phone) class Index: name = "people" ``` ::: :::{tab-item} Async Python :sync: async ```python from typing import Annotated from pydantic import BaseModel, Field from elasticsearch import dsl from elasticsearch.dsl.pydantic import AsyncBaseESModel class Phone(BaseModel): type: Annotated[str, dsl.Keyword()] = Field(default="Home") number: str class Person(AsyncBaseESModel): name: str main_phone: Phone # same as Object(Phone) other_phones: list[Phone] # same as Nested(Phone) class Index: name = "people" ``` ::: :::: Inner classes do not need to be defined with a custom base class; these should be standard Pydantic model classes. The attributes defined in these classes can include Elasticsearch annotations, as long as they are given in an `Annotated` type hint. All model classes that are created as described in this section function like normal Pydantic models and can be used anywhere standard Pydantic models are used, but they have some added attributes: - `_doc`: a class attribute that is a dynamically generated `Document` class to use with the Elasticsearch index. - `meta`: an attribute added to all models that includes Elasticsearch document metadata items such as `id`, `score`, etc. - `to_doc()`: a method that converts the Pydantic model to an Elasticsearch document. - `from_doc()`: a class method that accepts an Elasticsearch document as an argument and returns an equivalent Pydantic model. These are demonstrated in the examples below: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python # create a Pydantic model quote = Quote( quote="An unexamined life is not worth living.", author="Socrates", tags=["phillosophy"] ) # save the model to the Elasticsearch index quote.to_doc().save() # get a document from the Elasticsearch index as a Pydantic model quote = Quote.from_doc(Quote._doc.get(id=42)) # run a search and print the Pydantic models s = Quote._doc.search().query(Match(Quote._doc.quote, "life")) for doc in s: quote = Quote.from_doc(doc) print(quote.meta.id, quote.meta.score, quote.quote) ``` ::: :::{tab-item} Async Python :sync: async ```python async def example(): # create a Pydantic model quote = Quote( quote="An unexamined life is not worth living.", author="Socrates", tags=["phillosophy"] ) # save the model to the Elasticsearch index await quote.to_doc().save() # get a document from the Elasticsearch index as a Pydantic model quote = Quote.from_doc(await Quote._doc.get(id=42)) # run a search and print the Pydantic models s = Quote._doc.search().query(Match(Quote._doc.quote, "life")) async for doc in s: quote = Quote.from_doc(doc) print(quote.meta.id, quote.meta.score, quote.quote) ``` ::: :::: #### Analysis [_analysis] To specify `analyzer` values for `Text` fields you can use the name of the analyzer (as a string) and either rely on the analyzer being defined (like built-in analyzers) or define the analyzer yourself manually. Alternatively, you can create your own analyzer and have the persistence layer handle its creation, from our example earlier: ```python from elasticsearch.dsl import analyzer, tokenizer my_analyzer = analyzer('my_analyzer', tokenizer=tokenizer('trigram', 'nGram', min_gram=3, max_gram=3), filter=['lowercase'] ) ``` Each analysis object needs to have a name (`my_analyzer` and `trigram` in our example) and tokenizers, token filters and char filters also need to specify type (`nGram` in our example). Once you have an instance of a custom `analyzer` you can also call the [analyze API](https://www.elastic.co/docs/api/doc/elasticsearch/v8/group/endpoint-indices) on it by using the `simulate` method: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python response = my_analyzer.simulate('Hello World!') # ['hel', 'ell', 'llo', 'lo ', 'o w', ' wo', 'wor', 'orl', 'rld', 'ld!'] tokens = [t.token for t in response.tokens] ``` ::: :::{tab-item} Async Python :sync: async ```python response = my_analyzer.async_simulate('Hello World!') # ['hel', 'ell', 'llo', 'lo ', 'o w', ' wo', 'wor', 'orl', 'rld', 'ld!'] tokens = [t.token for t in response.tokens] ``` ::: :::: ::::{note} When creating a mapping which relies on a custom analyzer the index must either not exist or be closed. To create multiple `Document`-defined mappings you can use the `index` object. :::: #### Search [_search_2] To search for this document type, use the `search` class method: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python # by calling .search we get back a standard Search object s = Post.search() # the search is already limited to the index and doc_type of our document s = s.filter('term', published=True).query('match', title='first') results = s.execute() # when you execute the search the results are wrapped in your document class (Post) for post in results: print(post.meta.score, post.title) ``` ::: :::{tab-item} Async Python :sync: async ```python async def example(): # by calling .search we get back a standard Search object s = Post.search() # the search is already limited to the index and doc_type of our document s = s.filter('term', published=True).query('match', title='first') results = await s.execute() # when you execute the search the results are wrapped in your document class (Post) for post in results: print(post.meta.score, post.title) ``` ::: :::: Alternatively you can take a `Search` or `AsyncSearch` object and restrict it to return our document type, wrapped in correct class: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = Search() s = s.doc_type(Post) ``` ::: :::{tab-item} Async Python :sync: async ```python s = AsyncSearch() s = s.doc_type(Post) ``` ::: :::: You can also combine document classes with standard doc types (strings), which will be treated as before. You can also pass in multiple `Document` subclasses and each document in the response will be wrapped in it’s class. If you want to run suggestions, use the `suggest` method on the `Search` object: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python s = Post.search() s = s.suggest('title_suggestions', 'pyth', completion={'field': 'title_suggest'}) response = s.execute() for result in response.suggest.title_suggestions: print('Suggestions for %s:' % result.text) for option in result.options: print(' %s (%r)' % (option.text, option.payload)) ``` ::: :::{tab-item} Async Python :sync: async ```python async def example(): s = Post.search() s = s.suggest('title_suggestions', 'pyth', completion={'field': 'title_suggest'}) response = await s.execute() for result in response.suggest.title_suggestions: print('Suggestions for %s:' % result.text) for option in result.options: print(' %s (%r)' % (option.text, option.payload)) ``` ::: :::: #### `class Meta` options [_class_meta_options] In the `Meta` class inside your document definition you can define various metadata for your document: * `mapping`: optional instance of `Mapping` or `AsyncMapping` classes to use as base for the mappings created from the fields on the document class itself. Any attributes on the `Meta` class that are instance of `MetaField` will be used to control the mapping of the meta fields (`_all`, `dynamic` and so on). Name the parameter (without the leading underscore) as the field you wish to map and pass any parameters to the `MetaField` class: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python class Post(Document): title = Text() class Meta: all = MetaField(enabled=False) dynamic = MetaField('strict') ``` ::: :::{tab-item} Async Python :sync: async ```python class Post(AsyncDocument): title = Text() class Meta: all = MetaField(enabled=False) dynamic = MetaField('strict') ``` ::: :::: #### `class Index` options [_class_index_options] This section of the `Document` definition can contain any information about the index, its name, settings and other attributes: * `name`: name of the index to use, if it contains a wildcard (`*`) then it cannot be used for any write operations and an `index` kwarg will have to be passed explicitly when calling methods like `.save()`. * `using`: default connection alias to use, defaults to `'default'` * `settings`: dictionary containing any settings for the `Index` object like `number_of_shards`. * `analyzers`: additional list of analyzers that should be defined on an index (see `analysis` for details). * `aliases`: dictionary with any aliases definitions * `data_stream`: set to `True` to configure a data stream instead of an index. #### Document Inheritance [_document_inheritance] You can use standard Python inheritance to extend models, this can be useful in a few scenarios. For example if you want to have a `BaseDocument` defining some common fields that several different `Document` classes should share: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python class User(InnerDoc): username: str = mapped_field(Text(fields={'keyword': Keyword()})) email: str class BaseDocument(Document): created_by: User created_date: datetime last_updated: datetime def save(**kwargs): if not self.created_date: self.created_date = datetime.now() self.last_updated = datetime.now() return super(BaseDocument, self).save(**kwargs) class BlogPost(BaseDocument): class Index: name = 'blog' ``` ::: :::{tab-item} Async Python :sync: async ```python class User(InnerDoc): username: str = mapped_field(Text(fields={'keyword': Keyword()})) email: str class BaseDocument(AsyncDocument): created_by: User created_date: datetime last_updated: datetime async def save(**kwargs): if not self.created_date: self.created_date = datetime.now() self.last_updated = datetime.now() return await super(BaseDocument, self).save(**kwargs) class BlogPost(BaseDocument): class Index: name = 'blog' ``` ::: :::: Another use case would be using the [join type](elasticsearch://reference/elasticsearch/mapping-reference/parent-join.md) to have multiple different entities in a single index. You can see an [example](https://github.com/elastic/elasticsearch-py/blob/master/examples/dsl/parent_child.py) of this approach. In this case, if the subclasses don’t define their own Index classes, the mappings are merged and shared between all the subclasses. ### Index [_index] In typical scenario using `class Index` on a `Document` class is sufficient to perform any action. In a few cases though it can be useful to manipulate an `Index` object directly. `Index` is a class responsible for holding all the metadata related to an index in elasticsearch - mappings and settings. It is most useful when defining your mappings since it allows for easy creation of multiple mappings at the same time. This is especially useful when setting up your elasticsearch objects in a migration: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch.dsl import Index, Document, Text, analyzer blogs = Index('blogs') # define custom settings blogs.settings( number_of_shards=1, number_of_replicas=0 ) # define aliases blogs.aliases( old_blogs={} ) # register a document with the index blogs.document(Post) # can also be used as class decorator when defining the Document @blogs.document class Post(Document): title: str # You can attach custom analyzers to the index html_strip = analyzer('html_strip', tokenizer="standard", filter=["standard", "lowercase", "stop", "snowball"], char_filter=["html_strip"] ) blogs.analyzer(html_strip) # delete the index, ignore if it doesn't exist blogs.delete(ignore=404) # create the index in elasticsearch blogs.create() ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch.dsl import AsyncIndex, AsyncDocument, Text, analyzer blogs = AsyncIndex('blogs') # define custom settings blogs.settings( number_of_shards=1, number_of_replicas=0 ) # define aliases blogs.aliases( old_blogs={} ) # register a document with the index blogs.document(Post) # can also be used as class decorator when defining the Document @blogs.document class Post(AsyncDocument): title: str # You can attach custom analyzers to the index html_strip = analyzer('html_strip', tokenizer="standard", filter=["standard", "lowercase", "stop", "snowball"], char_filter=["html_strip"] ) blogs.analyzer(html_strip) async def example(): # delete the index, ignore if it doesn't exist await blogs.delete(ignore=404) # create the index in elasticsearch await blogs.create() ``` ::: :::: You can also set up a template for your indices and use the `clone` method to create specific copies: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python blogs = Index('blogs', using='production') blogs.settings(number_of_shards=2) blogs.document(Post) # create a copy of the index with different name company_blogs = blogs.clone('company-blogs') # create a different copy on different cluster dev_blogs = blogs.clone('blogs', using='dev') # and change its settings dev_blogs.setting(number_of_shards=1) ``` ::: :::{tab-item} Async Python :sync: async ```python blogs = AsyncIndex('blogs', using='production') blogs.settings(number_of_shards=2) blogs.document(Post) # create a copy of the index with different name company_blogs = blogs.clone('company-blogs') # create a different copy on different cluster dev_blogs = blogs.clone('blogs', using='dev') # and change its settings dev_blogs.setting(number_of_shards=1) ``` ::: :::: #### IndexTemplate [index-template] The DSL module also exposes an option to manage [index templates](docs-content://manage-data/data-store/templates.md) in elasticsearch using the `ComposableIndexTemplate` and `IndexTemplate` classes, which have a similar API to `Index`. ::::{note} Composable index templates should always be preferred over the legacy index templates. :::: Once an index template is saved in Elasticsearch its contents will be automatically applied to new indices (existing indices are completely unaffected by templates) that match the template pattern (any index starting with `blogs-` in our example), even if the index is created automatically upon indexing a document into that index. Potential workflow for a set of time based indices governed by a single template: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from datetime import datetime from elasticsearch.dsl import Document, Date, Text class Log(Document): content: str timestamp: datetime class Index: name = "logs-*" def save(self, **kwargs): # assign now if no timestamp given if not self.timestamp: self.timestamp = datetime.now() # override the index to go to the proper timeslot kwargs['index'] = self.timestamp.strftime('logs-%Y%m%d') return super().save(**kwargs) # once, as part of application setup, during deploy/migrations: logs = Log._index.as_composable_template('logs', priority=100) logs.save() # to perform search across all logs: search = Log.search() ``` ::: :::{tab-item} Async Python :sync: async ```python from datetime import datetime from elasticsearch.dsl import AsyncDocument, Date, Text class Log(AsyncDocument): content: str timestamp: datetime class Index: name = "logs-*" async def save(self, **kwargs): # assign now if no timestamp given if not self.timestamp: self.timestamp = datetime.now() # override the index to go to the proper timeslot kwargs['index'] = self.timestamp.strftime('logs-%Y%m%d') return await super().save(**kwargs) async def example(): # once, as part of application setup, during deploy/migrations: logs = Log._index.as_composable_template('logs', priority=100) await logs.save() # to perform search across all logs: search = Log.search() ``` ::: :::: ## Faceted Search [faceted_search] The library comes with a basic abstraction aimed at helping you develop faceted navigation for your data. ### Configuration [_configuration_2] You can provide several configuration options (as class attributes) when declaring a `FacetedSearch` subclass: * `index`: the name of the index (as string) to search through, defaults to `'_all'`. * `doc_types`: list of `Document` subclasses or strings to be used, defaults to `['_all']`. * `fields`: list of fields on the document type to search through. The list will be passes to `MultiMatch` query so can contain boost values (`'title^5'`), defaults to `['*']`. * `facets`: dictionary of facets to display/filter on. The key is the name displayed and values should be instances of any `Facet` subclass, for example: `{'tags': TermsFacet(field='tags')}` #### Facets [_facets] There are several different facets available: * `TermsFacet`: provides an option to split documents into groups based on a value of a field, for example `TermsFacet(field='category')` * `DateHistogramFacet`: split documents into time intervals, example: `DateHistogramFacet(field="published_date", calendar_interval="day")` * `HistogramFacet`: similar to `DateHistogramFacet` but for numerical values: `HistogramFacet(field="rating", interval=2)` * `RangeFacet`: allows you to define your own ranges for a numerical fields: `RangeFacet(field="comment_count", ranges=[("few", (None, 2)), ("lots", (2, None))])` * `NestedFacet`: is a basic facet that wraps another to provide access to nested documents: `NestedFacet('variants', TermsFacet(field='variants.color'))` By default facet results will only calculate document count, if you wish for a different metric you can pass in any single value metric aggregation as the `metric` kwarg (`TermsFacet(field='tags', metric=A('max', field=timestamp))`). When specifying `metric` the results will be, by default, sorted in descending order by that metric. To change it to ascending specify `metric_sort="asc"` and to sort by document count use `metric_sort=False`. #### Advanced [_advanced] If you require any custom behavior or modifications simply override one or more of the methods responsible for the class' functions: * `search(self)`: is responsible for constructing the `Search` object used. Override this if you want to customize the search object (for example by adding a global filter for published articles only). * `query(self, search)`: adds the query position of the search (if search input specified), by default using `MultiField` query. Override this if you want to modify the query type used. * `highlight(self, search)`: defines the highlighting on the `Search` object and returns a new one. Default behavior is to highlight on all fields specified for search. ### Usage [_usage] The custom subclass can be instantiated empty to provide an empty search (matching everything) or with `query`, `filters` and `sort`. * `query`: is used to pass in the text of the query to be performed. If `None` is passed in (default) a `MatchAll` query will be used. For example `'python web'` * `filters`: is a dictionary containing all the facet filters that you wish to apply. Use the name of the facet (from `.facets` attribute) as the key and one of the possible values as value. For example `{'tags': 'python'}`. * `sort`: is a tuple or list of fields on which the results should be sorted. The format of the individual fields are to be the same as those passed to `~elasticsearch.dsl.Search.sort`. #### Response [_response_2] the response returned from the `FacetedSearch` object (by calling `.execute()`) is a subclass of the standard `Response` class that adds a property called `facets` which contains a dictionary with lists of buckets -each represented by a tuple of key, document count and a flag indicating whether this value has been filtered on. ### Example [_example] ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from datetime import date from elasticsearch.dsl import FacetedSearch, TermsFacet, DateHistogramFacet class BlogSearch(FacetedSearch): doc_types = [Article, ] # fields that should be searched fields = ['tags', 'title', 'body'] facets = { # use bucket aggregations to define facets 'tags': TermsFacet(field='tags'), 'publishing_frequency': DateHistogramFacet(field='published_from', interval='month') } def search(self): # override methods to add custom pieces s = super().search() return s.filter('range', publish_from={'lte': 'now/h'}) bs = BlogSearch('python web', {'publishing_frequency': date(2015, 6)}) response = bs.execute() # access hits and other attributes as usual total = response.hits.total print('total hits', total.relation, total.value) for hit in response: print(hit.meta.score, hit.title) for (tag, count, selected) in response.facets.tags: print(tag, ' (SELECTED):' if selected else ':', count) for (month, count, selected) in response.facets.publishing_frequency: print(month.strftime('%B %Y'), ' (SELECTED):' if selected else ':', count) ``` ::: :::{tab-item} Async Python :sync: async ```python from datetime import date from elasticsearch.dsl import AsyncFacetedSearch, TermsFacet, DateHistogramFacet class BlogSearch(AsyncFacetedSearch): doc_types = [Article, ] # fields that should be searched fields = ['tags', 'title', 'body'] facets = { # use bucket aggregations to define facets 'tags': TermsFacet(field='tags'), 'publishing_frequency': DateHistogramFacet(field='published_from', interval='month') } def search(self): # override methods to add custom pieces s = super().search() return s.filter('range', publish_from={'lte': 'now/h'}) async def example(): bs = BlogSearch('python web', {'publishing_frequency': date(2015, 6)}) response = await bs.execute() # access hits and other attributes as usual total = response.hits.total print('total hits', total.relation, total.value) for hit in response: print(hit.meta.score, hit.title) for (tag, count, selected) in response.facets.tags: print(tag, ' (SELECTED):' if selected else ':', count) for (month, count, selected) in response.facets.publishing_frequency: print(month.strftime('%B %Y'), ' (SELECTED):' if selected else ':', count) ``` ::: :::: ## Update By Query [update_by_query] ### The `Update By Query` object [_the_update_by_query_object] The `Update By Query` object enables the use of the [_update_by_query](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-update-by-query) endpoint to perform an update on documents that match a search query. The object is implemented as a modification of the `Search` object, containing a subset of its query methods, as well as a script method, which is used to make updates. The `Update By Query` object implements the following `Search` query types: * queries * filters * excludes For more information on queries, see the `search_dsl` chapter. Like the `Search` object, the API is designed to be chainable. This means that the `Update By Query` object is immutable: all changes to the object will result in a shallow copy being created which contains the changes. This means you can safely pass the `Update By Query` object to foreign code without fear of it modifying your objects as long as it sticks to the `Update By Query` object APIs. You can define your client in a number of ways, but the preferred method is to use a global configuration. For more information on defining a client, see the `configuration` chapter. Once your client is defined, you can instantiate a copy of the `Update By Query` object as seen below: ```python from elasticsearch.dsl import UpdateByQuery ubq = UpdateByQuery().using(client) # or ubq = UpdateByQuery(using=client) ``` ::::{note} All methods return a *copy* of the object, making it safe to pass to outside code. :::: The API is chainable, allowing you to combine multiple method calls in one statement: ```python ubq = UpdateByQuery().using(client).query(Match("title", python")) ``` To send the request to Elasticsearch: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python response = ubq.execute() ``` ::: :::{tab-item} Async Python :sync: async ```python response = await ubq.execute() ``` ::: :::: It should be noted, that there are limits to the chaining using the script method: calling script multiple times will overwrite the previous value. That is, only a single script can be sent with a call. An attempt to use two scripts will result in only the second script being stored. Given the below example: ```python ubq = UpdateByQuery() \ .using(client) \ .script(source="ctx._source.likes++") \ .script(source="ctx._source.likes+=2") ``` This means that the stored script by this client will be `'source': 'ctx._source.likes{{plus}}=2'` and the previous call will not be stored. For debugging purposes you can serialize the `Update By Query` object to a `dict` explicitly: ```python print(ubq.to_dict()) ``` Also, to use variables in script see below example: ```python ubq.script( source="ctx._source.messages.removeIf(x -> x.somefield == params.some_var)", params={ 'some_var': 'some_string_val' } ) ``` #### Serialization and Deserialization [_serialization_and_deserialization_2] The search object can be serialized into a dictionary by using the `.to_dict()` method. You can also create a `Update By Query` object from a `dict` using the `from_dict` class method. This will create a new `Update By Query` object and populate it using the data from the dict: ```python ubq = UpdateByQuery.from_dict({"query": {"match": {"title": "python"}}}) ``` If you wish to modify an existing `Update By Query` object, overriding it’s properties, instead use the `update_from_dict` method that alters an instance **in-place**: ```python ubq = UpdateByQuery(index='i') ubq.update_from_dict({"query": {"match": {"title": "python"}}, "size": 42}) ``` #### Extra properties and parameters [_extra_properties_and_parameters_2] To set extra properties of the search request, use the `.extra()` method. This can be used to define keys in the body that cannot be defined via a specific API method like `explain`: ```python ubq = ubq.extra(explain=True) ``` To set query parameters, use the `.params()` method: ```python ubq = ubq.params(routing="42") ``` ### Response [_response_3] You can execute your search by calling the `.execute()` method that will return a `Response` object. The `Response` object allows you access to any key from the response dictionary via attribute access. It also provides some convenient helpers: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python response = ubq.execute() print(response.success()) # True print(response.took) # 12 ``` ::: :::{tab-item} Async Python :sync: async ```python async def example(): response = await ubq.execute() print(response.success()) # True print(response.took) # 12 ``` ::: :::: If you want to inspect the contents of the `response` objects, use its `to_dict` method to get access to the raw data for pretty printing. ## ES|QL Queries When working with `Document` classes, you can use the ES|QL query language to retrieve documents. For this you can use the `esql_from()` and `esql_execute()` methods available to all sub-classes of `Document`. Consider the following `Employee` document definition: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch.dsl import Document, InnerDoc, M class Address(InnerDoc): address: M[str] city: M[str] zip_code: M[str] class Employee(Document): emp_no: M[int] first_name: M[str] last_name: M[str] height: M[float] still_hired: M[bool] address: M[Address] class Index: name = 'employees' ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch.dsl import AsyncDocument, InnerDoc, M class Address(InnerDoc): address: M[str] city: M[str] zip_code: M[str] class Employee(AsyncDocument): emp_no: M[int] first_name: M[str] last_name: M[str] height: M[float] still_hired: M[bool] address: M[Address] class Index: name = 'employees' ``` ::: :::: The `esql_from()` method creates a base ES|QL query for the index associated with the document class. The following example creates a base query for the `Employee` class: ```python query = Employee.esql_from() ``` This query includes a `FROM` command with the index name, and a `KEEP` command that retrieves all the document attributes. To execute this query and receive the results, you can pass the query to the `esql_execute()` method: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python for emp in Employee.esql_execute(query): print(f"{emp.name} from {emp.address.city} is {emp.height:.2f}m tall") ``` ::: :::{tab-item} Async Python :sync: async ```python async for emp in Employee.esql_execute(query): print(f"{emp.name} from {emp.address.city} is {emp.height:.2f}m tall") ``` ::: :::: In this example, the `esql_execute()` class method runs the query and returns all the documents in the index, up to the maximum of 1000 results allowed by ES|QL. Here is a possible output from this example: ``` Kevin Macias from North Robert is 1.60m tall Drew Harris from Boltonshire is 1.68m tall Julie Williams from Maddoxshire is 1.99m tall Christopher Jones from Stevenbury is 1.98m tall Anthony Lopez from Port Sarahtown is 2.42m tall Tricia Stone from North Sueshire is 2.39m tall Katherine Ramirez from Kimberlyton is 1.83m tall ... ``` To search for specific documents you can extend the base query with additional ES|QL commands that narrow the search criteria. The next example searches for documents that include only employees that are taller than 2 meters, sorted by their last name. It also limits the results to 4 people: ```python query = ( Employee.esql_from() .where(Employee.height > 2) .sort(Employee.last_name) .limit(4) ) ``` When running this query with the same for-loop shown above, possible results would be: ``` Michael Adkins from North Stacey is 2.48m tall Kimberly Allen from Toddside is 2.24m tall Crystal Austin from East Michaelchester is 2.30m tall Rebecca Berger from Lake Adrianside is 2.40m tall ``` ### Additional fields ES|QL provides a few ways to add new fields to a query, for example through the `EVAL` command. The following example shows a query that adds an evaluated field: ```python from elasticsearch.esql import E, functions query = ( Employee.esql_from() .eval(height_cm=functions.round(Employee.height * 100)) .where(E("height_cm") >= 200) .sort(Employee.last_name) .limit(10) ) ``` In this example we are adding the height in centimeters to the query, calculated from the `height` document field, which is in meters. The `height_cm` calculated field is available to use in other query clauses, and in particular is referenced in `where()` in this example. Note how the new field is given as `E("height_cm")` in this clause. The `E()` wrapper tells the query builder that the argument is an ES|QL field name and not a string literal. This is done automatically for document fields that are given as class attributes, such as `Employee.height` in the `eval()`. The `E()` wrapper is only needed for fields that are not in the document. By default, the `esql_execute()` method returns only document instances. To receive any additional fields that are not part of the document in the query results, the `return_additional=True` argument can be passed to it, and then the results are returned as tuples with the document as first element, and a dictionary with the additional fields as second element: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python for emp, additional in Employee.esql_execute(query, return_additional=True): print(emp.name, additional) ``` ::: :::{tab-item} Async Python :sync: async ```python async for emp, additional in Employee.esql_execute(query, return_additional=True): print(emp.name, additional) ``` ::: :::: Example output from the query given above: ``` Michael Adkins {'height_cm': 248.0} Kimberly Allen {'height_cm': 224.0} Crystal Austin {'height_cm': 230.0} Rebecca Berger {'height_cm': 240.0} Katherine Blake {'height_cm': 214.0} Edward Butler {'height_cm': 246.0} Steven Carlson {'height_cm': 242.0} Mark Carter {'height_cm': 240.0} Joseph Castillo {'height_cm': 229.0} Alexander Cohen {'height_cm': 245.0} ``` ### Missing fields The base query returned by the `esql_from()` method includes a `KEEP` command with the complete list of fields that are part of the document. If any subsequent clauses added to the query remove fields that are part of the document, then the `esql_execute()` method will raise an exception, because it will not be able construct complete document instances to return as results. To prevent errors, it is recommended that the `keep()` and `drop()` clauses are not used when working with `Document` instances. If a query has missing fields, it can be forced to execute without errors by passing the `ignore_missing_fields=True` argument to `esql_execute()`. When this option is used, returned documents will have any missing fields set to `None`. ## Using asyncio with Elasticsearch Python DSL [asyncio] The DSL module supports async/await with [asyncio](https://docs.python.org/3/library/asyncio.html). To ensure that you have all the required dependencies, install the `[async]` extra: ```bash $ python -m pip install "elasticsearch[async]" ``` The DSL module also supports [Trio](https://trio.readthedocs.io/en/stable/) when using the Async HTTPX client. You do need to install Trio and HTTPX separately: ```bash $ python -m pip install "elasticsearch trio httpx" ``` ### Connections [_connections] Use the `connections` module to manage your synchronous connections. ```python from elasticsearch.dsl import connections connections.create_connection(hosts=['https://localhost:9200'], request_timeout=20) ``` For asynchronous connection management, use the `async_connections` module: ```python from elasticsearch.dsl import async_connections async_connections.create_connection(hosts=['https://localhost:9200'], request_timeout=20) ``` If you're using Trio, you need to explicitly request the Async HTTP client: ```python from elasticsearch.dsl import async_connections async_connections.create_connection(hosts=['https://localhost:9200'], node_class="httpxasync") ``` #### How to avoid *Unclosed client session / connector* warnings on exit [_how_to_avoid_unclosed_client_session_connector_warnings_on_exit] These warnings come from the `aiohttp` package, which is used internally by the `AsyncElasticsearch` client. They appear often when the application exits and are caused by HTTP connections that are open when they are garbage collected. To avoid these warnings, make sure that you close your connections. ```python es = async_connections.get_connection() await es.close() ``` ### Search DSL [_search_dsl] Use the `AsyncSearch` class to perform asynchronous searches. ```python from elasticsearch.dsl import AsyncSearch from elasticsearch.dsl.query import Match s = AsyncSearch().query(Match("title", "python")) async for hit in s: print(hit.title) ``` Instead of using the `AsyncSearch` object as an asynchronous iterator, you can explicitly call the `execute()` method to get a `Response` object. ```python s = AsyncSearch().query(Match("title", "python")) response = await s.execute() for hit in response: print(hit.title) ``` An `AsyncMultiSearch` is available as well. ```python from elasticsearch.dsl import AsyncMultiSearch from elasticsearch.dsl.query import Term ms = AsyncMultiSearch(index='blogs') ms = ms.add(AsyncSearch().filter(Term("tags", "python"))) ms = ms.add(AsyncSearch().filter(Term("tags", "elasticsearch"))) responses = await ms.execute() for response in responses: print("Results for query %r." % response.search.query) for hit in response: print(hit.title) ``` ### Asynchronous Documents, Indexes, and more [_asynchronous_documents_indexes_and_more] The `Document`, `BaseESModel`, `Index`, `IndexTemplate`, `Mapping`, `UpdateByQuery` and `FacetedSearch` classes all have asynchronous versions that use the same name with an `Async` prefix. These classes expose the same interfaces as the synchronous versions, but any methods that perform I/O are defined as coroutines. Auxiliary classes that do not perform I/O do not have asynchronous versions. The same classes can be used in synchronous and asynchronous applications. When using a custom analyzer in an asynchronous application, use the `async_simulate()` method to invoke the Analyze API on it. Consult the `api` section for details about each specific method. --- ## File: docs/reference/dsl_migrating.md # Migrating from the `elasticsearch-dsl` package [_migrating_from_elasticsearch_dsl_package] In the past the Elasticsearch Python DSL module was distributed as a standalone package called `elasticsearch-dsl`. This package is now deprecated as all its functionality has been integrated into the main Python client. We recommend all developers to migrate their applications and remove their dependency on the `elasticsearch-dsl` package. To migrate your application, all references to `elasticsearch_dsl` as a top-level package must be changed to `elasticsearch.dsl`. In other words, the underscore from the package name should be replaced by a period. Here are a few examples: ```python # from: from elasticsearch_dsl import Date, Document, InnerDoc, Text, connections # to: from elasticsearch.dsl import Date, Document, InnerDoc, Text, connections # from: from elasticsearch_dsl.query import MultiMatch # to: from elasticsearch.dsl.query import MultiMatch # from: import elasticsearch_dsl as dsl # to: from elasticsearch import dsl # from: import elasticsearch_dsl # to: from elasticsearch import dsl as elasticsearch_dsl # from: import elasticsearch_dsl # to: from elasticsearch import dsl # and replace all references to "elasticsearch_dsl" in the code with "dsl" ``` --- ## File: docs/reference/dsl_tutorials.md --- mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/_tutorials.html --- # Tutorials [_tutorials] ## Search [_search] Let’s have a typical search request written directly as a `dict`: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch import Elasticsearch client = Elasticsearch("https://localhost:9200") response = client.search( index="my-index", body={ "query": { "bool": { "must": [{"match": {"title": "python"}}], "must_not": [{"match": {"description": "beta"}}], "filter": [{"term": {"category": "search"}}] } }, "aggs" : { "per_tag": { "terms": {"field": "tags"}, "aggs": { "max_lines": {"max": {"field": "lines"}} } } } } ) for hit in response['hits']['hits']: print(hit['_score'], hit['_source']['title']) for tag in response['aggregations']['per_tag']['buckets']: print(tag['key'], tag['max_lines']['value']) ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch import AsyncElasticsearch client = AsyncElasticsearch("https://localhost:9200") async def example(): response = await client.search( index="my-index", body={ "query": { "bool": { "must": [{"match": {"title": "python"}}], "must_not": [{"match": {"description": "beta"}}], "filter": [{"term": {"category": "search"}}] } }, "aggs" : { "per_tag": { "terms": {"field": "tags"}, "aggs": { "max_lines": {"max": {"field": "lines"}} } } } } ) for hit in response['hits']['hits']: print(hit['_score'], hit['_source']['title']) for tag in response['aggregations']['per_tag']['buckets']: print(tag['key'], tag['max_lines']['value']) ``` ::: :::: The problem with this approach is that it is verbose, prone to syntax mistakes like incorrect nesting, hard to modify (for example adding another filter) and definitely not fun to write. Let’s rewrite the example using the DSL module: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch import Elasticsearch from elasticsearch.dsl import Search, query, aggs client = Elasticsearch("https://localhost:9200") s = Search(using=client, index="my-index") \ .query(query.Match("title", "python")) \ .filter(query.Term("category", "search")) \ .exclude(query.Match("description", "beta")) s.aggs.bucket('per_tag', aggs.Terms(field="tags")) \ .metric('max_lines', aggs.Max(field='lines')) response = s.execute() for hit in response: print(hit.meta.score, hit.title) for tag in response.aggregations.per_tag.buckets: print(tag.key, tag.max_lines.value) ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch import AsyncElasticsearch from elasticsearch.dsl import AsyncSearch, query, aggs client = AsyncElasticsearch("https://localhost:9200") async def example(): s = AsyncSearch(using=client, index="my-index") \ .query(query.Match("title", "python")) \ .filter(query.Term("category", "search")) \ .exclude(query.Match("description", "beta")) s.aggs.bucket('per_tag', aggs.Terms(field="tags")) \ .metric('max_lines', aggs.Max(field='lines')) response = await s.execute() for hit in response: print(hit.meta.score, hit.title) for tag in response.aggregations.per_tag.buckets: print(tag.key, tag.max_lines.value) ``` ::: :::: As you see, the DSL module took care of: * creating appropriate `Query` objects from classes * composing queries into a compound `bool` query * putting the `term` query in a filter context of the `bool` query * providing a convenient access to response data * no curly or square brackets everywhere ## Persistence [_persistence] Let’s have a simple Python class representing an article in a blogging system: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from datetime import datetime from elasticsearch.dsl import Document, Date, Integer, Keyword, Text, connections, mapped_field # Define a default Elasticsearch client connections.create_connection(hosts="https://localhost:9200") class Article(Document): title: str = mapped_field(Text(analyzer='snowball', fields={'raw': Keyword()})) body: str = mapped_field(Text(analyzer='snowball')) tags: list[str] = mapped_field(Keyword()) published_from: datetime lines: int class Index: name = 'blog' settings = { "number_of_shards": 2, } def save(self, **kwargs): self.lines = len(self.body.split()) return super(Article, self).save(** kwargs) def is_published(self): return datetime.now() > self.published_from # create the mappings in elasticsearch Article.init() # create and save and article article = Article(meta={'id': 42}, title='Hello world!', tags=['test']) article.body = ''' looong text ''' article.published_from = datetime.now() article.save() article = Article.get(id=42) print(article.is_published()) # Display cluster health print(connections.get_connection().cluster.health()) ``` ::: :::{tab-item} Async Python :sync: async ```python from datetime import datetime from elasticsearch.dsl import AsyncDocument, Date, Integer, Keyword, Text, async_connections, mapped_field # Define a default Elasticsearch client async_connections.create_connection(hosts="https://localhost:9200") class Article(AsyncDocument): title: str = mapped_field(Text(analyzer='snowball', fields={'raw': Keyword()})) body: str = mapped_field(Text(analyzer='snowball')) tags: list[str] = mapped_field(Keyword()) published_from: datetime lines: int class Index: name = 'blog' settings = { "number_of_shards": 2, } async def save(self, **kwargs): self.lines = len(self.body.split()) return await super(Article, self).save(** kwargs) def is_published(self): return datetime.now() > self.published_from async def example(): # create the mappings in elasticsearch await Article.init() # create and save and article article = Article(meta={'id': 42}, title='Hello world!', tags=['test']) article.body = ''' looong text ''' article.published_from = datetime.now() await article.save() article = await Article.get(id=42) print(article.is_published()) # Display cluster health print(await async_connections.get_connection().cluster.health()) ``` ::: :::: In this example you can see: * providing a default connection * defining fields with Python type hints and additional mapping configuration when necessary * setting index name * defining custom methods * overriding the built-in `.save()` method to hook into the persistence life cycle * retrieving and saving the object into Elasticsearch * accessing the underlying client for other APIs You can see more in the [persistence](dsl_how_to_guides.md#_persistence_2) chapter. ## Pre-built Faceted Search [_pre_built_faceted_search] If you have your `Document`s defined you can create a faceted search class to simplify searching and filtering. ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch.dsl import FacetedSearch, TermsFacet, DateHistogramFacet class BlogSearch(FacetedSearch): doc_types = [Article, ] # fields that should be searched fields = ['tags', 'title', 'body'] facets = { # use bucket aggregations to define facets 'tags': TermsFacet(field='tags'), 'publishing_frequency': DateHistogramFacet(field='published_from', interval='month') } # empty search bs = BlogSearch() response = bs.execute() for hit in response: print(hit.meta.score, hit.title) for (tag, count, selected) in response.facets.tags: print(tag, ' (SELECTED):' if selected else ':', count) for (month, count, selected) in response.facets.publishing_frequency: print(month.strftime('%B %Y'), ' (SELECTED):' if selected else ':', count) ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch.dsl import AsyncFacetedSearch, TermsFacet, DateHistogramFacet class BlogSearch(AsyncFacetedSearch): doc_types = [Article, ] # fields that should be searched fields = ['tags', 'title', 'body'] facets = { # use bucket aggregations to define facets 'tags': TermsFacet(field='tags'), 'publishing_frequency': DateHistogramFacet(field='published_from', interval='month') } async def example(): # empty search bs = BlogSearch() response = await bs.execute() for hit in response: print(hit.meta.score, hit.title) for (tag, count, selected) in response.facets.tags: print(tag, ' (SELECTED):' if selected else ':', count) for (month, count, selected) in response.facets.publishing_frequency: print(month.strftime('%B %Y'), ' (SELECTED):' if selected else ':', count) ``` ::: :::: You can find more details in the `faceted_search` chapter. ## Update By Query [_update_by_query] Let’s resume the simple example of articles on a blog, and let’s assume that each article has a number of likes. For this example, imagine we want to increment the number of likes by 1 for all articles that match a certain tag and do not match a certain description. Writing this as a `dict`, we would have the following code: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch import Elasticsearch client = Elasticsearch() response = client.update_by_query( index="my-index", body={ "query": { "bool": { "must": [{"match": {"tag": "python"}}], "must_not": [{"match": {"description": "beta"}}] } }, "script"={ "source": "ctx._source.likes++", "lang": "painless" } }, ) ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch import AsyncElasticsearch client = AsyncElasticsearch() async def example(): response = await client.update_by_query( index="my-index", body={ "query": { "bool": { "must": [{"match": {"tag": "python"}}], "must_not": [{"match": {"description": "beta"}}] } }, "script"={ "source": "ctx._source.likes++", "lang": "painless" } }, ) ``` ::: :::: Using the DSL, we can now express this query as such: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch import Elasticsearch from elasticsearch.dsl import Search, UpdateByQuery from elasticsearch.dsl.query import Match client = Elasticsearch() ubq = UpdateByQuery(using=client, index="my-index") \ .query(Match("title", "python")) \ .exclude(Match("description", "beta")) \ .script(source="ctx._source.likes++", lang="painless") response = ubq.execute() ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch import AsyncElasticsearch from elasticsearch.dsl import AsyncSearch, AsyncUpdateByQuery from elasticsearch.dsl.query import Match client = AsyncElasticsearch() async def example(): ubq = UpdateByQuery(using=client, index="my-index") \ .query(Match("title", "python")) \ .exclude(Match("description", "beta")) \ .script(source="ctx._source.likes++", lang="painless") response = await ubq.execute() ``` ::: :::: As you can see, the `Update By Query` object provides many of the savings offered by the `Search` object, and additionally allows one to update the results of the search based on a script assigned in the same manner. ## ES|QL Queries The DSL module features an integration with the ES|QL query builder, consisting of two methods available in all `Document` sub-classes: `esql_from()` and `esql_execute()`. Using the `Article` document from above, we can search for up to ten articles that include `"world"` in their titles with the following ES|QL query: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python from elasticsearch.esql import functions query = Article.esql_from().where(functions.match(Article.title, 'world')).limit(10) for a in Article.esql_execute(query): print(a.title) ``` ::: :::{tab-item} Async Python :sync: async ```python from elasticsearch.esql import functions async def example(): query = Article.esql_from().where(functions.match(Article.title, 'world')).limit(10) async for a in Article.esql_execute(query): print(a.title) ``` ::: :::: Review the [ES|QL query builder section](esql-query-builder.md) to learn more about building ES|QL queries in Python. ## Migration from the standard client [_migration_from_the_standard_client] You don’t have to port your entire application to get the benefits of the DSL module, you can start gradually by creating a `Search` object from your existing `dict`, modifying it using the API and serializing it back to a `dict`: ::::{tab-set} :group: sync_or_async :::{tab-item} Standard Python :sync: sync ```python body = {...} # insert complicated query here # Convert to Search object s = Search.from_dict(body) # Add some filters, aggregations, queries, ... s.filter(query.Term("tags", "python")) # Convert back to dict to plug back into existing code body = s.to_dict() ``` ::: :::{tab-item} Async Python :sync: async ```python body = {...} # insert complicated query here # Convert to Search object s = AsyncSearch.from_dict(body) # Add some filters, aggregations, queries, ... s.filter(query.Term("tags", "python")) # Convert back to dict to plug back into existing code body = s.to_dict() ``` ::: ::::