("Article"); await collection.data.insert({ // compiler error since 'body' field is missing in '.insert' title: "TS is awesome!", wordcount: 9001, }); ``` ## Iterator Method The cursor API has a new iterator method. To repeat an action over an entire collection, use `iterator()`. ```js const articles = client.collections.use("Article"); for await (const article of articles.iterator()) { // do something with article. console.log(article); // we print each object in the collection } ``` ## Type Safety The v3 client enables strong typing with custom TypeScript types and user-defined generics. You can find the type definitions in the folder that stores your Weaviate client package. The package is stored in a folder under the `node/` directory. Custom type definitions are stored in sub-folder for each bundle. For example, the `index.d.ts` file stores type definitions for the `cjs` bundle: ```bash node/cjs/index.d.ts ``` The v3 client also adds internal features that make JavaScript development more type-safe. ### Timeout values You can set timeout values, in seconds, for the client. Use the `timeout` property to configure the timeout values for initialization checks as well as query and insert operations. ```js import weaviate from 'weaviate-client'; const client = await weaviate.connectToLocal({ timeout: { query: 20, insert: 120, init: 10, } }) ``` :::tip Timeouts on `generate` queries If you see errors while using the `generate` submodule, try increasing the query timeout values (`query: 60`).

The `generate` submodule uses a large language model to generate text. The submodule is dependent on the speed of the language model and any API that serves the language model.

Increase the timeout values to allow the client to wait longer for the language model to respond. ::: --- ### Weaviate/Concepts/Cluster (docs/weaviate/concepts/cluster.md) --- title: Horizontal Scaling sidebar_position: 30 description: "Multi-node cluster architecture and horizontal scaling strategies for high-availability Weaviate deployments." image: og/docs/concepts.jpg # tags: ['architecture', 'horizontal scaling', 'cluster', 'replication', 'sharding'] --- Weaviate can be scaled horizontally by being run on a set of multiple nodes in a cluster. This section lays out various ways in which Weaviate can be scaled, as well as factors to consider while scaling, and Weaviate's architecture in relation to horizontal scaling. ## Basic concepts ### Shards A collection in Weaviate comprises of one or more "shards", which are the basic units of data storage and retrieval. A shard will contain its own vector index, inverted indexes, and object store. Each shard can be hosted on a different node, allowing for distributed data storage and processing. The number of unique shards in a single-tenant collection can only be set at collection creation time. In most cases, letting Weaviate manage the number of shards is sufficient. But in some cases, you may want to manually configure the number of shards for performance or data distribution reasons. In a multi-tenant collection, each tenant consists of one shard. This means that the number of unique shards in a multi-tenant collection is equal to the number of tenants. ### Replicas Depending on the setup, each shard can have one or more "replicas", to be hosted on different nodes. This is referred to as a "high availability" setup, where the same data is available on multiple nodes. This allows for better read throughput and fault tolerance. You can set the desired number of replicas, also called a replication factor, in Weaviate. This can be set a global cluster-level default using the [`REPLICATION_MINIMUM_FACTOR` environment variable](/docs/deploy/configuration/env-vars/index.md). It can also be set [per collection](/docs/weaviate/manage-collections/multi-node-setup.mdx#replication-settings), which will override the global default. ## Motivation to scale Weaviate Generally there are (at least) three distinct motivations to scale out horizontally which all will lead to different setups. ### Motivation 1: Maximum Dataset Size Due to the [memory footprint of an HNSW graph](./resources.md#the-role-of-memory) it may be desirable to spread a dataset across multiple servers ("nodes"). In such a setup, a single collection may be split into shards and shards are spread across nodes. The disk-based [HFresh index](./indexing/vector-index.md#hfresh-index) can also reduce the need to shard purely for memory reasons. Weaviate does the required orchestration at import and query time fully automatically. See [Sharding vs Replication](#sharding-vs-replication) below for trade-offs involved when running multiple shards. **Solution: Sharding across multiple nodes in a cluster** :::note The ability to shard across a cluster was added in Weaviate `v1.8.0`. ::: ### Motivation 2: Higher Query Throughput When you receive more queries than a single Weaviate node can handle, it is desirable to add more Weaviate nodes which can help in responding to your users' queries. Instead of sharding across multiple nodes, you can replicate (the same data) across multiple nodes. This process also happens fully automatically and you only need to specify the desired replication factor. Sharding and replication can also be combined. **Solution: Replicate your classes across multiple nodes in a cluster** ### Motivation 3: High Availability When serving critical loads with Weaviate, it may be desirable to be able to keep serving queries even if a node fails completely. Such a failure could be either due to a software or OS-level crash or even a hardware issue. Other than unexpected crashes, a highly available setup can also tolerate zero-downtime updates and other maintenance tasks. To run a highly available setup, classes must be replicated among multiple nodes. **Solution: Replicate your classes across multiple nodes in a cluster** ## Sharding vs Replication The motivation sections above outline when it is desirable to shard your classes across multiple nodes and when it is desirable to replicate your classes - or both. This section highlights the implications of a sharded and/or replicated setup. :::note All of the scenarios below assume that - as sharding or replication is increased - the cluster size is adapted accordingly. If the number of shards or the replication factor is lower than the number of nodes in the cluster, the advantages no longer apply.* ::: ### Advantages when increasing sharding * Run larger datasets * Speed up imports. To use multiple CPUs efficiently, create multiple shards for your collection. For the fastest imports, create multiple shards even on a single node. ### Disadvantages when increasing sharding * Query throughput does not improve when adding more sharded nodes ### Advantages when increasing replication * System becomes highly available * Increased replication leads to near-linearly increased query throughput ### Disadvantages when increasing replication * Import speed does not improve when adding more replicated nodes ### Sharding Keys ("Partitioning Keys") Weaviate uses specific characteristics of an object to decide which shard it belongs to. As of `v1.8.0`, a sharding key is always the object's UUID. The sharding algorithm is a 64bit Murmur-3 hash. Other properties and other algorithms for sharding may be added in the future. Note that in a multi-tenant collection, each tenant consists of one shard. ## Shard replica movement import ReplicaMovement from '/_includes/feature-notes/replica-movement.mdx'; A shard replica can be moved or copied from one node to another. This is useful when you want to balance the load across nodes or when you want to change the replication factor of a part of a collection. [See this page](/docs/deploy/configuration/replica-movement.mdx) for more details on how to move shard replicas. ### Use cases for moving shard replicas 1. **Load Balancing**: If certain nodes are experiencing higher loads than others, moving shard replicas can help distribute the load more evenly across the cluster. 2. **Scaling**: If you need to scale your cluster (e.g., adding more nodes to handle increased load), shard replicas can be moved to the new nodes to ensure that the data is evenly distributed across the cluster. 3. **Node Maintenance or Replacement**: If a node requires maintenance (e.g., hardware upgrades) or replacement, shard replicas can be moved to temporary or replacement nodes to ensure continuous availability during the maintenance window. ## Node Discovery By default, Weaviate nodes in a cluster use a gossip-like protocol through [Hashicorp's Memberlist](https://github.com/hashicorp/memberlist) to communicate node state and failure scenarios. Weaviate - especially when running as a cluster - is optimized to run on Kubernetes. The [Weaviate Helm chart](/deploy/installation-guides/k8s-installation.md#weaviate-helm-chart) makes use of a `StatefulSet` and a headless `Service` that automatically configures node discovery. All you have to do is specify the desired node count. ## Node affinity of shards and/or replication shards Weaviate tries to select the node with the most available disk space. This only applies when creating a new class, rather than when adding more data to an existing single class. ## Consistency and current limitations * From `v1.25`, Weaviate uses the [Raft consensus algorithm](https://raft.github.io/) for cluster metadata such as collection definitions and tenant activity statuses. Raft is a log-based algorithm coordinated by an elected leader, so cluster metadata changes remain consistent even if a minority of nodes fail, and concurrent schema changes are supported. For details, see [Replication architecture: Cluster metadata](/weaviate/concepts/replication-architecture/consistency.md#cluster-metadata).
If you are a Kubernetes user, see the [`1.25 migration guide`](/deploy/migration/weaviate-1-25.md) before you upgrade. To upgrade, you have to delete your existing StatefulSet. * Adding a node to an existing cluster does not by itself change the ownership of existing shards. To rebalance data across nodes, or to drain a node before you remove it, move its shard replicas with [replica movement](/deploy/configuration/replica-movement.mdx) as described in [Shard replica movement](#shard-replica-movement) above.

Behavior before `v1.25` and `v1.32`

Prior to `v1.25`, schema changes were broadcast across the cluster with a form of two-phase transaction that could not tolerate node failures during the lifetime of the transaction. Raft replaced this mechanism. See [Replication architecture: Cluster metadata](/weaviate/concepts/replication-architecture/consistency.md#cluster-metadata) for the comparison. Prior to `v1.32`, shard replicas could not be moved between nodes, so a node that still held data could not be removed from a cluster. [Replica movement](/deploy/configuration/replica-movement.mdx) removes that limitation.

## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Data Import (docs/weaviate/concepts/data-import.mdx) --- title: Data import sidebar_position: 11 description: "Theoretical explanation of client-side and server-side batch imports." image: og/docs/concepts.jpg --- Weaviate offers two flexible methods for importing data in bulk: **client-side batching** and **server-side batching**. This allows you to choose the best strategy based on your specific needs. - **Client-side batching**
In the client-side approach, **the Weaviate client library is responsible for grouping data into batches**. You define the batching mechanism and parameters, such as the size of each batch (e.g., 100 objects) using the appropriate [client library method](../manage-objects/import.mdx). The client then sends chunks to the Weaviate server accordingly. This method gives you direct control over the import process through manual tuning of parameters like the batch size and number of concurrent requests. However, the tuning must be done "blindly" on the client side, without knowledge of the server status. - **Server-side batching**
Server-side batching, or **automatic mode**, is a more robust and the recommended approach. Here, the client sends data at a rate based on **feedback from the Weaviate server**. Using an internal queue and a dynamic _[backpressure](https://en.wikipedia.org/wiki/Backpressure_routing)_ mechanism, the server tells the client how much data to send next based on its current workload. This simplifies your client code, eliminates the need for manual tuning, and results in a more efficient and resilient data import process. :::tip For **code examples**, check out the [How-to: Batch import](../manage-objects/import.mdx) guide. Server-side batch imports are supported by the Python, TypeScript, Java, and C# clients. The Go client does not yet support them; use client-side batching instead. ::: --- ## Server-side batching import SsbStatus from '/_includes/feature-notes/ssb-status.mdx'; Weaviate's server-side batching, also known as **automatic batching**, aims to provide a closed-loop system for simpler, faster, and more robust data ingestion. Instead of manually tuning batch parameters on the client side, you can let the server manage the data flow rate for optimal performance. ### How it works When an automatic batch import is initiated, the client opens a persistent connection to the server for the duration of the batch job. - **Client sends data**: Your client sends objects to the server in chunks, at a rate that is based on server-provided feedback. - **Server manages queues**: The server places incoming objects into an internal queue. This queue decouples the network communication from the actual database ingestion (like vectorization and storage). - **Dynamic backpressure**: The server continuously monitors its internal queue size. It calculates an exponential moving average (EMA) of its workload and tells the client the ideal number of objects to send in the next chunk. This feedback loop allows the system to self-regulate, maximizing throughput without overwhelming the server. - **Asynchronous errors**: If an error occurs while processing an object (e.g., validation fails), the server sends the error message back to the client over a separate, dedicated stream without interrupting the flow of objects. This architecture centralizes the complex batching logic on the server, resulting in a more efficient and stable data ingestion pipeline for all connected clients. :::info Why use automatic (server-side) batching? - **Simplified client code**: No need to tweak the batch size and the number of concurrent requests manually. The server determines the optimal batch size based on its current workload. - **Improved stability**: The system automatically applies **backpressure**. If the server is busy, it will instruct the client to send less data, preventing overloads and request timeouts, which is especially useful during long-running vectorization tasks. - **Enhanced resilience**: It's designed to handle cluster events like node scaling more gracefully, reducing the risk of interrupted batches. Because the server paces the client, the import load tracks the actual server capacity, which behaves well on autoscaling clusters and under memory pressure. ::: ## Further resources - [How-to: Batch import](../manage-objects/import.mdx) - [How-to: Create objects](../manage-objects/create.mdx) ## Questions and feedback import DocsFeedback from "/_includes/docs-feedback.mdx"; --- ### Weaviate/Concepts/Data (docs/weaviate/concepts/data.md) --- title: Data structure sidebar_position: 10 description: "Core data object concepts, schema design, and data organization principles in Weaviate." image: og/docs/concepts.jpg --- import SkipLink from '/src/components/SkipValidationLink' ## Data object concepts Each data object in Weaviate belongs to a `collection` and has one or more `properties`. Weaviate stores `data objects` in class-based collections. Data objects are represented as JSON-documents. Objects normally include a `vector` that is derived from a machine learning model. The vector is also called an `embedding` or a `vector embedding`. Each collection contains objects of the same `class`. The objects are defined by a common `schema`. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` import InitialCaps from '/_includes/schemas/initial-capitalization.md' ### JSON documents as objects Imagine we need to store information about an author named Alice Munro. In JSON format the data looks like this: ```json { "name": "Alice Munro", "age": 91, "born": "1931-07-10T00:00:00.0Z", "wonNobelPrize": true, "description": "Alice Ann Munro is a Canadian short story writer who won the Nobel Prize in Literature in 2013. Munro's work has been described as revolutionizing the architecture of short stories, especially in its tendency to move forward and backward in time." } ``` ### Vectors You can also attach `vector` representations to your data objects. Vectors are arrays of numbers that are stored under the `"vector"` property. In this example, the `Alice Munro` data object has a small vector. The vector is some information about Alice, maybe a story or an image, that a machine learning model has transformed into an array of numerical values. ```json { "id": "779c8970-0594-301c-bff5-d12907414002", "class": "Author", "properties": { "name": "Alice Munro", (...) }, "vector": [ -0.16147631, -0.065765485, -0.06546908 ] } ``` To generate vectors for your data, use one of Weaviate's vectorizer [modules](./modules.md). You can also use your own vectorizer. ### Collections Collections are groups of objects that share a schema definition. In this example, the `Author` collection holds objects that represent different authors. The collection looks like this: ```json [{ "id": "dedd462a-23c8-32d0-9412-6fcf9c1e8149", "class": "Author", "properties": { "name": "Alice Munro", "age": 91, "born": "1931-07-10T00:00:00.0Z", "wonNobelPrize": true, "description": "Alice Ann Munro is a Canadian short story writer who won the Nobel Prize in Literature in 2013. Munro's work has been described as revolutionizing the architecture of short stories, especially in its tendency to move forward and backward in time." }, "vector": [ -0.16147631, -0.065765485, -0.06546908 ] }, { "id": "779c8970-0594-301c-bff5-d12907414002", "class": "Author", "properties": { "name": "Paul Krugman", "age": 69, "born": "1953-02-28T00:00:00.0Z", "wonNobelPrize": true, "description": "Paul Robin Krugman is an American economist and public intellectual, who is Distinguished Professor of Economics at the Graduate Center of the City University of New York, and a columnist for The New York Times. In 2008, Krugman was the winner of the Nobel Memorial Prize in Economic Sciences for his contributions to New Trade Theory and New Economic Geography." }, "vector": [ -0.93070928, -0.03782172, -0.56288009 ] }] ``` Every collection has its own vector space. This means that different collections can have different embeddings of the same object. ### UUIDs Every object stored in Weaviate has a [UUID](https://en.wikipedia.org/wiki/Universally_unique_identifier). The UUID guarantees uniqueness across all collections. You can [use a deterministic UUID](../manage-objects/import.mdx#specify-an-id-value) to ensure that the same object always has the same UUID. This is useful when you want to update an object without changing its UUID. If you don't specify an ID, Weaviate generates a random UUID for you. In requests without any other ordering specified, Weaviate processes them in ascending UUID order. This means that requests to [list objects](../search/basics.md#list-objects), use of the [cursor API](../manage-objects/read-all-objects.mdx), or requests to [delete objects](../manage-objects/delete.mdx#delete-multiple-objects-by-id), without any other ordering specified, will be processed in ascending UUID order. ### Cross-references import CrossReferencePerformanceNote from '/_includes/cross-reference-performance-note.mdx'; If data objects are related, you can use [cross-references](../manage-collections/cross-references.mdx) to represent the relationships. Cross-references in Weaviate are like links that help you retrieve related information. Cross-references capture relationships, but they do not change the vectors of the underlying objects. To create a reference, use a property from one collection to specify the value of a related property in the other collection. #### Cross-reference example For example, *"Paul Krugman writes for the New York Times"* describes a relationship between Paul Krugman and the New York Times. To capture that relationship, create a cross-reference between the `Publication` object that represents the New York Times and the `Author` object that represents Paul Krugman. The New York Times `Publication` object looks like this. Note the UUID in the `"id"` field: ```json { "id": "32d5a368-ace8-3bb7-ade7-9f7ff03eddb6", "class": "Publication", "properties": { "name": "The New York Times" }, "vector": [...] } ``` The Paul Krugman `Author` object adds a new property, `writesFor`, to capture the relationship. ```json { "id": "779c8970-0594-301c-bff5-d12907414002", "class": "Author", "properties": { "name": "Paul Krugman", ... // highlight-start "writesFor": [ { "beacon": "weaviate://localhost/32d5a368-ace8-3bb7-ade7-9f7ff03eddb6", "href": "/v1/objects/32d5a368-ace8-3bb7-ade7-9f7ff03eddb6" } ], // highlight-end }, "vector": [...] } ``` The value of the `beacon` sub-property is the `id` value from the New York Times `Publication` object. Cross-reference relationships are directional. To make the link bi-directional, update the `Publication` collection to add a `hasAuthors` property points back to the `Author` collection. ### Multiple vector embeddings (named vectors) import MultiVectorSupport from '/_includes/multi-vector-support.mdx'; #### Adding a named vector after collection creation import AddNamedVectors from '/_includes/feature-notes/add-named-vectors.mdx'; A named vector can be added to an existing collection definition after collection creation. This allows you to add new vector representations for objects without having to delete and recreate the collection. When you add a new named vector to an existing collection definition, it's important to understand that **existing objects' new named vector will remain unpopulated**. Only objects created after the named vector addition will receive these new vector embeddings. This prevents any unintended side effects, such as incurring large vectorization time or costs for all existing objects in a collection. If you want to populate the new named vector for existing objects, delete and reinsert the objects manually. This will trigger the vectorization process for the new named vector. :::caution Not available for legacy (unnamed) vectorizers The ability to add a named vector after collection creation is only available for collections configured with named vectors. ::: ### Time to live (TTL) import TtlStatus from '/_includes/feature-notes/ttl-status.mdx'; Objects can be optionally set to expire after a predetermined amount of time using the Time to Live (TTL) feature. A TTL can be set at the collection level. The expiration time can be defined in relation to: - The time of object creation - The time of the last object update - A Weaviate `DATE` property value (a date & time property) The TTL value is specified in seconds. The TTL must be positive, except for those relative to a `DATE` property, which can also be negative to allow for expiration before the specified date. Expired objects are automatically deleted by Weaviate at a set of predetermined intervals. The default value can be overridden using the `OBJECTS_TTL_DELETE_SCHEDULE` environment variable or `objects_ttl_delete_schedule` configuration in the helm chart. Expired, but yet undeleted, objects can optionally be excluded from query results to prevent erroneous data retrieval before the deletion process runs. Note that for multi-tenant collections, deletions can only occur for active tenants. Deletion operations will be skipped for inactive or offloaded tenants; and the deletion will occur only when the tenant becomes active again. ## Data Schema Weaviate requires a data schema before you add data. However, you don't have to create a data schema manually. If you don't provide one, Weaviate generates a schema based on the incoming data. import SchemaDef from '/_includes/definition-schema.md'; :::note Schema vs. Taxonomy A Weaviate data schema is slightly different from a taxonomy. A taxonomy has a hierarchy. Read more about how taxonomies, ontologies and schemas are related in this Weaviate [blog post](https://medium.com/semi-technologies/taxonomies-ontologies-and-schemas-how-do-they-relate-to-weaviate-9f76739fc695). ::: Schemas fulfill several roles: 1. Schemas define collections and properties. 1. Schemas define cross-references that link collections, even collections that use different embeddings. 1. Schemas let you configure module behavior, ANN index settings, reverse indexes, and other features on a collection level. For details on configuring your schema, see the [schema tutorial](../starter-guides/managing-collections/index.mdx) or [How-to: Manage collections](../manage-collections/index.mdx). ## Multi-tenancy To separate data within a cluster, use multi-tenancy. Weaviate partitions the cluster into shards. Each shard holds data for a single tenant. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Sharding has several benefits: - Data isolation - Fast, efficient querying - Easy and robust setup and clean up Tenant shards are more lightweight. You can easily have 50,000, or more, active shards per node. This means that you can support 1M concurrently active tenants with just 20 or so nodes. Multi-tenancy is especially useful when you want to store data for multiple customers, or when you want to store data for multiple projects. :::caution Tenant deletion == Tenant data deletion Deleting a tenant deletes the associated shard. As a result, deleting a tenant also deletes all of its objects. ::: ### Tenant states Tenants have an activity status (also called a tenant state) that reflects their availability and storage location. A tenant can be `ACTIVE`, `INACTIVE`, `OFFLOADED`, `OFFLOADING`, or `ONLOADING`. - `ACTIVE` tenants are loaded and available for read and write operations. - In all other states, the tenant is not available for read or write access. Access attempts return an error message. - `INACTIVE` tenants are stored on local disk storage for quick activation. - `OFFLOADED` tenants are stored on cloud storage. This status is useful for long-term storage for tenants that are not frequently accessed. - `OFFLOADING` tenants are being moved to cloud storage. This is a transient status, and therefore not user-specifiable. - `ONLOADING` tenants are being loaded from cloud storage. This is a transient status, and therefore not user-specifiable. An `ONLOADING` tenant may be being warmed to a `ACTIVE` status or a `INACTIVE` status. For more details on managing tenants, see [Multi-tenancy operations](../manage-collections/multi-tenancy.mdx). | Status | Available | Description | User-specifiable | | :-- | :-- | :-- | :-- | | `ACTIVE` | Yes | Loaded and available for read/write operations. | Yes | | `INACTIVE` | No | On local disk storage, no read / write access. Access attempts return an error message. | Yes | | `OFFLOADED` | No | On cloud storage, no read / write access. Access attempts return an error message. | Yes | | `OFFLOADING` | No | Being moved to cloud storage, no read / write access. Access attempts return an error message. | No | | `ONLOADING` | No | Being loaded from cloud storage, no read / write access. Access attempts return an error message. | No | :::info Tenant status renamed in `v1.26` In `v1.26`, the `HOT` status was renamed to `ACTIVE` and the `COLD` status was renamed to `INACTIVE`. ::: :::info Tenant state propagation A tenant state change may take some time to propagate across a cluster, especially a multi-node cluster.
For example, data may not be immediately available after reactivating an offloaded tenant. Similarly, data may not be immediately unavailable after offloading a tenant. This is because the [tenant states are eventually consistent](../concepts/replication-architecture/consistency.md#tenant-states-and-data-objects), and the change must be propagated to all nodes in the cluster. ::: #### Offloaded tenants import OffloadingLimitation from '/_includes/offloading-limitation.mdx'; Offloading tenants requires the relevant `offload-` module to be [enabled](../configuration/modules.md) in the Weaviate cluster. When a tenant is offloaded, the entire tenant shard is moved to cloud storage. This is useful for long-term storage of tenants that are not frequently accessed. Offloaded tenants are not available for read or write operations until they are loaded back into the cluster. ### Backups :::caution Backups do not include inactive or offloaded tenants Backups of multi-tenant collections will only include `active` tenants, and not `inactive` or `offloaded` tenants. [Activate tenants](../manage-collections/multi-tenancy.mdx#manage-tenant-states) before creating a backup to ensure all data is included. ::: ### Tenancy and IDs Each tenancy is like a namespace, so different tenants could, in theory, have objects with the same IDs. To avoid naming problems, object IDs in multi-tenant clusters combine the tenant ID and the object ID to create an ID that is unique across tenants. ### Tenancy and cross-references Multi-tenancy supports some cross-references. Cross-references like these are supported: - From a multi-tenancy object to a non-multi-tenancy object. - From a multi-tenancy object to another multi-tenancy object, as long as they belong to the same tenant. Cross-references like these are not supported: - From a non-multi-tenancy object to a multi-tenancy object. - From a multi-tenancy object to another multi-tenancy object if they belong to different tenants. ### Key features - Each tenant has a dedicated, high-performance vector index. Dedicated indexes mean faster query speeds. Instead of searching a shared index space, each tenant responds as if it was the only user on the cluster. - Each tenant's data is isolated on a dedicated shard. This means that deletes are fast and do not affect other tenants. - To scale out, add a new node to your cluster. Weaviate does not redistribute existing tenants, however Weaviate adds new tenants to the node with the least resource usage. :::info Related pages - [How-to: Manage Data | Multi-tenancy operations](../manage-collections/multi-tenancy.mdx) - [Multi-tenancy blog](https://weaviate.io/blog/multi-tenancy-vector-search) ::: ### Monitoring metrics To group tenants together for monitoring, set [`PROMETHEUS_MONITORING_GROUP = true`](/deploy/configuration/env-vars/index.md) in your system configuration file. ### Number of tenants per node The number of tenants per node is limited by operating system constraints. The number of tenants cannot exceed the Linux open file limit per process. For example, a 9-node test cluster built on `n1-standard-8` machines holds around 170k active tenants. There are 18,000 to 19,000 tenants per node. Note that these numbers relate to active tenants only. If you [set unused tenants as `inactive`](../manage-collections/multi-tenancy.mdx#manage-tenant-states), the open file per process limit does not apply. ## Related pages For more information, see the following: - [How-to: Multi-tenancy operations](../manage-collections/multi-tenancy.mdx) - References: REST API: Schema - [How-to: Manage collections](../manage-collections/index.mdx) ## Summary * The schema defines collections and properties. * Collections contain data objects that are describe in JSON documents. * Data objects can contain a vector and properties. * Vectors come from machine learning models. * Different collections represent different vector spaces. * Cross-references link objects between schemas. * Multi-tenancy isolates data for each tenant. ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Filtering (docs/weaviate/concepts/filtering.md) --- title: Filtering sidebar_position: 26 description: "Filtered vector search capabilities combining semantic similarity with structured scalar filtering." image: og/docs/concepts.jpg # tags: ['architecture', 'filtered vector search', 'pre-filtering'] --- Weaviate provides powerful filtered vector search capabilities, allowing you to combine vector searches with structured, scalar filters. This enables you to find the closest vectors to a query vector that also match certain conditions. Filtered vector search in Weaviate is based on the concept of pre-filtering. This means that the filter is constructed before the vector search is performed. Unlike some pre-filtering implementations, Weaviate's pre-filtering does not require a brute-force vector search and is highly efficient. Starting in `v1.34`, Weaviate uses the [`ACORN`](#acorn) filter strategy as the default. This filtering method significantly improves performance for large datasets, especially when the filter has low correlation with the query vector. Because the [HFresh index](./indexing/vector-index.md#hfresh-index) uses an HNSW index for its centroids, it also benefits from ACORN when routing filtered queries. ## Post-Filtering vs Pre-Filtering Systems that cannot make use of pre-filtering typically have to make use of post-filtering. This is an approach where a vector search is performed first and then some results are removed which do not match the filter. This leads to two major disadvantages: 1. You cannot easily predict how many elements will be contained in the search, as the filter is applied to an already reduced list of candidates. 2. If the filter is very restrictive, i.e. it matches only a small percentage of data points relative to the size of the data set, there is a chance that the original vector search does not contain any match at all. The limitations of post-filtering are overcome by pre-filtering. Pre-Filtering describes an approach where eligible candidates are determined before a vector search is started. The vector search then only considers candidates that are present on the "allow" list. :::note Some authors make a distinction between "pre-filtering" and "single-stage filtering" where the former implies a brute-force search and the latter does not. We do not make this distinction, as Weaviate does not have to resort to brute-force searches, even when pre-filtering due to the its combined inverted index and HNSW index. ::: ## Efficient Pre-Filtered Searches in Weaviate In the section about Storage, [we have described in detail which parts make up a shard in Weaviate](./storage.md). Most notably, each shard contains an inverted index right next to the HNSW index. This allows for efficient pre-filtering. The process is as follows: 1. An inverted index (similar to a traditional search engine) is used to create an allow-list of eligible candidates. This list is essentially a list of `uint64` ids, so it can grow very large without sacrificing efficiency. 2. A vector search is performed where the allow-list is passed to the HNSW index. The index will move along any node's edges normally, but will only add ids to the result set that are present on the allow list. The exit conditions for the search are the same as for an unfiltered search: The search will stop when the desired limit is reached and additional candidates no longer improve the result quality. ## Filter strategy Weaviate supports two filter strategies: `sweeping` and `acorn` specifically for the HNSW index type. ### ACORN The Weaviate filtering algorithm `ACORN` is based on the paper [ACORN: Performant and Predicate-Agnostic Search Over Vector Embeddings and Structured Data](https://arxiv.org/html/2403.04871v1). We refer to this as `ACORN`, but the actual implementation in Weaviate is a custom implementation that is inspired by the paper. (References to `ACORN` in this document refer to the Weaviate implementation.) The `ACORN` algorithm is designed to speed up filtered searches with the [HNSW index](./indexing/vector-index.md#hierarchical-navigable-small-world-hnsw-index) by the following: - Objects that do not meet the filters are ignored in distance calculations. - The algorithm reaches the relevant part of the HNSW graph faster, by using a multi-hop approach to evaluate the neighborhood of candidates. - Additional entrypoints matching the filter are randomly seeded to speed up convergence to the filtered zone. The `ACORN` algorithm is especially useful when the filter has low correlation with the query vector. In other words, when a filter excludes many objects in the region of the graph most similar to the query vector. Our internal testing indicates that for lowly correlated, restrictive filters, the `ACORN` algorithm can be significantly faster, especially for large datasets. If this has been a bottleneck for your use case, we recommend enabling the `ACORN` algorithm. ### Sweeping The `sweeping` strategy is based on the concept of "sweeping" through the HNSW graph. The algorithm starts at the root node and traverses the graph, evaluating the distance to the query vector at each node, while keeping the "allow list" of the filter as context. If the filter is not met, the node is skipped and the traversal continues. This process is repeated until the desired number of results is reached. The `sweeping` algorithm can be enabled by setting the `filterStrategy` field for the relevant HNSW vector index [in the collection configuration](../manage-collections/vector-config.mdx#set-vector-index-parameters). ## `indexFilterable` {#indexFilterable} The `indexFilterable` index speeds up match-based filtering through use of Roaring Bitmaps. Roaring Bitmaps employ various strategies to add efficiencies, whereby it divides data into chunks and applies an appropriate storage strategy to each one. This enables high data compression and set operations speeds, resulting in faster filtering speeds for Weaviate. If you are dealing with a large dataset, this will likely improve your filtering performance significantly and we therefore encourage you to migrate and re-index. In addition, our team maintains our underlying Roaring Bitmap library to address any issues and make improvements as needed. #### `indexFilterable` for `text` properties A roaring bitmap index for `text` properties is implemented using two separate (`filterable` & `searchable`) indexes, which replaces the existing single index. You can configure the new `indexFilterable` and `indexSearchable` parameters to determine whether to create the roaring set index and the BM25-suitable Map index, respectively. (Both are enabled by default.) :::info Read more To learn more about Weaviate's roaring bitmaps implementation, see the [in-line documentation](https://pkg.go.dev/github.com/weaviate/weaviate/adapters/repos/db/lsmkv/roaringset). ::: ## `indexRangeFilters` The `indexRangeFilters` index is a range-based index for filtering by numerical ranges. This index is available for `int`, `number`, or `date` properties. The index is not available for arrays of these data types. Internally, rangeable indexes are implemented as roaring bitmap slices. This data structure limits the index to values that can be stored as 64 bit integers. `indexRangeFilters` is only available for new properties. Existing properties cannot be converted to use the rangeable index. ## Recall on Pre-Filtered Searches Thanks to Weaviate's custom HNSW implementation, which persists in following all links in the HNSW graph normally and only applying the filter condition when considering the result set, graph integrity is kept intact. The recall of a filtered search is typically not any worse than that of an unfiltered search. The graphic below shows filters of varying levels of restrictiveness. From left (100% of dataset matched) to right (1% of dataset matched) the filters become more restrictive without negatively affecting recall on `k=10`, `k=15` and `k=20` vector searches with filters. ## Flat-Search Cutoff Weaviate offers an option to automatically switch to a flat (brute-force) vector search when a filter becomes too restrictive. This scenario only applies to combined vector and scalar searches. For a detailed explanation of why HNSW requires switching to a flat search on certain filters, see this article at [medium](https://medium.com/data-science/effects-of-filtered-hnsw-searches-on-recall-and-latency-434becf8041c). In short, if a filter is very restrictive (i.e. a small percentage of the dataset is matched), an HNSW traversal becomes exhaustive. In other words, the more restrictive the filter becomes, the closer the performance of HNSW is to a brute-force search on the entire dataset. However, with such a restrictive filter, we have already narrowed down the dataset to a small fraction. So if the performance is close to brute-force anyway, it is much more efficient to only search on the matching subset as opposed to the entire dataset. The following graphic shows filters with varying restrictiveness. From left (0%) to right (100%), the filters become more restrictive. The **cut-off is configured at ~15% of the dataset** size. This means the right side of the dotted line uses a brute-force search. As a comparison, with pure HNSW - without the cutoff - the same filters would look like the following: The cutoff value can be configured as [part of the `vectorIndexConfig` settings in the schema](/weaviate/config-refs/indexing/vector-index.mdx#hnsw-index) for each collection separately. ## Further resources - [References: GraphQL API - Filters](../api/graphql/filters.md) ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Index (docs/weaviate/concepts/index.md) --- title: Concepts sidebar_position: 0 description: "Foundational concepts and architecture principles behind Weaviate's vector search and AI database capabilities." image: og/docs/concepts.jpg # tags: ['getting started'] --- The **Concepts** section explains various aspects related to Weaviate and its architecture to help you get the most out of it. You can read these sections in any order. :::info Quickstart If you are after a practical guide with code examples, check out the [quickstart tutorial](/weaviate/quickstart/index.md). ::: import AcademyAdmonition from '@site/src/components/AcademyAdmonition'; ## Core concepts **[Data structure](./data.md)** - How Weaviate deals with data objects, including how they are stored, represented, and linked to each other. **[Modules](./modules.md)** - An overview of Weaviate's module system, including what can be done with modules, existing module types, and custom modules. **[Indexing](./indexing/index.md)** - Read how data is indexed within Weaviate using inverted and ANN indexes, and about configurable settings. **[Vector indexing](./indexing/vector-index.md)** - Read more about Weaviate's vector indexing architecture, such as the HNSW algorithm, distance metrics, and configurable settings. **[Vector quantization](./vector-quantization.md)** - Read more about Weaviate's vector quantization options. ## Weaviate Architecture The figure below gives a 30,000 feet view of Weaviate's architecture. [](./img/weaviate-architecture-overview.svg) You can learn more about the individual components in this figure by following these guides: **[Learn about storage inside a shard](./storage.md)** * How Weaviate stores data * How Weaviate makes writes durable * How an inverted index, a vector index and an object store interact with each other **[Ways to scale Weaviate horizontally](./cluster.md)** * Different motivations to scale * Sharding vs. Replication * Configuring a cluster * Consistency **[How to plan resources](./resources.md)** * The roles of CPU, Memory and GPUs * How to size a cluster correctly * Speeding up specific processes * Preventing bottlenecks **[Filtered vector search](./filtering.md)** * Combine vector search with filters * Learn how combining an HNSW with an inverted index leads to high-recall, high-speed filtered queries **[User-facing interfaces](./interface.md)** * Design philosophy behind user-facing APIs * Role of the REST and GraphQL APIs **[Replication architecture](./replication-architecture/index.md)** * About replication * Weaviate's implementation * Use cases ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Interface (docs/weaviate/concepts/interface.md) --- title: Interface sidebar_position: 85 description: "RESTful, GraphQL and gRPC API interfaces with client library support for Weaviate integration." image: og/docs/concepts.jpg # tags: ['architecture', 'interface', 'API design'] --- You can manage and use Weaviate through its APIs. Weaviate has RESTful, GraphQL, and gRPC APIs. The client libraries broadly mirror this API surface, although feature coverage can vary by language; see the [client library pages](/weaviate/client-libraries/index.mdx) for what each one supports. Some clients, e.g. the Python client, have additional functionality, such as full schema management and batching operations. This way, Weaviate is easy to use in custom projects. Additionally, the APIs are intuitive, so it is easy to integrate into your existing data landscape. This page contains information on how Weaviate's APIs are designed, and how you can use Weaviate Console to search through your Weaviate instance with GraphQL. ## API Design ### Design: UX & Weaviate Features User Experience (UX) is one of our most valuable principles. Weaviate should be easy to understand, intuitive to use and valuable, desirable and usable to the community. The interaction with Weaviate is naturally very important for its UX. Weaviate's APIs are designed from the perspective of user needs, keeping the software features in mind. We do user research, user testing and prototyping to make sure all features resonate with our users. User requirements are continuously gathered during collaborative discussions. We match user needs with the functions of Weaviate. When there is a strong need from the user or application perspective, we may extend Weaviate's functions and APIs. When there is a new Weaviate function, this will naturally be accessible via (new) API functions. The UX of Weaviate's APIs is designed following the UX Honeycomb usability rules, defined by Peter Morville. ### RESTful API and GraphQL API Weaviate has both a RESTful API and a GraphQL API. Currently, there is no feature parity between both APIs (this will be implemented later, there is an [issue](https://github.com/weaviate/weaviate/issues/1540) on GitHub). The RESTful APIs are mostly used for DB management and CRUD operations. The GraphQL API is mostly used to access data objects in Weaviate, whether it's a simple lookup or a combination of scalar and vector search. The APIs support the following user needs, roughly speaking: - **Adding, retrieving, updating and deleting data CRUD** -> RESTful API - **Weaviate management operations** -> RESTful API - **Data search** -> GraphQL API - **Explorative data search** -> GraphQL API - **Data analysis (meta data)** -> GraphQL API - **Near real time on very large datasets in production** -> Client libraries (Python, Go, Java, JavaScript, C#) using both APIs under the hood - **Easy to integrate in applications** -> Client libraries (Python, Go, Java, JavaScript, C#) using both APIs under the hood ## GraphQL ### Why GraphQL? We have chosen to use a GraphQL API, for multiple reasons: - **Data structure**. - Data in Weaviate follows a class-property structure. Data objects can be queried by their class and properties with GraphQL. - It is possible to link data in Weaviate with cross-references. A Graph query language like GraphQL is very useful here. - **Performance**. - With GraphQL, there is no over/under-fetching. You get back exactly the information about data objects that you query, nothing more and nothing less. This is beneficial for performance. - Reducing the number of requests. With GraphQl, you can make highly efficient and precise queries that usually require many more queries with a traditional RESTful API for the same results. - **User Experience** - Reducing complexity. - Less error-prone (because of its typed schema) - Custom design - Data exploration and fuzzy search is possible ### GraphQL Design Principles GraphQL queries are designed to be intuitive and fit Weaviate's features. [This article on Hackernoon](https://hackernoon.com/how-weaviates-graphql-api-was-designed-t93932tl) tells you more about how GraphQL API was designed (note that examples show an older Weaviate and GraphQL API version). The following three points are key in the design: - **Natural language**. The GraphQL queries follow a natural language pattern as much as possible. The function of a query is easy to understand and queries are easy to write and remember. An example query where you can recognize human language is: "_Get_ the _title_ of the _Articles_ where the _wordcount_ is _greater than_ _1000_. The most important words in this query are also used in the GraphQL query: ```graphql { Get { Article( where: { path: ["wordCount"] # Path to the property that should be used operator: GreaterThan # operator valueInt: 1000 # value (which is always = to the type of the path property) } ) { title } } } ``` There are currently three main functions in a GraphQL request: "Get{}", "Explore{}" and "Aggregate{}". - **Classes & properties**. Data in Weaviate has a class-property structure, where cross-references may appear between data object. The class name of the data to return is written one layer deeper than the 'main function'. The next layer consists of the properties and cross-reference properties to return per class: ```graphql { { {

{ ... on { } } _ { } } } } ``` - **Query filters (search arguments) dependent on database setup**. You can add filters on class level to filter objects. Scalar (`where` filters) can be combined with vector (`near<...>`) filters. Depending on your Weaviate setup (which modules you have connected), additional filters may be used. A filter can look like (using the [`qna-transformers` module](/weaviate/modules/qna-transformers.md)): ```graphql { Get { Article( ask: { question: "Who is the king of the Netherlands?" properties: ["summary"] } limit: 1 ) { title _additional { answer { result } } } } } ``` ### GraphQL Design of Main Functions 1. **Data search: `Get {}`**: to search for data objects when you know the class name of the data objects you're looking for. 2. **Explorative & fuzzy search: `Explore {}`**: to search in a fuzzy way, when you don't know the data schema and class names. 3. **Data analysis (meta data): `Aggregate {}`**: to search for meta data, and do data analysis of data aggregations. ## gRPC API support Alongside the RESTful and GraphQL APIs, Weaviate serves a gRPC API. gRPC is built on HTTP/2 and Protocol Buffers, which makes it faster and more efficient than sending the equivalent request as JSON over HTTP. It was introduced in Weaviate `v1.19.0` and has been considered stable since `v1.23.7`. gRPC carries most of the search and batch import traffic that the client libraries generate, so a Weaviate deployment usually exposes a gRPC port (`50051` by default, configurable with the `GRPC_PORT` [environment variable](/deploy/configuration/env-vars/index.md)) in addition to the REST port. Client coverage is not uniform: the [Python](/weaviate/client-libraries/python/index.mdx), [TypeScript](/weaviate/client-libraries/typescript/index.mdx), [Java](/weaviate/client-libraries/java/index.mdx), and [C#](/weaviate/client-libraries/csharp.mdx) clients use gRPC for queries and batch operations, while the [Go](/weaviate/client-libraries/go.md) client uses it for batch imports and reaches gRPC search through its experimental API. For the Protobuf definitions and for ways to call the API without a client library, see the [gRPC API reference](../api/grpc.md). ## Weaviate Console The [Weaviate Console](/go/console?utm_content=others) is a dashboard to manage Weaviate clusters from WCD, and access Weaviate instances running elsewhere. You can use the Query Module to make GraphQL queries. ## Weaviate Clients Weaviate has several client libraries: in [C#](/weaviate/client-libraries/csharp.mdx), [Go](/weaviate/client-libraries/go.md), [Java](/weaviate/client-libraries/java/index.mdx), [Python](/weaviate/client-libraries/python/index.mdx), and [TypeScript/JavaScript](/weaviate/client-libraries/typescript/index.mdx). The client libraries broadly mirror the server API surface, although feature coverage varies by language. See the [client library pages](/weaviate/client-libraries/index.mdx) for what each one supports. Some clients, e.g. the Python client, have additional functionality, such as full schema management and batching operations. This way, Weaviate is easy to use in custom projects. The APIs are intuitive to use, so it is easy to integrate Weaviate into your existing data landscape. ## Further resources :::info Related pages - [References: GraphQL API](../api/graphql/index.md) - [References: RESTful API](/weaviate/api/rest). - [References: Client Libraries](../client-libraries/index.mdx). ::: ## Questions and feedback import DocsFeedback from '/\_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Modules (docs/weaviate/concepts/modules.md) --- title: Modules sidebar_position: 15 description: "Modular architecture overview for extending Weaviate functionality with specialized add-on components." image: og/docs/concepts.jpg # tags: ['modules'] --- Weaviate has a modularized structure. Functionality such as vectorization or backups is handled by *optional* modules. The core of Weaviate, without any modules attached, is a pure vector-native database. [](./img/weaviate-module-diagram.svg) Data is stored in Weaviate as the combination of an object and its vector, and these vectors are searchable by the provided [vector index algorithm](../concepts/indexing/vector-index.md). Without any vectorizer modules attached, Weaviate does not know how to *vectorize* an object, i.e. *how* to calculate the vectors given an object. Depending on the type of data you want to store and search (text, images, etc.), and depending on the use case (like search, question answering, etc., depending on language, classification, ML model, training set, etc.), you can choose and attach a vectorizer module that best fits your use case. Or, you can "bring your own" vectors to Weaviate. This page explains what modules are, and what purpose they serve in Weaviate. ## Available module types This graphic displays the available modules for the latest Weaviate version (||site.weaviate_version||) . Modules are grouped into these categories: - Vectorization modules - Vectorization and additional functionality modules - Other modules ### Vectorizer & Ranker modules Vectorizer modules, like the `text2vec-*`, `multi2vec-*` or `img2vec-*` modules, transform data into vectors. Ranker modules, like the `rerank-*` modules, rank the results. ### Reader & Generator modules Reader or Generator modules can be used on top of a Vectorizer module. These modules take the set of relevant documents that are retrieved, and performs another operation, such as question answering, or a generative task. An example Reader module is [`qna-transformers`](../modules/qna-transformers.md) module, which extracts an answer directly from a document. A Generator module would, on the other hand, use *language generation* to generate an answer from the given document. ### Other modules These include those such as `backup-gcs` or `text-spellcheck`. ## Dependencies Modules can be dependent on other modules to be present. For example, to use the [`qna-transformers`](../modules/qna-transformers.md) module, *exactly one* text vectorization module is required. ## Weaviate without modules Weaviate can also be used without any modules, as pure vector native database and search engine. If you choose not to include any modules, you will need to enter a vector for each data entry. You can then search through the objects by a vector as well. ## Custom modules It is possible for anyone to create a custom module for use with Weaviate. Click [here](../modules/custom-modules.md) to see how you can create and use your own modules. ## Further resources :::info Related pages - [Configuration: Modules](../configuration/modules.md) - [References: Modules](../modules/index.md) ::: ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Reranking (docs/weaviate/concepts/reranking.md) --- title: Reranking sidebar_position: 28 description: "Search result reordering techniques using alternative models to improve search relevance and accuracy." image: og/docs/concepts.jpg # tags: ['basics'] --- Reranking seeks to improve search relevance by reordering the result set returned by a search with a different model. Reranking computes a relevance score between the query and each data object, and returns the list of objects sorted from the most to the least relevant. Computing this score for all `(query, data_object)` pairs would typically be prohibitively slow, which is why reranking is used as a second stage after retrieving the relevant objects first. As the reranker works on a smaller subset of data after retrieval, different, potentially more computationally expensive approaches can be used to improve search relevance. :::info Learn how to [set up a reranker for your collection](../manage-collections/generative-reranker-models.mdx#specify-a-reranker-model-integration) and [apply reranking to your search results](../search/rerank.md). ::: ## Reranking in Weaviate With our reranker modules, you can conveniently perform [multi-stage searches](https://weaviate.io/blog/cross-encoders-as-reranker) without leaving Weaviate. In other words, you can perform a search - for example, a vector search - and then use a reranker to re-rank the results of that search. Our reranker modules are compatible with all of vector, bm25, and hybrid searches. ### An example GraphQL query with a reranker You can use reranking in a GraphQL query as follows: ```graphql { Get { JeopardyQuestion( nearText: { concepts: "flying" } limit: 10 ) { answer question _additional { distance rerank( property: "answer" query: "floating" ) { score } } } } } ``` This query retrieves 10 results from the `JeopardyQuestion` class, using a hybrid search with the query “flying”. It then re-ranks the results using the `answer` property, and the query “floating”. You can specify which `property` of the `JeopardyQuestion` class you want to pass to the reranker. Note that here, the returned `score` will include the score from the reranker. ## Further resources :::info Related pages - [API References: GraphQL - Additional properties](../api/graphql/additional-properties.md#rerank) - [How-to search: Rerank](../search/rerank.md) - [Cohere reranker integration](../model-providers/cohere/reranker.md) - [Transformers reranker integration](../model-providers/transformers/reranker.md) - [VoyageAI reranker integration](../model-providers/voyageai/reranker.md) ::: ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Resources (docs/weaviate/concepts/resources.md) --- title: Resource Planning sidebar_position: 90 description: "CPU, memory, and GPU resource planning guidelines for optimal Weaviate performance at scale." image: og/docs/concepts.jpg # tags: ['architecture', 'resource', 'cpu', 'memory', 'gpu'] --- Weaviate scales well for large projects. Smaller projects, less than 1M objects, do not require resource planning. For medium and large-scale projects, you should plan how to get the best performance from your resources. While you design you system, keep in mind CPU and memory management. CPU and memory are the primary resources for Weaviate instances. Depending on the modules you use, GPUs may also play a role. ## Limit available resources You can set [environment variables](/deploy/configuration/env-vars/index.md) to manage Weaviate's resource usage, as to prevent Weaviate from using all available resources. The following environment variables are available: - [`LIMIT_RESOURCES`](/deploy/configuration/env-vars/index.md#LIMIT_RESOURCES): When set to true, Weaviate automatically limits its resource usage. It sets memory usage to 80% of the total memory and uses all but one CPU core. It overrides any `GOMEMLIMIT` values but respects `GOMAXPROCS` settings. - [`GOMEMLIMIT`](/deploy/configuration/env-vars/index.md#GOMEMLIMIT): This sets the memory limit for the Go runtime, which should be around 80-90% of the total memory available for Weaviate. It controls the aggressiveness of the Garbage Collector as memory usage approaches this limit. - [`GOMAXPROCS`](/deploy/configuration/env-vars/index.md#GOMAXPROCS): This sets the maximum number of threads for concurrent execution. If set, it's respected by `LIMIT_RESOURCES`, allowing users to specify the exact number of CPU cores Weaviate should use. These settings help in optimizing Weaviate's performance by balancing resource utilization with the available system resources. ## The role of CPUs :::tip Rule of thumb The CPU has a direct effect on query and import speed, but does not affect dataset size. ::: Vector search is the most CPU intensive process in Weaviate operations. Queries are CPU-bound, but imports are also CPU-bound because imports rely on vector search for indexing. Weaviate uses the HNSW (Hierarchical Navigable Small World) algorithm to index vectors. You can [tune the HNSW index](../config-refs/indexing/vector-index.mdx) on a per collection basis in order to maximize performance for your primary use case. To use multiple CPUs efficiently, create multiple shards for your collection. For the fastest imports, create multiple shards even on a single node. Each insert, or search, is single-threaded. However, if you make multiple searches or inserts at the same time, Weaviate can make use of multiple threads. [Batch inserts](/weaviate/manage-objects/import) use multiple threads to process data in parallel. ### When to add more CPUs When CPU utilization is high during importing, add CPUs to increases import speed. When search throughput is limited, add CPUs to increase the number of queries per second. ## The role of memory :::tip Rule of thumb Memory determines the maximum supported dataset size. Memory does not directly influence query speed. ::: The HNSW index must be stored in memory. The memory required is directly related to the size of your dataset. There is no correlation between the size of your dataset and the current query load. You can use [`product quantization (PQ)`](/weaviate/concepts/vector-quantization#product-quantization) to compress the vectors in your dataset and increase the number of vectors you can hold in memory. If memory is your main constraint, the disk-based [HFresh index](/weaviate/concepts/vector-index#hfresh-index) is another option. Weaviate lets you configure a limit to the number of vectors held in memory in order to prevent unexpected Out-of-Memory ("OOM") situations. The default value is one trillion (`1e12`) objects per collection. To adjust the number of objects, update the value of [`vectorCacheMaxObjects`](../config-refs/indexing/vector-index.mdx) in your index settings. Weaviate also uses [memory-mapped files](https://en.wikipedia.org/wiki/Memory-mapped_file) for data stored on disks. Memory-mapped files are efficient, but disk storage is much slower than in-memory storage. ### Which factors drive memory usage? The HNSW vector index is the primary driver of memory usage. These factors influence the amount of memory Weaviate uses: - **The total number of object vectors**. The number of vectors is important, but the raw size of the original objects is not important. Only the vector is stored in memory. The size of the original text or other data is not a limiting factor. - **The `maxConnections` HNSW index setting**. Each object in memory has at most [`maxConnections`](../config-refs/indexing/vector-index.mdx) connections per layer. Each of the connections uses 8-10B of memory. Note that the base layer allows for `2 * maxConnections`. ### An example calculation :::note The following calculation assumes that you want to hold all vectors in memory. For a hybrid approach that combines in memory and on-disk storage, see [Vector Cache](#vector-cache) below. ::: To estimate your memory needs, use the following rule of thumb: `Memory usage = 2 * (the memory footprint of all vectors)` For example, consider a model that has one million 384-dimensional vectors of type `float32`. - The memory requirement for a single vector is: `384 * 4 B = 1536 B`. - The memory requirement for one million objects is: `1e6 * 1536 B = 1.5G B` The rule of thumb says to double the memory requirement. The total memory requirement for one million 384-dimensional vectors of type `float32` is: `2 * 1e6 * 1536 B = 3 GB`. For a more accurate calculation, include a factor for the [`maxConnections`](../config-refs/indexing/vector-index.mdx) setting instead of multiplying the base requirement by two. For example, if `maxConnections` is 64 and the other values are the same, a more accurate memory estimate is `1e6 * (1536B + (64 * 10)) = 2.2 GB`. The estimate that includes `maxConnections` is smaller than the rule of thumb estimate. However, the `maxConnections` estimate doesn't account for garbage collection. Garbage collection adds overhead that is explained in the next section. ## Effects of garbage collection Weaviate is written in Go, which is a garbage-collected language. This means some memory is not immediately available for reuse when it is no longer needed. The application has to wait for an asynchronous process, the garbage collector, to free up the memory. This has two distinct effects on memory use: - [Memory overhead](#memory-overhead-for-the-garbage-collector) - [Out-of-memory issues](#out-of-memory-issues-due-to-garbage-collection) ### Memory overhead for the garbage collector The memory calculation that includes `maxConnections` describes the system state at rest. However, while Weaviate imports vectors, additional memory is allocated and eventually freed by the garbage collector. Since garbage collection is an asynchronous process, this additional memory must also be accounted for. The 'rule of thumb' formula accounts for garbage collection. ### Out-of-Memory issues due to garbage collection In rare situations - typically on large machines with very high import speeds - Weaviate can allocate memory faster than the garbage collector can free it. When this happens, the system kernel can trigger an `out of memory kill (OOM-Kill)`. This is a known issue that Weaviate is actively working on. ### Data import To avoid out-of-memory issues during imports, set `LIMIT_RESOURCES` to `True` or configure the `GOMEMLIMIT` environment variable. For details, see [Environment variables](/deploy/configuration/env-vars/index.md). ## Strategies to reduce memory usage The following tactics can help to reduce Weaviate's memory usage: - **Use vector compression**. Product quantization (PQ) is a technique that reduces the size of vectors. Vector compression impacts recall performance, so we recommend testing PQ on your dataset before using it in production.

For more information, see [Product Quantization](/weaviate/concepts/vector-quantization).
To configure PQ, see [Compression](../configuration/compression/pq-compression.md). - **Reduce the dimensionality of your vectors.** The most effective approach to reducing memory size, is to reduce the number of dimensions per vector. If you have high dimension vectors, consider using a model that uses fewer dimensions. For example, a model that has 384 dimensions uses far less memory than a model with 1536 dimensions. - **Reduce the number of [`maxConnections`](../config-refs/indexing/vector-index.mdx) in your HNSW index settings**. Each object in memory has up to `maxConnections` connections. Each of those connections uses 8-10B of memory. To reduce the overall memory footprint, reduce `maxConnections`. Reducing `maxConnections` adversely affects HNSW recall performance. To mitigate this effect, increase one or both of the `efConstruction` and `ef` parameters. - Increasing `efConstruction` increases import time without affecting query times. - Increasing `ef` increases query times without affecting import times. - **Use a vector cache that is smaller than the total amount of your vectors (not recommended)**. This strategy is described under [Vector Cache](#vector-cache) below. It has a significant performance impact, and is only recommended in specific, limited situations. ## Vector Cache For optimal search and import performance, all previously imported vectors need to be held in memory. The size of the vector cache is specified by the [`vectorCacheMaxObjects`](../config-refs/indexing/vector-index.mdx) parameter in the collection definition. By default this limit is set to one trillion (`1e12`) objects when you create a new collection. You can reduce the size of `vectorCacheMaxObjects`, but a disk lookup for a vector is orders of magnitudes slower than memory lookup. Only reduce the size of `vectorCacheMaxObjects` with care and as a last resort. Generally we recommend that: - During import set `vectorCacheMaxObjects` high enough that all vectors can be held in memory. Each import requires multiple searches. Import performance drop drastically when there isn't enough memory to hold all of the vectors in the cache. - After import, when your workload is mostly querying, experiment with vector cache limits that are less than your total dataset size. Vectors that aren't currently in cache are added to the cache if there is still room. If the cache fills, Weaviate drops the whole cache. All future vectors have to be read from disk for the first time. Then, subsequent queries runs against the cache, until it fills again and the procedure repeats. Note that the cache can be a very valuable tool if you have a large dataset, and a large percentage of users only query a specific subset of vectors. In this case you might be able to serve the largest user group from cache while requiring disk lookups for "irregular" queries. ### When to add more Memory to your Weaviate machine or cluster Consider adding more memory if: - You want to import a larger dataset (more common). - Exact lookups are disk-bound and more memory will improve page-caching (less common). ## The role of GPUs in Weaviate Weaviate Database itself does not make use of GPUs. However, some of the models that Weaviate includes as modules are meant to run with GPUs, for example `text2vec-transformers`, `qna-transformers`, and `ner-transformers`. These modules run in isolated containers, so you can run the module containers on GPU-accelerated hardware while running Weaviate Database on low-cost CPU-only hardware. ## Disks: SSD vs Spinning Disk Weaviate is optimized to work with Solid-State Disks (SSDs). However, spinning hard-disks can also be used with some performance penalties. ## File system For optimal performance and reliability, avoid using `NFS` or similar file systems for the Weaviate persistent volume ([`PERSISTENCE_DATA_PATH`](/deploy/configuration/env-vars/index.md)). Instead, use file systems like `Ext4` or `XFS` in combination with SAN storage (e.g. `EBS`) to ensure the best performance. ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Storage (docs/weaviate/concepts/storage.md) --- title: Storage sidebar_position: 18 description: "Persistent, fault-tolerant storage architecture for objects, vectors, and inverted index management, including HNSW snapshots and commit log compaction." image: og/docs/concepts.jpg # tags: ['architecture', 'storage'] --- Weaviate is a persistent and fault-tolerant database. This page gives you an overview of how objects and vectors are stored within Weaviate and how an inverted index is created at import time. The components mentioned on this page aid Weaviate in creating some of its unique features: * Each write operation is immediately persisted and also tolerant to application and system crashes. * On a vector search query, Weaviate returns the entire object (in other databases sometimes called a "document"), not just a reference, such as an ID. * When combining structured search with vector search, filters are applied prior to performing the vector search. This means that you will always receive the specified number of elements as opposed to post-filtering when the final result count is unpredictable. * Objects and their vectors can be updated or deleted at will, even while reading from the database. ## Logical Storage Units: Indexes, Shards, Stores Each class in Weaviate's user-defined schema leads to the creation of an index internally. An index is a wrapper type that is comprised of one or many shards. Shards within an index are self-contained storage units. Multiple shards can be used to distribute the load among multiple server nodes automatically. ### Components of a Shard Each shard houses three main components: * An object store, essentially a key-value store * An [inverted index](https://en.wikipedia.org/wiki/Inverted_index) * A vector index store (plugable, currently a [custom implementation of HNSW](/weaviate/config-refs/indexing/vector-index.mdx#hnsw-index)) #### Object and Inverted Index Store Since version `v1.5.0`, the object and inverted store are implemented using an [LSM-Tree approach](https://en.wikipedia.org/wiki/Log-structured_merge-tree). This means that data can be ingested at the speed of memory and after meeting a configured threshold, Weaviate will write the entire (sorted) memtable into a disk segment. When a read request comes in, Weaviate will first check the Memtable for the latest update for a specific object. If it is not present in the memtable, Weaviate will then check all previously written segments starting with the newest. To avoid checking segments which don't contain the desired objects, [Bloom filters](https://en.wikipedia.org/wiki/Bloom_filter) are used. Weaviate periodically merges smaller, older segments to make larger segments. Since the segments are already sorted, this is a relatively cheap operation. It happens constantly in the background. Fewer, larger segments make lookups more efficient. In the inverted index data is rarely replaced, but it is often appended. Merging means that, instead of checking all past segments and aggregating potential results, Weaviate can check a single segment (or a few large segments) and immediately find all the relevant object pointers. In addition, segments are used to remove earlier versions of an object that are out-dated because of a delete or a more recent update. Considerations Object storage and inverted index storage implement the LSM algorithm, they use segmentation. The vector index uses a different storage algorithm. The vector index does not use segmentation. Weaviate versions before `v1.5.0` use a B+Tree storage mechanism. The LSM method is faster, it works in constant time, and it improves write performance. To learn more about Weaviate's LSM store, see the LSM library documentation in the [Go package repository](https://pkg.go.dev/github.com/weaviate/weaviate/adapters/repos/db/lsmkv) #### HNSW Vector Index Storage Each shard contains a vector index that corresponds to the object and inverted index stores. The vector store and the other stores are independent. The vector store does not have to manage segmentation. By grouping a vector index with the object storage within a shard, Weaviate can make sure that each shard is a fully self-contained unit which can independently serve requests for the data it owns. By placing the vector index next to the object store (instead of within), Weaviate can avoid the downsides of a segmented vector index. Furthermore, its persistence and loading at startup are optimized through a combination of Write-Ahead-Logging and HNSW snapshots, detailed in the [Persistence and Crash Recovery](#persistence-and-crash-recovery) section. ### Shard Components Optimizations Weaviate's storage mechanisms use segmentation for structured/object data. Segments are cheap to merge and even unmerged segments can be navigated efficiently thanks to Bloom filters. In turn, ingestion speed is high and does not degrade over time. Weaviate keeps the vector index as large as possible within a shard. HNSW indexes cannot be merged efficiently. Querying a single large index is more efficient than sequentially querying many small indexes. To use multiple CPUs efficiently, create multiple shards for your collection. For the fastest imports, create multiple shards even on a single node. ### Lazy shard loading When Weaviate starts, it loads data from all of the shards in your deployment. This process can take a long time. Since every tenant is a shard, multi-tenant deployments with many tenants can have reduced availability after a restart. Lazy shard loading allows you to start working with your data sooner. After a restart, shards load in the background. If the shard you want to query is already loaded, you can get your results sooner. If the shard is not loaded yet, Weaviate prioritizes loading that shard and returns a response when it is ready. #### Dynamic lazy shard loading :::info Added in `v1.36.6` ::: Starting in v1.36.6, Weaviate automatically decides **per collection** whether to use lazy shard loading. Auto-detection only applies to **multi-tenant** collections and is based on two thresholds: - **Shard count threshold** ([`LAZY_LOAD_SHARD_COUNT_THRESHOLD`](/docs/deploy/configuration/env-vars/index.md#LAZY_LOAD_SHARD_COUNT_THRESHOLD)): Number of shards (tenants) in a collection. Default: `1000`. - **Shard size threshold** ([`LAZY_LOAD_SHARD_SIZE_THRESHOLD_GB`](/docs/deploy/configuration/env-vars/index.md#LAZY_LOAD_SHARD_SIZE_THRESHOLD_GB)): Total shard size for a collection. Default: `100` GB. If either threshold is exceeded, that collection's shards are lazy-loaded at startup. Otherwise, shards are loaded eagerly (synchronously) before Weaviate reports ready. Single-tenant collections are always eagerly loaded unless `LAZY_LOAD_SHARD_COUNT_THRESHOLD` is set to `0`, which forces lazy loading for all collections. This change improves reliability during rolling restarts and upgrades. Eager loading eliminates the increased query and ingestion latency that lazy loading can introduce for smaller deployments during rollouts. #### Vector cache prefill behavior The [`HNSW_STARTUP_WAIT_FOR_VECTOR_CACHE`](/deploy/configuration/env-vars#HNSW_STARTUP_WAIT_FOR_VECTOR_CACHE) environment variable controls whether vector cache prefill is synchronous (blocking) or asynchronous (background) at startup. Its default changed to `true` in v1.36.6. For collections where lazy shard loading is active, vector cache prefill is always **asynchronous**: the `HNSW_STARTUP_WAIT_FOR_VECTOR_CACHE` value is overridden to `false` regardless of the configured value. For eagerly-loaded collections, the configured value applies (default: `true`, meaning synchronous prefill). :::note Behavior change from v1.36.6 Prior to v1.36.6, lazy shard loading was enabled by default for all collections. From v1.36.6 onward, shards are **eagerly loaded by default** until a multi-tenant collection crosses the count or size threshold. This may increase startup time for smaller deployments but provides better reliability during rollouts. ::: ## Persistence and Crash Recovery ### Write-Ahead-Log Both the LSM stores used for object and inverted storage, as well as the HNSW vector index store make use of memory at some point of the ingestion journey. To prevent data loss on a crash, each operation is additionally written into a **[Write-Ahead-Log (WAL)](https://martinfowler.com/articles/patterns-of-distributed-systems/wal.html)** (also known as a *commit log*). WALs are append-only files that are very efficient to write to and that are rarely a bottleneck for ingestion. By the time Weaviate has responded with a successful status to your ingestion request, an LSM store WAL entry will have been created. If a WAL entry could not be created - for example because the disks are full - Weaviate will respond with an error to the insert or update request. The HNSW vector index keeps its own commit log, described [below](#hnsw-snapshots). It is written on the same request path, and the two differ in when they are synced to disk. The LSM stores will try to flush a segment on an orderly shutdown. Only if the operation is successful, will the WAL be marked as "complete". This means that if an unexpected crash happens and Weaviate encounters an "incomplete" WAL, it will recover from it. As part of the recovery process, Weaviate will flush a new segment based on the WAL and mark it as complete. As a result, future restarts will no longer have to recover from this WAL. For the HNSW vector index, the Write-Ahead-Log (WAL) is a critical component for disaster recovery and persisting the most recent changes. The cost in building up an HNSW index is in figuring out where to place a new object and how to link it with its neighbors. The WAL contains only the result of those calculations. The entire HNSW index state can be reconstructed by replaying these WAL entries. For very large indexes of tens or hundreds of millions of objects, this can be time-consuming. To avoid replaying the entire commit log on every restart, Weaviate writes **[HNSW snapshots](#hnsw-snapshots)**. ### HNSW snapshots import HnswSnapshots from '/_includes/feature-notes/hnsw-snapshots.mdx'; A snapshot represents a point-in-time state of the HNSW index. When Weaviate starts, it loads the most recent snapshot and replays only the commit log entries written after it. This significantly reduces startup time, because the number of entries that have to be replayed no longer grows with the age of the index. The commit log records every change to the index as it happens. Entries are written to the log as batches are processed, and a log file is synced to disk when it is rotated. Even with a fresh snapshot, Weaviate typically still has to load at least one subsequent commit log file. Starting in `v1.39`, snapshots are part of how the vector index is stored rather than an optional speedup. A background process called the commit log compactor owns the on-disk lifecycle of the index: it compacts newly flushed commit logs, merges them together, and writes a new snapshot when doing so is worthwhile. Snapshots and commit logs live in the same directory, and a snapshot replaces the commit logs it covers rather than duplicating them, so the commit logs left on disk hold only the delta since the last snapshot. This keeps the disk footprint proportional to the size of the index. Snapshots are also written as a stream. Weaviate still loads the snapshot it supersedes into memory, but the commit log delta and the new snapshot itself are streamed rather than also held there, as they were before `v1.39`. Upgrading to `v1.39` reduces the disk space the vector index uses, in some cases substantially. Earlier versions keep the full commit log alongside the snapshot, and a snapshot is a more compact representation of the same index than the commit logs it replaces, because compaction keeps only the final state of each vector's connections instead of every change made to them. A few caveats apply. The saving appears once the compactor has run its first cycles on each loaded shard rather than at the moment you upgrade, and inactive tenants do not shrink until they are next activated. Plan headroom for the peak rather than the steady state: while a snapshot is being written, the directory transiently holds the previous snapshot, the files being merged, and the new snapshot as it is assembled, so disk usage during snapshot creation is meaningfully above the size the index settles at. Weaviate protects this on-disk state in several ways. Snapshots and compacted commit logs are written to a temporary path and atomically renamed into place, so an interrupted write can never be mistaken for a complete file, and orphaned temporary files are cleaned up on the next startup. When a new snapshot is written, the snapshot it supersedes and the commit logs it covers are removed only after the new one is durably on disk. Commit logs are self-healing. If a crash leaves the last entry of a log incomplete, the file is truncated back to its last valid entry. The entries written before the tear are retained and the file becomes valid again for later compaction, so only the incomplete tail is lost. Snapshots are handled differently. A snapshot is stored in a checksummed block format and every block is verified when it is read, but unlike a commit log, a snapshot is not truncated or repaired. In the rare case that the current snapshot cannot be read, restore the affected data from a [backup](/deploy/configuration/backups.md), which includes the snapshot. Weaviate does not load a partial index, and because the commit logs the snapshot covers have already been removed, nothing remains on the node to replay in its place. That failure is scoped to the shard that owns the snapshot: the shard fails to load, and so does every other vector index on it. If that shard uses [dynamic lazy shard loading](#dynamic-lazy-shard-loading), the node stays up and requests to the shard return an error. If the shard is loaded eagerly, which is the default for single-tenant collections and for multi-tenant collections below the auto-detection thresholds, node startup fails instead. Weaviate creates and maintains snapshots automatically, so there is nothing to enable, disable, schedule, or tune. [`PERSISTENCE_HNSW_MAX_LOG_SIZE`](/deploy/configuration/env-vars/index.md#PERSISTENCE_HNSW_MAX_LOG_SIZE) still influences the size at which commit log files are rotated, and therefore how often there is new material to compact, but it does not configure snapshots. The environment variables that configured snapshots before `v1.39` are deprecated. That version and later still recognize `PERSISTENCE_HNSW_DISABLE_SNAPSHOTS` and the `PERSISTENCE_HNSW_SNAPSHOT_*` variables, so an existing deployment starts without a configuration error, but their values are ignored. For each of these variables that is set, Weaviate logs a warning at startup stating that the variable has no effect and will be removed in a future version. If these options are set through a configuration file rather than as environment variables, they are ignored in the same way, but no startup warning is logged. Remove the variables from your deployment configuration to clear the warnings. #### Snapshot configuration before `v1.39` {#pre-v1-39-configuration} In `v1.31` through `v1.38`, snapshots are an optional feature layered on top of the commit log rather than part of it, and the `PERSISTENCE_HNSW_SNAPSHOT_*` environment variables control when Weaviate creates them. Snapshots are enabled by default starting in `v1.36`, and disabled by default in `v1.31` through `v1.35`. Weaviate can create one at startup and periodically thereafter, once enough new commit log data has accumulated since the last snapshot. Only commit log files that have been rotated count toward that threshold, so changes still in the active file are not considered until the next rotation. If a snapshot cannot be read in these versions, it is discarded and Weaviate replays the full commit log instead. For the variables themselves, including their defaults and deprecation status, see [`PERSISTENCE_HNSW_DISABLE_SNAPSHOTS`](/deploy/configuration/env-vars/index.md#PERSISTENCE_HNSW_DISABLE_SNAPSHOTS) and the rows that follow it.

Periodic snapshot conditions and memory requirements

Periodic snapshot creation is governed by three variables, and **all** of the following conditions must be met before Weaviate creates a snapshot: - `PERSISTENCE_HNSW_SNAPSHOT_INTERVAL_SECONDS` — the minimum time since the previous snapshot has elapsed (default `21600` seconds, or six hours). - `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_NUMBER` — enough new commit log files have been created since the last snapshot (default `1`). - `PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_SIZE_PERCENTAGE` — the new commit logs are large enough, measured as a percentage of the previous snapshot's size (default `5`). This condition does not apply to the first snapshot, when there is no previous snapshot to measure against. Meeting these conditions makes a snapshot eligible rather than guaranteed. The background process that condenses and combines commit log files is also the one that writes the snapshot, so a snapshot can be created on a later pass than the one where the conditions are first met. In these versions, before creating a new snapshot, Weaviate loads the previous snapshot and the commit log difference into memory, so the node needs enough memory to accommodate both.

## Conclusions This page introduced you to the storage mechanisms of Weaviate. It outlined how all writes are persisted to a log before they are acknowledged and outlined the patterns used within Weaviate to make datasets scale well. For structured data, Weaviate makes use of segmentation to keep the write times constant. For the HNSW vector index, Weaviate avoids segmentation to keep query times efficient. ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Vector Quantization (docs/weaviate/concepts/vector-quantization.md) --- title: Compression (Vector Quantization) sidebar_position: 19 description: "Vector compression techniques reducing memory footprint and costs while improving search speed performance." image: og/docs/concepts.jpg # tags: ['vector compression', 'quantization'] --- import Rq8bit from '/_includes/feature-notes/rq-8bit.mdx'; import Rq1bit from '/_includes/feature-notes/rq-1bit.mdx'; **Vector quantization** reduces the memory footprint of the [vector index](./indexing/vector-index.md) by compressing the vector embeddings, and thus reduces deployment costs and improves the speed of the vector similarity search process. Weaviate currently offers four vector quantization techniques: - [Binary quantization (BQ)](#binary-quantization) - [Product quantization (PQ)](#product-quantization) - [Scalar quantization (SQ)](#scalar-quantization) - [Rotational quantization (RQ)](#rotational-quantization) import CompressionByDefault from '/\_includes/compression-by-default.mdx'; ## What is quantization? In general, quantization techniques reduce the memory footprint by representing numbers with lower precision numbers, like rounding a number to the nearest integer. In neural networks, quantization reduces the values of the weights or activations of the model stored as a 32-bit floating-point number (4 bytes) to a lower precision number, such as an 8-bit integer (1 byte). ### What is vector quantization? Vector quantization is a technique that reduces the memory footprint of vector embeddings. Vector embeddings have been typically represented as 32-bit floating-point numbers. Vector quantization techniques reduce the size of the vector embeddings by representing them as smaller numbers, such as 8-bit integers or binary numbers. Some quantization techniques also reduce the number of dimensions in the vector embeddings. ## Product quantization [Product quantization](https://ieeexplore.ieee.org/document/5432202) is a multi-step quantization technique that is available for use with `hnsw` indexes in Weaviate. PQ reduces the size of each vector embedding in two steps. First, it reduces the number of vector dimensions to a smaller number of "segments", and then each segment is quantized to a smaller number of bits from the original number of bits (typically a 32-bit float). import PQTradeoffs from '/\_includes/configuration/pq-compression/tradeoffs.mdx' ; In PQ, the original vector embedding is represented as a product of smaller vectors that are called 'segments' or 'subspaces.' Then, each segment is quantized independently to create a compressed vector embedding. After the segments are created, there is a training step to calculate `centroids` for each segment. By default, Weaviate clusters each segment into 256 centroids. The centroids make up a codebook that Weaviate uses in later steps to compress the vector embeddings. Once the codebook is ready, Weaviate uses the id of the closest centroid to compress each vector segment. The new vector embedding reduces memory consumption significantly. Imagine a collection where each vector embedding has 768 four byte elements. Before PQ compression, each vector embeddingrequires `768 x 4 = 3072` bytes of storage. After PQ compression, each vector requires `128 x 1 = 128` bytes of storage. The original representation is almost 24 times as large as the PQ compressed version. (It is not exactly 24x because there is a small amount of overhead for the codebook.) To enable PQ compression, see [Enable PQ compression](/weaviate/configuration/compression/pq-compression#enable-pq-compression) ### Segments The PQ `segments` controls the tradeoff between memory and recall. A larger `segments` parameter means higher memory usage and recall. An important thing to note is that the segments must divide evenly the original vector dimension. Below is a list segment values for common vectorizer modules: | Module | Model | Dimensions | Segments | | ----------- | --------------------------------------- | ---------- | ---------------------- | | openai | text-embedding-ada-002 | 1536 | 512, 384, 256, 192, 96 | | cohere | multilingual-22-12 | 768 | 384, 256, 192, 96 | | huggingface | sentence-transformers/all-MiniLM-L12-v2 | 384 | 192, 128, 96 | ### PQ compression process PQ has a training stage where it creates a codebook. We recommend using 10,000 to 100,000 records per shard to create the codebook. The training step can be triggered manually or automatically. See [Configuration: Product quantization](../configuration/compression/pq-compression.md) for more details. When the training step is triggered, a background job converts the index to the compressed index. While the conversion is running, the index is read-only. Shard status returns to `READY` when the conversion finishes. Weaviate uses a maximum of `trainingLimit` objects (per shard) for training, even if there are more objects available. After the PQ conversion completes, query and write to the index as normal. Distances may be slightly different due to the effects of quantization. :::info Which objects are used for training? - (`v1.27` and later) If the collection has more objects than the training limit, Weaviate randomly selects objects from the collection to train the codebook. - (`v1.26` and earlier) Weaviate uses the first `trainingLimit` objects in the collection to train the codebook. - If the collection has fewer objects than the training limit, Weaviate uses all objects in the collection to train the codebook. ::: ### Encoders In the configuration above you can see that you can set the `encoder` object to specify how the codebook centroids are generated. Weaviate's PQ supports using two different encoders. The default is `kmeans` which maps to the traditional approach used for creating centroid. Alternatively, there is also the `tile` encoder. This encoder is currently experimental but does have faster import times and better recall on datasets like SIFT and GIST. The `tile` encoder has an additional `distribution` parameter that controls what distribution to use when generating centroids. You can configure the encoder by setting `type` to `tile` or `kmeans` the encoder creates the codebook for product quantization. For configuration details, see [Configuration: Vector index](../config-refs/indexing/vector-index.mdx). ### Distance calculation With product quantization, distances are then calculated asymmetrically with a query vector with the goal being to keep all the original information in the query vector when calculating distances. :::tip Learn more about [how to configure product quantization in Weaviate](../configuration/compression/pq-compression.md).

You might be also interested in our blog post [How to Reduce Memory Requirements by up to 90%+ using Product Quantization](https://weaviate.io/blog/pq-rescoring). ::: ## Binary quantization **Binary quantization (BQ)** is a quantization technique that converts each vector embedding to a binary representation. The binary representation is much smaller than the original vector embedding. Usually each vector dimension requires 32 bits, but the binary representation only requires 1 bit, representing a 32x reduction in storage requirements. This works to speed up vector search by reducing the amount of data that needs to be read from disk, and simplifying the distance calculation. The tradeoff is that BQ is lossy. The binary representation by nature omits a significant amount of information, and as a result the distance calculation is not as accurate as the original vector embedding. Some vectorizers work better with BQ than others. Anecdotally, we have seen encouraging recall with Cohere's V3 models (e.g. `embed-multilingual-v3.0` or `embed-english-v3.0`), and OpenAI's `ada-002` model with BQ enabled. We advise you to test BQ with your own data and preferred vectorizer to determine if it is suitable for your use case. Note that when BQ is enabled, a vector cache can be used to improve query performance. The vector cache is used to speed up queries by reducing the number of disk reads for the quantized vector embeddings. Note that it must be balanced with memory usage considerations, with each vector taking up `n_dimensions` bits. ## Scalar quantization **Scalar quantization (SQ)** The dimensions in a vector embedding are usually represented as 32 bit floats. SQ transforms the float representation to an 8 bit integer. This is a 4x reduction in size. SQ compression, like BQ, is a lossy compression technique. However, SQ has a much greater range. The SQ algorithm analyzes your data and distributes the dimension values into 256 buckets (8 bits). SQ compressed vectors are more accurate than BQ compressed vectors. They are also significantly smaller than uncompressed vectors. The bucket boundaries are derived by determining the minimum and maximum values in a training set, and uniformly distributing the values between the minimum and maximum into 256 buckets. The 8 bit integer is then used to represent the bucket number. The size of the training set is configurable. The default is 100,000 objects per shard. When SQ is enabled, Weaviate boosts recall by over-fetching compressed results. After Weaviate retrieves the compressed results, it compares the original, uncompressed vectors that correspond to the compressed result against the query. The second search is very fast because it only searches a small number of vectors rather than the whole database. ## Rotational quantization **Rotational quantization (RQ)** provides significant compression while maintaining high recall. Unlike SQ, RQ requires no training phase and can be enabled immediately at index creation. RQ is available in: **8-bit** and **1-bit** variants. ### 8-bit RQ 8-bit RQ provides 4x compression while maintaining 98-99% recall in internal testing. The method works as follows: 1. **Fast pseudorandom rotation**: The input vector is transformed using a fast rotation based on the Walsh Hadamard Transform. This rotation takes approximately 7-10 microseconds for a 1536-dimensional vector. The output dimension is rounded up to the nearest multiple of 64. 2. **Scalar quantization**: Each entry of the rotated vector is quantized to an 8-bit integer. The minimum and maximum values of each individual rotated vector define the quantization interval. ### 1-bit RQ 1-bit RQ is an asymmetric quantization method that provides close to 32x compression as dimensionality increases. **1-bit RQ serves as a more robust and accurate alternative to BQ** with only a slight performance trade-off (approximately 10% decrease in throughput in internal testing compared to BQ). While more performant than PQ in terms of encoding time and distance calculations, 1-bit RQ typically offers slightly lower recall than well-tuned PQ. The method works as follows: 1. **Fast pseudorandom rotation**: The same rotation process as 8-bit RQ is applied to the input vector. For 1-bit RQ, the output dimension is always padded to at least 256 bits to improve performance on low-dimensional data. 2. **Asymmetric quantization**: - **Data vectors**: Quantized using 1 bit per dimension by storing only the sign of each entry - **Query vectors**: Scalar quantized using 5 bits per dimension during search This asymmetric approach improves recall compared to symmetric 1-bit schemes (such as BQ) by using more precision for query vectors during distance calculation. On datasets well-suited for BQ (like OpenAI embeddings), 1-bit RQ essentially matches BQ recall. It also works well on datasets where BQ performs poorly (such as [SIFT](https://arxiv.org/abs/2504.09081)). ### RQ characteristics The rotation step provides multiple benefits. It tends to reduce the quantization interval and decrease quantization error by distributing values more uniformly. It also distributes the distance information more evenly across all dimensions, providing a better starting point for distance estimation. Both RQ variants round up the number of dimensions to multiples of 64, which means that low-dimensional data (< 64 or 128 dimensions) might result in less than optimal compression. Additionally, several factors affect the actual compression rates: - **Auxiliary data storage**: 16 bytes for 8-bit RQ and 8 bytes for 1-bit RQ are stored with the compressed codes - **Dimension rounding**: Dimensionality is rounded up to the nearest multiple of 64 and 1-bit RQ is also padded to at least 256 bits Due to these factors, the 4x and 32x compression rates are only approached as dimensionality increases. These effects are more pronounced for low-dimensional vectors. While inspired by extended [RaBitQ](https://arxiv.org/abs/2405.12497), this implementation differs significantly for performance reasons. It uses fast pseudorandom rotations instead of truly random rotations. :::tip Learn more about how to [configure rotational quantization](../configuration/compression/rq-compression.md) in Weaviate or dive deer into the [implementation details and theoretical background](https://weaviate.io/blog/8-bit-rotational-quantization). ::: ## Over-fetching / re-scoring Weaviate over-fetches results and then re-scores them when you use SQ, RQ, or BQ. This is because the distance calculation on the compressed vectors is not as accurate as the same calculation on the original vector embedding. When you run a query, Weaviate compares the query limit against a configurable `rescoreLimit` parameter. The query retrieves compressed objects until the object count reaches whichever limit is greater. Then, Weaviate fetches the original, uncompressed vector embeddings that correspond to the compressed vectors. The uncompressed vectors are used to recalculate the query distance scores. For example, if a query is made with a limit of 10, and a rescore limit of 200, Weaviate fetches 200 objects. After rescoring, the query returns top 10 objects. This process offsets the loss in search quality (recall) that is caused by compression. :::note RQ optimization With RQ's high native recall of 98-99%, you can often disable rescoring (set `rescoreLimit` to 0) for maximum query performance with minimal impact on search quality. ::: ## Vector compression with vector indexing ### With an HNSW index An [HNSW index](./indexing/vector-index.md#hierarchical-navigable-small-world-hnsw-index) can be configured using [PQ](#product-quantization), [SQ](#scalar-quantization), [RQ](#rotational-quantization), or [BQ](#binary-quantization). Since HNSW is in memory, compression can reduce your memory footprint or allow you to store more data in the same amount of memory. If memory is your main constraint, you can also consider the disk-based [HFresh index](./indexing/vector-index.md#hfresh-index) as an index-level alternative to compression: it keeps only a compressed centroid index in memory and stores the rest on disk. :::tip You might be also interested in our blog post [HNSW+PQ - Exploring ANN algorithms Part 2.1](https://weaviate.io/blog/ann-algorithms-hnsw-pq). ::: ### With a flat index [RQ](#rotational-quantization) and [BQ](#binary-quantization) can be applied to a [flat index](./indexing/vector-index.md#flat-index). As a flat index search is a brute-force method, compression reduces the amount of data Weaviate has to read and increases speed. ## Rescoring Quantization inherently involves some loss information due to the reduction in information precision. To mitigate this, Weaviate uses a technique called rescoring, using the uncompressed vectors that are also stored alongside compressed vectors. Rescoring recalculates the distance between the original vectors of the returned candidates from the initial search. This ensures that the most accurate results are returned to the user. In some cases, rescoring also includes over-fetching, whereby additional candidates are fetched to ensure that the top candidates are not omitted in the initial search. ## Further resources :::info Related pages - [Concepts: Indexing](./indexing/index.md) - [Concepts: Vector Indexing](./indexing/vector-index.md) - [Configuration: Vector index](../config-refs/indexing/vector-index.mdx) - [Configuration: Schema (Configure semantic indexing)](../config-refs/indexing/vector-index.mdx#configure-semantic-indexing) - [How to configure: Binary quantization (compression)](../configuration/compression/bq-compression.md) - [How to configure: Product quantization (compression)](../configuration/compression/pq-compression.md) - [How to configure: Scalar quantization (compression)](../configuration/compression/sq-compression.md) - [How to configure: Rotational quantization (compression)](../configuration/compression/rq-compression.md) ::: ## Questions and feedback import DocsFeedback from '/\_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Indexing/Index (docs/weaviate/concepts/indexing/index.md) --- title: Indexing sidebar_position: 0 description: "Overview of Weaviate's indexing systems for optimized search performance and data retrieval efficiency." image: og/docs/concepts.jpg # tags: ['basics'] --- Weaviate supports several types of indexes. 1. **[Vector indexes](./vector-index.md)** - a vector index (e.g. HNSW or flat) is used to serve all vector-search queries. - **HNSW** - an approximate nearest neighbor (ANN) search-based vector index. HNSW indexes scale well with large datasets. - **Flat** - a vector index that is used for brute-force searches. This is useful for small datasets. - **Dynamic** - a vector index that is flat when the dataset is small and switches to HNSW when the dataset is large. 1. **[Inverted indexes](./inverted-index.md)** - inverted indexes enable BM25 queries or speed up filtering. You can configure indexes in Weaviate per collection. :::tip Tips for indexing Especially for large datasets, configuring the indexes is important because the more you index, the more storage is needed. A rule of thumb - if you don't query over a specific field or vector space, don't index it. ::: --- ### Weaviate/Concepts/Indexing/Inverted Index (docs/weaviate/concepts/indexing/inverted-index.md) --- title: Inverted indexes sidebar_position: 2 description: "Inverted index architecture for efficient keyword search and filtering with performance improvements." image: og/docs/concepts.jpg # tags: ['basics'] --- Inverted indexes in Weaviate map values (like words or numbers) to the objects that contain them, enabling fast keyword search and filtering operations. ## How Weaviate creates inverted indexes Understanding Weaviate's indexing architecture is crucial for optimizing performance and resource usage. Weaviate creates **individual inverted indexes for each property and each index type**. This means: - Each property in your collection gets its own dedicated inverted index(es) - Meta properties (like creation timestamps) also get their own separate inverted indexes - A single property can have multiple inverted indexes if it supports multiple index types - All aggregations and combinations across properties happen at query time, not at index time **Example**: A `title` property with both `indexFilterable: true` and `indexSearchable: true` will result in two separate inverted indexes - one optimized for search operations and another for filtering operations. This architecture provides flexibility and performance optimization but also means that enabling multiple index types increases storage requirements and indexing overhead. For `text` properties specifically, the indexing process follows these steps: 1. **Tokenization**: The text is first tokenized according to the [tokenization method](#tokenization) configured for that property. 3. **Index entry creation**: Each processed token gets an entry in the inverted index, pointing to the object containing it. This process ensures that your text searches and filters can quickly locate relevant objects based on the tokens they contain.

Performance improvements added in Oct 2024

In Weaviate versions `v1.24.26`, `v1.25.20`, `v1.26.6` and `v1.27.0`, we introduced performance improvements and bugfixes for the BM25F scoring algorithm: - The BM25 segment merging algorithm was made faster - Improved WAND algorithm to remove exhausted terms from score computation and only do a full sort when necessary - Solved a bug in BM25F multi-prop search that could lead to not summing all the query term score for all segments - The BM25 scores are now calculated concurrently for multiple segments As always, we recommend upgrading to the latest version of Weaviate to benefit from improvements such as these.

## BlockMax WAND algorithm import BlockmaxWand from '/_includes/feature-notes/blockmax-wand.mdx'; The BlockMax WAND algorithm is a variant of the WAND algorithm that is used to speed up BM25 and hybrid searches. It organizes the inverted index in blocks to enable skipping over blocks that are not relevant to the query. This can significantly reduce the number of documents that need to be scored, improving search performance. If you are experiencing slow BM25 (or hybrid) searches and use a Weaviate version prior to `v1.30`, try migrating to a newer version that uses the BlockMax WAND algorithm to see if it improves performance. If you need to migrate existing data from a previous version of Weaviate, follow the [v1.30 migration guide](/deploy/migration/weaviate-1-30.md). :::note Scoring changes with BlockMax WAND Due to the nature of the BlockMax WAND algorithm, the scoring of BM25 and hybrid searches may differ slightly from the default WAND algorithm. Additionally BlockMax WAND scores on single and multiple property search may be different due to different IDF and property length normalization calculations. This is expected behavior and is not a bug. ::: ## Configure inverted indexes There are three inverted index types in Weaviate: - `indexSearchable` - a searchable index for BM25 or hybrid search - `indexFilterable` - a match-based index for fast [filtering](../filtering.md) by matching criteria - `indexRangeFilters` - a range-based index for [filtering](../filtering.md) by numerical ranges Each inverted index can be set to `true` (on) or `false` (off) on a property level. The `indexSearchable` and `indexFilterable` indexes are on by default, while the `indexRangeFilters` index is off by default. The filterable indexes are only capable of [filtering](../filtering.md), while the searchable index can be used for both searching and filtering (though not as fast as the filterable index). So, setting `"indexFilterable": false` and `"indexSearchable": true` (or not setting it at all) will have the trade-off of worse filtering performance but faster imports (due to only needing to update one index) and lower disk usage. See the [related how-to section](../../manage-collections/vector-config.mdx#property-level-settings) to learn how to enable or disable inverted indexes on a property level. A rule of thumb to follow when determining whether to switch off indexing is: _if you will never perform queries based on this property, you can turn it off._ #### Inverted index types summary import InvertedIndexTypesSummary from '/_includes/inverted-index-types-summary.mdx'; - Enable one or both of `indexFilterable` and `indexRangeFilters` to index a property for faster filtering. - If only one is enabled, the respective index is used for filtering. - If both are enabled, `indexRangeFilters` is used for operations involving comparison operators, and `indexFilterable` is used for equality and inequality operations. This chart shows which filter makes the comparison when one or both index type is `true` for an applicable property. | Operator | `indexRangeFilters` only | `indexFilterable` only | Both enabled | | :- | :- | :- | :- | | Equal | `indexRangeFilters` | `indexFilterable` | `indexFilterable` | | Not equal | `indexRangeFilters` | `indexFilterable` | `indexFilterable` | | Greater than | `indexRangeFilters` | `indexFilterable` | `indexRangeFilters` | | Greater than equal | `indexRangeFilters` | `indexFilterable` | `indexRangeFilters` | | Less than | `indexRangeFilters` | `indexFilterable` | `indexRangeFilters` | | Less than equal | `indexRangeFilters` | `indexFilterable` | `indexRangeFilters` | #### Inverted index for timestamps You can also enable an inverted index to search [based on timestamps](/weaviate/config-refs/indexing/inverted-index.mdx#indextimestamps). Timestamps are currently indexed using the `indexFilterable` index. ## Collections without indexes If you don't want to set an index at all, this is possible too. To create a collection without any indexes, skip indexing on the collection and on the properties.

Example collection configuration without inverted indexes - JSON object

An example of a complete collection object without inverted indexes: ``` /* Detailed source-code truncated for AI context efficiency. */ ```

## Tokenization Tokenization is the process of breaking text into smaller units called tokens. This process is fundamental to how inverted indexes work - the tokens produced determine what can be searched and how matching occurs. ### How tokenization works When you add an object to Weaviate, text in each property is tokenized according to that property's configured tokenization method. For example, the text: '"Ankh-Morpork's police captain"' could be tokenized using different tokenization methods: 1. `'word'`: `["ankh", "morpork", "s", "police", "captain"]` - splits on non-alphanumeric characters, lowercased 2. `'lowercase'`: `["ankh-morpork's", "police", "captain"]` - splits on whitespace only, lowercased 3. `'whitespace'`: `["Ankh-Morpork's", "police", "captain"]` - splits on whitespace, preserves case 4. `'field'`: `["Ankh-Morpork's police captain"]` - treats entire text as single token Each tokenization method serves different use cases and directly impacts search and filter behavior. ### Tokenization and the inverted index The inverted index maps each token to the objects containing it. When you perform a keyword search or filter: 1. Your query/filter text is tokenized using the **same method** as the indexed property 2. The inverted index looks up which objects contain those tokens 3. For searches, BM25f ranks results based on token matches 4. For filters, exact token matches determine inclusion This means the tokenization method controls the "granularity" of matching. For example, with `word` tokenization, searching for `"clark"` will match an object containing `"Clark:"` because both tokenize to `["clark"]`. With `field` tokenization, only exact matches succeed. ### Available tokenization methods Weaviate provides several tokenization methods optimized for different data types: **Standard methods:** - **`word`** (default): Splits on non-alphanumeric characters, lowercases. Best for typical text. - **`lowercase`**: Splits on whitespace, lowercases. Preserves symbols like `@`, `_`, `-`. - **`whitespace`**: Splits on whitespace, preserves case and symbols. For case-sensitive data. - **`field`**: No splitting - entire value is one token. For exact matching. **Language-specific methods** (for languages without word boundaries): - **`gse`**: Japanese text segmentation using the [`gse`](https://pkg.go.dev/github.com/go-ego/gse) tokenizer (Japanese dictionary) - **`gse_ch`**: Chinese text segmentation using the same `gse` tokenizer with a Chinese dictionary - **`trigram`**: Splits into character trigrams for CJK languages - **`kagome_ja`**: Japanese morphological analysis - **`kagome_kr`**: Korean morphological analysis These language-specific tokenizers are not loaded by default. Enable them with the corresponding environment variables (`ENABLE_TOKENIZER_GSE`, `ENABLE_TOKENIZER_GSE_CH`, `ENABLE_TOKENIZER_KAGOME_JA`, `ENABLE_TOKENIZER_KAGOME_KR`). See the [tokenization configuration reference](../../config-refs/collections.mdx#tokenization) for detailed specifications and behavior examples. ### Accent folding import TokenizerPreview from '/_includes/feature-notes/tokenizer.mdx'; Text properties can opt in to **accent folding** via the `textAnalyzer` block. When `asciiFold` is set to `true`, the analyzer normalizes accented Latin characters and any other character carrying combining marks or diacritics to their ASCII equivalents during both indexing and querying. A document containing "Café Crème" becomes searchable as "cafe creme", and vice versa. The same normalization is applied to filters (`Equal`, `Like`), so what you can search for is exactly what you can filter on. ```json { "name": "description", "dataType": ["text"], "tokenization": "word", "textAnalyzer": { "asciiFold": true } } ``` The implementation uses [Unicode NFD decomposition](https://unicode.org/reports/tr15/) (covering acute, grave, circumflex, tilde, dieresis, caron, cedilla, ogonek, macron, breve, ring, and more) plus an explicit replacement table for single-codepoint letters that do not decompose, such as `ł`, `æ`, `ø`, `ð`, `þ`, `đ`, and `ß`. Together this covers 20+ Latin-script languages, including French, Portuguese, Spanish, German, Polish, Czech, Croatian, and Icelandic. Accent folding composes with every tokenization method: `word`, `lowercase`, `whitespace`, `field`, and `trigram`. #### Per-character exceptions If you want most accents folded but need to preserve specific characters (for example, an `é` that distinguishes two product names) use `asciiFoldIgnore`: ```json { "name": "name", "dataType": ["text"], "tokenization": "word", "textAnalyzer": { "asciiFold": true, "asciiFoldIgnore": ["é", "Ł"] } } ``` Because `asciiFoldIgnore` changes which tokens are written to disk, it is **immutable** after the property is created. Schema updates that change the ignore list are rejected. To change it, create a new property and reindex. See the [accent folding tutorial](../../tutorials/tokenization.md#example-4-accent-folding) for a worked example and the [textAnalyzer configuration reference](../../config-refs/indexing/inverted-index.mdx#textanalyzer) for all options. ### Impact on search and filtering #### Filters Filters perform binary matching - an object either matches or doesn't. Tokenization determines what counts as a match: | Query | Indexed text | `word` | `lowercase` | `whitespace` | `field` | |-------|--------------|--------|-------------|--------------|---------| | `"clark"` | `"Clark:"` | ✅ | ❌ | ❌ | ❌ | | `"variable_name"` | `"variable_name"` | ✅ | ✅ | ✅ | ✅ | | `"variable_name"` | `"variable_new_name"` | ✅ | ❌ | ❌ | ❌ | With `word` tokenization, `"variable_name"` matches `"variable_new_name"` because both contain the tokens `["variable", "name"]`. #### Keyword searches Keyword searches use BM25f to rank results. Tokenization affects: 1. **Result inclusion**: Only objects with matching tokens appear 2. **Ranking scores**: More matching tokens = higher scores For example, searching for `"lois clark"` with `word` tokenization will rank objects containing both words higher than those with just one. ### Stop words Stop words are common words (like "a", "the", "is") that are typically ignored during search. By default, Weaviate uses a standard English stop words list. After tokenization, stop words in queries behave as if they're not present for matching purposes: - Filter for `"a computer mouse"` behaves like `"computer mouse"` - Stop words still affect BM25f ranking scores You can [configure custom stop words](../../config-refs/indexing/inverted-index.mdx#stopwords) in your collection definition. **Note**: With `field` tokenization, stop words don't apply since the entire field is one token. #### Custom stopword presets Beyond the built-in `en` and `none` presets, you can declare custom stopword presets on the collection's `invertedIndexConfig.stopwordPresets`. Each preset has a name and a flat word list. A preset name that matches a built-in (`en`, `none`) replaces the built-in for this collection. ```json { "invertedIndexConfig": { "stopwordPresets": { "fr": ["le", "la", "les", "un", "une", "des", "du", "de", "et"], "de": ["der", "die", "das", "und", "oder", "aber"] } } } ``` #### Per-property stopword overrides Each text property can override the collection-level stopword behavior via `textAnalyzer.stopwordPreset`. This is useful for multilingual collections where different properties contain text in different languages. The override is only supported on properties with `tokenization: "word"`. Schema validation rejects it on other tokenizers. ```json "properties": [ { "name": "name_en", "dataType": ["text"], "tokenization": "word", "textAnalyzer": { "stopwordPreset": "en" } }, { "name": "name_fr", "dataType": ["text"], "tokenization": "word", "textAnalyzer": { "stopwordPreset": "fr" } } ] ``` Stopwords are still **indexed**: they are only filtered at query time. Changing the stopword configuration does **not** require reindexing your data. See the [custom stopwords tutorial](../../tutorials/tokenization.md#example-5-custom-and-per-property-stopword-presets) for a worked example and the [stopwordPresets configuration reference](../../config-refs/indexing/inverted-index.mdx#stopwordpresets) for all options. ### Choosing a tokenization method The choice of tokenization method should match your data characteristics and search requirements. Here are some general guidelines: - **General text** (articles, descriptions): Use `word` (default) - **Technical data with symbols** (code, emails): Use `lowercase` - **Case-sensitive data** (names, acronyms): Use `whitespace` - **Unique identifiers** (URLs, IDs): Use `field` - **CJK languages**: Use language-specific methods For detailed guidance and practical examples, see the [tokenization tutorial](../../tutorials/tokenization.md). ## Further resources :::info Related pages - [Configuration: Inverted index](../../config-refs/indexing/inverted-index.mdx) - [How-to: Configure collections](../../manage-collections/vector-config.mdx#property-level-settings) - [Configuration: Tokenization](../../config-refs/collections.mdx#tokenization) - [Tutorial: Configure tokenization](../../tutorials/tokenization.md) ::: ## Questions and feedback import DocsFeedback from '/\_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Indexing/Vector Index (docs/weaviate/concepts/indexing/vector-index.md) --- title: Vector Indexing sidebar_position: 1 description: "Dynamic vector indexing with HNSW, flat, HFresh indexes for speed-optimized similarity search operations." image: og/docs/concepts.jpg slug: /weaviate/concepts/vector-index # tags: ['vector index plugins'] --- What is vector indexing? It's a key component of vector databases that helps to [significantly **increase the speed** of the search process of similarity search](https://weaviate.io/blog/vector-search-explained) with only a minimal tradeoff in search accuracy ([HNSW index](#hierarchical-navigable-small-world-hnsw-index)), or efficiently store many subsets of data in a small memory footprint ([flat index](#flat-index)). The [dynamic index](#dynamic-index) can even start off as a flat index and then dynamically switch to the HNSW index as it scales past a threshold. Weaviate's vector-first storage system takes care of all storage operations with a vector index. Storing data in a vector-first manner not only allows for semantic or context-based search, but also makes it possible to store _very_ large amounts of data without decreasing performance (assuming scaled well horizontally or having sufficient shards for the indexes). Weaviate supports these vector index types: - [HNSW index](#hierarchical-navigable-small-world-hnsw-index): a more complex index that is slower to build, but it scales well to large datasets as queries have a logarithmic time complexity. - [Flat index](#flat-index): a simple, lightweight index that is designed for small datasets. - [Dynamic index](#dynamic-index): allows you to automatically switch from a flat index to an HNSW index as object count scales - [HFresh index](#hfresh-index): a cluster-based index that uses HNSW for the centroid index, providing strong memory efficiency by keeping most of the data on disk This page explains what vector indexes are, and what purpose they serve in the Weaviate vector database. :::info What is a vector index? In vector databases, a vector index is a data structure that organizes vector embeddings to enable efficient similarity search. Indexing vector databases properly is crucial for performance, and different index types serve different purposes - from the simple flat index to more sophisticated approaches like HNSW and HFresh. ::: ## Why do you need vector indexing? [Vector embeddings](https://weaviate.io/blog/vector-embeddings-explained) are a great way to represent meaning. Understanding how to index a vector is crucial for working with vector databases effectively. Vectors embeddings are arrays of elements that can capture meaning from different data types, such as texts, images, videos, and other content. The number of elements are called dimensions. High dimension vectors capture more information, but they are harder to work with. Vector databases make it easier to work with high dimensional vectors. Consider search; Vector databases efficiently measure semantic similarity between data objects. When you run a [similarity search](../../search/similarity.md), a vector database like Weaviate uses a vectorized version of the query to find objects in the database that have vectors similar to the query vector. Vectors are like coordinates in a multi-dimensional space. A very simple vector might represent objects, _words_ in this case, in a 2-dimensional space. In the graph below, the words `Apple` and `Banana` are shown close to each other. `Newspaper` and `Magazine` are also close to each other, but they are far away from `Apple` and `Banana` in the same vector space. Within each pair, the distance between words is small because the objects have similar vector embeddings. The distance between the pairs is larger because the difference between the vectors is larger. Intuitively, fruits are similar to each other, but fruits are not similar to reading material. For more details of this representation, see: ([GloVe](https://github.com/stanfordnlp/GloVe)) and [vector embeddings](https://weaviate.io/blog/vector-embeddings-explained#what-exactly-are-vector-embeddings). Another way to think of this is how products are placed in a supermarket. You'd expect to find `Apples` close to `Bananas`, because they are both fruit. But when you are searching for a `Magazine`, you would move away from the `Apples` and `Bananas`, more towards the aisle with, for example, `Newspapers`. This is how the semantics of concepts can be stored in Weaviate as well, depending on the module you're using to calculate the numbers in the vectors. Not only words or text can be indexed as vectors, but also images, video, DNA sequences, etc. Read more about which model to use [here](/weaviate/modules/index.md). :::tip You might be also interested in our blog post [Vector search explained](https://weaviate.io/blog/vector-search-explained). ::: Let's explore how to index a vector using different approaches supported by Weaviate. ## Vector index types Many different types of vector indexes exist. A majority of them are designed to speed up searches by reducing the number of vectors that need to be compared. However, they do this in different ways, and each has its own strengths and weaknesses. ### Graph indexes Graph indexes form a network of vectors, such that similar vectors are connected to each other. This allows for fast "traversal" of the graph to find similar vectors to a query vector. HNSW, or "Hierarchical Navigable Small World", is the most common graph index type. It creates a set of "layers" of vectors, to enable fast traversal of the graph. They are very scalable, allow incremental updates, and efficient for high-dimensional vectors. This is the default index type in Weaviate. ### Tree-based indexes Tree-based indexes divide the vectors into a tree structure. ANNOY, or "Approximate Nearest Neighbors Oh Yeah", is a well-known tree-based index. It divides the vectors into a binary tree structure. They can be memory-efficient, and are good for low-dimensional vectors. However, it may be costly to update the index over time, as the tree may need to be rebuilt. ### Cluster-based indexes Cluster-based indexes group vectors based on their similarity. As a result, the search space is reduced to only the cluster(s) that is most likely to contain the nearest neighbors. Their search accuracy (recall and precision) may generally be lower than graph-based indexes, but they can be more memory-efficient. ### Flat index A flat index is the simplest type of index. It stores all vectors in a single list, and searches through all of them to find the nearest neighbors. This is extremely memory-efficient, but does not scale well, as the search time grows linearly with the number of vectors. The first method is the HNSW index. ## Hierarchical Navigable Small World (HNSW) index **Hierarchical Navigable Small World (HNSW)** is an algorithm that works on multi-layered graphs. It is also an index type, and refers to vector indexes that are created using the HNSW algorithm. HNSW indexes enable very fast queries, but rebuilding the index when you add new vectors can be resource intensive. Weaviate's `hnsw` index is a [custom implementation](../../more-resources/faq.md#q-does-weaviate-use-hnswlib) of the Hierarchical Navigable Small World ([HNSW](https://arxiv.org/abs/1603.09320)) algorithm that offers full [CRUD-support](https://db-engines.com/en/blog_post/87). At build time, the HNSW algorithm creates a series of layers. At query time, the HNSW algorithm uses the layers to build a list of approximate nearest neighbors (ANN) quickly and efficiently. Consider this diagram of a vector index using HNSW. An individual object can exist in more than one layer, but every object in the database is represented in the lowest layer (layer zero in the picture). The layer zero data objects are very well connected to each other. Each layer above the lowest layer has fewer data object, and fewer connections. The data objects in the higher layers correspond to the objects in the lower layers, but each higher layer has exponentially fewer objects than the layer below. The HNSW algorithm takes advantage of the layers to efficiently process large amounts of data. When a search query comes in, the HNSW algorithm finds the closest matching data points in the highest layer. Then, HNSW goes one layer deeper, and finds the closest data points in that layer to the ones in the higher layer. These are the nearest neighbors. The algorithm searches the lower layer to create a new list of nearest neighbors. Then, HNSW uses the new list and repeats the process on the next layer down. When it gets to the deepest layer, the HNSW algorithm returns the data objects closest to the search query. Since there are relatively few data objects on the higher layers, HNSW has to search fewer objects. This means HNSW 'jumps' over large amounts of data that it doesn't need to search. When a data store has only one layer, the search algorithm can't skip unrelated objects. It has to search significantly more data objects even though they are unlikely to match. ### Resource requirements HNSW is an in-memory index, where each node in the graph as well as each edge between nodes are stored in memory. This means that the size of the index in memory is directly proportional to the number of vectors in the index, as well as the number of connections between vectors. The size of an HNSW index is dominated by the number of vectors; take a look at the table below for an example: | Component | Size derivation | Typical size | Size @1M vectors | Size @100M vectors | | --- | --- | --- | --- | --- | | Node | 4B (float) x N dimensions | 2-12kB | 2-12GB | 200-1200GB | | Edge | 10B x 20 connections | 200B | 200MB | 20GB | As you can see, the memory requirements of an HNSW index can quickly become a bottleneck. This is where [quantization](../vector-quantization.md) can be used to reduce the size of the index in memory. Alternatively, the disk-based [HFresh index](#hfresh-index) keeps only a compressed centroid index in memory, which can dramatically reduce the memory footprint. HNSW is very fast, memory efficient, approach to similarity search. The memory cache only stores the highest layer instead of storing all of the data objects in the lowest layer. When the search moves from a higher layer to a lower one, HNSW only adds the data objects that are closest to the search query. This means HNSW uses a relatively small amount of memory compared to other search algorithms. Have another look at the diagram; it demonstrates how the HNSW algorithm searches. The search vector in the top layer connects to a partial result in layer one. The objects in layer one lead HNSW to the result set in layer zero. This allows HNSW to skip objects that are unrelated to the search query. Inserting a vector into an HNSW index works in a similar way. The HNSW algorithm finds the closest data objects in the highest layer, and then moves down to the next layer. It continues until it finds the best place to insert the new vector. The HNSW algorithm then connects the new vector to the existing vectors in that layer. ### Managing search quality vs speed tradeoffs HNSW parameters can be adjusted to adjust search quality against speed. The `ef` parameter is a critical setting for balancing the trade-off between search speed and quality. The `ef` parameter dictates the size of the dynamic list used by the HNSW algorithm during the search process. A higher `ef` value results in a more extensive search, enhancing accuracy but potentially slowing down the query. In contrast, a lower `ef` makes the search faster but might compromise on accuracy. This balance is crucial in scenarios where either speed or accuracy is a priority. For instance, in applications where rapid responses are critical, a lower `ef` might be preferable, even at the expense of some accuracy. Conversely, in analytical or research contexts where precision is paramount, a higher `ef` would be more suitable, despite the increased query time. `ef` can be configured explicitly or dynamically. This feature is particularly beneficial in environments with varying query patterns. When `ef` is configured dynamically, Weaviate optimizes the balance between speed and recall based on real-time query requirements. To enable dynamic `ef`, set `ef`: -1. Weaviate adjusts the size of the ANN list based on the query response limit. The calculation also takes into account the values of `dynamicEfMin`, `dynamicEfMax`, and `dynamicEfFactor`. ### Dynamic ef The `ef` parameter controls the size of the ANN list at query time. You can configure a specific list size or else let Weaviate configure the list dynamically. If you choose dynamic `ef`, Weaviate provides several options to control the size of the ANN list. The length of the list is determined by the query response limit that you set in your query. Weaviate uses the query limit as an anchor and modifies the size of ANN list according to the values you set for the `dynamicEf` parameters. - `dynamicEfMin` sets a lower bound on the list length. - `dynamicEfMax` sets an upper bound on the list length. - `dynamicEfFactor` sets a range for the list. The dynamic list size will be set as the query limit multiplied by `dynamicEfFactor`, modified by a minimum of `dynamicEfMin` and a maximum of `dynamicEfMax`. In code, this can be expressed as: ```python ef = min(max(dynamicEfMin, queryLimit * dynamicEfFactor), dynamicEfMax) ``` To keep search recall high, the actual dynamic `ef` value stays above `dynamicEfMin` even if the query limit is small enough to suggest a lower value. To keep search speed reasonable even when retrieving large result sets, the dynamic `ef` value is limited to `dynamicEfMax`. Weaviate doesn't exceed `dynamicEfMax` even if the query limit is large enough to suggest a higher value. If the query limit is higher than `dynamicEfMax`, `dynamicEfMax` does not have any effect. In this case, dynamic `ef` value is equal to the query limit. To determine the length of the ANN list, Weaviate multiples the query limit by `dynamicEfFactor`. The list range is modified by `dynamicEfMin` and `dynamicEfMax`. Consider this GraphQL query that sets a limit of 4. ```graphql { Get { JeopardyQuestion(limit: 4) { answer question } } } ``` Imagine the collection has dynamic `ef` configured. ```json "vectorIndexConfig": { "ef": -1, "dynamicEfMin": 5 "dynamicEfMax": 25 "dynamicEfFactor": 10 } ``` The resulting search list has these characteristics. - A potential length of 40 objects ( ("dynamicEfFactor": 10) \* (limit: 4) ). - A minimum length of 5 objects ("dynamicEfMin": 5). - A maximum length of 25 objects ("dynamicEfMax": 25). - An actual size of 5 to 25 objects. If you use the [`docker-compose.yml` file from Weaviate](/deploy/installation-guides/docker-installation.md) to run your local instance, the `QUERY_DEFAULTS_LIMIT` environment variable sets a reasonable default query limit. To prevent out of memory errors,`QUERY_DEFAULTS_LIMIT` is significantly lower than `QUERY_MAXIMUM_RESULTS`. To change the default limit, edit the value for `QUERY_DEFAULTS_LIMIT` when you configure your Weaviate instance. ### Deletions Cleanup is an async process runs that rebuilds the HNSW graph after deletes and updates. Prior to cleanup, objects are marked as deleted, but they are still connected to the HNSW graph. During cleanup, the edges are reassigned and the objects are deleted for good. ## Flat index The **flat index** is one of the fundamental ways to implement vector indexing in databases. As the name suggests, it's a simple, lightweight index that is fast to build and has a very small memory footprint. This index type is a good choice for use cases where each end user (i.e. tenant) has their own, isolated, dataset, such as in a SaaS product for example, or a database of isolated record sets. As the name suggests, the flat index is a single layer of disk-backed data objects and thus a very small memory footprint. The flat index is a good choice for small collections, such as for multi-tenancy use cases. A drawback of the flat index is that it does not scale well to large collections as it has a linear time complexity as a function of the number of data objects, unlike the `hnsw` index which has a logarithmic time complexity. ## Dynamic index :::caution Experimental feature Available starting in `v1.25`. This is an experimental feature. Use with caution. ::: import DynamicAsyncRequirements from '/\_includes/dynamic-index-async-req.mdx'; The flat index is ideal for use cases with a small object count and provides lower memory overhead and good latency. As the object count increases the HNSW index provides a more viable solution as HNSW speeds up search. The goal of the dynamic index is to shorten latencies during querying time at the cost of a larger memory footprint as you scale. When memory is your main constraint, the [HFresh index](#hfresh-index) is another option: it avoids keeping the full index in memory, trading some query speed for much lower memory use. By configuring a dynamic index, you can automatically switch from flat to HNSW indexes. This switch occurs when the object count exceeds a specified threshold (by default 10,000). This functionality only works with async indexing enabled. When the threshold is hit while importing, all the data piles up in the async queue, the HNSW index is constructed in the background and when ready the swap from flat to HNSW is completed. Currently, this is only a one-way upgrade from a flat to an HNSW index, it does not support changing back to a flat index even if the object count goes below the threshold due to deletion. This is particularly useful in a multi-tenant setup where building an HNSW index per tenant would introduce extra overhead. With a dynamic index, as individual tenants grow their index will switch from flat to HNSW, while smaller tenants' indexes remain flat. ## HFresh index import HFreshStatus from '/_includes/feature-notes/hfresh_status.mdx'; The **HFresh index** is a cluster-based vector index that uses HNSW for the centroid index. It is based on the SPFresh algorithm, which organizes vectors into posting lists (clusters) for efficient retrieval. Read more about SPFresh in [SPFresh: Incremental In-Place Update for Billion-Scale Vector Search](https://arxiv.org/pdf/2410.14452). HFresh works by: 1. **Partitioning vectors into clusters**, each with a centroid vector. 2. **Using an HNSW index** to efficiently search the centroids. 3. **Searching only the most relevant posting lists** (clusters) for a query. This approach can provide memory efficiency benefits for large datasets while maintaining good search performance. The key trade-off is between memory usage and search recall, controlled by parameters like `searchProbe` (number of posting lists to search) and `replicas` (number of posting lists each vector is added to). Only the centroid index stays in memory, compressed with 8-bit [rotational quantization (RQ)](../vector-quantization.md#rotational-quantization). The posting lists are stored on disk, compressed with 1-bit RQ. A query searches the in-memory centroid index to pick a few relevant postings, reads only those postings from disk, and rescores the top candidates against the uncompressed vectors. Because only a small subset of the data is read per query, disk I/O stays bounded and latency stays predictable as the dataset grows. HFresh also rebalances its posting lists in the background as data changes, so it does not require periodic full index rebuilds. HFresh is particularly well-suited for: - **Memory-constrained deployments**: Reduces memory usage while maintaining good search performance, from small collections up to very large ones - **High-dimensional vectors**: Particularly effective with high-dimensional embedding models - **Cost-sensitive deployments**: Lower memory requirements can reduce infrastructure costs HFresh is not designed to beat HNSW on raw query throughput. It targets deployments where memory efficiency matters more than peak QPS, and where the application can tolerate slightly higher query latencies. :::note Supported distance metrics HFresh only supports `cosine` and `l2-squared` distance metrics. Dot product is not supported. ::: For configuration details, see the [HFresh index parameters](../../config-refs/indexing/vector-index.mdx#hfresh-index-parameters). ## Vector cache considerations For optimal search and import performance, previously imported vectors need to be in memory. A disk lookup for a vector is orders of magnitudes slower than memory lookup, so the disk cache should be used sparingly. However, Weaviate can limit the number of vectors in memory. By default, this limit is set to one trillion (`1e12`) objects when a new collection is created. During import set `vectorCacheMaxObjects` high enough that all vectors can be held in memory. Each import requires multiple searches. Import performance drops drastically when there isn't enough memory to hold all of the vectors in the cache. After import, when your workload is mostly querying, experiment with vector cache limits that are less than your total dataset size. Vectors that aren't currently in cache are added to the cache if there is still room. If the cache fills, Weaviate drops the whole cache. All future vectors have to be read from disk for the first time. Then, subsequent queries run against the cache until it fills again and the procedure repeats. Note that the cache can be a very valuable tool if you have a large dataset, and a large percentage of users only query a specific subset of vectors. In this case you might be able to serve the largest user group from cache while requiring disk lookups for "irregular" queries. ## Asynchronous indexing This feature relates to the vector index, specifically only to the HNSW and HFresh indexes. Asynchronous indexing can be enabled by opting in as follows: - Open-source users can do this by setting the `ASYNC_INDEXING` environment variable to `true`. - Weaviate Cloud users can do this by toggling the `Enable async indexing` switch in the Weaviate Cloud Console. With synchronous indexing, the vector index is updated in lockstep with the object store. Updating an HNSW index can be an expensive operation, especially as the size of the index grows. As a result, the indexing operation can be the bottleneck in the system, slowing down the time for user requests to be completed. When asynchronous indexing is enabled, all vector indexing operations go through a queue. This applies to not only batch imports, but also to single object imports, deletions, and updates. This means that the object store can be updated quickly to finish performing user requests while the vector index updates in the background. Asynchronous indexing is especially useful for importing large amounts of data. This means that there will be a short delay between object creation and the object being available for vector search using the HNSW index. The number of objects in the queue can be monitored per node [as shown here](/deploy/configuration/status.md#cluster-node-data). :::info Changes in `v1.28` In Weaviate `v1.22` to `v1.27`, the async indexing feature only affected batch import operations, using an in-memory queue.
Starting in `v1.28`, the async indexing feature has been expanded to include single object imports, deletions, and updates. Additionally, the in-memory queue has been replaced with a persistent, on-disk queue. This change allows for more robust handling of indexing operations, and improves performance though reduction of lock contention and memory usage.
The use of an on-disk queue may result in a slight increase in disk usage, however this is expected to be a small percentage of the total disk usage. ::: ## Vector indexing FAQ ### Can I use vector indexing with vector quantization? Yes, you can read more about it in [vector quantization (compression)](../vector-quantization.md). ### Which vector index is right for me? Here's a quick guide to choosing the right index: - **Flat index**: Best for SaaS products where each end user (tenant) has their own isolated, small dataset. Fast for small collections with a known size for minimal memory overhead. - **HNSW index**: Best for large collections requiring high query throughput and low latency. Requires more memory but provides excellent search performance. - **Dynamic index**: Best for collections that start small but may grow significantly over time. Automatically transitions from flat to HNSW as data scales. - **HFresh index**: Best when memory efficiency is the priority, especially with high-dimensional vectors. Suitable from small collections up to very large ones. #### Comparison between index types | Feature | Flat | HNSW | HFresh | | ----------------------------- | -------------------------------- | --------------------------- | --------------------------------------------------- | | Memory usage | Very low | High | Low | | Search speed (small datasets) | Fast | Very fast | Moderate | | Search speed (large datasets) | Slow | Very fast | Fast | | Disk usage | Low | Moderate | Moderate to high | | Maintenance | None | Costlier as the graph grows | Self-balancing in the background, no full rebuilds | | Best for | Small collections, multi-tenancy | Large collections, high QPS | Memory-constrained deployments, any size (disk-backed) | Note that the vector index type parameter only specifies how the vectors of data objects are _indexed_. The index is used for data retrieval and similarity search. The `vectorizer` parameter determines how the data vectors are created (which numbers the vectors contain). `vectorizer` specifies a [module](/weaviate/modules/index.md), such as `text2vec-contextionary`, that Weaviate uses to create the vectors. (You can also set to `vectorizer` to `none` if you want to import your own vectors). To learn more about configuring the collection, see [this how-to page](../../manage-collections/vector-config.mdx). ### Which distance metrics can I use with vector indexing? All of [the distance metrics](/weaviate/config-refs/distances.md), such as cosine similarity, can be used with most vector index types. The HFresh index only supports `cosine` and `l2-squared` distance metrics. ### How to configure the vector index type in Weaviate? The index type can be specified per data collection via the [collection definition](../../manage-collections/vector-config.mdx#set-vector-index-type) settings, according to available [vector index settings](../../config-refs/indexing/vector-index.mdx). ### When to skip indexing There are situations where it doesn't make sense to vectorize a collection. For example, if the collection consists solely of references between two other collections, or if the collection contains mostly duplicate elements. Importing duplicate vectors into HNSW is very expensive. The import algorithm checks early on if a candidate vector's distance is greater than the worst candidate's distance. When there are lots of duplicate vectors, this early exit condition is never met so each import or query results in an exhaustive search. To avoid indexing a collection, set `"skip"` to `"true"`. By default, collections are indexed. ### What ANN algorithms exist? There are different ANN algorithms, you can find a nice overview of them on this website. ### Are there indicative benchmarks for Weaviate's ANN performance? The [ANN benchmark page](/weaviate/benchmarks/ann.md) contains a wide variety of vector search use cases and relative benchmarks. This page is ideal for finding a dataset similar to yours and learning what the most optimal settings are. ## Further resources :::info Related pages - [Concepts: Vector quantization (compression)](../vector-quantization.md) - [Configuration: Vector index](../../config-refs/indexing/vector-index.mdx) - [Configuration: Schema (Configure semantic indexing)](../../config-refs/indexing/vector-index.mdx#configure-semantic-indexing) ::: ## Questions and feedback import DocsFeedback from '/\_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Replication Architecture/ Category .Json (docs/weaviate/concepts/replication-architecture/_category_.json) { "label": "Replication Architecture", "position": 35 } --- ### Weaviate/Concepts/Replication Architecture/Cluster Architecture (docs/weaviate/concepts/replication-architecture/cluster-architecture.md) --- title: Cluster Architecture sidebar_position: 3 description: "Node behavior and cluster coordination mechanisms in Weaviate's distributed replication system." image: og/docs/concepts.jpg # tags: ['architecture'] --- This page describes how the nodes or clusters in Weaviate's replication design behave. In Weaviate, metadata replication and data replication are separate. For the metadata, Weaviate uses the [Raft](https://raft.github.io/) consensus algorithm. For data replication, Weaviate uses a leaderless design with eventual consistency. ## Node Discovery By default, Weaviate nodes in a cluster use a gossip-like protocol through [Hashicorp's Memberlist](https://github.com/hashicorp/memberlist) to communicate node state and failure scenarios. Weaviate is optimized to run on Kubernetes, especially when operating as a cluster. The [Weaviate Helm chart](/deploy/installation-guides/k8s-installation.md#weaviate-helm-chart) makes use of a `StatefulSet` and a headless `Service` that automatically configures node discovery. ## Metadata replication: Raft Weaviate uses the [Raft consensus algorithm](https://raft.github.io/) for metadata replication, implemented with Hashicorp's [raft library](https://pkg.go.dev/github.com/hashicorp/raft). Metadata in this context includes collection definition and shard/tenant states. Raft ensures that metadata changes are consistent across the cluster. A metadata change is forwarded to the leader node, which applies the change to its log before replicating it to the follower nodes. Once a majority of nodes have acknowledged the change, the leader commits the change to the log. The leader then notifies the followers, which apply the change to their logs. This architecture ensures that metadata changes are consistent across the cluster, even in the event of (a minority of) node failures. As a result, a Weaviate cluster will include a leader node that is responsible for metadata changes. The leader node is elected by the Raft algorithm and is responsible for coordinating metadata changes. ## Data replication: Leaderless Weaviate uses a leaderless architecture for data replication. This means there is no central leader or primary node that will replicate to follower nodes. Instead, all nodes can accept writes and reads from the client, which can offer better availability. There is no single point of failure. A leaderless replication approach, also known as [Dynamo-style](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf) data replication (after Amazon's implementation), has been adopted by other open-source projects like [Apache Cassandra](https://cassandra.apache.org). In Weaviate, a coordination pattern is used to relay a client's read and write requests to the correct nodes. Unlike in a leader-based database, a coordinator node does not enforce any ordering of the operations. The following illustration shows a leaderless replication design in Weaviate. There is one coordination node, which leads traffic from the client to the correct replicas. There is nothing special about this node; it was chosen to be the coordinator because this node received the request from the load balancer. A future request for the same data may be coordinated by a different node.

Replication Architecture

The main advantage of a leaderless replication design is improved fault tolerance. Without a leader that handles all requests, a leaderless design offers better availability. In a single-leader design, all writes need to be processed by this leader. If this node cannot be reached or goes down, no writes can be processed. With a leaderless design, all nodes can receive write operations, so there is no risk of one master node failing. On the flipside of high availability, a leaderless database tends to be less consistent. Because there is no leader node, data on different nodes may temporarily be out of date. Leaderless databases tend to be eventually consistent. Consistency in Weaviate is [tunable](./consistency.md), but this occurs at the expense of availability. ## Replication Factor import RaftRFChangeWarning from '/_includes/1-25-replication-factor.mdx'; In Weaviate, data replication is enabled and controlled per collection. This means you can have different replication factors for different collections. The replication factor (RF or n) determines how many copies of data are stored in the distributed setup. A replication factor of 1 means that there is only 1 copy of each data entry in the database setup, in other words there is no replication. A replication factor of 2 means that there are two copies of each data entry, which are present on two different nodes (replicas). Naturally, the replication factor cannot be higher than the number of nodes. Any node in the cluster can act as a coordinating node to lead queries to the correct target node(s). A replication factor of 3 is commonly used, since this provides a right balance between performance and fault tolerance. An odd number of nodes is generally preferred, as it makes it easier to resolve conflicts. In a 3-node setup, a quorum can be reached with 2 nodes. Therefore the fault tolerance is 1 node. In a 2-node setup, on the other hand, no node failures can be tolerated while still reaching consensus across nodes. In a 4-node setup, respectively, 3 nodes would be required to reach a consensus. Thus, a 3-node setup has a better fault-tolerance to cost ratio than either a 2-node or 4-node setup.

Replication Factor

## Write operations On a write operation, the client's request will be sent to any node in the cluster. The first node which receives the request is assigned as the coordinator. The coordinator node sends the request to a number of predefined replicas and returns the result to the client. So, any node in the cluster can be a coordinator node. A client will only have direct contact with this coordinator node. Before sending the result back to the client, the coordinator node waits for a number of write acknowledgments from different nodes depending on the configuration. How many acknowledgments Weaviate waits for, depends on the [consistency configuration](./consistency.md). **Steps** 1. The client sends data to any node, which will be assigned as the coordinator node 2. The coordinator node sends the data to more than one replica node in the cluster 3. The coordinator node waits for acknowledgment from a specified proportion (let's call it `x`) of cluster nodes. Starting with v1.18, `x` is [configurable](./consistency.md), and defaults to `QUORUM` nodes. 4. When `x` ACKs are received by the coordinator node, the write is successful. As an example, consider a cluster size of 3 with replication factor of 3. So, all nodes in the distributed setup contain a copy of the data. When the client sends new data, this will be replicated to all three nodes.

Replication Factor 3 with cluster size 3

With a cluster size of 8 and a replication factor of 3, a write operation will not be sent to all 8 nodes, but only to those three containing the replicas. The coordinating node will determine which nodes the data will be written to. Which nodes store which collections (and therefore shards) is determined by the setup of Weaviate, which is known by each node and thus each coordinator node. Where something is replicated is deterministic, so all nodes know on which shard which data will land.

Replication Factor 3 with cluster size 8

## Read operations Read operations are also coordinated by a coordinator node, which directs a query to the correct nodes that contain the data. Since one or more nodes may contain old (stale) data, the read client will determine which of the received data is the most recent before sending it to the user. **Steps** 1. The client sends a query to Weaviate, any node in the cluster that receives the request first will act as the coordinator node 2. The coordinator node sends the query to more than one replica node in the cluster 3. The coordinator waits for a response from x nodes. *x is [configurable](./consistency.md) (`ALL`, `QUORUM` or `ONE`, available from v1.18, Get-Object-By-ID type requests have tunable consistency from v1.17).* 4. The coordinator node resolves conflicting data using some metadata (e.g. timestamp, id, version number) 5. The coordinator returns the latest data to the client If the cluster size is 3 and the replication factor is also 3, then all nodes can serve the query. The consistency level determines how many nodes will be queried. If the cluster size is 10 and the replication factor is 3, the 3 nodes which contain that data (collection) can serve queries, coordinated by the coordinator node. The client waits until x (the consistency level) nodes have responded. ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Replication Architecture/Consistency (docs/weaviate/concepts/replication-architecture/consistency.md) --- title: Consistency sidebar_position: 4 description: "Replication factor configuration and data consistency models across Weaviate cluster replicas." image: og/docs/concepts.jpg # tags: ['architecture'] --- import SkipLink from '/src/components/SkipValidationLink' The replication factor in Weaviate determines how many copies of shards (also called replicas) will be stored across a Weaviate cluster.

Replication factor

When the replication factor is > 1, consistency models balance the system's reliability, scalability, and/or performance requirements. Weaviate uses multiple consistency models. One for its cluster metadata and another for its data objects. ### Consistency models in Weaviate Weaviate uses the [Raft](https://raft.github.io/) consensus algorithm for [cluster metadata replication](./cluster-architecture.md#metadata-replication-raft). Cluster metadata in this context includes the collection definitions and tenant activity statuses. This allows cluster metadata updates to occur even when some nodes are down. Data objects are replicated using a [leaderless design](./cluster-architecture.md#data-replication-leaderless) using tunable consistency levels. So, data operations can be tuned to be more consistent or more available, depending on the desired tradeoff. These designs reflect the trade-off between consistency and availability that is described in the [CAP Theorem](./index.md#cap-theorem). :::tip Rule of thumb on consistency The strength of consistency can be determined by applying the following conditions: * If r + w > n, then the system is strongly consistent. * r is the consistency level of read operations * w is the consistency level of write operations * n is the replication factor (number of replicas) * If r + w <= n, then eventual consistency is the best that can be reached in this scenario. ::: ## Cluster metadata The cluster metadata in Weaviate makes use of the Raft algorithm. Weaviate uses the [Raft](https://raft.github.io/) consensus algorithm for cluster metadata replication. Raft is a consensus algorithm with an elected leader node that coordinates replication across the cluster using a log-based approach. As a result, each request that changes the cluster metadata will be sent to the leader node. The leader node will apply the change to its logs, then propagate the changes to the follower nodes. Once a quorum of nodes has acknowledged the cluster metadata change, the leader node will commit the change and confirm it to the client. This architecture ensures that cluster metadata changes are consistent across the cluster, even in the event of (a minority of) node failures.

Pre-v1.25 cluster metadata consensus algorithm

Prior to using Raft, a cluster metadata update was done via a [Distributed Transaction](https://en.wikipedia.org/wiki/Distributed_transaction) algorithm. This is a set of operations that is done across databases on different nodes in the distributed network. Weaviate used a [two-phase commit (2PC)](https://en.wikipedia.org/wiki/Two-phase_commit_protocol) protocol, which replicates the cluster metadata updates in a short period of time (milliseconds). A clean (without fails) execution has two phases: 1. The commit-request phase (or voting phase), in which a coordinator node asks each node whether they are able to receive and process the update. 2. The commit phase, in which the coordinator commits the changes to the nodes.

### Collection definition requests in queries Some queries require the collection definition. Prior to the introduction of this feature, every such query led to the local (requesting) node to fetch the collection definition from the leader node. This meant that the definition was strongly consistent, but it could lead to additional traffic and load. Where available, the `COLLECTION_RETRIEVAL_STRATEGY` [environment variable](/deploy/configuration/env-vars/index.md#multi-node-instances) can be set to `LeaderOnly`, `LocalOnly`, or `LeaderOnMismatch`. - `LeaderOnly` (default): Always requests the definition from the leader node. This is the most consistent behavior but can lead to higher intra-cluster traffic. - `LocalOnly`: Always use the local definition; leading to eventually consistent behavior while reducing intra-cluster traffic. - `LeaderOnMismatch`: Checks if the local definition is outdated, and requests the definition if necessary. Balances consistency and intra-cluster traffic. The default behavior is `LeaderOnly` to achieve strong consistency. However, `LocalOnly` and `LeaderOnMismatch` can be used to reduce intra-cluster traffic according to the desired consistency level. ## Data objects Weaviate uses two-phase commits for objects, adjusted for the consistency level. For example for a `QUORUM` write (see below), if there are 5 nodes, 3 requests will be sent out, each of them using a 2-phase commit under the hood. As a result, data objects in Weaviate are eventually consistent. Eventual consistency provides BASE semantics: * **Basically available**: reading and writing operations are as available as possible * **Soft-state**: there are no consistency guarantees since updates might not yet have converged * **Eventually consistent**: if the system functions long enough, after some writes, all nodes will be consistent. Weaviate uses eventual consistency to improve availability. Read and write consistency are tunable, so you can tradeoff between availability and consistency to match your application needs. *The animation below is an example of how a write or a read is performed with Weaviate with a replication factor of 3 and 8 nodes. The blue node acts as the coordinator node. The consistency level is set to `QUORUM`, so the coordinator node only waits for two out of three responses before sending the result back to the client.*

Write consistency QUORUM

### Tunable write consistency Adding or changing data objects are **write** operations. :::note Write operations are tunable starting with Weaviate v1.18, to `ONE`, `QUORUM` (default) or `ALL`. In v1.17, write operations are always set to `ALL` (highest consistency). ::: The main reason for introducing configurable write consistency in v1.18 is because that is also when automatic repairs are introduced. A write will always be written to n (replication factor) nodes, regardless of the chosen consistency level. The coordinator node however waits for acknowledgments from `ONE`, `QUORUM` or `ALL` nodes before it returns. To guarantee that a write is acknowledged everywhere before the request returns, without relying on read-time repairs, set write consistency to `ALL`. Possible settings in v1.18+ are: * **ONE** - a write must receive an acknowledgment from at least one replica node. This is the fastest (most available), but least consistent option. * **QUORUM** - a write must receive an acknowledgment from at least `QUORUM` replica nodes. `QUORUM` is calculated as _n / 2 + 1_, where _n_ is the number of replicas (replication factor). For example, using a replication factor of 6, the quorum is 4, which means the cluster can tolerate 2 replicas down. * **ALL** - a write must receive an acknowledgment from all replica nodes. This is the most consistent, but 'slowest' (least available) option. *Figure below: a replicated Weaviate setup with write consistency of ONE. There are 8 nodes in total out of which 3 replicas.*

Write consistency ONE

*Figure below: a replicated Weaviate setup with Write Consistency of `QUORUM` (n/2+1). There are 8 nodes in total, out of which 3 replicas.*

Write consistency QUORUM

*Figure below: a replicated Weaviate setup with Write Consistency of `ALL`. There are 8 nodes in total, out of which 3 replicas.*

Write consistency ALL

### Tunable read consistency Read operations are GET requests to data objects in Weaviate. Like write, read consistency is tunable, to `ONE`, `QUORUM` (default) or `ALL`. :::note Prior to `v1.18`, read consistency was tunable only for [requests that obtained an object by id](../../manage-objects/read.mdx#get-an-object-by-id), and all other read requests had a consistency of `ALL`. ::: The following consistency levels are applicable to most read operations: - Starting with `v1.18`, consistency levels are applicable to REST endpoint operations. - Starting with `v1.19`, consistency levels are applicable to GraphQL `Get` requests. - All gRPC based read and write operations support tunable consistency levels. * **ONE** - a read response must be returned by at least one replica. This is the fastest (most available), but least consistent option. * **QUORUM** - a response must be returned by `QUORUM` amount of replica nodes. `QUORUM` is calculated as _n / 2 + 1_, where _n_ is the number of replicas (replication factor). For example, using a replication factor of 6, the quorum is 4, which means the cluster can tolerate 2 replicas down. * **ALL** - a read response must be returned by all replicas. The read operation will fail if at least one replica fails to respond. This is the most consistent, but 'slowest' (least available) option. Examples: * **ONE**
In a single datacenter with a replication factor of 3 and a read consistency level of ONE, the coordinator node will wait for a response from one replica node.

Write consistency ONE

* **QUORUM**
In a single datacenter with a replication factor of 3 and a read consistency level of `QUORUM`, the coordinator node will wait for n / 2 + 1 = 3 / 2 + 1 = 2 replicas nodes to return a response.

Write consistency QUORUM

* **ALL**
In a single datacenter with a replication factor of 3 and a read consistency level of `ALL`, the coordinator node will wait for all 3 replicas nodes to return a response.

Write consistency ALL

### Tunable consistency strategies Depending on the desired tradeoff between consistency and speed, below are three common consistency level pairings for write / read operations. These are _minimum_ requirements that guarantee eventually consistent data: * `QUORUM` / `QUORUM` => balanced write and read latency * `ONE` / `ALL` => fast write and slow read (optimized for write) * `ALL` / `ONE` => slow write and fast read (optimized for read) ### Tunable consistency and queries Note that tunable consistency levels for read operations do not affect consistency of the list of objects returned by a query. In other words, the list of object UUIDs returned by a query depends only on the coordinator node's (and any other required shards') local index, and is independent of the read consistency level. This is due to the fact that each query is performed by the coordinator node and any other shards required to answer the query. Even if the read consistency level is set to `ALL`, it does not mean that multiple replicas will be queried and the results merged together. Where the read consistency level is applied is in retrieving the identified objects from the replicas. For example, if the read consistency level is set to `ALL`, the coordinator node will wait for all replicas to return the identified objects. And if the read consistency level is set to `ONE`, the coordinator node may simply return the objects from itself. In other words, the read consistency level only affects which versions of the objects are retrieved, but it does not lead to a more (or less) consistent query result. :::note When might this occur? By default, Weaviate writes to all nodes on an insert/update/delete. So, most of the time this won't matter as all shards will have identical local indexes to each other. This is a rare care which may only occur if there is a problem, such as a node being down, or there is a network problem. ::: ### Tenant states and data objects Each tenant in a [multi-tenant collection](../data.md#multi-tenancy) has a configurable [tenant state](../../starter-guides/managing-resources/tenant-states.mdx), which determines the availability and location of the tenant's data. The tenant state can be set to `active`, `inactive`, or `offloaded`. An `active` tenant's data should be available for queries and updates, while `inactive` or `offloaded` tenants are not. However, there can be a delay between the time a tenant state is set, and when the tenant's data reflects the (declarative) tenant state. As a result, a tenant's data may be available for queries for a period of time even if the tenant state is set to `inactive` or `offloaded`. Conversely, a tenant's data may not be available for queries and updates for a period of time even if the tenant state is set to `active`. :::info Why is this not addressed by repair-on-read? For speed, data operations on a tenant occur independently of any tenant activity status operations. As a result, tenant states are not updated by repair-on-read operations. ::: ## Repairs In distributed systems like Weaviate, object replicas can become inconsistent due to any number of reasons - network issues, node failures, or timing conflicts. When Weaviate detects inconsistent data across replicas, it attempts to repair the out of sync data. Weaviate uses [async replication](#async-replication), [deletion resolution](#deletion-resolution-strategies) and [repair-on-read](#repair-on-read) strategies to maintain consistency across replicas. ### Async replication Async replication is a background synchronization process in Weaviate that ensures eventual consistency across nodes storing the same data. When each shard is replicated across multiple nodes, async replication guarantees that all nodes holding copies of the same data remain in sync by periodically comparing and propagating data. It uses a Merkle tree (hash tree) algorithm to monitor and compare the state of nodes within a cluster. If the algorithm identifies an inconsistency, it resyncs the data on the inconsistent node. Repair-on-read works well with one or two isolated repairs. Async replication is effective in situations where there are many inconsistencies. For example, if an offline node misses a series of updates, async replication quickly restores consistency when the node returns to service. Async replication supplements the repair-on-read mechanism. If a node becomes inconsistent between sync checks, the repair-on-read mechanism catches the problem at read time. As of Weaviate `v1.38`, async replication is enabled by default for any collection with a replication factor greater than `1`, there is no per-collection flag to enable it. To turn it off cluster-wide, set the `ASYNC_REPLICATION_DISABLED` environment variable to `true`. Visit the [How-to: Replication](/deploy/configuration/replication.md#async-replication-settings) page to learn more about the available async replication settings. #### Memory and performance considerations for async replication Async replication uses a hash tree to compare and synchronize data between the database cluster nodes, based on objects' latest update time. The additional memory required for this process is determined by the height of the hash tree (`H`). A higher hash tree uses more memory but allows faster hashing, reducing the time required to detect and repair inconsistencies. The trade-offs can be summarized like this: - **Higher** `H`: Higher memory usage, faster replication. - **Lower** `H`: Lower memory usage, slower replication. :::tip Memory management for multi-tenancy Each tenant is backed by a shard. Therefore, when there is a high number of tenants, the memory consumption of async replication can be significant. (e.g. 1,000 tenants with a hash tree height of 16 will require an extra ~2 GB of memory per node, while a height of 20 will require ~34 GB per node).
As of `v1.36`, multi-tenant collections default to a hash tree height of `10` (~16KB per tenant per node), significantly reducing memory overhead compared to the single-tenant default of `16`.
To further reduce memory consumption, reduce the hash tree height. Keep in mind that this will result in slower hashing and potentially slower replication. ::: Use the following formulas and examples as a quick reference: ##### Memory calculation - **Total number of nodes in the hash tree:** For a hash tree with height `H`, the total number of nodes is: ``` Number of hash tree nodes = 2^(H+1) - 1 ≈ 2^(H+1) ``` - **Total memory required (per shard/tenant on each node):** Each hash tree node uses approximately **16 bytes** of memory. ``` Memory Required ≈ 2^(H+1) * 16 bytes ``` ##### Examples - Hash tree with height `16`: - `Total hash tree nodes ≈ 2^(16+1) = 131,072` - `Memory required ≈ 131072 * 16 bytes ≈ 2,097,152 bytes (~2 MB)` - Hash tree with height `20`: - `Total hash tree nodes ≈ 2^(20+1) = 2,097,152` - `Memory required ≈ 2,097,152 * 16 bytes ≈ 33,554,432 bytes (~33 MB)` ##### Performance Consideration: Number of Leaves The objects in a shard (e.g. tenant) are distributed among the leaves of the hash tree. A larger hash tree means less data for each leaf to hash, leading to faster comparisons and faster replication. - **Number of Leaves in the hash tree:** ``` Number of leaves = 2^H ``` ##### Examples - Hash tree with height `16`: - `Number of Leaves = 2^16 = 65,536` - Hash tree with height `20`: - `Number of Leaves = 2^20 = 1,048,576` :::note Default settings The default hash tree height is `16` for single-tenant collections and `10` for multi-tenant collections. These defaults balance memory consumption with replication performance. As of `v1.36`, these parameters can be configured per-collection via the [`asyncConfig`](/weaviate/config-refs/collections#async-config) object in `replicationConfig`. Worker concurrency is no longer a per-collection setting and as of `v1.38` the cluster shares a single async replication worker pool sized by [`ASYNC_REPLICATION_SCHEDULER_WORKERS`](/deploy/configuration/env-vars/index.md#async-replication). ::: ### Deletion resolution strategies When an object is present on some replicas but not others, this can be because a creation has not yet been propagated to all replicas, or because a deletion has not yet been propagated to all replicas. It is important to distinguish between these two cases. Deletion resolution works alongside async replication and repair-on-read to ensure consistent handling of deleted objects across the cluster. For each collection, [you can set one of the following](../../manage-collections/multi-node-setup.mdx#replication-settings) deletion resolution strategies: - `NoAutomatedResolution` - `DeleteOnConflict` - `TimeBasedResolution` Deletion resolution strategies are mutable. [Read more about how to update collection definitions](../../manage-collections/collection-operations.mdx#update-a-collection-definition). #### `NoAutomatedResolution` In this mode, Weaviate does not treat deletion conflicts as a special case. If an object is present on some replicas but not others, Weaviate may potentially restore the object on the replicas where it is missing. #### `DeleteOnConflict` A deletion conflict in `deleteOnConflict` is always resolved by deleting the object on all replicas. To do so, Weaviate updates an object as a deleted object on a replica upon receiving a deletion request, rather than removing all traces of the object. #### `TimeBasedResolution` This is the default setting from `v1.36` onwards. A deletion conflict in `timeBasedResolution` is resolved based on the timestamp of the deletion request, in comparison to any subsequent updates to the object such as a creation or an update. If the deletion request has a timestamp that is later than the timestamp of any subsequent updates, the object is deleted on all replicas. If the deletion request has a timestamp that is earlier than the timestamp of any subsequent updates, the later updates are applied to all replicas. For example: - If an object is deleted at timestamp 100 and then recreated at timestamp 110, the recreation wins - If an object is deleted at timestamp 100 and then recreated at timestamp 90, the deletion wins #### Choosing a strategy - Use `NoAutomatedResolution` when you want maximum control and handle conflicts manually - Use `DeleteOnConflict` when you want to ensure deletions are always honored - Use `TimeBasedResolution` when you want the most recent operation to take precedence ### Repair-on-read If your read consistency is set to `All` or `Quorum`, the read coordinator will receive responses from multiple replicas. If these responses differ, the coordinator can attempt to repair the inconsistency, as shown in the examples below. This process is called "repair-on-read", or "read repairs". | Problem | Action | | :- | :- | | Object never existed on some replicas. | Propagate the object to the missing replicas. | | Object is out of date. | Update the object on stale replicas. | | Object was deleted on some replicas. | Returns an error. Deletion may have failed, or the object may have been partially recreated. When using the `TimeBasedResolution` deletion strategy, the most recent version wins based on timestamps. | The read repair process also depends on the read and write consistency levels used. | Write consistency level | Read consistency level | Action | | :- | :- | | `ONE` | `ALL` | Weaviate has to verify all nodes to guarantee repair. | | `QUORUM` | `QUORUM` or `ALL` | Weaviate attempts to fix the sync issues. | | `ALL` | - | This situation should not occur. The write should have failed. | Repairs only happen on read, so they do not create a lot of background overhead. While nodes are in an inconsistent state, read operations with consistency level of `ONE` may return stale data. ## Replica movement import ReplicaMovement from '/_includes/feature-notes/replica-movement.mdx'; A shard represents a part of the collection in a single-tenant collection, or a whole tenant in a multi-tenant collection. Weaviate allows users to manually move or copy individual shard replicas from a source node to a destination node in a Weaviate cluster. This capability addresses operational scenarios such as cluster rebalancing after scaling, node decommissioning, optimizing data locality for improved performance, or increasing data availability. Replica movement operates as a state machine with stages that ensure data integrity throughout the process. The feature works for both single-tenant collections and multi-tenant collections. Unlike the static replication factor configured at collection creation, replica movement allows the replication factor to be adjusted for specific shards as replicas are moved or copied across the cluster. When a copy operation is performed, the newly created replica increases the replication factor for that specific shard. While a collection may have a default replication factor, individual shards within that collection can have a higher replication factor. However, shards can't have a replication factor lower then the one set on the collection level. :::info Replica movement must be enabled by setting the [`REPLICA_MOVEMENT_ENABLED` environment variable](/docs/deploy/configuration/env-vars/index.md#REPLICA_MOVEMENT_ENABLED) to `true`. When disabled (the default). The [`REPLICATION_ENGINE_MAX_WORKERS` environment variable](/docs/deploy/configuration/env-vars/index.md#REPLICATION_ENGINE_MAX_WORKERS) can be used to adjust the number of workers that process replica movements in parallel. ::: ### Movement states Each replica movement operation progresses through a workflow designed to maintain data consistency and availability. The workflow comprises of the following states: - **REGISTERED**: The movement operation has been initiated and logged by the Raft leader. The request has been received and the operation is queued for processing. - **HYDRATING**: A new replica is being created on the destination node. Data segments are transferred from an existing replica (usually the source replica, or another available peer) to establish the new replica. - **FINALIZING**: The bulk data transfer is complete, and the new replica is catching up on any writes that occurred during the transfer. This ensures the replica is fully synchronized with the latest data. You can use the [`REPLICA_MOVEMENT_MINIMUM_ASYNC_WAIT` environment variable](/docs/deploy/configuration/env-vars/index.md#REPLICA_MOVEMENT_MINIMUM_ASYNC_WAIT) to adjust the wait time which ensures that any in progress writes have been completed and replicated to the target node. - **INTEGRATING** (added in `v1.38.0`): The new replica has joined the shard's replica set and is being brought into the write path on every node. The operation waits until all nodes agree that the new replica is a write target, so that no node can acknowledge a write that skips it. Each node reports that it reached this state only after its own in-flight writes to the shard have drained, which prevents a write that was already accepted from being lost. Once the cluster agrees, the last writes recorded on the source during the transition are applied to the new replica, and the source stops recording further changes for this operation. For copy operations, the next state is **READY**. For move operations, the next state is **DEHYDRATING**. - **DEHYDRATING**: For move operations, after the new replica is ready, the original replica on the source node is being removed. - **READY**: The operation has completed successfully. The new replica is fully synchronized and ready to serve traffic. For move operations, the source replica has been removed. - **CANCELLED**: The operation has been cancelled before completion. This can happen either through manual intervention or if the operation encounters an unrecoverable error. Replica movement supports two distinct operation modes: - **Move**: Move a replica from one node to another, maintaining the same replication factor - **Copy**: Copy a replica from one node to another and increase the shard replication factor by one for that specific shard :::note Replication factor and quorum When a shard replica is copied, the increased replication factor may become an even number. This can make achieving a quorum more difficult, as it now requires `(n/2 + 1)` nodes instead of `(n/2 + 0.5)` nodes. For example, going from `RF=3` to `RF=4` increases the required nodes for quorum from 2 to 3 (67% to 75% of replicas). ::: ## Related pages - [API References | GraphQL | Get | Consistency Levels](../../api/graphql/get.md#consistency-levels) - API References | REST | Objects ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Replication Architecture/Index (docs/weaviate/concepts/replication-architecture/index.md) --- title: Replication Architecture sidebar_position: 0 description: "Multi-node data replication design for high availability, reliability, and improved database performance." image: og/docs/concepts.jpg # tags: ['architecture'] --- Weaviate allows data replication across a multi-node cluster by [setting a replication factor](../../manage-collections/multi-node-setup.mdx#replication-settings) > 1. This enables a variety of [benefits](./motivation.md) such as [high availability](./motivation.md#high-availability-redundancy). Database replication improves reliability, scalability, and/or performance. Weaviate utilizes multiple replication architectures: - [Cluster metadata replication](./consistency.md#cluster-metadata) is managed by the [Raft](https://raft.github.io/) consensus algorithm. - [Data replication](./consistency.md#data-objects) is [tunable](./consistency.md) and leaderless.

What is the cluster metadata?

Weaviate cluster `metadata` includes collection definitions and tenant activity statuses.
All cluster metadata is always replicated across all nodes, regardless of the replication factor.
Note that this is different to object metadata, such as the object creation time. Object metadata is stored alongside the object data according to the specified replication factor.

In this Replication Architecture section, you will find information about: * **General Concepts**, on this page * What is replication? * CAP Theorem * Why replication for Weaviate? * Replication vs. Sharding * How does replication work in Weaviate? * Roadmap * **[Use Cases](./motivation.md)** * Motivation * High Availability * Increased (Read) Throughput * Zero Downtime Upgrades * Regional Proximity * **[Philosophy](./philosophy.md)** * Typical Weaviate use cases * Reasons for a leaderless architecture * Gradual rollout * Large-scale testing * **[Cluster Architecture](./cluster-architecture.md)** * Leaderless design * Replication Factor * Write and Read operations * **[Consistency](./consistency.md)** * Cluster metadata * Data objects * Repairs * **[Multi-DataCenter](./multi-dc.md)** * Regional Proximity ## What is replication?

Example setup with replication

Database replication refers to keeping a copy of the same data point on multiple nodes of a cluster. The resulting system is a distributed database. A distributed database consists of multiple nodes, all of which can contain a copy of the data. So if one node (server) goes down, users can still access data from another node. In addition, query throughput can be improved with replication. ## CAP Theorem The primary goal of introducing replication is to improve reliability. [Eric Brewer](https://en.wikipedia.org/wiki/Eric_Brewer_(scientist)) states that there are some limits on reliability for distributed databases, described by the [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem). The CAP theorem states that a distributed database can only provide two of the following three guarantees: * **Consistency (C)** - Every read receives the most recent write or an error, ensuring all nodes see the same data at the same time. * **Availability (A)** - Every request receives a non-error response all the time, without the guarantee that it contains the most recent write. * **Partition tolerance (P)** - The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes.

CAP Theorem

Ideally, you want a database, like Weaviate, to have the highest reliability as possible, but this is limited by the tradeoff between consistency, availability and partition tolerance. ### Consistency vs Availability :::tip Only two out of Consistency (C), Availability (A), and Partition tolerance (P) can be guaranteed simultaneously Given that partition tolerance is required, consider which of the other two are more important for your system. ::: Only two out of consistency, availability, and partition tolerance can be guaranteed. Since by definition a cluster is a distributed system in which network partitions are present, only two options are left for designing the system: **consistency (C)** or **availability (A)**. When you prioritize **consistency** over availability, the database will return an error or timeout when it cannot be guaranteed that the data is up to date due to network partitioning. When prioritizing **availability** over consistency, the database will always process the query and try to return the most recent version of data even if it cannot guarantee it is up to date due to network partitioning. C over A is preferred when the database contains critical data, such as transactional bank account data. For transactional data, you want the data to always be consistent (otherwise your bank balance is not guaranteed to be correct if you make transactions while some nodes (e.g. ATMs) are down). When a database involves less-critical data, A over C can be preferred. An example can be a messaging service, where you can tolerate showing some old data but the application should be highly available and handle large amounts of writes with minimal latency. Weaviate generally follows this latter design, since Weaviate typically deals with less critical data and is used for approximate search as a secondary database in use cases with more critical data. More about this design decision in [Philosophy](./philosophy.md). However, you can use Weaviate's [tunable consistency](./consistency.md#tunable-consistency-strategies) options according to your needs. ## Why replication for Weaviate? Weaviate, as a database, must provide reliable answers to users' requests. As discussed above, database reliability consists of various parts. Below are Weaviate use cases in which replication is desired. For detailed information, visit the [Replication Use Cases (Motivation) page](./motivation.md). 1. **High availability (redundancy)**
With a distributed (replicated) database structure, service will not be interrupted if one server node goes down. The database can still be available, read queries will just be (unnoticeably) redirected to an available node. 2. **Increased (read) throughput**
Adding extra server nodes to your database setup means that the throughput scales with it. The more server nodes, the more users (read operations) the system will be able to handle. When reading with consistency level of `ONE`, then scaling the replication factor (i.e. how many database server nodes) increases the throughput linearly. 3. **Zero downtime upgrades**
Without replication, there is a window of downtime when you update a Weaviate instance. This is because the single node needs to stop, update and restart before it's ready to serve again. With replication, upgrades are done using a rolling update, in which at most one node is unavailable at any point in time while the other nodes can still serve traffic. 4. **Regional proximity**
When users are located in different regional areas (e.g. Iceland and Australia as extreme examples), you cannot ensure low latency for all users due to the physical distance between the database server and the users. With a distributed database, you can place nodes in different local regions to decrease this latency. This depends on the Multi-Datacenter feature of replication. ## Replication vs. Sharding Replication is not the same as [sharding](../cluster.md). Sharding refers to horizontal scaling, and was introduced to Weaviate in v1.8. * **Replication** copies the data to different server nodes. For Weaviate, this increases data availability and provides redundancy in case a single node fails. Query throughput can be improved with replication. * **Sharding** handles horizontal scaling across servers by dividing the data and sending the pieces of data (shards) to multiple replica sets. The data is thus divided, and all shards together form the entire set of data. You can use sharding with Weaviate to run larger datasets and speed up imports.

Replication vs Sharding

Replication and sharding can be combined in a setup, to improve throughput and availability as well as import speed and support for large datasets. For example, you can have 3 replicas of the database and shards set to 3, which means you have 9 shards in total, where each server node holds 3 different shards. ## How does replication work in Weaviate? ### Cluster metadata replication Weaviate’s cluster metadata changes are managed through Raft to provide consistency across the cluster. (This includes collection definitions and tenant activity statuses.) From Weaviate `v1.25`, cluster metadata changes are committed using the Raft consensus algorithm. Raft is a leader-based consensus algorithm. A leader node is responsible for cluster metadata changes. Raft ensures that these changes are consistent across the cluster, even in the event of (a minority of) node failures.

Metadata replication pre-v1.25

Prior to Weaviate `v1.25`, each cluster metadata change was recorded via a distributed transaction with a two-phase commit.
This is a synchronous process, which means that the cluster metadata change is only committed when all nodes have acknowledged the change. In this architecture, any node downtime would temporarily prevent metadata operations. Additionally, only one such operation could be processed at a time. If you are using Weaviate `v1.24` or earlier, you can [upgrade to `v1.25`](/deploy/migration/weaviate-1-25.md) to benefit from the Raft consensus algorithm for cluster metadata changes.

### Data replication In Weaviate, availability is generally favored over consistency. Weaviate's data replication uses a leaderless design, which means there are no primary and secondary nodes. When writing and reading data, the client contacts one or more nodes. A load balancer exists between the user and the nodes, so the user doesn't know which node they are talking to (Weaviate will forward internally if a user is requesting a wrong node). The number of nodes that need to acknowledge the read or write (from v1.18) operation is tunable, to `ONE`, `QUORUM` (n/2+1) or `ALL`. When write operations are carried out with consistency level `ALL`, the database works synchronously. If write is not set to `ALL` (possible from v1.18), writing data is asynchronous from the user's perspective. The number of replicas doesn't have to match the number of nodes (cluster size). It is possible to split data in Weaviate based on collections. Note that this is [different from Sharding](#replication-vs-sharding). Read more about how replication works in Weaviate in [Philosophy](./philosophy.md), [Cluster Architecture](./cluster-architecture.md) and [Consistency](./consistency.md). ## How do I use replication in Weaviate? See [how to configure replication](/deploy/configuration/replication.md). You can enable replication in the collection definition. In queries, you can [specify the desired consistency level](../../search/basics.md#replication). ## Related pages - [Configuration: Replication](/deploy/configuration/replication.md) ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Replication Architecture/Motivation (docs/weaviate/concepts/replication-architecture/motivation.md) --- title: Use Cases (Motivation) sidebar_position: 1 description: "Four key use cases demonstrating the benefits and configuration requirements for Weaviate replication." image: og/docs/concepts.jpg # tags: ['architecture'] --- On this page you will find four use cases which motivate replication for Weaviate. Each of them serves a different purpose and, as a result, may require different configuration. ## High Availability (Redundancy) High availability of a database means that the database is designed to operate continuously without service interruptions. That means that the database system should be tolerant to failures and errors, which should be handled automatically. This is solved by replication, where redundant nodes can handle requests when other nodes fail. Weaviate considers cluster metadata operations critical, so it treats them differently than it does data operations. From Weaviate `v1.25`, Weaviate uses the Raft consensus algorithm for cluster metadata replication. This is a leader-based consensus algorithm, where a leader node is responsible for cluster metadata changes. Use of Raft ensures that cluster metadata changes are consistent across the cluster, even in the event of (a minority of) node failures. Prior to Weaviate `v1.25`, Weaviate used a leaderless design with two-phase commit for cluster metadata operations. This required all nodes for a cluster metadata operation such as a collection definition update, or a tenant state update. This meant that one or more nodes being down temporarily prevented cluster metadata operations. Additionally, only one cluster metadata operation could be processed at a time. Regarding data operations, read or write queries may still be available in a distributed database structure even when a node goes down, so single points of failure are eliminated. Users' queries will be automatically (unnoticeably) redirected to an available replica node. Examples of applications where High Availability is desired are emergency services, enterprise IT systems, social media, and website search. Nowadays, users are used to highly available applications, so they expect little to no downtime. For e.g. website search, service (read queries) should not be interrupted if a node goes down. In that case, if writing is temporarily unavailable, it is acceptable and in the worst case scenario the site search will be stale, but still available for read requests.

High Availability

High Availability can be illustrated by the following configuration examples: 1. Write `ALL`, Read `ONE` - There is no High Availability during writing because `ALL` nodes need to respond to write requests. There is High Availability on read requests: all nodes can go down except one while reading and the read operations are still available. 2. Write `QUORUM`, Read `QUORUM` (n/2+1) - A minority of nodes could go down, the majority of nodes should be up and running, and you can still do both reading and writing. 3. Write `ONE`, Read `ONE` - This is the most available configuration. All but one node can go down and both read and write operations are still possible. Note that this super High Availability comes with a cost of Low Consistency guarantees. Due to eventual consistency, your application must be able to deal with temporarily showing out-of-date data. ## Increased (Read) Throughput When you have many read requests on your Weaviate instance, for example because you're building an application for many users, the database setup should be able to support high throughput. Throughput is measured in Queries Per Second (QPS). Adding extra server nodes to your database setup means that the throughput scales with it. The more server nodes, the more users (read operations) the system will be able to handle. Thus, replicating your Weaviate instance increases throughput. When reading is set to a low consistency level (i.e. `ONE`), then scaling the replication factor (i.e. the number of database server nodes) increases the throughput linearly. For example, when the read consistency level is `ONE`, if one node can reach 10,000 QPS, then a setup with 3 replica nodes can receive 30,000 QPS.

Increased Throughput

## Zero Downtime Upgrades Without replication, there is a window of downtime when you update a Weaviate instance. The single node needs to stop, update and restart before it's ready to serve again. With replication, upgrades are done using a rolling update, in which at most one node is unavailable at the same time while the other nodes can still serve traffic. As an example, consider you're updating the version of a Weaviate instance from v1.19 to v1.20. Without replication there is a window of downtime: 1. Node is ready to serve traffic 2. Node is stopped, no requests can be served 3. Node image is replaced with newer version 4. Node is restarted 5. Node takes time to be ready 6. Node is ready to serve traffic. From step 2 until step 6 the Weaviate server cannot receive and respond to any requests. This leads to bad user experience. With replication (e.g. replication factor of 3), upgrades to the Weaviate version are done using a rolling update. At most one node will be unavailable at the same time, so all other nodes can still serve traffic. 1. 3 nodes ready to serve traffic 2. node 1 being replaced, nodes 2,3 can serve traffic 3. node 2 being replaced, nodes 1,3 can serve traffic 4. node 3 being replaced, nodes 1,2 can serve traffic

Zero downtime upgrades

## Regional Proximity When users are located in different regional areas (e.g. Iceland and Australia as extreme examples), you cannot ensure low latency for all users due to the physical distance between the database server and the users. You can only place the database server at one geographical location, so the question arises where you put the server: 1. Option 1 - Put the cluster in the middle (e.g. India).
All users will have relatively high latency, since data needs to travel between Iceland and India, and Australia and India.

Cluster in the geographical middle

2. Option 2 - Put the cluster close to one user group (e.g. Iceland)
Users from Iceland have very low latency while users from Australia experience relatively high latency since data needs to travel a long distance. Another option arises when you have the option to replicate your data cluster to two different geographical locations. This is called Multi-Datacenter (Multi-DC) replication. 3. Option 3 - Multi-DC replication with server clusters in both Iceland and Australia.
Users from Iceland and Australia now both experience low latency, because each user group is served from local clusters.

Replication multi-dc

Multi-DC replication also comes with the additional benefit that data is redundant on more physical locations, which means that in the rare case of an entire datacenter going down, data can still be served from another location. :::note Regional Proximity is enabled by [running a Weaviate cluster across data centers](./multi-dc.md). ::: ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Replication Architecture/Multi Dc (docs/weaviate/concepts/replication-architecture/multi-dc.md) --- title: Multi-Data center sidebar_position: 5 description: "Run a Weaviate cluster across multiple data centers (WAN) for lower latency and cross-region redundancy." image: og/docs/concepts.jpg # tags: ['architecture'] --- Multi-Data center (Multi-DC) replication enables you to have multiple copies of the data on multiple servers across more than one data center. Weaviate supports running a single cluster across multiple data centers (since `v1.31`), so you can place nodes in different geographic regions for lower latency and keep your data redundant across locations. Multi-DC replication is beneficial if you have user groups spread over different geographical locations (e.g. Iceland and Australia). When you place nodes in different local regions of user groups, latency can be decreased. Multi-DC replication also comes with the additional benefit that data is redundant on more physical locations, which means that in the rare case of an entire data center going down, data can still be served from another location. If all replica nodes are in the same data center, network requests between nodes are cheap and fast, but the whole deployment is at risk if that data center goes down. Spreading a cluster's nodes across data centers removes this single point of failure, so your data remains available even if an entire data center becomes unreachable. To run a cluster across data centers, tune Weaviate's inter-node networking for the higher-latency, lower-reliability links between regions. See [Running a single cluster across data centers (WAN)](#running-a-single-cluster-across-data-centers-wan) below.

Replication multi-dc

## Running a single cluster across data centers (WAN) Weaviate can operate a single cluster across data centers (a wide-area network, or WAN) by adjusting its inter-node communication for high-latency, lower-reliability links. ### Enabling WAN mode Set the `CLUSTER_ADVERTISE_ADDR` environment variable to enable WAN mode. When it is set, Weaviate switches its internal [memberlist](https://github.com/hashicorp/memberlist) configuration to `DefaultWANConfig`, which increases timeouts and relaxes failure-detection thresholds so they are suitable for cross-data center communication. ### Key environment variables | Variable | Description | Required | | --- | --- | --- | | `CLUSTER_ADVERTISE_ADDR` | The public IP address that other nodes should use to reach this node. Setting this enables WAN mode. Must be a valid IP address (hostnames are rejected). | Yes (for WAN) | | `CLUSTER_ADVERTISE_PORT` | The port to advertise to other nodes. If not set, it defaults to `CLUSTER_GOSSIP_BIND_PORT`. If set, it must be between `1024` and `65535`. | No | | `CLUSTER_BIND_ADDR` | The local address to bind to. Defaults to `0.0.0.0`. | No | | `CLUSTER_GOSSIP_BIND_PORT` | The port used for gossip (memberlist) traffic. Defaults to `7946`. | No | | `CLUSTER_DATA_BIND_PORT` | The port used for data traffic. Defaults to `CLUSTER_GOSSIP_BIND_PORT + 1`. | No | | `CLUSTER_JOIN` | A comma-separated list of `host:port` addresses of existing cluster members to join. | Yes (for joining) | ### Example configuration Below is an example environment configuration for a node intended to participate in a cross-data center cluster: ```yaml environment: - CLUSTER_HOSTNAME=weaviate-0 - CLUSTER_ADVERTISE_ADDR=203.0.113.10 # Public IP of this node - CLUSTER_JOIN=203.0.113.20:7946 # Public IP:Port of a node in another DC - CLUSTER_BIND_ADDR=0.0.0.0 # Optional; this is already the default bind address - CLUSTER_GOSSIP_BIND_PORT=7946 - CLUSTER_DATA_BIND_PORT=7947 - RAFT_BOOTSTRAP_EXPECT=3 # Use an odd number > 1 across DCs for faster, more reliable elections - RAFT_JOIN=203.0.113.20:8300 # Join an existing Raft voter (default Raft port is 8300) ``` ### Notes - **Latency**: Cross-data center operations inherently have higher latency. Make sure your application timeouts account for this. - **Security**: Gossip and data traffic are not encrypted by default. For cross-data center communication over the public internet, use a VPN or other secure tunnel, or make sure `CLUSTER_ADVERTISE_ADDR` is only reachable over a private network (for example, VPC peering). ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Replication Architecture/Philosophy (docs/weaviate/concepts/replication-architecture/philosophy.md) --- title: Philosophy sidebar_position: 2 description: "Design principles and user-centric approach behind Weaviate's replication architecture decisions." image: og/docs/concepts.jpg # tags: ['architecture'] --- ## A design modeled after how our users use Weaviate The architecture that guides the principles for Weaviate’s replication systems is modeled after how users typically use Weaviate. Weaviate powers site search, recommendation, knowledge extraction, and other information retrieval cases. These cases all have a few things in common: * They are often **very-large-scale** (with datasets in the billions of objects and vectors) * They often incur **large parallel usage with strict latency requirements** (i.e. high throughput with low p99 latencies) * It is vital that the service is **highly-available** and resilient to unplanned outages or planned maintenance, such as version upgrades. * It is often tolerable if data is temporarily out of sync, as long as **consistency is reached eventually**. * Weaviate is sometimes used alongside strongly consistent, transactional databases if transactional data exists in a use case. In cases where Weaviate is used as the primary database, data is typically not transactional. * Weaviate’s users have a lot of experience working with cloud-native technologies, including NoSQL databases, and know how an application needs to be structured to deal with eventually consistent systems correctly. Based on the above usage patterns, and keeping the [CAP theorem](./index.md#cap-theorem) trade-offs in mind, Weaviate implements two different architectures for cluster metadata and data replication. 1. **Cluster metadata replication** is based on the [Raft consensus algorithm](https://raft.github.io/), which provides log-based consistency operations coordinated by an elected leader. This means that cluster metadata changes can be made even in the event of (a minority of) node failures. 2. **Data replication** is based on a leaderless design with tunable consistency. This means there is no central leader or primary node that will replicate to follower nodes. Weaviate’s data replication architecture **prefers availability over consistency**. Nevertheless, individual requests might have stricter consistency requirements than others. For those cases, Weaviate offers both [tunable read and write consistency](./consistency.md). ## Reasons for a leaderless architecture Weaviate’s replication architecture is inspired by other modern, Internet-scale databases that serve similar goals; [Apache Cassandra](https://cassandra.apache.org/_/index.html) is a notable example. Weaviate’s [Replication Architecture](./cluster-architecture.md) has significant similarities to Cassandra's. Unsurprisingly, a leaderless pattern was chosen to achieve both availability and throughput goals. In a leaderful setup, leader nodes have two significant disadvantages: Firstly, leaders become a performance bottleneck, e.g., because every write request needs to pass through the leader. Secondly, a leader's failure involves the election of a new leader, which can be a complex and costly process that can lead to temporary unavailability. The main advantage of a leaderful system is that it may be easier to provide specific consistency guarantees. As outlined in the motivation above, Weaviate prefers large-scale use cases, linear scaling, and availability over strict consistency. ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Search/Hybrid Search (docs/weaviate/concepts/search/hybrid-search.md) --- title: Hybrid search sidebar_position: 60 description: "Combined vector (semantic) and keyword search leveraging semantic similarity and exact keyword matching strengths." image: og/docs/concepts.jpg # tags: ['concepts', 'search', 'hybrid search', 'vector search', 'keyword search', 'bm25'] --- Hybrid search combines [vector search](./vector-search.md) and [keyword search (BM25)](./keyword-search.md) to leverage the strengths of both approaches. This takes into account results' semantic similarity (vector search) and exact keyword relevance (BM25), providing more comprehensive search results. A hybrid search runs both search types in parallel and combines their scores to produce a final ranking of results. This makes it versatile and robust, suitable for a wide range of search use cases. ## How hybrid search works In Weaviate, a hybrid search performs the following steps: 1. Executes both searches in parallel: - Vector search to find semantically similar content - BM25 search to find keyword matches 1. Combines the normalized scores using a [fusion method](#fusion-strategies) 1. Returns results ranked by the combined scores ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Fusion strategies Weaviate supports two strategies (`relativeScoreFusion` and `rankedFusion`) for combining vector and keyword search scores: With `relativeScoreFusion` (default from `v1.24`), each object is scored by *normalizing* the metrics output by the vector search and keyword search respectively. The highest value becomes 1, the lowest value becomes 0, and others end up in between according to this scale. The total score is thus calculated by a scaled sum of normalized vector distance and normalized BM25 score. With `rankedFusion` (default for `v1.23` and lower), each object is scored according to its position in the results for the given search, starting from the highest score for the top-ranked object and decreasing down the order. The total score is calculated by adding these rank-based scores from the vector and keyword searches. Generally, `relativeScoreFusion` might be a good choice, which is why it is the default. The main reason is that `relativeScoreFusion` retains more information from the original searches than `rankedFusion`, which only retains the rankings. More generally we believe that the nuances captured in the vector and keyword search metrics are more likely to be reflected in rankings produced by `relativeScoreFusion`. We include a concrete example of the two fusion strategies below. ### Fusion example Let's say that a search returns **five objects** with **document id** (from 0 to 4), and **scores** from **keyword** and **vector search**, **ordered by score**:

Search Type	(id): score	(id): score	(id): score	(id): score	(id): score
Keyword	(1): 5	(0): 2.6	(2): 2.3	(4): 0.2	(3): 0.09
Vector	(2): 0.6	(4): 0.598	(0): 0.596	(1): 0.594	(3): 0.009

#### Ranked fusion The score depends on the rank of each result and is computed according to `1/(RANK + 60)`, resulting in:

Search Type	(id): score	(id): score	(id): score	(id): score	(id): score
Keyword	(1): 0.0154	(0): 0.0160	(2): 0.0161	(4): 0.0167	(3): 0.0166
Vector	(2): 0.016502	(4): 0.016502	(0): 0.016503	(1): 0.016503	(3): 0.016666

As you can see, the results for each rank are identical, regardless of the input score. #### Relative score fusion In relative score fusion, the largest score is set to 1 and the lowest to 0, and all entries in-between are scaled according to their **relative distance** to the **maximum** and **minimum values**.

Search Type	(id): score	(id): score	(id): score	(id): score	(id): score
Keyword	(1): 1.0	(0): 0.511	(2): 0.450	(4): 0.022	(3): 0.0
Vector	(2): 1.0	(4): 0.996	(0): 0.993	(1): 0.986	(3): 0.0

The scores therefore reflect the relative distribution of the original scores. For example, the vector search scores of the first 4 documents were almost identical, which is still the case for the normalized scores. #### Comparison For the vector search, the scores for the top 4 objects (**IDs 2, 4, 0, 1**) were almost identical, and all of them were good results. While for the keyword search, one object (**ID 1**) was much better than the rest. This is captured in the final result of `relativeScoreFusion`, which identified the object **ID 1** the top result. This is justified because this document was the best result in the keyword search with a big gap to the next-best score and in the top group of vector search. In contrast, for `rankedFusion`, the object **ID 2** is the top result, closely followed by objects **ID 1** and **ID 0**. ### Alpha parameter The alpha value determines the weight of the vector search results in the final hybrid search results. The alpha value can range from 0 to 1: - `alpha = 0`: Keyword search only - `alpha < 0.5`: More weight to keyword search - `alpha = 0.5`: Equal weight to both searches - `alpha > 0.5`: More weight to vector search (`0.75` is the default) - `alpha = 1`: Vector search only Lower `alpha` towards `0` to give the keyword component more influence. :::caution Set `alpha` explicitly `0.75` is the server default. It applies only when a request reaches Weaviate with no `alpha` value, which is the case for GraphQL, and over gRPC from Weaviate `v1.36.7` and later, which added the ability for a client to leave `alpha` unset. Client libraries do not all leave `alpha` unset. Depending on your client and your server version, the effective weighting can differ from `0.75`, and in some cases can be a pure keyword search. Set `alpha` explicitly whenever the weighting matters, and check your client library page for its behavior. ::: ## Search thresholds Hybrid search supports a maximum vector distance threshold through the `max vector distance` parameter. This threshold applies only to the vector search component of the hybrid search, allowing you to filter out results that are too dissimilar in vector space, regardless of their keyword search scores. For example, consider a maximum vector distance of `0.3`. This means objects with a vector distance higher than `0.3` will be excluded from the hybrid search results, even if they have high keyword search scores. This can be useful when you want to ensure semantic similarity meets a minimum standard while still taking advantage of keyword matching. There is no equivalent threshold parameter for the keyword (BM25) component of hybrid search or the final combined scores. This is because BM25 scores are not normalized or bounded like vector distances, making a universal threshold less meaningful. ## Keyword (BM25) search parameters Hybrid search in Weaviate supports all the parameters available for keyword (BM25) search. This includes, for example, the ability to set the tokenization method, stopwords, BM25 parameters (k1, b), [search operators](./keyword-search.md#keyword-search-operators) (`and`, `or`, or `and_cross`), specific properties to search and/or to boost particular properties. For more information on these parameters, see the [keyword search page](./keyword-search.md). ## Further resources - [How-to: Search](../../search/index.mdx) - [How-to: Hybrid search](../../search/hybrid.md) - [Blog: A deep dive into Weaviate's fusion algorithms](https://weaviate.io/blog/hybrid-search-fusion-algorithms) ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Search/Index (docs/weaviate/concepts/search/index.md) --- title: Search sidebar_position: 5 description: "Overview of search capabilities designed for billion-scale datasets and real-time queries." image: og/docs/concepts.jpg # tags: ['concepts', 'search'] --- Weaviate performs flexible, fast and scalable searches to help users to find the right data quickly even with billion-scale datasets. With Weaviate, you can perform variety of search types to suit your needs, and configure search settings to optimize performance and accuracy. The following sections provide a conceptual overview of search in Weaviate, including [an overview of the search process and types](#search-process). ## Search process The following table and figure illustrate the search process in Weaviate. Around the core search process, there are several steps that can be taken to improve and manipulate the search results. | Step | Description | Optional | |------|-------------|----------| | 1. [Retrieval](#retrieval-filter) | [Filter](#retrieval-filter): Narrow result sets based on criteria
[Search](#retrieval-search): Find the most relevant entries, using one of [keyword](#keyword-search), [vector](#vector-search) or [hybrid](#hybrid-search) search types
| Required | | 2. [Rerank](#rerank) | Reorder results using a different (e.g. more complex) model | Optional | | 3. [Retrieval augmented generation](#retrieval-augmented-generation-rag) | Send retrieved data and a prompt to a generative AI model. Also called retrieval augmented generation, or RAG. | Optional | ```mermaid flowchart LR %% Node definitions Query[/"🔍 Query"/] Filter["Filter"] Key["Keyword Search
(BM25F)"] Vec["Vector Search
(Embeddings)"] Hyb["Hybrid Search
(Combined)"] Rerank["Rerank
(Optional)"] RAG["RAG
(Optional)"] Results[/"📊 Results"/] %% Main flow grouping subgraph retrieval ["Retrieval"] direction LR Filter search end subgraph search ["Search"] direction LR Key Vec Hyb end %% Connections Query --> retrieval Filter --> search retrieval --> Results retrieval --> Rerank Rerank --> RAG RAG --> Results %% Node styles style Query fill:#ffffff,stroke:#B9C8DF,color:#130C49 style Filter fill:#ffffff,stroke:#B9C8DF,color:#130C49 style Key fill:#ffffff,stroke:#B9C8DF,color:#130C49 style Vec fill:#ffffff,stroke:#B9C8DF,color:#130C49 style Hyb fill:#ffffff,stroke:#B9C8DF,color:#130C49 style Rerank fill:#ffffff,stroke:#B9C8DF,color:#130C49 style RAG fill:#ffffff,stroke:#B9C8DF,color:#130C49 style Results fill:#ffffff,stroke:#B9C8DF,color:#130C49 %% Subgraph styles style retrieval fill:#ffffff,stroke:#130C49,stroke-width:2px,color:#130C49 style search fill:#ffffff,stroke:#7AD6EB,stroke-width:2px,color:#130C49 ``` Here is a brief overview of each step: ### Retrieval: Filter :::info In one sentence A filter reduces the number of objects based on specific criteria. ::: Filters reduce the number of objects based on specific criteria. This can include: - Text matches - Numerical thresholds - Date ranges - Categorical values - Geographical locations Effective filtering can significantly improve search relevance. This is due to filters' ability to precisely reduce the result set based on exact criteria. :::info How do filters interact with searches? Weaivate applies [pre-filtering](../filtering.md), where filters are performed before searches.
This ensures that search results overlap with the filter criteria to make sure that the right objects are retrieved. :::

Filter: Example

In a dataset such as `animal_objs` below, you could filter by a specific color to retrieve only objects that match this criterion.
```json [ {"description": "brown dog"}, {"description": "small domestic black cat"}, {"description": "orange cheetah"}, {"description": "black bear"}, {"description": "large white seagull"}, {"description": "yellow canary"}, ] ``` A filter for `"black"` in the `"description"` would return only the objects with a black color. - `{'description': 'black bear'}` - `{'description': 'small domestic black cat'}`
In Weaviate, the order of these results are based on the UUIDs of the objects, if no other ranking is applied. As a result, the order of these objects would be essentially random, as the filter only passes or blocks objects based on the criteria.

### Retrieval: Search :::info In one sentence A search produces an ordered list of objects based on relevance to a query. ::: Search is about finding the closest, or most relevant data objects. Weaviate supports three primary search types: [keyword search](#keyword-search), [vector search](#vector-search), and [hybrid search](#hybrid-search). Here's a summary of these search types: | Search Type | Description | |-------------|-------------| | Keyword Search | Traditional text-based search using "token" frequency. | | Vector Search | Similarity-based search using vector embeddings. | | Hybrid Search | Combines vector and keyword search results. | :::tip Search vs Filter A filter simply passes or blocks objects based on criteria. Therefore, there is no ranking of results.
Unlike filters, Search results will be **ranked** based on their relevance to the query. ::: Let's review these search types in more detail. #### Keyword Search Keyword search ranks results based on keyword match "scores". These scores are based on how often tokens in the query appear in each data object. These metrics are combined using the BM25 algorithm to produce a score.

Keyword Search: Example

In a dataset such as `animal_objs` below, you could perform keyword searches by a specific color to retrieve how significant they are.
```json [ {"description": "brown dog"}, {"description": "small domestic black cat"}, {"description": "orange cheetah"}, {"description": "black bear"}, {"description": "large white seagull"}, {"description": "yellow canary"}, ] ``` A keyword search for `"black"` would return only the objects with a black color, as before. But here, the results are ranked based on the BM25 algorithm. 1. `{'description': 'black bear'}` 1. `{'description': 'small domestic black cat'}`
Here `{"description": "black bear"}` has a higher score than `{"description": "small domestic black cat"}` because the term "black" is a larger proportion of the text.

When to use keyword search

Keyword search is great where occurrences of certain words strongly indicate the text's relevance. For example: - Find medical, or legal literature containing specific terms. - Search for technical documentation or API references where exact terminology is crucial. - Locating specific product names or SKUs in an e-commerce database. - Finding code snippets or error messages in a programming context.

:::info Read more See the [keyword search](./keyword-search.md) page for more details on how keyword search works in Weaviate. ::: #### Vector Search Similarity-based search using vector embeddings. This method compares vector embeddings of the query against those of the stored objects to find the closest matches, based on a predefined [distance metric](../../config-refs/distances.md). In Weaviate, you can perform vector searches in multiple ways. You can search for similar objects based on [a text input](../../search/similarity.md#search-with-text), [a vector input](../../search/similarity.md#search-with-a-vector), or [an exist object](../../search/similarity.md#search-with-an-existing-object). You can even search for similar objects with other modalities such as [with images](../../search/image.md).

Vector Search: Example

In a dataset such as `animal_objs` below, you could perform vector searches with words that are semantically similar to retrieve how significant they are.
```json [ {"description": "brown dog"}, {"description": "small domestic black cat"}, {"description": "orange cheetah"}, {"description": "black bear"}, {"description": "large white seagull"}, {"description": "yellow canary"}, ] ``` A search for `"black"` here would work similarly to the keyword search. But, a vector search would also produce similar results for queries such as `"very dark"`, `"noir"`, or `"ebony"`.
This is because vector search is based on the extracted meaning of the text, rather than the exact words used. The vector embeddings capture the semantic meaning of the text, allowing for more flexible search queries.
As a result, the top 3 results are: 1. `{'description': 'black bear'}` 1. `{'description': 'small domestic black cat'}` 1. `{'description': 'orange cheetah'}`

When to use vector search

Vector search is best suited where a human-like concept of "similarity" can be a good measure of result quality. For example: - Semantic text search: Locating documents with similar meanings, even if they use different words. - Multi-lingual search: Finding relevant content across different languages. - Image similarity search: Finding visually similar images in a large database.

:::info Read more See the [vector search](./vector-search.md) page for more details on how vector search works in Weaviate. ::: #### Hybrid Search Combines vector and keyword search to leverage the strengths of both approaches. Both searches are carried out and the results are combined using the selected parameters, such as the hybrid fusion method and the alpha value.

Hybrid Search: Example

In a dataset such as `animal_objs` below, you could perform hybrid searches to robustly find relevant objects, taking a best-of-both-worlds approach.
```json [ {"description": "brown dog"}, {"description": "small domestic black cat"}, {"description": "orange cheetah"}, {"description": "black bear"}, {"description": "large white seagull"}, {"description": "yellow canary"}, ] ``` A hybrid search for `"black canine"` would match well the objects with `"black"` in the description due to its match with the keyword search. So it would surface `{"description": "small domestic black cat"}` and `{"description": "black bear"}` towards the top.
But it would also boost objects with `"dog"` in the description, such as `{"description": "brown dog"}`. This is because the vector search would find a high similarity between the query and the word `"dog"`, even though the word `"dog"` is not in the query.
As a result, the top 3 results are: 1. `{"description": "black bear"}` 1. `{"description": "small domestic black cat"}` 1. `{"description": "brown dog"}`

When to use hybrid search

Hybrid search is great as a starting point, as it is a robust search type. It tends to boost results that perform well in at least one of the two searches. For example: - Academic paper search: Finding research papers based on both keyword relevance and semantic similarity to the query. - Job matching: Identifying suitable candidates by combining keyword matching of skills with semantic understanding of job descriptions. - Recipe search: Locating recipes that match specific ingredients (keywords) while also considering overall dish similarity (vector). - Customer support: Finding relevant support tickets or documentation using both exact term matching and conceptual similarity.

:::info Read more See the [hybrid search](./hybrid-search.md) page for more details on how hybrid search works in Weaviate. ::: ### Retrieval: Unordered Queries can be formulated without any ranking mechanisms. For example, a query may simply consist of a filter, or you may wish to iterate through the entire dataset, using the [cursor API](../../manage-objects/read-all-objects.mdx). In such cases of unordered retrieval requests, Weaviate will retrieve objects in order of their UUIDs. This retrieval method will result in an essentially randomly-ordered object list. ### Rerank :::info In one sentence A reranker reorders initial retrieval results with a more complex model or different criteria. ::: Reranking improves search relevance by reordering initial results. If a collection is [configured with a reranker integration](../../model-providers/index.md), Weaviate will use the configured reranker model to reorder the initial search results. This allows you to use a more computationally expensive model on a smaller subset of results, improving the overall search quality. Typically, reranking models such as [Cohere Rerank](../../model-providers/cohere/reranker.md) or [Hugging Face Reranker](../../model-providers/transformers/reranker.md) models are cross-encoder models that can provide a more nuanced understanding of the text. A reranker can also be used to provide a different input query to that used for retrieval, allowing for more complex search strategies.

When to use reranking

Reranking is useful when you want to improve the quality of search results by applying a more complex model to a smaller subset of results. This may be necessary when the object set is very subtle or specific, such as in particular industries or use cases. For example, searches in legal, medical, or scientific literature may require a more nuanced understanding of the text. Reranking can help to ensure that the most relevant results are surfaced.

### Retrieval augmented generation (RAG) :::info In one sentence Retrieval Augmented Generation combines search with a generative AI model to produce new content based on the search results. ::: Retrieval augmented generation (RAG), also called generative search, combines search with a generative AI model to produce new content based on the search results. It is a powerful technique that can leverage the generative capabilities of AI models and the search capabilities of Weaviate. Weaviate integrates with many popular [generative model providers](../../model-providers/index.md) such as [AWS](../../model-providers/aws/generative.md), [Cohere](../../model-providers/cohere/generative.md), [Google](../../model-providers/google/generative.md), [OpenAI](../../model-providers/openai/generative.md) and [Ollama](../../model-providers/ollama/generative.md). As a result, Weaviate makes RAG easy to [set up](../../manage-collections/generative-reranker-models.mdx#specify-a-generative-model-integration), and easy to [execute as an integrated, single query](../../search/generative.md#grouped-task-search).

RAG: Example

In a dataset such as `animal_objs` below, you could combine retrieval augmented generation with any other search method to find relevant objects and then transform it.
```json [ {"description": "brown dog"}, {"description": "small domestic black cat"}, {"description": "orange cheetah"}, {"description": "black bear"}, {"description": "large white seagull"}, {"description": "yellow canary"}, ] ``` Take an example of a keyword search for `"black"`, and a RAG request `"What do these animal descriptions have in common?"`.
The search results consist of `{"description": "black bear"}` and `{"description": "small domestic black cat"}` as you saw before. Then, the generative model would produce an output based on our query. In one example, it produced:
```text "What these descriptions have in common are: * **Color:** Both describe animals with a **black** color. * **Species:** One is an **animal**, the other describes a **breed** of animal (domesticated)." ```

## Search scores and metrics Weaviate uses a variety of metrics to rank search results of a given query. The following metrics are used in Weaviate: - Vector distance: A vector distance measure between the query and the object. - BM25F score: A keyword search score calculated using the BM25F algorithm. - Hybrid score: A combined score from vector and keyword searches. ## Named vectors ### Query a specific named vector To do a vector search on a collection with named vectors, specify the vector space to search. Use named vectors with [vector similarity searches](/weaviate/search/similarity#named-vectors) (`near_text`, `near_object`, `near_vector`, `near_image`) and [hybrid search](/weaviate/search/hybrid#named-vectors). Named vector collections support hybrid search, but only for one vector at a time. [Keyword search](/weaviate/search/bm25) syntax does not change if a collection has named vectors. ### Query multiple named vectors Where multiple named vectors are defined in a collection, you can query them in a single search. This is useful for comparing the similarity of an object to multiple named vectors. This is called a "multi-target vector search". In a multi-target vector search, you can specify: - The target vectors to search - The query(ies) to compare to the target vectors - The weights to apply to each distance (raw, or normalized) for each target vector Read more in [How-to: Multi-target vector search](../../search/multi-vector.md). ## Further resources For more details, see the respective pages for: - [Concepts: Vector search](./vector-search.md) - [Concepts: Keyword search](./keyword-search.md) - [Concepts: Hybrid search](./hybrid-search.md). For code snippets on how to use these search types, see the [How-to: search](../../search/index.mdx) page. ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Search/Keyword Search (docs/weaviate/concepts/search/keyword-search.md) --- title: Keyword Search (BM25) sidebar_position: 40 description: "Exact token-based matching using BM25 algorithm for precise keyword and phrase searching." image: og/docs/concepts.jpg # tags: ['concepts', 'search', 'keyword search', 'bm25', 'keyword'] --- import ThemedImage from '@theme/ThemedImage'; Keyword search is an exact matching-based search using "tokens", or strings of characters. It uses the BM25 algorithm, which ranks matching documents according to their relevance to a given search query. At a high level, the BM25 algorithm uses the count of query terms in the document (term frequency) against the overall frequency of the term in the dataset (inverse document frequency) to calculate a relevance score. More specifically, Weaviate uses the BM25F algorithm, which extends BM25 to support using multiple fields in the search index. A keyword search determines the best matches based on the matches of exact tokens contained in the query against those of the stored objects. As a result, a keyword search is a good choice when exact matches (e.g. exact domain-specific language, precise categories or tags) are important. For example: - Searching for documents containing specific technical terms - Identifying articles by precise keywords or tags This differs from vector search, which finds semantically similar content even when the exact words don't match. You might use keyword search when precision is more important than finding related concepts. ## Keyword search in Weaviate In Weaviate, a keyword search will return the objects best matching the query, as measured by [BM25F](https://en.wikipedia.org/wiki/Okapi_BM25) "score". :::info BM25F vs BM25 The "F" in BM25F stands for "field", indicating that it is a field-specific version of BM25. This allows for different weights for different fields, or properties, of the objects.
In Weaviate, they are used interchangeably, as the BM25F algorithm is used to calculate the scores for keyword searches. Here we will refer to it generally as BM25. ::: A BM25 score is calculated based on the frequency of the query tokens in the object properties, as well as the length of the object properties and the query. When an input string such as `"A red Nike shoe"` is provided as the query, Weaviate will: 1. [Tokenize](#tokenization) the input (e.g. to `["a", "red", "nike", "shoe"]`) 2. Remove any [stopwords](#stopwords) (e.g. remove `a`, to produce `["red", "nike", "shoe"]`) 3. Determine the BM25 scores against [selected properties](#selected-properties) of the database objects, based on the [BM25 parameters](#bm25-parameters) and any [property boosting](#property-boosting). 4. Return the objects with the highest BM25 scores as the search results ### Tokenization Tokenization for keyword searches refers to how each source text is split up into individual "tokens" to be compared and matched. The default tokenization method is `word`. Other tokenization methods such as `whitespace`, `lowercase`, and `field` are available, as well as specialized ones such as `gse` or `kagome_kr` for other languages ([more details](../../config-refs/collections.mdx#tokenization)). Set the tokenization option [in the inverted index configuration](../../search/bm25.md#set-tokenization) for a collection. :::info Tokenization in different contexts The term "tokenization" is used in other contexts such as vectorization, or language generation. Note that each of these typically use different tokenizers to meet different requirements. This results in different sets of tokens, even from the same input text. ::: Text properties can also enable **accent folding** via `textAnalyzer.asciiFold`, which normalizes accented characters before tokens enter the inverted index. A document containing "Café Crème" becomes searchable as "cafe creme" (and vice versa), and the same rule applies to `Equal` and `Like` filters. See [Inverted index: Accent folding](../indexing/inverted-index.md#accent-folding) for details. ### Stopwords Stopwords are words that are filtered out before processing text. Weaviate uses configurable stopwords in calculating the BM25 score. Any tokens that are contained in the stopword list will be ignored from the BM25 score calculation. See the [reference page](../../config-refs/indexing/inverted-index.mdx#stopwords) for more details. Stopword lists are also configurable per collection **and** per property. You can define custom presets on `invertedIndexConfig.stopwordPresets` and assign them to individual text properties via `textAnalyzer.stopwordPreset`. This is useful for multilingual collections. For example, English and French properties can use different stopword lists. Stopwords are still indexed and only filtered at query time, so changing your stopword configuration does not require reindexing. See [Inverted index: Custom stopword presets](../indexing/inverted-index.md#custom-stopword-presets) for details. ### BM25 parameters BM25 is a scoring function used to rank documents based on the query terms appearing in them. It has two main parameters that control its behavior: - `k1` (default: 1.2): Controls term frequency saturation. Higher values mean that multiple occurrences of a term continue to increase the score more - `b` (default: 0.75): Controls document length normalization. Values closer to 1 mean more normalization for document length ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Set custom `k1` and `b` values [for a collection](../../manage-collections/inverted-index.mdx#set-inverted-index-parameters). ### Keyword search operators import SearchOperators from '/_includes/feature-notes/search-operators.mdx'; Search operators define how many of the query [tokens](../../search/bm25.md#set-tokenization) must match, and whether they must all match within a single searched property. Conceptually, it works as though a filter is applied to the results of the BM25 score calculation. The available operators are: - `and`: All tokens must be present within a single searched property - `or`: At least one token must be present within a single searched property, with the minimum number of tokens being configurable (`minimumOrTokensMatch`) - `and_cross`: Every token must be matched by at least one of the searched properties, so the tokens can be spread across different properties. All searched properties must share the same tokenization and analyzer settings, otherwise the query fails with an error. (available from `v1.38.8`) As an example, a BM25 query of `computer networking guide` with the `and` operator would only return objects where all of the tokens `computer`, `networking`, and `guide` appear together within a single searched property. If the tokens are spread across different properties (for example, `computer` in `title` and `networking guide` in `description`), the object does not match under `and`. That restriction is specific to `and`; the same object does match under `and_cross`, which requires each token to appear in at least one of the searched properties rather than all of them in the same one. In contrast, the same query with the `or` operator would return objects where at least one of those tokens appears in a searched property. If the `or` operator is used with a `minimumOrTokensMatch` of `2`, then at least two of the tokens must be present within a single searched property. If not specified, the default operator is `or`, with a `minimumOrTokensMatch` of `1`. This means that at least one token must be present in a searched property for the object to be returned. import BM25OperatorsLight from '../img/bm25_operators_light.png'; import BM25OperatorsDark from '../img/bm25_operators_dark.png'; See the [how-to page](../../search/bm25.md#search-operators) for details on usage. ### Selected properties A BM25 query can optionally specify which object properties are to be included in the score calculations. By default, all `text` properties are included in a BM25 calculation. There are two ways to vary this: - In the collection configuration, [set `indexSearchable` for a property to `false`](../../manage-collections/vector-config.mdx#property-level-settings). This property will then be ignored in all BM25 searches. - [Specify which properties to search at query time](../../search/bm25.md#search-on-selected-properties-only). This will only apply for that particular query. ### Property Boosting Property boosting allows a query apply different weights to different properties when calculating the final BM25 score. This is useful when certain properties are more important for search than others. For example, when searching an e-commerce catalog, you could boost the title property and its categories over the product description. [Set the property weights](../../search/bm25.md#use-weights-to-boost-properties) at query time. ## Combining with Vector Search Keyword search can be combined with vector search in Weaviate to perform a hybrid search. This allows you to leverage both: - Exact matching capabilities of keyword search - Semantic understanding of vector search See [Hybrid Search](./hybrid-search.md) for more information. ## Notes and Best Practices Here are some key considerations when using keyword search: 1. **Tokenization Choice** - Choose based on your data and search requirements. For example, use `word` tokenization for natural language text, but consider `field` for URLs or email addresses that need exact matching as a whole. - For multilingual content, consider specialized tokenizers like `gse` for Chinese/Japanese or `kagome_kr` for Korean - Consider special characters and case sensitivity needs - Test your tokenization choice with subsets of your data and queries to ensure it handles special characters and case sensitivity as expected. You could perform these experiments with vectorization disabled to save resources/costs, as the two processes are independent. 2. **Performance Optimization** - Index only the properties you need for search - Consider combining keyword search with vector search (i.e. perform a [hybrid search](./hybrid-search.md)) as a starting point, especially where you cannot anticipate users' behavior 3. **Query Optimization** - Consider boosting properties that are more important for search (e.g. title, category) over others (e.g. description) - Only modify `k1` and `b` values if you have a good reason to do so, as the defaults are generally well-suited for most use cases 4. **Debugging Tokenization** - Use the [`/v1/tokenize` endpoint](../../config-refs/indexing/inverted-index.mdx#tokenize-endpoint) to inspect how text is tokenized before committing to a schema configuration. This is useful when experimenting with accent folding or custom stopword presets. ### Further resources - [How-to: Search](../../search/index.mdx) - [How-to: Keyword search](../../search/bm25.md) ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Concepts/Search/Vector Search (docs/weaviate/concepts/search/vector-search.md) --- title: Vector Search sidebar_position: 20 description: "Similarity-based semantic search using vector embeddings for text, images, audio, and multimodal data." image: og/docs/concepts.jpg # tags: ['concepts', 'search', 'vector search', 'vector'] --- Vector search is a similarity-based search using vector embeddings, or embeddings. Vector search is also referred to as "semantic search" due to its ability to find semantically similar objects. It should be noted, however, that vector search is not limited to text data. Vector search can be used with other types of data, such as images, videos, and audio. A vector embedding captures semantic meaning of an object in a vector space. It consists of a set of numbers that represent the object's features. Vector embeddings are generated by a vectorizer model, which is a machine learning model that is trained for this purpose. A vector search compares [vectors of the stored objects](#object-vectors) against the [query vector(s)](#query-vectors) to find the closest matches, before returning the top `n` results. :::tip An introduction to vector search New to vector search? Check out our blog, ["Vector Search Explained"](https://weaviate.io/blog/vector-search-explained) for an introduction to vector search concepts and use cases. ::: ## Object vectors For vector search, each object must have representative vector embeddings. The model used to generate vectors is called a vectorizer model, or an embedding model. A user can populate Weaviate with objects and their vectors in one of two ways: - Use Weaviate's [vectorizer model provider integrations](#model-provider-integration) to generate vectors - [Provide vectors directly](#bring-your-own-vector) to Weaviate ### Model provider integration Weaviate provides [first-party integrations with popular vectorizer model providers](../../model-providers/index.md) such as [Cohere](../../model-providers/cohere/index.md), [Ollama](../../model-providers/ollama/index.md), [OpenAI](../../model-providers/openai/index.md), and more. In this workflow, the user can [configure a vectorizer for a collection](../../manage-collections/vector-config.mdx#specify-a-vectorizer) and Weaviate will automatically generate vectors as needed, such as when inserting objects or performing searches. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` This integration abstracts the vector generation process from the user, allowing the user to focus on building applications and performing searches without worrying about the vector generation process. :::info Vectorizer configuration is immutable Once it is set, the vectorizer cannot be changed for a collection. This ensures that the vectors are generated consistently and stay compatible. If you need to change the vectorizer, you must create a new collection with the desired vectorizer, and [migrate the data to the new collection](../../manage-collections/migrate.mdx). ::: #### Manual vectors when vectorizer is configured Even when a vectorizer model is configured for a collection, a user can still provide vectors directly when inserting objects or performing a query. In this case, Weaviate will use the provided vector instead of generating a new one. This is useful when the user already has vectors generated by the same model, such as when importing objects from another system. Re-using the same vectors will save time and resources, as Weaviate will not need to generate new vectors. ### Bring your own vector A user can directly upload vectors to Weaviate when inserting objects. This is useful when the user already has vectors generated by a model, or if the user wants to use a specific vectorizer model that does not have an integration with Weaviate. ```mermaid %%{init: { 'theme': 'base', 'themeVariables': { 'primaryColor': '#4a5568', 'primaryTextColor': '#2d3748', 'primaryBorderColor': '#718096', 'lineColor': '#718096', 'secondaryColor': '#f7fafc', 'tertiaryColor': '#edf2f7', 'fontFamily': 'Inter, system-ui, sans-serif', 'fontSize': '14px', 'lineHeight': '1.4', 'nodeBorder': '1px', 'mainBkg': '#ffffff', 'clusterBkg': '#f8fafc' } }}%% flowchart LR %% Style definitions classDef systemBox fill:#f8fafc,stroke:#3182ce,stroke-width:1.5px,color:#2d3748,font-weight:bold classDef weaviateBox fill:#f8fafc,stroke:gray,stroke-width:0.5px,color:#2d3748,font-weight:bold classDef component fill:white,stroke:#a0aec0,stroke-width:1px,color:#2d3748 classDef edgeLabel fill:white,stroke:#e2e8f0,stroke-width:1px,color:#4a5568 %% Weaviate section subgraph weaviate["Weaviate"] core["💾 Data &
vector store"] end %% User System subgraph user["🖥️ User System"] data["📄 Data"] end %% Connections with curved edges data --->|"1\. Insert objects
(with vectors)"| core %% Apply styles class user systemBox class weaviate weaviateBox class cloud cloudBox class provider providerBox class data,core,vectorizer,inference component %% Linkstyle for curved edges linkStyle default stroke:#718096,stroke-width:3px,fill:none,background-color:white ``` In this workflow, the user has the flexibility to use any vectorizer model and process independently of Weaviate. If using your own model, we recommend explicitly setting the vectorizer as `none` in the vectorizer configuration, such that you do not accidentally generate incompatible vectors with Weaviate. ### Named vectors A collections can be configured to allow each object to be represented by more than one vector embedding. Each such vector works as its distinct vector space that is independent of each other, referred to as a "named vector". A named vector can be configured with a [vectorizer model integration](#model-provider-integration), and may be provided using the ["bring your own vector"](#bring-your-own-vector) integration. ## Query vectors In Weaviate, you can specify the query vector using: - A query vector (called `nearVector`), - A query object (called `nearObject`), - A query text (called `nearText`), or - A query media (called `nearImage` or `nearVideo`). In each of these cases, the search will return the most similar objects to the query, based on the vector embeddings of the query and the stored objects. However, they differ in how the query vector is specified to Weaviate. ### `nearVector` In a `nearVector` query, the user provides a vector directly to Weaviate. This vector is compared to the vectors of the stored objects to find the most similar objects. ### `nearObject` In a `nearObject` query, the user provides an object ID to Weaviate. Weaviate retrieves the vector of the object and compares it to the vectors of the stored objects to find the most similar objects. ### `nearText` (and `nearImage`, `nearVideo`) In a `nearText` query, the user provides an input text to Weaviate. Weaviate uses the specified vectorizer model to generate a vector for the text, and compares it to the vectors of the stored objects to find the most similar objects. As a result, a `nearText` query is only available for collections where a vectorizer model is configured. A `nearImage` or `nearVideo` query works similarly to a `nearText` query, but with an image or video input instead of text. ## Multi-target vector search In a multi-target vector search, Weaviate performs multiple, concurrent, single-target vector searches. These searches will produce multiple sets of results, each with a vector distance score. Weaviat combines these result sets, using a ["join strategy"](#available-join-strategies) to produce final scores for each result. If an object is within the search limit or the distance threshold of any of the target vectors, it will be included in the search results. If an object does not contain vectors for any selected target vector, Weaviate ignores that object and does not include it in the search results. ### Available join strategies. - **minimum** (*default*) Use the minimum of all vector distances. - **sum** Use the sum of the vector distances. - **average** Use the average of the vector distances. - **manual weights** Use the sum of weighted distances, where the weight is provided for each target vector. - **relative score** Use the sum of weighted normalized distances, where the weight is provided for each target vector. ## Vector index and search Weaviate uses vector indexes to facilitate efficient vector searches. Like other types of indexes, a vector index organizes vector embeddings in a way that allows for fast retrieval while optimizing for other needs such as search quality (e.g. recall), search throughput, and resource use (e.g. memory). In Weaviate, multiple types of vector indexes are available such as `hnsw`, `flat` and `dynamic` indexes. Each [collection](../data.md#collections) or [tenant](../data.md#multi-tenancy) in Weaviate will have its own vector index. Additionally, each collection or tenant can have [multiple vector indexes](../data.md#multiple-vector-embeddings-named-vectors), each with different configurations. :::info Read more about: - [Collections](../data.md#collections) - [Multi-tenancy](../data.md#multi-tenancy) - [Vector indexes](../indexing/vector-index.md) - [Multiple named vectors](../data.md#multiple-vector-embeddings-named-vectors) ::: ### Distance metrics There are many ways to measure vector distances, such as cosine distance, dot product, and Euclidean distance. Weaviate supports a variety of these distance metrics, as listed on the [distance metrics](../../config-refs/distances.md) page. Each vectorizer model is trained with a specific distance metric, so it is important to use the same distance metric for search as was used for training the model. Weaviate uses cosine distance as the default distance metric for vector searches, as this is the typical distance metric for vectorizer models. :::tip Distance vs Similarity In a "distance", the lower the value, the closer the vectors are to each other. In a "similarity", or "certainty" score, the higher the value, the closer the vectors are to each other. Some metrics, such as cosine distance, can also be expressed as a similarity score. Others, such as Euclidean distance, are only expressable as a distance. ::: ## Diversity selection (MMR) import V137Preview from '/_includes/feature-notes/v137-preview.mdx'; Standard vector search returns the closest matches to a query, which often means a cluster of near-duplicate results. For example, searching for "Italian food" in a product catalog might return five images of pizza instead of a diverse spread of Italian dishes. **Maximum Marginal Relevance (MMR)** solves this by reranking results to balance two objectives: - **Relevance**: how well does the item match the query? - **Diversity**: how different is the item from the results already selected? The algorithm works iteratively. It selects the most relevant item first, then for each subsequent pick it scores candidates by weighing their query similarity against their maximum similarity to any already-selected result. The `balance` parameter (λ) controls the trade-off: - **λ = 0.0**: Pure diversity (maximizes difference from already-selected items) - **λ = 0.5**: Balanced (each result must be both relevant and distinct) - **λ = 1.0**: Pure relevance (equivalent to standard vector search) MMR is applied at query time as a reranking step on top of standard search. No reindexing is needed. The typical pattern is to retrieve a larger candidate set via regular vector search, then rerank a subset using MMR. :::note Result ordering Results are ordered by MMR score, not query similarity. The first result is always the most relevant, but subsequent results may have lower query similarity because they were chosen for the diversity they add. ::: See the [how-to guide](../../search/similarity.md#diversity-selection-mmr) for configuration details and code examples. ## Notes and best practices All compatible vectors are similar to some degree search. This has two effects: 1. There will always be some "top" search results regardless of relevance. 1. The entire dataset is always returned. If you search a vector database containing vectors for colors "Red", "Crimson" and "LightCoral" with a query vector for "SkyBlue", the search will still return a result (e.g. "Red"), even if it is not semantically similar to the query. The search is simply returning the closest match, even if it is not a good match in the absolute sense. As a result, Weaviate provides multiple ways to limit the search results: - **Limit**: Specify the maximum number of results to return. - If not provided, defaults to system-defined [`QUERY_DEFAULTS_LIMIT`](/deploy/configuration/env-vars/index.md#general) of 10. - **AutoCut**: Limit results based on discontinuities in result metrics such as vector distance or search score. - **Threshold**: Specify a minimum similarity score (e.g. maximum cosine distance) for the results. - **Apply filters**: Use [filters](../filtering.md) to exclude results based on other criteria, such as metadata or properties. Use a combination of these methods to ensure that the search results are meaningful and relevant to the user. Generally, start with a `limit` to a maximum number of results to provide to the user, and adjust the `threshold` such that irrelevant results are unlikely to be returned. This will cause the search to return up to the specified (`limit`) number of results, but only if they are above the specified (`threshold`) similarity score. ### Further resources - [How-to: Search](../../search/index.mdx) - [How-to: Vector similarity search](../../search/similarity.md) ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Config Refs/Collections (docs/weaviate/config-refs/collections.mdx) --- title: Collection definition description: Reference for Weaviate collection parameters. --- import SkipLink from "/src/components/SkipValidationLink"; import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem"; import FilteredTextBlock from "@site/src/components/Documentation/FilteredTextBlock"; import PyCode from "!!raw-loader!/_includes/code/config-refs/reference.collections.py"; import TSCode from "!!raw-loader!/_includes/code/howto/manage-data.collections.ts"; import GoCode from "!!raw-loader!/_includes/code/howto/go/docs/manage-data.classes_test.go"; import PyCodeMultiTenancy from "!!raw-loader!/_includes/code/howto/manage-data.multi-tenancy.py"; import TSCodeMultiTenancy from "!!raw-loader!/_includes/code/howto/manage-data.multi-tenancy.ts"; import GoCodeMultiTenancy from "!!raw-loader!/_includes/code/howto/go/docs/manage-data.multi-tenancy_test.go"; A **collection definition** specifies how to store and index a set of data objects in Weaviate. This page discusses the available parameters for configuring a collection. ## Collection definition parameters These are the top-level parameters you can set when creating a collection. | Parameter | Type | Description | Default | Mutable | | :------------------------------------------- | :----- | :-------------------------------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------ | | [`class`](#class) | String | The name of the collection. | (Required) | No | | [`description`](#description) | String | A description of the collection. | `""` | Yes | | [`properties`](#properties) | Array | An array of property objects defining the data schema. | `[]` | Partially\* | | [`invertedIndexConfig`](#inverted-index) | Object | Configuration for the inverted index, affecting filtering and keyword search. | See [Inverted Index reference](./indexing/inverted-index.mdx#inverted-index-parameters) | Yes | | [`vectorConfig`](#vector-configuration) | Object | Configure multiple named vectors each with their own `vectorizer`, `vectorIndexType`, and `vectorIndexConfig` fields. | `null` | Partially\*\* | | [`vectorizer`](#vector-configuration) | String | The vectorizer module to use. | Default vectorizer defined by [environment variable](/deploy/configuration/env-vars#DEFAULT_VECTORIZER_MODULE). See [Model provider](../model-providers/index.md) for module-specific config defaults | No | | [`vectorIndexType`](#vector-configuration) | String | The type of vector index to use (`hnsw`, `flat`, `dynamic`, `hfresh`). | `hnsw` | No | | [`moduleConfig`](#module-configuration) | Object | Module-specific configuration settings. | See [Module configuration](#module-configuration) | Partially | | [`vectorIndexConfig`](#vector-configuration) | Object | Configuration settings specific to the chosen `vectorIndexType`. | See [Vector index reference](./indexing/vector-index.mdx) | Partially | | [`shardingConfig`](#sharding) | Object | Controls sharding behavior in a multi-node cluster. | See [Sharding section](#sharding) | No | | [`replicationConfig`](#replication) | Object | Controls data replication settings for fault tolerance. | See [Replication section](#replication) | Partially | | [`multiTenancyConfig`](#multi-tenancy) | Object | Configuration to enable multi-tenancy for the collection. | See [Multi-tenancy section](#multi-tenancy) | Partially | \* [New properties can be added](../manage-collections/collection-operations.mdx#add-a-property); existing properties cannot be modified
\*\* [New named vectors can be added](../manage-collections/vector-config.mdx#add-new-named-vectors); some vector index settings are mutable

Example collection configuration - JSON object

An example of a complete collection object including properties: ``` /* Detailed source-code truncated for AI context efficiency. */ ```

#### Code example - How to create a collection This code example shows how to configure the collection parameters through a client library: :::tip Further resources For more code examples and configuration guides visit the [How-to: Manage collections](../manage-collections/index.mdx) section. ::: #### `class` The `class` is the name of the collection. The collection name starts with an upper case letter. The upper case letter distinguishes collection names from primitive data types when the name is used as a property value. Consider these examples that use the `dataType` property: - `dataType: ["text"]` is a `text` data type. - `dataType: ["Text"]` is a cross-reference type to a collection named `Text`. After the first letter, collection names may use any GraphQL-compatible characters. The collection name validation regex is `/^[A-Z][_0-9A-Za-z]*$/`. import InitialCaps from "/_includes/schemas/initial-capitalization.md"; #### `description` A description of the collection. This is for your reference and can also provide additional information to the [Query Agent](/query-agent/index.md). --- ### Properties | Parameter | Type | Description | Default | Mutable | | :--------------------------------------- | :------ | :--------------------------------------------------------------------------------------------------------------------------- | :--------- | :------ | | [`name`](#name) | String | The name of the property. | (Required) | No | | [`dataType`](./datatypes.md) | Array | An array containing one or more data types. For cross-references, use the capitalized collection name (e.g., `["Article"]`). | (Required) | No | | `description` | String | A description of the property for your reference. | `null` | Yes | | [`tokenization`](#tokenization) | String | For `text` properties, specifies how the text is split into tokens for inverted indexing. | `word` | No | | [`indexInverted`](#inverted-index) | Boolean | If `true`, inverted index is enabled for this property. | `true` | No | | [`indexFilterable`](#inverted-index) | Boolean | If `true`, builds a roaring bitmap index for this property to allow for efficient filtering. | `true` | No | | [`indexSearchable`](#inverted-index) | Boolean | If `true`, builds a searchable map index for this property, suitable for BM25 or hybrid search. | `true` | No | | [`indexRangeFilters`](#inverted-index) | Boolean | If `true`, builds a roaring bitmap index for numerical range-based filtering. | `false` | No | | [`invertedIndexConfig`](#inverted-index) | Object | Property-level overrides for inverted index settings, such as `bm25` parameters. | `{}` | No | | `moduleConfig` | Object | Module-specific settings, such as skipping vectorization for this property. | `{}` | No |

Example property configuration - JSON object

An example of a complete property object: ```json { "name": "title", // The name of the property "description": "title of the article", // A description for your reference "dataType": [ // The data type of the object as described above. When creating cross-references, a property can have multiple dataTypes. "text" ], "tokenization": "word", // Split field contents into word-tokens when indexing into the inverted index. See "Property Tokenization" below for more detail. "moduleConfig": { // Module-specific settings "text2vec-contextionary": { "skip": true, // If true, the whole property is NOT included in vectorization. Default is false, meaning that the object will be NOT be skipped. "vectorizePropertyName": true // Whether the name of the property is used in the calculation for the vector position of data objects. Default false. } }, "indexFilterable": true, // Optional, default is true. By default each property is indexed with a roaring bitmap index where available for efficient filtering. "indexSearchable": true // Optional, default is true. By default each property is indexed with a searchable index for BM25-suitable Map index for BM25 or hybrid searching. } ```

#### Code example - How to configure collection properties This code example shows how to configure the property parameters through a client library: :::tip Further resources For more code example and configuration guides visit the [How-to: Manage collections](../manage-collections/index.mdx) section. ::: #### `name` Property names can contain the following characters: `/[_A-Za-z][_0-9A-Za-z]*/`. ##### Reserved words The following words are reserved and cannot be used as property names: - `_additional` - `id` - `_id` Additionally, we strongly recommend that you do not use the following words as property names, due to potential conflicts with future reserved words: - `vector` - `_vector` ##### Reserved suffixes A property name may also not *end* in one of the following suffixes, because each would collide with an internal index that Weaviate derives from another property: - `_searchable` - `_rangeable` - `_temp` - `__meta_count` - `_propertyLength` - `_nullState` A property whose name ends in one of these suffixes, such as `comments_temp`, is rejected with a validation error: `'comments_temp' is not a valid property name: suffix '_temp' is reserved for internal indices`. This check runs when you create a collection or add a property to an existing collection. It is not applied when an existing collection definition is loaded, so a collection created before the check was introduced continues to work, and a backup that contains such a property still restores. The check was added in `v1.38.0`, and backported to `v1.35.20`, `v1.36.15`, and `v1.37.5`. #### `tokenization` You can customize how `text` data is tokenized and indexed in the inverted index. Tokenization influences the results returned by the [`bm25`](../api/graphql/search-operators.md#bm25) and [`hybrid`](../api/graphql/search-operators.md#hybrid) operators, and [`where` filters](../api/graphql/filters.md). Tokenization is a property-level configuration for `text` properties. [See how to set the tokenization option using a client library](../manage-collections/vector-config.mdx#property-level-settings)

Example property configuration - JSON object

```json { "classes": [ { "class": "Question", "properties": [ { "dataType": ["text"], "name": "question", // highlight-start "tokenization": "word" // highlight-end }, ], ... "vectorizer": "text2vec-openai" } ] } ```

##### Standard tokenization methods ###### `word` tokenization **Description**: Splits text by any non-alphanumeric characters, then lowercases each token. This is the default setting. **Behavior examples**: | Text | Tokens | | ---- | ------ | | `"Why, hello there!"` | `["why", "hello", "there"]` | | `"Lois & Clark: The New Adventures of Superman"` | `["lois", "clark", "the", "new", "adventures", "of", "superman"]` | | `"variable_name"` | `["variable", "name"]` | | `"Email: john.doe@example.com"` | `["email", "john", "doe", "example", "com"]` | **When to use**: - Recommended for most general text data (articles, descriptions). - When case-insensitivity and ignoring punctuation is desired for more forgiving searches. --- ###### `lowercase` tokenization **Description**: Splits text by whitespace only, then lowercases each token. It preserves symbols that `word` tokenization would remove. **Behavior examples**: | Text | Tokens | | ---- | ------ | | `"Why, hello there!"` | `["why,", "hello", "there!"]` | | `"Lois & Clark: The New Adventures of Superman"` | `["lois", "&", "clark:", "the", "new", "adventures", "of", "superman"]` | | `"variable_name"` | `["variable_name"]` | | `"Email: john.doe@example.com"` | `["email:", "john.doe@example.com"]` | **When to use**: - For technical data where symbols like `&`, `@`, or `_` are meaningful (e.g., code snippets, email addresses). - When you need case-insensitive matching but must preserve symbols. --- ###### `whitespace` tokenization **Description**: Splits text by whitespace only, preserving both case and symbols. **Behavior examples**: | Text | Tokens | | ---- | ------ | | `"Why, hello there!"` | `["Why,", "hello", "there!"]` | | `"Lois & Clark: The New Adventures of Superman"` | `["Lois", "&", "Clark:", "The", "New", "Adventures", "of", "Superman"]` | | `"variable_name"` | `["variable_name"]` | | `"Email: john.doe@example.com"` | `["Email:", "john.doe@example.com"]` | **When to use**: - When case-sensitivity is important (e.g., for proper nouns, acronyms, or specific codes). - Requires careful query construction to match case. --- ###### `field` tokenization **Description**: Treats the entire value of the property as a single token. No splitting occurs. **Behavior examples**: | Text | Tokens | | ---- | ------ | | `"Why, hello there!"` | `["Why, hello there!"]` | | `"variable_name"` | `["variable_name"]` | | `"Email: john.doe@example.com"` | `["Email: john.doe@example.com"]` | **When to use**: - When you need to match the entire field value exactly. - For properties containing unique identifiers like URLs, UUIDs, or email addresses. - Limited use for keyword searches but powerful for exact filtering. --- ##### Language-specific tokenization The standard tokenization methods work well for English and other languages that use spaces to separate words. For languages like Chinese, Japanese, and Korean that don't rely on spaces, Weaviate provides specialized tokenization methods.

`gse` and `trigram` tokenization methods

For Japanese and Chinese text, we recommend use of `gse` or `trigram` tokenization methods. These methods work better with these languages than the other methods as these languages are not easily able to be tokenized using whitespaces. The `gse` tokenizer is not loaded by default to save resources. To use it, set the environment variable `ENABLE_TOKENIZER_GSE` to `true` on the Weaviate instance. `gse` tokenization examples: - `"素早い茶色の狐が怠けた犬を飛び越えた"`: `["素早", "素早い", "早い", "茶色", "の", "狐", "が", "怠け", "けた", "犬", "を", "飛び", "飛び越え", "越え", "た", "素早い茶色の狐が怠けた犬を飛び越えた"]` - `"すばやいちゃいろのきつねがなまけたいぬをとびこえた"`: `["すばや", "すばやい", "やい", "いち", "ちゃ", "ちゃい", "ちゃいろ", "いろ", "のき", "きつ", "きつね", "つね", "ねが", "がな", "なま", "なまけ", "まけ", "けた", "けたい", "たい", "いぬ", "を", "とび", "とびこえ", "こえ", "た", "すばやいちゃいろのきつねがなまけたいぬをとびこえた"]` :::note `trigram` for fuzzy matching While originally designed for Asian languages, `trigram` tokenization is also highly effective for fuzzy matching and typo tolerance in other languages. :::

`kagome_ja` tokenization method

For Japanese text, `kagome_ja` tokenization method is also available. This uses the [`Kagome` tokenizer](https://github.com/ikawaha/kagome?tab=readme-ov-file) with a Japanese [MeCab IPA](https://github.com/ikawaha/kagome-dict/) dictionary to split the property text. The `kagome_ja` tokenizer is not loaded by default to save resources. To use it, set the environment variable `ENABLE_TOKENIZER_KAGOME_JA` to `true` on the Weaviate instance. `kagome_ja` tokenization examples: - `"春の夜の夢はうつつよりもかなしき夏の夜の夢はうつつに似たり秋の夜の夢はうつつを超え冬の夜の夢は心に響く山のあなたに小さな村が見える川の音が静かに耳に届く風が木々を通り抜ける音星空の下、すべてが平和である"`: - [`"春", "の", "夜", "の", "夢", "は", "うつつ", "より", "も", "かなしき", "\n\t", "夏", "の", "夜", "の", "夢", "は", "うつつ", "に", "似", "たり", "\n\t", "秋", "の", "夜", "の", "夢", "は", "うつつ", "を", "超え", "\n\t", "冬", "の", "夜", "の", "夢", "は", "心", "に", "響く", "\n\n\t", "山", "の", "あなた", "に", "小さな", "村", "が", "見える", "\n\t", "川", "の", "音", "が", "静か", "に", "耳", "に", "届く", "\n\t", "風", "が", "木々", "を", "通り抜ける", "音", "\n\t", "星空", "の", "下", "、", "すべて", "が", "平和", "で", "ある"`] - `"素早い茶色の狐が怠けた犬を飛び越えた"`: - `["素早い", "茶色", "の", "狐", "が", "怠け", "た", "犬", "を", "飛び越え", "た"]` - `"すばやいちゃいろのきつねがなまけたいぬをとびこえた"`: - `["すばやい", "ちゃ", "いろ", "の", "きつね", "が", "なまけ", "た", "いぬ", "を", "とびこえ", "た"]`

`kagome_kr` tokenization method

For Korean text, we recommend use of the `kagome_kr` tokenization method. This uses the [`Kagome` tokenizer](https://github.com/ikawaha/kagome?tab=readme-ov-file) with a Korean MeCab ([mecab-ko-dic](https://bitbucket.org/eunjeon/mecab-ko-dic/src/master/)) dictionary to split the property text. The `kagome_kr` tokenizer is not loaded by default to save resources. To use it, set the environment variable `ENABLE_TOKENIZER_KAGOME_KR` to `true` on the Weaviate instance. `kagome_kr` tokenization examples: - `"아버지가방에들어가신다"`: - `["아버지", "가", "방", "에", "들어가", "신다"]` - `"아버지가 방에 들어가신다"`: - `["아버지", "가", "방", "에", "들어가", "신다"]` - `"결정하겠다"`: - `["결정", "하", "겠", "다"]`

Limit the number of `gse` and `Kagome` tokenizers

The `gse` and `Kagome` tokenizers can be resource intensive and affect Weaviate's performance. You can limit the combined number of `gse` and `Kagome` tokenizers running at the same time using the [`TOKENIZER_CONCURRENCY_COUNT` environment variable](/deploy/configuration/env-vars/index.md).

Fuzzy matching with `trigram` tokenization

The `trigram` tokenization method provides fuzzy matching capabilities by breaking text into overlapping 3-character sequences. This enables BM25 searches to find matches even with spelling errors or variations. **Use cases for trigram fuzzy matching:** - **Typo tolerance**: Find matches despite spelling errors (e.g., "Reliace" matches "Reliance") - **Name reconciliation**: Match entity names with variations across datasets - **Search-as-you-type**: Build autocomplete functionality - **Partial matching**: Find objects with partial string matches **How it works:** When text is tokenized with `trigram`, it's broken into all possible 3-character sequences: - `"hello"` → `["hel", "ell", "llo"]` - `"world"` → `["wor", "orl", "rld"]` Similar strings share many trigrams, enabling fuzzy matching: - `"Morgan Stanley"` and `"Stanley Morgn"` share trigrams like `"sta", "tan", "anl", "nle", "ley"` **Performance considerations:** - Filtering behavior will change significantly, as text filtering will be done based on trigram-tokenized text, instead of whole words - Creates larger inverted indexes due to more tokens - May impact query performance for large datasets :::tip Use trigram tokenization selectively on fields where fuzzy matching is preferred. Keep exact-match fields with `word` or `field` tokenization for precision. :::

##### Decision guide Use this table to quickly identify the right tokenization method for your data. | If your data is... | Consider | Because | |-------------------|----------|---------| | General text (articles, descriptions) | `word` | Case-insensitive, ignores punctuation, forgiving | | Code, technical IDs with `_` or `-` | `lowercase` | Preserves symbols, case-insensitive | | Names, acronyms where case matters | `whitespace` | Case-sensitive, preserves symbols | | Email addresses, URLs, unique IDs | `field` | Requires exact matches | | Chinese text | `gse` or `trigram` | Proper word segmentation | | Japanese text | `kagome_ja` or `trigram` | Proper morphological analysis | | Korean text | `kagome_kr` or `trigram` | Proper morphological analysis | ##### Performance considerations **Indexing speed** - `word`, `lowercase`, `whitespace`: Fast, with similar performance. - `field`: Fastest, as no splitting is required. - `gse`, `trigram`, `kagome_*`: Slower due to more complex segmentation algorithms. **Query performance** - Simple tokenization methods (`word`, `lowercase`, `whitespace`): Fast. - `field` with wildcard filters: Can be slow and should be used judiciously. - Language-specific methods: Performance is similar to simple methods for queries. **Index size** - More tokens result in a larger index. - `field`: Creates the smallest index (one token per value). - `trigram`: Creates the largest index due to many overlapping trigrams. --- ### Inverted index {#inverted-index} Weaviate uses **inverted indexes** to enable fast and efficient filtering and searching. The inverted index maps values (like words or numbers) to the objects that contain them in order to speed-up all attribute-based filtering (`where` filters) and keyword searching (`bm25`, `hybrid`). Disabling indexing for properties you will never query can speed up data imports and reduce disk usage. More details about the `indexFilterable`, `indexSearchable`, `indexRangeFilters` and `invertedIndexConfig` parameters can be found in [Reference: Inverted index](./indexing/inverted-index.mdx). --- ### Vector configuration Weaviate supports two approaches for vector configuration: - **Single vector collections**: One vector space per object using top-level parameters (`vectorizer`, `vectorIndexType`, `vectorIndexConfig`) - **Multiple named vectors**: Multiple vector spaces per object using the `vectorConfig` parameter (**recommended**) You cannot combine both approaches in the same collection. :::tip We recommend using `vectorConfig` Using the `vectorConfig` parameter allows you to start with one vector per collection and adding [new named vectors](../manage-collections/vector-config.mdx#add-new-named-vectors) afterward. ::: #### Vector configuration parameters | Parameter | Type | Description | Default | Mutable | | :----------------------------------------------------------------------- | :----- | :------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------- | :------------ | | `vectorizer` | String | The vectorizer module to use (e.g., `text2vec-cohere`). Set to `none` to disable auto-vectorization. [Available model providers](../model-providers/index.md) | Module-specific default | No | | [`vectorIndexType`](./indexing/vector-index.mdx) | String | Vector index type: `hnsw` (default), `flat`, `dynamic`, or `hfresh` | `hnsw` | No | | [`vectorIndexConfig`](./indexing/vector-index.mdx) | Object | Configuration settings for your chosen `vectorIndexType` | Index-specific defaults | Partially\* | | `vectorConfig` | Object | **Alternative to above**: Define multiple named vector spaces | `null` | Partially\*\* | | ↪ `vectorConfig..vectorizer` | Object | Vectorizer config for this named vector (e.g., `{"text2vec-openai": {"properties": ["title"]}}`) | (Required) | No | | [↪ `vectorConfig..vectorIndexType`](./indexing/vector-index.mdx) | String | Index type for this named vector | `hnsw` | No | | [↪ `vectorConfig..vectorIndexConfig`](./indexing/vector-index.mdx) | Object | Index configuration for this named vector | Index-specific defaults | Partially\* | \* See [vector index mutable parameters](./indexing/vector-index.mdx) \*\* [New named vectors can be added](../manage-collections/vector-config.mdx#add-new-named-vectors) after collection creation #### Single vector collections If you don't explicitly define a [named vector](#named-vectors) in your collection definition, Weaviate automatically creates what's known as a _single vector_ collection. These vectors are stored internally under the named vector `default` (which is a reserved vector name). To learn which properties of your data are vectorized, refer to the [Configure semantic indexing](./indexing/vector-index.mdx#configure-semantic-indexing) section. ##### Code example - How to create single vector collection This code example shows how to configure the vectorizer parameters for a single vector collection through a client library: :::tip Further resources For more code example and configuration guides visit the [How-to: Vectorizer and vector index config](../manage-collections/vector-config.mdx) guide. ::: #### Multiple vector embeddings (named vectors) {#named-vectors} import MultiVectorSupport from "/_includes/multi-vector-support.mdx"; ##### Code example - How to create multiple named vectors This code example shows how to configure multiple named vectors through a client library: :::tip Further resources For more code example and configuration guides visit the [How-to: Vectorizer and vector index config](../manage-collections/vector-config.mdx) guide. ::: --- ### Module configuration The `moduleConfig` parameter allows you to specify if the vectorizers will include or exclude the collection name in vector calculations (default `true`). It is also used to specify reranker and generative [model providers](../model-providers/index.md) at a collection level.

Example module configuration - JSON object

An example of a complete `moduleConfig` object: ```json "moduleConfig": { "text2vec-contextionary": { "vectorizeClassName": true // Include the collection name in vector calculation (default true) } }, ```

--- ### Vector index {#vector-index} Vector indexing organizes vector data to make similarity searches fast and efficient. Instead of comparing a query to every vector, an index builds a structure that rapidly narrows the search to the most relevant candidates. More details about the `vectorIndexType` and `vectorIndexConfig` parameters can be found in [Reference: Vector index](./indexing/vector-index.mdx). --- ### Replication [Replication](/deploy/configuration/replication.md) configurations can be set using the definition, through the `replicationConfig` parameter. | Parameter | Type | Description | Default | Mutable | | :----------------- | :------ | :-------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------ | :-------------------------------------- | | `factor` | Integer | The number of copies (replicas) to maintain for each shard. A factor of `3` means one primary and two replicas. | `1` | No | | `deletionStrategy` | String | Strategy for handling deletions in replication. Can be `NoAutomatedResolution`, `DeleteOnConflict` or `TimeBasedResolution`. | `"TimeBasedResolution"` | Yes | | `asyncConfig` | Object | Configuration for async replication tuning. See [`asyncConfig` parameters](#async-config) below. Added in `v1.36` | See below | Yes | :::note Async replication is on by default (`v1.38`) The `asyncEnabled` flag has been removed. As of Weaviate `v1.38`, async replication runs automatically for any collection with a `factor` greater than `1`. To turn it off, set the [`ASYNC_REPLICATION_DISABLED`](/deploy/configuration/env-vars/index.md#async-replication) environment variable to `true`. :::

Example replication configuration - JSON object

An example of a complete `replicationConfig` object: ```json { "class": "Article", "vectorizer": "text2vec-openai", // highlight-start "replicationConfig": { "factor": 3, "deletionStrategy": "TimeBasedResolution", "asyncConfig": { "hashtreeHeight": 16, "frequency": 30000 } } // highlight-end } ```

#### `asyncConfig` parameters {#async-config} import AsyncConfigCollection from '/_includes/feature-notes/async-config-collection.mdx'; :::note Multi-tenant vs single-tenant defaults Some `asyncConfig` parameters have different defaults depending on whether the collection uses multi-tenancy. These differences are noted in the table below. ::: | Parameter | Type | Description | Default (single-tenant) | Default (multi-tenant) | | :--- | :--- | :--- | :--- | :--- | | `hashtreeHeight` | Integer | Height of the hash tree used for data comparison between nodes. Min: `0`, Max: `20` | `16` | `10` | | `frequency` | Integer | Frequency of periodic data comparison between nodes, in milliseconds. | `30000` | `30000` | | `frequencyWhilePropagating` | Integer | Frequency of data comparison while propagation is active, in milliseconds. | `5000` | `5000` | | `loggingFrequency` | Integer | How often the async replication process logs its activity, in seconds. | `60` | `60` | | `diffBatchSize` | Integer | Number of object keys fetched per request during comparison. Min: `1`, Max: `10000` | `1000` | `1000` | | `diffPerNodeTimeout` | Integer | Timeout for a comparison response from a remote node, in seconds. | `10` | `10` | | `prePropagationTimeout` | Integer | Overall timeout for the pre-propagation phase, in seconds. | `300` | `300` | | `propagationTimeout` | Integer | Timeout for a propagation request to a remote node, in seconds. | `60` | `60` | | `propagationLimit` | Integer | Maximum number of objects propagated in a single iteration. Min: `1`, Max: `100000` | `1000` | `1000` | | `propagationDelay` | Integer | Delay before considering an object for propagation, in milliseconds. | `30000` | `30000` | | `propagationConcurrency` | Integer | Number of concurrent workers for propagation. Min: `1`, Max: `20` | `1` | `1` | | `propagationBatchSize` | Integer | Maximum number of objects per propagation batch. Min: `1`, Max: `1000` | `100` | `100` | :::note Values changed in `v1.34.19`, `v1.35.14`, `v1.36.4` and `v1.37.0` Three of the defaults above were changed in the patch releases `v1.34.19`, `v1.35.14` and `v1.36.4`, and apply to every release from `v1.37.0` onwards. On earlier releases of each of those lines, `frequencyWhilePropagating` defaults to `3000`, `propagationLimit` defaults to `10000`, and `propagationConcurrency` defaults to `5`. The maximum value for `propagationLimit` was lowered from `1000000` to `100000` one patch later, in `v1.34.20`, `v1.35.15` and `v1.36.6`. ::: #### Code example - How to configure replication This code example shows how to configure the replication parameters through a client library: :::tip Further resources For more code example and configuration guides visit the [How-to: Manage collections](../manage-collections/index.mdx) section. ::: --- ### Sharding Sharding is configured via the `shardingConfig` object in the collection definition. These parameters are immutable and cannot be changed after the collection is created. | Parameter | Type | Description | Default | Mutable | | :-------------------- | :------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------- | :------ | | `desiredCount` | Integer | The desired number of physical shards for the collection. If this value is larger than the number of cluster nodes, some nodes will host multiple shards. | Number of nodes | No | | `virtualPerPhysical` | Integer | The number of virtual shards per physical shard. Virtual shards aid in reducing data movement during rebalancing. | `128` | No | | `strategy` | String | The strategy for determining which shard an object belongs to. Only `"hash"` is currently supported. The hash is based on the `key` property. | `"hash"` | No | | `key` | String | The property used for hashing to determine the target shard. Currently, only the object's internal UUID (`_id`) can be used. | `"_id"` | No | | `function` | String | The hashing function used on the `key`. Only `"murmur3"` is supported, which creates a 64-bit hash, making collisions highly unlikely. | `"murmur3"` | No | | `actualCount` | Integer | **(Read-only)** The actual number of physical shards created. This typically matches `desiredCount` unless an issue occurred during creation. | `1` | No | | `desiredVirtualCount` | Integer | **(Read-only)** A calculated value representing `desiredCount * virtualPerPhysical`. | `128` | No | | `actualVirtualCount` | Integer | **(Read-only)** The actual number of virtual shards that were created. | `128` | No |

Example sharding configuration - JSON object

An example of a complete `shardingConfig` object: ```json "shardingConfig": { "virtualPerPhysical": 128, "desiredCount": 1, // defaults to the amount of Weaviate nodes in the cluster "actualCount": 1, "desiredVirtualCount": 128, "actualVirtualCount": 128, "key": "_id", "strategy": "hash", "function": "murmur3" } ```

#### Code example - How to configure sharding This code example shows how to configure the sharding parameters through a client library: :::tip Further resources For more code example and configuration guides visit the [How-to: Manage collections](../manage-collections/index.mdx) section. ::: --- ### Multi-tenancy Multi-tenancy allows you to isolate data within a single collection, where objects are associated with specific tenants. This is a useful feature for building SaaS applications or any system requiring strict data partitioning. :::note Why use multi-tenancy? It provides data isolation at a lower overhead than creating a separate collection for each tenant, making it more scalable when you have a large number of tenants. ::: To enable multi-tenancy, set the `enabled` key to `true` in the `multiTenancyConfig` object. This parameter is immutable and must be set at creation time. | Parameter | Type | Description | Default | Mutable | | :--------------------- | :------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------- | :------ | :------ | | `enabled` | Boolean | If `true`, enables multi-tenancy for the collection. | `false` | No | | `autoTenantCreation` | Boolean | If `true`, a new tenant is created if you try to insert an object into a non-existent tenant. | `false` | Yes | | `autoTenantActivation` | Boolean | If `true`, automatically activate `INACTIVE` or `OFFLOADED` tenants if a search, read, update, or delete operation is performed on them. | `false` | Yes | #### Code example - How to configure multi-tenancy This code example shows how to configure the multi-tenancy parameters through a client library: :::tip Further resources For more code example and configuration guides visit the [How-to: Manage collections](../manage-collections/index.mdx) section. ::: ## Mutability Some, but not all, parameters are mutable after you create your collection. To modify immutable parameters, export your data, create a new collection, and import your data into it.

Mutable parameters

import RaftRFChangeWarning from "/_includes/1-25-replication-factor.mdx"; import CollectionMutableParameters from "/_includes/collection-mutable-parameters.mdx";

After you create a collection, you can [add new properties](../manage-collections/collection-operations.mdx#add-a-property). You cannot modify existing properties after you create the collection. You can also [add new named vectors](../concepts/data.md#adding-a-named-vector-after-collection-creation). ## Auto-schema The "Auto-schema" feature generates a collection definition automatically by inferring parameters from data being added. It is enabled by default, and can be disabled (e.g. in `docker-compose.yml`) by setting the environment variable [`AUTOSCHEMA_ENABLED`](/docs/deploy/configuration/env-vars/index.md#AUTOSCHEMA_ENABLED) to `'false'`. It will: - Create a collection if an object is added to a non-existent collection. - Add any missing property from an object being added. - Infer array data types, such as `int[]`, `text[]`, `number[]`, `boolean[]`, `date[]` and `object[]`. - Infer nested properties for `object` and `object[]` data types. - Throw an error if an object being added contains a property that conflicts with an existing schema type. (e.g. trying to import text into a field that exists in the schema as `int`). :::tip Define the collection manually for production use Generally speaking, we recommend that you disable auto-schema for production use. - A manual collection definition will provide more precise control. - There is a performance penalty associated with inferring the data structure at import time. This may be a costly operation in some cases, such as complex nested properties. ::: #### Auto-schema data types Additional configurations are available to help the auto-schema infer properties to suit your needs. - `AUTOSCHEMA_DEFAULT_NUMBER=number` - create `number` columns for any numerical values (as opposed to `int`, etc). - `AUTOSCHEMA_DEFAULT_DATE=date` - create `date` columns for any date-like values. The following are not allowed: - Any map type is forbidden, unless it clearly matches one of the two supported types `phoneNumber` or `geoCoordinates`. - Any array type is forbidden, unless it is clearly a reference-type. In this case, Weaviate needs to resolve the beacon and see what collection the resolved beacon is from, since it needs the collection name to be able to alter the schema. ## Collections count limit {#collections-count-limit} import CollectionsLimit from '/_includes/feature-notes/collections-limit.mdx'; Each collection adds overhead in terms of indexing, definition management, and storage. It is possible to **limit the number of collections per instance**. This helps maintain optimal performance and resource utilization. - **Default limit**: `-1` (no limit). - **Modify the limit**: Use the [`MAXIMUM_ALLOWED_COLLECTIONS_COUNT`](/deploy/configuration/env-vars/index.md#MAXIMUM_ALLOWED_COLLECTIONS_COUNT) environment variable to adjust the collection count limit. :::note If your instance already exceeds the limit, Weaviate will not allow the creation of any new collections. Existing collections will not be deleted. ::: :::tip **Instead of raising the collections count limit, consider rethinking your architecture**. For more details, see [Starter Guides: Scaling limits with collections](../starter-guides/managing-collections/collections-scaling-limits.mdx). ::: ## Collection aliases import CollectionAliases from '/_includes/feature-notes/collection-aliases.mdx'; Collection aliases are alternative names for Weaviate collections that allow you to reference a collection by an alternative name. Alias names must be unique (can't match existing collections or other aliases) and multiple aliases can point to the same collection. You can set up collection aliases [programmatically through client libraries](../manage-collections/collection-aliases.mdx) or by using the REST endpoints. In order to manage collection aliases, you need to posses the right [`Collection aliases`](../configuration/rbac/index.mdx#available-permissions) permissions. To manage the underlying collection the alias references, you also need the [`Collections`](../configuration/rbac/index.mdx#available-permissions) permissions for that specific collection. **Collection aliases cannot be used to update collection definitions**, including: - Updating and adding properties - Updating vector and inverted indexes - Configuring sharding and multi-tenancy - Modifying vectorizer, generative and reranker configurations import CollectionAliasUsage from "/_includes/collection-alias-usage.mdx"; ## Further resources - [Starter guides: Collection definition](/weaviate/starter-guides/managing-collections) - [How to: Manage collections](../manage-collections/index.mdx) - [Concepts: Data structure](/weaviate/concepts/data) - REST API: Collection definition (schema) ## Questions and feedback import DocsFeedback from "/_includes/docs-feedback.mdx"; --- ### Weaviate/Config Refs/Datatypes (docs/weaviate/config-refs/datatypes.md) --- title: Property data types sidebar_label: Data types description: Weaviate schema data types reference for defining object properties and field specifications. image: og/docs/configuration.jpg # tags: ['Data types'] --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import FilteredTextBlock from '@site/src/components/Documentation/FilteredTextBlock'; import SkipLink from '/src/components/SkipValidationLink' When [creating a property](../manage-collections/collection-operations.mdx#add-a-property), you must specify a data type. Weaviate accepts the following types. ## Available data types :::note Array types Arrays of a data type are specified by adding `[]` to the type (e.g. `text` ➡ `text[]`). Note that not all data types support arrays. ::: import DataTypes from '/\_includes/datatypes.mdx'; Further details on each data type are provided below. ## `text` Use this type for any text data. - Properties with the `text` type is used for vectorization and keyword search unless specified otherwise [in the property settings](../manage-collections/vector-config.mdx#property-level-settings). - If using [named vectors](../concepts/data.md#multiple-vector-embeddings-named-vectors), the property vectorization is defined in the [named vector definition](../manage-collections/vector-config.mdx#define-named-vectors). - Text properties are tokenized prior to being indexed for keyword/BM25 searches. See [collection definition: tokenization](../config-refs/collections.mdx#tokenization) for more information.

string is deprecated

Prior to `v1.19`, Weaviate supported an additional datatype `string`, which was differentiated by tokenization behavior to `text`. As of `v1.19`, this type is deprecated and will be removed in a future release. Use `text` instead of `string`. `text` supports the tokenization options that are available through `string`.

### Examples import TextTypePy from '!!raw-loader!/\_includes/code/python/config-refs.datatypes.text.py'; import TextTypeTs from '!!raw-loader!/\_includes/code/typescript/config-refs.datatypes.text.ts'; #### Property definition #### Object insertion ## `boolean` / `int` / `number` The `boolean`, `int`, and `number` types are used for storing boolean, integer, and floating-point numbers, respectively. ### Examples import NumericalTypePy from '!!raw-loader!/\_includes/code/python/config-refs.datatypes.numerical.py'; import NumericalTypeTs from '!!raw-loader!/\_includes/code/typescript/config-refs.datatypes.numerical.ts'; #### Property definition #### Object insertion ### Note: GraphQL and `int64` Although Weaviate supports `int64`, GraphQL currently only supports `int32`, and does not support `int64`. This means that currently _integer_ data fields in Weaviate with integer values larger than `int32`, will not be returned using GraphQL queries. We are working on solving this [issue](https://github.com/weaviate/weaviate/issues/1563). As current workaround is to use a `string` instead. ## `date` A `date` in Weaviate is represented by an [RFC 3339](https://datatracker.ietf.org/doc/rfc3339/) timestamp in the `date-time` format. The timestamp includes the time and an offset. For example: - `"1985-04-12T23:20:50.52Z"` - `"1996-12-19T16:39:57-08:00"` - `"1937-01-01T12:00:27.87+00:20"` To add a list of dates as a single entity, use an array of `date-time` formatted strings. For example: `["1985-04-12T23:20:50.52Z", "1937-01-01T12:00:27.87+00:20"]` In specific client libraries, you may be able to use the native date object as shown in the following examples. ### Examples import DateTypePy from '!!raw-loader!/\_includes/code/python/config-refs.datatypes.date.py'; import DateTypeTs from '!!raw-loader!/\_includes/code/typescript/config-refs.datatypes.date.ts'; #### Property definition #### Object insertion ## `uuid` The dedicated `uuid` and `uuid[]` data types efficiently store [UUIDs](https://en.wikipedia.org/wiki/Universally_unique_identifier). - Each `uuid` is a 128-bit (16-byte) number. - The filterable index uses roaring bitmaps. :::note Aggregate/sort currently not possible It is currently not possible to aggregate or sort by `uuid` or `uuid[]` types. ::: ### Examples import UUIDTypePy from '!!raw-loader!/\_includes/code/python/config-refs.datatypes.uuid.py'; import UUIDTypeTs from '!!raw-loader!/\_includes/code/typescript/config-refs.datatypes.uuid.ts'; #### Property definition #### Object insertion ## `geoCoordinates` Geo coordinates can be used to find objects in a radius around a query location. A geo coordinate value stored as a float, and is processed as [decimal degree](https://en.wikipedia.org/wiki/Decimal_degrees) according to the [ISO standard](https://www.iso.org/standard/39242.html#:~:text=For%20computer%20data%20interchange%20of,minutes%2C%20seconds%20and%20decimal%20seconds). To supply a `geoCoordinates` property, specify the `latitude` and `longitude` as floating point decimal degrees. ### Examples import GeoTypePy from '!!raw-loader!/\_includes/code/python/config-refs.datatypes.geocoordinates.py'; import GeoTypeTs from '!!raw-loader!/\_includes/code/typescript/config-refs.datatypes.geocoordinates.ts'; #### Property definition #### Object insertion import GeoLimitations from '/\_includes/geo-limitations.mdx'; ## `phoneNumber` A `phoneNumber` input will be normalized and validated, unlike the single fields as `number` and `string`. The data field is an object with multiple fields. ```yaml { "phoneNumber": { "input": "020 1234567", // Required. Raw input in string format "defaultCountry": "nl", // Required if only a national number is provided, ISO 3166-1 alpha-2 country code. Only set if explicitly set by the user. "internationalFormatted": "+31 20 1234567", // Read-only string "countryCode": 31, // Read-only unsigned integer, numerical country code "national": 201234567, // Read-only unsigned integer, numerical representation of the national number "nationalFormatted": "020 1234567", // Read-only string "valid": true // Read-only boolean. Whether the parser recognized the phone number as valid } } ``` There are two fields that accept input. `input` must always be set, while `defaultCountry` must only be set in specific situations. There are two scenarios possible: - When you enter an international number (e.g. `"+31 20 1234567"`) to the `input` field, no `defaultCountry` needs to be entered. The underlying parser will automatically recognize the number's country. - When you enter a national number (e.g. `"020 1234567"`), you need to specify the country in `defaultCountry` (in this case, `"nl"`), so that the parse can correctly convert the number into all formats. The string in `defaultCountry` should be an [ISO 3166-1 alpha-2](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2) country code. Weaviate will also add further read-only fields such as `internationalFormatted`, `countryCode`, `national`, `nationalFormatted` and `valid` when reading back a field of type `phoneNumber`. ### Examples import PhoneTypePy from '!!raw-loader!/\_includes/code/python/config-refs.datatypes.phonenumber.py'; import PhoneTypeTs from '!!raw-loader!/\_includes/code/typescript/config-refs.datatypes.phonenumber.ts'; #### Property definition #### Object insertion ## `blob` The datatype blob accepts any binary data. The data should be `base64` encoded, and passed as a `string`. Characteristics: - Weaviate doesn't make assumptions about the type of data that is encoded. A module (e.g. `img2vec`) can investigate file headers as it wishes, but Weaviate itself does not do this. - When storing, the data is `base64` decoded (so Weaviate stores it more efficiently). - When serving, the data is `base64` encoded (so it is safe to serve as `json`). - There is no max file size limit. - This `blob` field is always skipped in the inverted index, regardless of setting. This mean you can not search by this `blob` field in a Weaviate GraphQL `where` filter, and there is no `valueBlob` field accordingly. Depending on the module, this field can be used in module-specific filters (e.g. `nearImage`{} in the `img2vec-neural` filter). To obtain the base64-encoded value of an image, you can run the following command - or use the helper methods in the Weaviate clients - to do so: ```bash cat my_image.png | base64 ``` ### Examples import BlobTypePy from '!!raw-loader!/\_includes/code/python/config-refs.datatypes.blob.py'; import BlobTypeTs from '!!raw-loader!/\_includes/code/typescript/config-refs.datatypes.blob.ts'; #### Property definition #### Object insertion ## `blobHash` :::info Added in `v1.37` ::: The `blobHash` data type accepts base64-encoded data (same as [`blob`](#blob)) but stores only a SHA-256 hash on disk. This reduces storage space while still allowing modules (such as `multi2vec-google`) to vectorize the original media content during import. **How it works:** - During validation, the base64 input is validated but kept as-is. - The raw data flows through the vectorization pipeline so modules can vectorize the actual media content. - After vectorization, the base64 data is converted to a SHA-256 hex hash before being persisted. - When an object is updated, the incoming base64 data is hashed before being compared against the stored hash to determine whether re-vectorization is needed. **Behavior:** identical to `blob` for indexing restrictions (no `indexFilterable`), sorting (string comparator), API serialization (GraphQL string, gRPC blob value), and inverted index exclusion. ```json { "properties": [ { "name": "image", "dataType": ["blobHash"] } ] } ``` Use `blobHash` when you need a vectorizer to see the raw media at import time but don't need to retrieve the original bytes afterwards: only the hash is stored. ## `object` The `object` type allows you to store nested data as a JSON object that can be nested to any depth. For example, a `Person` collection could have an `address` property as an object. It could in turn include nested properties such as `street` and `city`: :::note Indexing and filtering `object` and `object[]` properties are not vectorized by default, and only their leaf scalars are stored in the inverted index. If you list an object property in the vector configuration's [`properties` field](indexing/vector-index.mdx#specify-which-properties-to-vectorize), it is converted to a string (its JSON representation) and concatenated into the vectorizer's input text. From Weaviate `v1.38` (preview), you can filter on nested-object leaves using a dotted path syntax. See [Filter on nested object properties](../search/filters.md#filter-on-nested-object-properties). ::: ### Examples import ObjectTypePy from '!!raw-loader!/\_includes/code/python/config-refs.datatypes.object.py'; import ObjectTypeTs from '!!raw-loader!/\_includes/code/typescript/config-refs.datatypes.object.ts'; #### Property definition #### Object insertion ## `cross-reference` import CrossReferencePerformanceNote from '/\_includes/cross-reference-performance-note.mdx'; The `cross-reference` type allows a link to be created from one object to another. This is useful for creating relationships between collections, such as linking a `Person` collection to a `Company` collection. The `cross-reference` type objects are `arrays` by default. This allows you to link to any number of instances of a given collection (including zero). For more information on cross-references, see the [cross-references](../concepts/data.md#cross-references). To see how to work with cross-references, see [how to manage data: cross-references](../manage-collections/cross-references.mdx). ## Notes #### Formatting in payloads In raw payloads (e.g. JSON payloads for REST), data types are specified as an array (e.g. `["text"]`, or `["text[]"]`), as it is required for some cross-reference specifications. ## Further resources - [How-to: Manage collections](../manage-collections/index.mdx) - [Concepts: Data structure](../concepts/data.md) - References: REST API: Schema ## Questions and feedback import DocsFeedback from '/\_includes/docs-feedback.mdx'; --- ### Weaviate/Config Refs/Distances (docs/weaviate/config-refs/distances.md) --- title: Distance metrics description: "Distance metric options for vector similarity calculations and search result ranking algorithms." image: og/docs/configuration.jpg --- import SkipLink from '/src/components/SkipValidationLink' ## Available distance metrics If not specified explicitly, the default distance metric in Weaviate is `cosine`. It can be [set in the vectorIndexConfig](/weaviate/config-refs/indexing/vector-index.mdx#hnsw-index) field as part of the schema ([example](../manage-collections/vector-config.mdx#specify-a-distance-metric)) to any of the following types: :::tip Comparing distances In all cases, larger distance values indicate lower similarity. Conversely, smaller distance values indicate higher similarity. ::: | Name | Description | Definition | Range | Examples | | ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ------------------------------------------------------------- | ----------------------------------------------------------------- | | `cosine` | Cosine (angular) distance.
_{[See note 1 below]} | `1 - cosine_sim(a,b)` | `0 <= d <= 2` | `0`: identical vectors

`2`: Opposing vectors. | | `dot` | A dot product-based indication of distance.

More precisely, the negative dot product.
_{[See note 2 below]} | `-dot(a,b)` | `-∞ < d < ∞` | `-3`: more similar than `-2`

`2`: more similar than `5` | | `l2-squared` | The squared euclidean distance between two vectors. | `sum((a_i - b_i)^2)` | `0 <= d < ∞` | `0`: identical vectors | | `hamming` | Number of differences between vectors at each dimensions. | sum(|a_i != b_i|) | `0 <= d < dims` | `0`: identical vectors | | `manhattan` | The distance between two vector dimensions measured along axes at right angles. | sum(|a_i - b_i|) | `0 <= d < ∞` | `0`: identical vectors | If you're missing your favorite distance type and would like to contribute it to Weaviate, we'd be happy to review your [PR](https://github.com/weaviate/weaviate). :::note Additional notes 1. If `cosine` is chosen, all vectors are normalized to length 1 at read time and dot product is used to calculate the distance for computational efficiency. 2. Dot Product on its own is a similarity metric, not a distance metric. As a result, Weaviate returns the negative dot product to stick with the intuition that a smaller value of a distance indicates a more similar result and a higher distance value indicates a less similar result. 3. The [HFresh index](/weaviate/config-refs/indexing/vector-index.mdx#hfresh-index) only supports `cosine` and `l2-squared` distance metrics. ::: ## Distance implementations and optimizations On a typical Weaviate use case the largest portion of CPU time is spent calculating vector distances. Even with an approximate nearest neighbor index - which leads to far fewer calculations - the efficiency of distance calculations has a major impact on [overall performance](/weaviate/benchmarks/ann.md). Weaviate uses SIMD (Single Instruction, Multiple Data) instructions for the following distance metrics and architectures. The available optimizations are resolved in the shown order (e.g. SVE -> Neon). | Distance | `arm64` | `amd64` | | ----------------------------- | ----------- | --------------------------------------------- | | `cosine`, `dot`, `l2-squared` | SVE or Neon | Sapphire Rapids with AVX512, or Any with AVX2 | | `hamming`, `manhattan` | No SIMD | No SIMD | If you like dealing with Assembly programming, SIMD, and vector instruction sets we would love to receive your contribution for one of the combinations that have not yet received an SIMD-specific optimization. ## Distance fields in the APIs The `distance` is exposed in the APIs in two ways: - Whenever a [vector search](../search/similarity.md#set-a-similarity-threshold) is involved, the distance can be displayed as part of the results, for example using `_additional { distance }` - Whenever a [vector search](../search/similarity.md#set-a-similarity-threshold) is involved, the distance can be specified as a limiting criterion, for example using `nearVector({distance: 1.5, vector: ... })` ## Distance vs Certainty Prior to version `v1.14` only `certainty` was available in the APIs. The original ideas behind certainty was to normalize the distance score into a value between `0 <= certainty <= 1`, where 1 would represent identical vectors and 0 would represent opposite vectors. This concept is however unique to `cosine` distance. With other distance metrics, scores may be unbounded. As a result the preferred way is to use `distance` in favor of `certainty`. For backward compatibility, `certainty` can still be used when the distance is `cosine`. If any other distance is selected `certainty` cannot be used. See also [Search API: Additional properties (metadata)](../api/graphql/additional-properties.md). ## Further resources - [How-to: Manage collections](../manage-collections/index.mdx) - REST API: Collection definition (schema) - [Concepts: Data structure](../concepts/data.md) ## Questions and feedback import DocsFeedback from '/\_includes/docs-feedback.mdx'; --- ### Weaviate/Config Refs/Index (docs/weaviate/config-refs/index.mdx) --- title: Configuration description: "Configuration reference guides covering collections, data types, distances, and optimization settings." image: og/docs/configuration.jpg hide_table_of_contents: true --- Use the following reference guides to understand Weaviate's core concepts and configurations. These guides cover collection definitions, data types, distance metrics, environment variables, and more to help you optimize your Weaviate deployment. import CardsSection from "/src/components/CardsSection"; export const mainReferencesData = [ { title: "Collection definition", description: "An overview of all top-level collection parameters and concepts.", link: "/weaviate/config-refs/collections", icon: "fas fa-cube", }, { title: "Vector Index", description: "Tune HNSW, Flat, or Dynamic indexes to balance search speed and recall.", link: "/weaviate/config-refs/indexing/vector-index", icon: "fas fa-sitemap", }, { title: "Inverted Index", description: "Optimize filtering and keyword search with BM25, stopwords, and other settings.", link: "/weaviate/config-refs/indexing/inverted-index", icon: "fas fa-filter", }, { title: "Data Types", description: "Reference for supported data types and their handling in Weaviate.", link: "/weaviate/config-refs/datatypes", icon: "fas fa-list-ul", }, { title: "Distance Metrics", description: "Learn about the distance metrics used for similarity calculations.", link: "/weaviate/config-refs/distances", icon: "fas fa-ruler", }, { title: "Environment Variables", description: "Learn about the environment variables used for configuration.", link: "/deploy/configuration/env-vars", icon: "fas fa-gear", }, ];

:::info Deployment documentation For deployment related topics like security, backups, replication, cluster information and advanced configuration options, visit the [deployment documentation](/docs/deploy/configuration/index.mdx). ::: --- ### Weaviate/Config Refs/Indexing/Inverted Index (docs/weaviate/config-refs/indexing/inverted-index.mdx) --- title: Inverted index description: Reference for inverted index parameters in Weaviate. --- import SkipLink from "/src/components/SkipValidationLink"; import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem"; import FilteredTextBlock from "@site/src/components/Documentation/FilteredTextBlock"; import PyCode from "!!raw-loader!/_includes/code/config-refs/reference.collections.py"; import TSCode from "!!raw-loader!/_includes/code/howto/manage-data.collections.ts"; import GoCode from "!!raw-loader!/_includes/code/howto/go/docs/manage-data.classes_test.go"; import PyTokenizeEndpoint from "!!raw-loader!/_includes/code/tutorials/tokenization/tokenize_endpoint.py"; The **[inverted index](../../concepts/indexing/inverted-index.md)** maps values (like words or numbers) to the objects that contain them. It is the backbone for all attribute-based filtering (`where` filters) and keyword searching (`bm25`, `hybrid`). ## Inverted index types Multiple [inverted index types](../../concepts/indexing/inverted-index.md) are available in Weaviate. Not all inverted index types are available for all data types. The available inverted index types are: import InvertedIndexTypesSummary from "/_includes/inverted-index-types-summary.mdx"; - Enable one or both of `indexFilterable` and `indexRangeFilters` to index a property for faster filtering. - If only one is enabled, the respective index is used for filtering. - If both are enabled, `indexRangeFilters` is used for operations involving comparison operators, and `indexFilterable` is used for equality and inequality operations. ## Inverted index parameters These parameters are set within the `invertedIndexConfig` object in your collection definition. | Parameter | Type | Default | Details | | :-------------------------------------------- | :-------- | :------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------- | | [`bm25`](#bm25) | `object` | `{ "k1": 1.2, "b": 0.75 }` | Sets the `k1` and `b` parameters for the BM25 ranking algorithm. Can be overridden at the property level. See [**BM25 Configuration**](#bm25) below. | | [`stopwords`](#stopwords) | `object` | (Varies) | Defines the stopword list to exclude common words from search queries. See [**Stopwords Configuration**](#stopwords) below. | | [`indexTimestamps`](#indextimestamps) | `boolean` | `false` | If `true`, indexes object creation and update timestamps, enabling filtering by `creationTimeUnix` and `lastUpdateTimeUnix`. | | [`indexNullState`](#indexnullstate) | `boolean` | `false` | If `true`, indexes the null/non-null state of each property, enabling filtering for `null` values. | | [`indexPropertyLength`](#indexpropertylength) | `boolean` | `false` | If `true`, indexes the length of each property, enabling filtering by property length. | :::caution Performance Impact Enabling `indexTimestamps`, `indexNullState`, or `indexPropertyLength` adds overhead as these additional indexes must be created and maintained. Only enable them if you require these specific filtering capabilities. ::: #### Code example This code example shows how to configure inverted index parameters through a client library: --- #### `bm25` Part of `invertedIndexConfig`. The settings for BM25 are the [free parameters `k1` and `b`](https://en.wikipedia.org/wiki/Okapi_BM25#The_ranking_function), and they are optional. The defaults (`k1` = 1.2 and `b` = 0.75) work well for most cases. They can be configured per collection, and can optionally be overridden per property.

Example `bm25` configuration - JSON object

An example of a complete collection object with `bm25` configuration: ```json { "class": "Article", // Configuration of the sparse index "invertedIndexConfig": { "bm25": { "b": 0.75, "k1": 1.2 } }, "properties": [ { "name": "title", "description": "title of the article", "dataType": ["text"], // Property-level settings override the collection-level settings "invertedIndexConfig": { "bm25": { "b": 0.75, "k1": 1.2 } }, "indexFilterable": true, "indexSearchable": true } ] } ```

#### `stopwords` Part of `invertedIndexConfig`. `text` properties may contain words that are very common and don't contribute to search results. Ignoring them speeds up queries that contain stopwords, as they can be automatically removed from queries as well. This speedup is very notable on scored searches, such as `BM25`. The stopword configuration uses a preset system. You can select a preset to use the most common stopwords for a particular language (e.g. [`"en"` preset](https://github.com/weaviate/weaviate/blob/main/adapters/repos/db/inverted/stopwords/presets.go)). If you need more fine-grained control, you can add additional stopwords or remove stopwords that you believe should not be part of the list. Alternatively, you can create your custom stopword list by starting with an empty (`"none"`) preset and adding all your desired stopwords as additions.

Example `stopwords` configuration - JSON object

An example of a complete collection object with `stopwords` configuration: ```json "invertedIndexConfig": { "stopwords": { "preset": "en", "additions": ["star", "nebula"], "removals": ["a", "the"] } } ```

This configuration allows stopwords to be configured by collection. If not set, these values are set to the following defaults: | Parameter | Default value | Acceptable values | | ------------- | ------------- | -------------------------- | | `"preset"` | `"en"` | `"en"`, `"none"` | | `"additions"` | `[]` | _any list of custom words_ | | `"removals"` | `[]` | _any list of custom words_ | :::note - If `preset` is `none`, then the collection only uses stopwords from the `additions` list. - If the same item is included in both `additions` and `removals`, Weaviate returns an error. ::: As of `v1.18`, stopwords are indexed. Thus stopwords are included in the inverted index, but not in the tokenized query. As a result, when the BM25 algorithm is applied, stopwords are ignored in the input for relevance ranking but will affect the score. Stopwords can now be configured at runtime. You can use the RESTful API to update the list of stopwords after your data has been indexed. :::info Stopwords are only removed when [tokenization](../collections.mdx#tokenization) is set to `word`. ::: #### `stopwordPresets` import TokenizerPreview from "/_includes/feature-notes/tokenizer.mdx"; Part of `invertedIndexConfig`. Defines named stopword presets at the collection level. Each preset is a flat list of words. Properties can then reference a preset by name via [`textAnalyzer.stopwordPreset`](#textanalyzer). A preset name that matches a built-in (`"en"`, `"none"`) fully replaces the built-in for properties of this collection. Preset names must not be empty or whitespace-only; each word list must contain at least one word; individual words must not be empty or whitespace-only.

Example stopwordPresets configuration - JSON object

```json "invertedIndexConfig": { "stopwordPresets": { "fr": ["le", "la", "les", "un", "une", "des", "du", "de", "et"], "de": ["der", "die", "das", "und", "oder", "aber"] } } ```

The existing [`stopwords`](#stopwords) configuration remains as the default for properties that do not specify a `textAnalyzer.stopwordPreset` override. For extending a built-in preset with `additions`/`removals`, use [`stopwords`](#stopwords) instead. It is the only stopword config that accepts that object form. #### `textAnalyzer` {#textanalyzer} Part of a **property definition** (not `invertedIndexConfig`). Configures text analysis behavior for individual `text` properties. The accent-folding options (`asciiFold`, `asciiFoldIgnore`) are supported on properties with tokenization `word`, `lowercase`, `whitespace`, `field`, or `trigram`. They are not supported on the language-specific tokenizers (`gse`, `gse_ch`, `kagome_ja`, and `kagome_kr`). The `stopwordPreset` option is only supported on properties with `tokenization: "word"`. | Parameter | Type | Default | Details | | :---------------- | :--------- | :------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `asciiFold` | `boolean` | `false` | Normalizes accented Latin characters to ASCII equivalents during indexing and querying. Uses Unicode NFD decomposition. **Immutable** after the property is created. | | `asciiFoldIgnore` | `string[]` | `[]` | Characters exempt from ASCII folding. Each entry must be a single character. **Immutable** after the property is created. | | `stopwordPreset` | `string` | (none) | Name of a built-in (`en`, `none`) or collection-level stopword preset to use for this property, overriding the default `stopwords` config. **Only supported on properties with `tokenization: "word"`**. Schema validation rejects it on other tokenizers. |

Example textAnalyzer configuration - JSON object

```json { "name": "description", "dataType": ["text"], "tokenization": "word", "textAnalyzer": { "asciiFold": true, "asciiFoldIgnore": ["é"], "stopwordPreset": "fr" } } ```

:::note `asciiFoldIgnore` changes which tokens are written to disk. It cannot be modified after the property is created. Schema updates that change the ignore list are rejected. To change it, create a new property and reindex. ::: #### `indexTimestamps` Part of `invertedIndexConfig`. To perform queries that are filtered by timestamps, configure the target collection to maintain an inverted index based on the objects' internal timestamps. Currently the timestamps include `creationTimeUnix` and `lastUpdateTimeUnix`. To configure timestamp based indexing, set `indexTimestamps` to `true` in the `invertedIndexConfig` object. #### `indexNullState` Part of `invertedIndexConfig`. To perform queries that filter on `null`, configure the target collection to maintain an inverted index that tracks `null` values for each property in a collection . To configure `null` based indexing, setting `indexNullState` to `true` in the `invertedIndexConfig` object. #### `indexPropertyLength` Part of `invertedIndexConfig`. To perform queries that filter by the length of a property, configure the target collection to maintain an inverted index based on the length of the properties. To configure indexing based on property length, set `indexPropertyLength` to `true` in the `invertedIndexConfig` object. :::note Using these features requires more resources, as the additional inverted indexes must be created and maintained. ::: ## Drop an inverted index You can drop (delete) an inverted index from a property. This is a destructive operation: the index data is removed from disk. To use the index again, it must be regenerated. The following index types can be dropped: `searchable`, `filterable`, `rangeFilters`. See [How-to: Drop an inverted index](../../manage-collections/inverted-index.mdx#drop-an-inverted-index) for code examples. ## How Weaviate creates inverted indexes Weaviate creates **separate inverted indexes for each property and each index type**. For example, if you have a `title` property that is both searchable and filterable, Weaviate will create two separate inverted indexes for that property - one optimized for search operations and another for filtering operations. Find out more in [Concepts: Inverted index](../../concepts/indexing/inverted-index.md#how-weaviate-creates-inverted-indexes). ### Adding a property after collection creation Adding a property after importing objects can lead to limitations in inverted-index related behavior, such as filtering by the new property's length or null status. This is caused by the inverted index being built at import time. If you add a property after importing objects, the inverted index for metadata such as the length or the null status will not be updated to include the new properties. This means that the new property will not be indexed for existing objects. This can lead to unexpected behavior when querying. To avoid this, you can either: - Add the property before importing objects. - Delete the collection, re-create it with the new property and then re-import the data. We are working on a re-indexing API to allow you to re-index the data after adding a property. This will be available in a future release. ## How tokenization affects inverted indexing For `text` properties, Weaviate first **[tokenizes](../collections.mdx#tokenization)** the text before creating inverted index entries. Tokenization is the process of breaking text into individual tokens (words, phrases, or characters) that can be indexed and searched. See the related [concepts page](../../concepts/indexing/inverted-index.md#tokenization) for more details. ## Tokenize endpoint Two REST endpoints let you test tokenization without modifying your schema. ### Freeform tokenization `POST /v1/tokenize` tokenizes arbitrary text with an explicit tokenizer and analyzer config. **Request body:** | Parameter | Type | Required | Details | | :---------------- | :------- | :------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `text` | `string` | yes | The text to tokenize. Maximum length 10,000 characters. | | `tokenization` | `string` | yes | Tokenization method (`word`, `lowercase`, `whitespace`, `field`, `trigram`, `gse`, `gse_ch`, `kagome_ja`, `kagome_kr`). | | `analyzerConfig` | `object` | no | Analyzer settings: `asciiFold` (`boolean`), `asciiFoldIgnore` (`string[]`), `stopwordPreset` (`string`). | | `stopwords` | `object` | no | Fallback stopword configuration (same shape as [`invertedIndexConfig.stopwords`](#stopwords)). Applied when `analyzerConfig.stopwordPreset` is not set. With `word` tokenization, defaults to preset `en` when omitted. | | `stopwordPresets` | `object` | no | Named stopword presets (same shape as [`invertedIndexConfig.stopwordPresets`](#stopwordpresets)). Reference one via `analyzerConfig.stopwordPreset`. | :::note `stopwords` and `stopwordPresets` are mutually exclusive. Pass one or the other, not both. Use `stopwords` for a single preset optionally tweaked with additions/removals; use `stopwordPresets` to define named presets and select one via `analyzerConfig.stopwordPreset`. ::: **Example:** ```bash curl -X POST http://localhost:8080/v1/tokenize -d '{ "text": "The organic café crème blend", "tokenization": "word", "analyzerConfig": { "asciiFold": true, "stopwordPreset": "en" } }' ``` **Response:** ```json { "indexed": ["the", "organic", "cafe", "creme", "blend"], "query": ["organic", "cafe", "creme", "blend"] } ``` - `indexed`: tokens as stored in the inverted index - `query`: tokens after stopword filtering (what BM25 scores at search time) **Example with a custom stopword preset:** Define a named preset on the request via `stopwordPresets` and reference it from `analyzerConfig.stopwordPreset`. This is useful for previewing a non-English preset before adding it to a collection. ```bash curl -X POST http://localhost:8080/v1/tokenize -d '{ "text": "La Tasse Bleue et le Bol", "tokenization": "word", "analyzerConfig": { "stopwordPreset": "fr" }, "stopwordPresets": { "fr": ["le", "la", "les", "un", "une", "des", "du", "de", "et"] } }' ``` **Response:** ```json { "indexed": ["la", "tasse", "bleue", "et", "le", "bol"], "query": ["tasse", "bleue", "bol"] } ``` ### Property-based tokenization `POST /v1/schema/{className}/properties/{propertyName}/tokenize` resolves the full analyzer config from an existing property. The property's tokenization method, `textAnalyzer` settings, and the collection's stopword configuration are applied automatically. Nothing else needs to be passed. **Request body:** | Parameter | Type | Required | Details | | :-------- | :------- | :------- | :------------------- | | `text` | `string` | yes | The text to tokenize | The response format is the same as freeform tokenization. Class and property names are case-insensitive, and collection aliases are resolved automatically. **Example:** ```bash curl -X POST http://localhost:8080/v1/schema/TokenizeDemo/properties/name_fr/tokenize -d '{ "text": "La Tasse Bleue et le Bol" }' ``` See the [tokenization tutorial](../../tutorials/tokenization.md#example-6-inspecting-tokenization-with-the-tokenize-endpoint) for worked examples. ## Further resources - [Concepts: Inverted index](../../concepts/indexing/inverted-index.md) - [How-to: Set inverted index parameters](../../manage-collections/inverted-index.mdx#set-inverted-index-parameters) - [Reference: Tokenization options](../collections.mdx#tokenization) - Learn about different tokenization methods and how they affect text indexing ## Questions and feedback import DocsFeedback from "/_includes/docs-feedback.mdx"; --- ### Weaviate/Config Refs/Indexing/Vector Index (docs/weaviate/config-refs/indexing/vector-index.mdx) --- title: Vector index description: Reference for vector index types and parameters in Weaviate. --- **[Vector indexes](../../concepts/indexing/vector-index.md)** facilitate efficient, vector-first data storage and retrieval. There are four supported vector index types: - **[HNSW index](#hnsw-index)** - **[Flat index](#flat-index)** - **[Dynamic index](#dynamic-index)** - **[HFresh index](#hfresh-index)** ## Index configuration parameters :::caution Experimental feature Available starting in `v1.25`. Dynamic indexing is an experimental feature. Use with caution. ::: Use these parameters to configure the index type and their properties. They can be set in the [collection configuration](../../manage-collections/vector-config.mdx#set-vector-index-type). | Parameter | Type | Default | Details | | :------------------ | :----- | :------ | :------------------------------------------------------------------------ | | `vectorIndexType` | string | `hnsw` | Optional. The index type - can be `hnsw`, `flat`, `dynamic`, or `hfresh`. | | `vectorIndexConfig` | object | - | Optional. Set parameters that are specific to the vector index type. |

How to select the index type

Generally, the `hnsw` index type is recommended for most use cases. The `flat` index type is recommended for use cases where the data the number of objects per index is low, such as in multi-tenancy cases. You can also opt for the `dynamic` index which will initially configure a `flat` index and once the object count exceeds a specified threshold it will automatically convert to an `hnsw` index. The `hfresh` index is a cluster-based index that uses HNSW for the centroid index. It can provide significant memory efficiency benefits while maintaining good search performance. See [this section](../../concepts/indexing/vector-index.md#which-vector-index-is-right-for-me) for more information about the different index types and how to choose between them.

If faster import speeds are desired, [asynchronous indexing](#asynchronous-indexing) allows de-coupling of indexing from object creation. ## HNSW index HNSW indexes are scalable and super fast at query time, but HNSW algorithms are costly when you add data during the index building process. ### HNSW index parameters Some HNSW parameters are mutable, but others cannot be modified after you create your collection. | Parameter | Type | Description | Default | Mutable | | :----------------------- | :------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------- | :------ | | `cleanupIntervalSeconds` | integer | Cleanup frequency. This value does not normally need to be adjusted. A higher value means cleanup runs less frequently, but it does more in a single batch. A lower value means cleanup is more frequent, but it may be less efficient on each run. | 300 | Yes | | `distance` | string | Distance metric. The metric that measures the distance between two arbitrary vectors. For available distance metrics, see [supported distance metrics](/weaviate/config-refs/distances.md). | `cosine` | No | | `ef` | integer | Balance search speed and recall. `ef` is the size of the dynamic list that the HNSW uses during search. Search is more accurate when `ef` is higher, but it is also slower. `ef` values greater than 512 show diminishing improvements in recall.

Dynamic `ef`. Weaviate automatically adjusts the `ef` value and creates a dynamic `ef` list when `ef` is set to -1. For more details, see [dynamic ef](../../concepts/indexing/vector-index.md#dynamic-ef). | -1 | Yes | | `efConstruction` | integer | Balance index search speed and build speed. A high `efConstruction` value means you can lower your `ef` settings, but importing is slower.

`efConstruction` must be greater than 0. | 128 | No | | `HNSWGeoIndexEF` | integer | Balance geo index search speed and recall. This value controls the search depth for geo-based queries. | 800 | Yes | | `maxConnections` | integer | Maximum number of connections per element. `maxConnections` is the connection limit per layer for layers above the zero layer. The zero layer can have (2 \* `maxConnections`) connections.

`maxConnections` must be greater than 0. | 32 | No | | `dynamicEfMin` | integer | Lower bound for [dynamic `ef`](../../concepts/indexing/vector-index.md#dynamic-ef). Protects against a creating search list that is too short.

This setting is only used when `ef` is -1. | 100 | Yes | | `dynamicEfMax` | integer | Upper bound for [dynamic `ef`](../../concepts/indexing/vector-index.md#dynamic-ef). Protects against creating a search list that is too long.

If `dynamicEfMax` is higher than the limit, `dynamicEfMax` does not have any effect. In this case, `ef` is the limit.

This setting is only used when `ef` is -1. | 500 | Yes | | `dynamicEfFactor` | integer | Multiplier for [dynamic `ef`](../../concepts/indexing/vector-index.md#dynamic-ef). Sets the potential length of the search list.

This setting is only used when `ef` is -1. | 8 | Yes | | `filterStrategy` | string | The filter strategy to use for filtering the search results. The filter strategy can be set to [`acorn`](../../concepts/filtering.md#acorn) (default as of `v1.34`) or [`sweeping`](../../concepts/filtering.md#sweeping). | `acorn` | Yes | | `flatSearchCutoff` | integer | Optional. Threshold for the [flat-search cutoff](/weaviate/concepts/filtering.md#flat-search-cutoff). To force a vector index search, set `"flatSearchCutoff": 0`. | 40000 | Yes | | `skip` | boolean | When true, do not index the collection.

Weaviate decouples vector creation and vector storage. If you skip vector indexing, but a vectorizer is configured (or a vector is provided manually), Weaviate logs a warning each import.

To skip indexing and vector generation, set `"vectorizer": "none"` when you set `"skip": true`.

See [When to skip indexing](../../concepts/indexing/vector-index.md#when-to-skip-indexing). | `false` | No | | `vectorCacheMaxObjects` | integer | Maximum number of objects in the memory cache. By default, this limit is set to one trillion (`1e12`) objects when a new collection is created. For sizing recommendations, see [Vector cache considerations](../../concepts/indexing/vector-index.md#vector-cache-considerations). | `1e12` | Yes | | `rq` | object | Enable and configure [rotational quantization (RQ)](/weaviate/concepts/indexing/vector-index.md) compression.

For RQ configuration details, see [RQ configuration parameters](#rq-parameters). | -- | Yes | | `pq` | object | Enable and configure [product quantization (PQ)](/weaviate/concepts/indexing/vector-index.md) compression.

PQ assumes some data has already been loaded. You should have 10,000 to 100,000 vectors per shard loaded before you enable PQ.

For PQ configuration details, see [PQ configuration parameters](#pq-parameters). | -- | Yes | | `bq` | object | Enable and configure [binary quantization (BQ)](/weaviate/concepts/indexing/vector-index.md) compression.

For BQ configuration details, see [BQ configuration parameters](#bq-parameters). | -- | Yes | | `sq` | object | Enable and configure [scalar quantization (SQ)](/weaviate/concepts/indexing/vector-index.md) compression.

For SQ configuration details, see [SQ configuration parameters](#sq-parameters). | -- | Yes | ### Database parameters for HNSW Note that some database-level parameters are available to configure HNSW indexing behavior. - [`PERSISTENCE_HNSW_MAX_LOG_SIZE`](/deploy/configuration/env-vars/index.md#PERSISTENCE_HNSW_MAX_LOG_SIZE) is a database-level parameter that sets the maximum size of the HNSW write-ahead-log. The default value is `500MiB`. Increase this value to improve efficiency of the compaction process, but be aware that this will increase the memory usage of the database. Conversely, decreasing this value will reduce memory usage but may slow down the compaction process. Preferably, the `PERSISTENCE_HNSW_MAX_LOG_SIZE` should set to a value close to the size of the HNSW graph. - [`DEFAULT_QUANTIZATION`](/deploy/configuration/env-vars/index.md#DEFAULT_QUANTIZATION) is a database-level parameter that defines which quantization technique will be used by default when creating new collections. ### Tombstone cleanup parameters :::info Environment variable availability - `TOMBSTONE_DELETION_CONCURRENCY` is available in `v1.24.0` and up. - `TOMBSTONE_DELETION_MIN_PER_CYCLE` and `TOMBSTONE_DELETION_MAX_PER_CYCLE` are available in `v1.24.15` / `v1.25.2` and up. ::: Tombstones are records that mark deleted objects. In an HNSW index, tombstones are regularly cleaned up, triggered periodically by the `cleanupIntervalSeconds` parameter. As the index grows in size, the cleanup process may take longer to complete and require more resources. For very large indexes, this may cause performance issues. To control the number of tombstones deleted per cleanup cycle and prevent performance issues, set the [`TOMBSTONE_DELETION_MAX_PER_CYCLE` and `TOMBSTONE_DELETION_MIN_PER_CYCLE` environment variables](/deploy/configuration/env-vars/index.md#general). - Set `TOMBSTONE_DELETION_MIN_PER_CYCLE` to prevent occurrences of unnecessary cleanup cycles. - Set `TOMBSTONE_DELETION_MAX_PER_CYCLE` to prevent the cleanup process from taking too long and consuming too many resources. As an example, for a cluster with 300 million objects per shard, a `TOMBSTONE_DELETION_MIN_PER_CYCLE` value of 1000000 (1 million) and a `TOMBSTONE_DELETION_MAX_PER_CYCLE` value of 10000000 (10 million) may be good starting points. You can also set the `TOMBSTONE_DELETION_CONCURRENCY` environment variable to limit the number of threads used for tombstone cleanup. This can help prevent prevent the cleanup process from unnecessarily consuming too many resources, or the cleanup process from taking too long. The default value for `TOMBSTONE_DELETION_CONCURRENCY` is set to half the number of CPU cores available to Weaviate. In a cluster with a large number of cores, you may want to set `TOMBSTONE_DELETION_CONCURRENCY` to a lower value to prevent the cleanup process from consuming too many resources. Conversely, in a cluster with a small number of cores and a large number of deletions, you may want to set `TOMBSTONE_DELETION_CONCURRENCY` to a higher value to speed up the cleanup process. ### HNSW Configuration tips To determine reasonable settings for your use case, consider the following questions and compare your answers in the table below: 1. How many queries do you expect per second? 1. Do you expect a lot of imports or updates? 1. How high should the recall be? | Number of queries | Many imports or updates | Recall level | Configuration suggestions | | ----------------- | ----------------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | not many | no | low | This is the ideal scenario. Keep both the `ef` and `efConstruction` settings low. You don't need a big machine and you will still be happy with the results. | | not many | no | high | Here the tricky thing is that your recall needs to be high. Since you're not expecting a lot of requests or imports, you can increase both the `ef` and `efConstruction` settings. Keep increasing them until you are happy with the recall. In this case, you can get pretty close to 100%. | | not many | yes | low | Here the tricky thing is the high volume of imports and updates. Be sure to keep `efConstruction` low. Since you don't need a high recall, and you're not expecting a lot of queries, you can adjust the `ef` setting until you've reached the desired recall. | | not many | yes | high | The trade-offs are getting harder. You need high recall _and_ you're dealing with a lot of imports or updates. This means you need to keep the `efConstruction` setting low, but you can significantly increase your `ef` setting because your queries per second rate is low. | | many | no | low | Many queries per second means you need a low `ef` setting. Luckily you don't need high recall so you can significantly increase the `efConstruction` value. | | many | no | high | Many queries per second means a low `ef` setting. Since you need a high recall but you are not expecting a lot of imports or updates, you can increase your `efConstruction` until you've reached the desired recall. | | many | yes | low | Many queries per second means you need a low `ef` setting. A high number of imports and updates also means you need a low `efConstruction` setting. Luckily your recall does not have to be as close to 100% as possible. You can set `efConstruction` relatively low to support your input or update throughput, and you can use the `ef` setting to regulate the query per second speed. | | many | yes | high | Aha, this means you're a perfectionist _or_ you have a use case that needs the best of all three worlds. Increase your `efConstruction` value until you hit the time limit of imports and updates. Next, increase your `ef` setting until you reach your desired balance of queries per second versus recall.

While many people think they need maximize all three dimensions, in practice that's usually not the case. We leave it up to you to decide, and you can always ask for help in [our forum](https://forum.weaviate.io). | :::tip This set of values is a good starting point for many use cases. | Parameter | Value | | :--------------- | :---- | | `ef` | `64` | | `efConstruction` | `128` | | `maxConnections` | `32` | ::: ## Flat index Flat indexes are recommended for use cases where the number of objects per index is low, such as in multi-tenancy use cases. | Parameter | Type | Default | Changeable | Details | | :---------------------- | :------ | :------ | :--------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `vectorCacheMaxObjects` | integer | `1e12` | Yes | Maximum number of objects in the memory cache. By default, this limit is set to one trillion (`1e12`) objects when a new collection is created. For sizing recommendations, see [Vector cache considerations](../../concepts/indexing/vector-index.md#vector-cache-considerations). | | `bq` | object | -- | No | Enable and configure [binary quantization (BQ)](../../concepts/vector-quantization.md#binary-quantization) compression.

For BQ configuration details, see [BQ configuration parameters](#bq-parameters). | ## Dynamic index :::caution Experimental feature Available starting in `v1.25`. Dynamic indexing is an experimental feature. Use with caution. ::: import DynamicAsyncRequirements from "/_includes/dynamic-index-async-req.mdx"; Using the `dynamic` index will initially create a flat index and once the number of objects exceeds a certain threshold (by default 10,000 objects) it will automatically switch you over to an HNSW index. This is only a one-way switch that converts a flat index to a HNSW, the index does not support changing back to a flat index even if the object count goes below the threshold due to deletion. The goal of `dynamic` indexing is to shorten latencies during query time at the cost of a larger memory footprint. If your priority is the opposite (keeping memory low), consider the [HFresh index](#hfresh-index) instead. ### Dynamic index parameters | Parameter | Type | Default | Details | | :---------- | :------ | :----------- | :------------------------------------------------------------------------------------ | | `distance` | string | `cosine` | Distance metric. The metric that measures the distance between two arbitrary vectors. | | `hnsw` | object | default HNSW | [HNSW index configuration](#hnsw-index-parameters) to be used. | | `flat` | object | default Flat | [Flat index configuration](#flat-index) to be used. | | `threshold` | integer | 10000 | Threshold object count at which `flat` to `hnsw` conversion happens | ## HFresh index import HFreshStatus from "/_includes/feature-notes/hfresh_status.mdx"; HFresh is a cluster-based vector index based on the SPFresh algorithm. It uses an HNSW index for centroid search, providing a balance between memory efficiency and search performance. :::note Supported distance metrics HFresh only supports `cosine` and `l2-squared` distance metrics. Dot product is not supported. ::: ### HFresh index parameters | Parameter | Type | Default | Mutable | Details | | :----------------- | :------ | :------- | :------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `distance` | string | `cosine` | No | Distance metric. Only `cosine` and `l2-squared` are supported. | | `maxPostingSizeKB` | integer | `48` | Yes | Maximum size in KB for a posting list. Weaviate uses this value along with the vector dimensions to calculate the maximum number of vectors per posting. Min: `8`, Max: `1024`. Best set when you create the collection: an update is accepted but only affects newly-indexed data. Data that is already indexed is not re-partitioned. | | `replicas` | integer | `4` | No | Number of posting lists in which a vector is added. Min: `1`, Max: `10`. | | `searchProbe` | integer | `256` | Yes | Number of posting lists to search during a query. The default is `256` in `v1.36.20`, `v1.37.10`, `v1.38.2` and later. Earlier releases on each of those lines default to `64`. | | `rq` | object | -- | Partial | Rotational quantization (RQ) compression configuration. RQ is mandatory for HFresh and cannot be turned off. Its `rescoreLimit` (default `350`), the number of candidates rescored against uncompressed vectors, is mutable at runtime. | :::tip Tuning HFresh recall Start with the defaults. If recall is too low, increase `searchProbe` (search more posting lists per query) or the RQ `rescoreLimit` (rescore more candidates with full-precision vectors). Both are mutable at runtime and take effect **without reindexing**. ::: ## Quantization parameters ### RQ parameters The following parameters are available for RQ compression, under `vectorIndexConfig`: import RQParameters from "/_includes/configuration/rq-compression-parameters.mdx"; ### SQ parameters The following parameters are available for SQ compression, under `vectorIndexConfig`: import SQParameters from "/_includes/configuration/sq-compression-parameters.mdx"; ### PQ parameters The following parameters are available for PQ compression, under `vectorIndexConfig`: import PQParameters from "/_includes/configuration/pq-compression/parameters.mdx"; ### BQ parameters The following parameters are available for BQ compression, under `vectorIndexConfig`: import BQParameters from "/_includes/configuration/bq-compression-parameters.mdx"; ## Default quantization for new collections {#default-quantization} import DefaultQuantization from '/_includes/feature-notes/default-quantization.mdx'; Starting with Weaviate v1.33, you can set a quantization method that will be enabled by default for all new collections\*\*. Existing collections (for example restored from backups) are not affected and retain their original configuration. Set the [`DEFAULT_QUANTIZATION` environment variable](/docs/deploy/configuration/env-vars/index.md#DEFAULT_QUANTIZATION) before starting Weaviate to change the default quantization technique or to disable it. ## Configure semantic indexing Weaviate can generate vector embeddings for objects using [model provider integrations](/weaviate/model-providers/). For instance, text embedding integrations (e.g. `text2vec-cohere` for Cohere, or `text2vec-ollama` for Ollama) can generate vectors from text objects. Weaviate follows the collection configuration and a set of predetermined rules to vectorize objects. Unless specified otherwise in the collection definition, the default behavior is to: - Only vectorize properties with a string value (`text`, `text[]`, and `blob`, which is a base64-encoded string) unless [skipped](../../manage-collections/vector-config.mdx#property-level-settings). Other data types (such as `number`, `int`, `boolean`, `date`, and `object`) are not vectorized unless they are listed in `source_properties` (see [below](#specify-which-properties-to-vectorize)). - Sort properties in alphabetical (a-z) order before concatenating values - If `vectorizePropertyName` is `true` (`false` by default) prepend the property name to each property value - Join the (prepended) property values with spaces - Prepend the class name (unless `vectorizeClassName` is `false`) - Convert the produced string to lowercase For example, this data object, ```js Article = { summary: "Cows lose their jobs as milk prices drop", text: "As his 100 diary cows lumbered over for their Monday...", }; ``` will be vectorized as: ```md article cows lose their jobs as milk prices drop as his 100 diary cows lumbered over for their monday... ``` By default, the calculation includes the collection name and all property values, but the property names are not indexed. To configure vectorization behavior on a per-collection basis, use `vectorizeClassName`. To configure vectorization on a per-property basis, use `skip` and `vectorizePropertyName`. ### Specify which properties to vectorize To vectorize only a specific set of properties, set `source_properties` (the `properties` field of the vector configuration). Only the listed properties are then vectorized, in the order given. When `source_properties` is set, listed properties that are **not** text are also vectorized: `number`, `int`, `boolean`, `date`, `object`, and their array variants are converted to a string and concatenated into the input text. (Without `source_properties`, only `text`, `text[]`, and `blob` properties are vectorized. `uuid`, geo-coordinates, and phone-number properties are never vectorized.) :::caution `blob` properties are vectorized as text A `blob` value is a base64-encoded string, so an indexed `blob` property is vectorized like text, even without `source_properties`. To avoid sending a blob's base64 data to a text vectorizer, exclude it with `source_properties` or [`skip`](../../manage-collections/vector-config.mdx#property-level-settings). ::: ## Asynchronous indexing To enable asynchronous indexing, set the `ASYNC_INDEXING` environment variable to `true` in your Weaviate configuration (the `docker-compose.yml` file if you use Docker Compose). This setting enables asynchronous indexing for all collections.

Example Docker Compose configuration

```yaml --- services: weaviate: command: - --host - 0.0.0.0 - --port - "8080" - --scheme - http image: cr.weaviate.io/semitechnologies/weaviate:||site.weaviate_version|| restart: on-failure:0 ports: - 8080:8080 - 50051:50051 environment: QUERY_DEFAULTS_LIMIT: 25 QUERY_MAXIMUM_RESULTS: 10000 AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: "true" PERSISTENCE_DATA_PATH: "/var/lib/weaviate" CLUSTER_HOSTNAME: "node1" AUTOSCHEMA_ENABLED: "false" ASYNC_INDEXING: "true" ```

To get the index status, check the [node status](/deploy/configuration/status.md#cluster-node-data) endpoint.

Node status example usage

The `nodes/shards/vectorQueueLength` field shows the number of objects that still have to be indexed. import Nodes from "/_includes/code/nodes.mdx"; Then, you can check the status of the vector index queue by inspecting the output.
The `vectorQueueLength` field will show the number of remaining objects to be indexed. In the example below, the vector index queue has 425 objects remaining to be indexed on the `TestArticle` shard, out of a total of 1000 objects. ```json { "nodes": [ { "batchStats": { "ratePerSecond": 0 }, "gitHash": "e6b37ce", "name": "weaviate-0", "shards": [ { "class": "TestArticle", "name": "nq1Bg9Q5lxxP", "objectCount": 1000, // highlight-start "vectorIndexingStatus": "INDEXING", "vectorQueueLength": 425 // highlight-end } ], "stats": { "objectCount": 1000, "shardCount": 1 }, "status": "HEALTHY", "version": "1.22.1" } ] } ```

## Multiple vector embeddings (named vectors) import MultiVectorSupport from "/_includes/multi-vector-support.mdx"; ## Further resources - [Concepts: Vector index](../../concepts/indexing/vector-index.md) - [How-to: Configure collections](../../manage-collections/vector-config.mdx) ## Questions and feedback import DocsFeedback from "/_includes/docs-feedback.mdx"; --- ### Weaviate/Configuration/Authz Authn (docs/weaviate/configuration/authz-authn.md) --- title: Authentication and authorization sidebar_position: 30 image: og/docs/configuration.jpg # tags: ['authentication'] --- :::info Authentication and authorization Authentication and authorization are closely related concepts, and sometimes abbreviated as `AuthN` and `AuthZ`. Authentication (`AuthN`) is the process of verifying the identity of a user, while authorization (`AuthZ`) is the process of determining what permissions the user has. ::: ## Authentication Weaviate controls access through user authentication via API keys or OpenID Connect (OIDC), with an option for anonymous access. Users can then be assigned different [authorization](/deploy/configuration/authorization.md) levels, as shown in the diagram below. ```mermaid flowchart LR %% Define main nodes Request["Client
Request"] AuthCheck{"AuthN
Enabled?"} AccessCheck{"Check
AuthZ"} Access["✅ Access
Granted"] Denied["❌ Access
Denied"] %% Define authentication method nodes subgraph auth ["AuthN"] direction LR API["API Key"] OIDC["OIDC"] AuthResult{"Success?"} end %% Define connections Request --> AuthCheck AuthCheck -->|"No"| AccessCheck AuthCheck -->|"Yes"| auth API --> AuthResult OIDC --> AuthResult AuthResult -->|"Yes"| AccessCheck AuthResult -->|"No"| Denied AccessCheck -->|"Pass"| Access AccessCheck -->|"Fail"| Denied %% Style nodes style Request fill:#ffffff,stroke:#B9C8DF,color:#130C49 style AuthCheck fill:#ffffff,stroke:#B9C8DF,color:#130C49 style AccessCheck fill:#ffffff,stroke:#B9C8DF,color:#130C49 style Access fill:#ffffff,stroke:#B9C8DF,color:#130C49 style Denied fill:#ffffff,stroke:#B9C8DF,color:#130C49 style API fill:#ffffff,stroke:#B9C8DF,color:#130C49 style OIDC fill:#ffffff,stroke:#B9C8DF,color:#130C49 style AuthResult fill:#ffffff,stroke:#B9C8DF,color:#130C49 %% Style subgraph style auth fill:#ffffff,stroke:#130C49,stroke-width:2px,color:#130C49 ``` For example, a user logging in with the API key `jane-secret` may be granted administrator permissions, while another user logging in with the API key `ian-secret` may be granted read-only permissions. API key and OIDC authentication can be both enabled at the same time. We recommend using a client library to authenticate against Weaviate. See [How-to: Connect](docs/weaviate/connections/index.mdx) pages for more information. :::info What about Weaviate Cloud (WCD)? For Weaviate Cloud (WCD) instances, authentication is pre-configured with API key access. You can [authenticate against Weaviate](../connections/connect-cloud.mdx) by [creating new API keys](/cloud/manage-clusters/connect.mdx). ::: ### API key API key authentication is the simplest method to authenticate against Weaviate. Each user is assigned a unique API key, which is passed in the request header. For details on configuring API key authentication, see the [authentication guide](/deploy/configuration/authentication.md#api-key-authentication). ### OpenID Connect (OIDC) [OpenID Connect (OIDC)](/deploy/configuration/authentication.md#oidc-authentication) enables authentication through an external identity provider (e.g., Okta, Azure AD, Google). OIDC supports multiple flows such as client credentials, resource owner password, and hybrid flow. For details on configuring OIDC and working with tokens, see the [OIDC configuration guide](/deploy/configuration/oidc). ### Anonymous access [Anonymous access](/deploy/configuration/authentication.md#anonymous-access) allows unauthenticated requests. This is **strongly discouraged** except for local development or evaluation purposes, as it bypasses all identity verification. ## Authorization Weaviate provides differentiated access through authorization levels, based on the user's [authentication](#authentication) status. The following authorization schemes are available: ### RBAC (recommended) [Role-Based Access Control (RBAC)](./rbac/index.mdx) provides fine-grained control over user permissions. With RBAC, you define roles with specific permissions and assign them to users. This is the **recommended authorization scheme** for production deployments. RBAC supports: - **Predefined roles**: `root` (full access) and `viewer` (read-only access) - **Custom roles**: Create roles with specific permissions for collections, objects, tenants, backups, and more - **Granular permissions**: Control access at the collection, tenant, and operation level using name filters and regex patterns See [RBAC Overview](./rbac/index.mdx) for the full permissions model, and [Configuring RBAC](/deploy/configuration/configuring-rbac.md) for setup instructions. ### Admin list (legacy) :::caution Prefer RBAC over Admin list The Admin list authorization scheme only provides coarse-grained access control (admin or read-only). Use [RBAC](#rbac-recommended) instead for production deployments, as it provides much more flexible and secure permission management. ::: The [Admin list](../../deploy/configuration/authorization.md#admin-list) scheme assigns users as either admin (full access) or read-only. [Anonymous users](../../deploy/configuration/authorization.md#anonymous-users) can optionally be granted permissions. ### Undifferentiated access With [undifferentiated access](../../deploy/configuration/authorization.md#undifferentiated-access), all authenticated users have full access. This is only suitable for development or trusted single-user environments. ## Further resources - [Configuration: Authentication](/deploy/configuration/authentication.md) - [Configuration: Authorization](/deploy/configuration/authorization.md) - [Configuration: OIDC](/deploy/configuration/oidc.md) - [Configuration: RBAC](/weaviate/configuration/rbac/index.mdx) - [Configuration: Environment variables - Authentication and Authorization](/deploy/configuration/env-vars/index.md#authentication-and-authorization) - [Weaviate MCP server](/weaviate/configuration/mcp-server.mdx) (authenticates via API key and respects RBAC permissions) ## Questions and feedback import DocsFeedback from '/\_includes/docs-feedback.mdx'; --- ### Weaviate/Configuration/Index (docs/weaviate/configuration/index.mdx) --- title: How to configure Weaviate sidebar_position: 0 image: og/docs/configuration.jpg hide_table_of_contents: true # tags: ['configuration'] --- Configure and manage key operational aspects of your Weaviate instance with these guides: import CardsSection from "/src/components/CardsSection"; export const configOpsData = [ { title: "Authentication and authorization", description: "Set up methods to authenticate and to verify user identities before granting access.", link: "/weaviate/configuration/authz-authn", icon: "fas fa-key", }, { title: "Compression/Quantization (PQ/BQ/RQ/SQ)", description: "Reduce memory usage with Product, Binary, Rotational or Scalar Quantization techniques.", link: "/weaviate/configuration/compression", icon: "fas fa-compress-alt", }, { title: "Modules", description: "Explore Weaviate's ecosystem of vectorizer, generative (RAG), and other modules.", link: "/weaviate/configuration/modules", icon: "fas fa-puzzle-piece", }, { title: "MCP server", description: "Enable the built-in Model Context Protocol server so LLMs and IDE assistants can interact with your Weaviate instance.", link: "/weaviate/configuration/mcp-server", icon: "fas fa-robot", }, ];

## Deployment configuration These guides cover server-level configuration for your Weaviate deployment: export const deployConfigData = [ { title: "Database configuration", description: "Configure environment variables, ports, and runtime settings.", link: "/deploy/configuration/env-vars", icon: "fas fa-cog", }, { title: "Monitoring and logging", description: "Collect metrics, configure logging, and monitor cluster health.", link: "/deploy/configuration/monitoring", icon: "fas fa-chart-bar", }, { title: "RBAC", description: "Configure Role-Based Access Control for fine-grained permissions.", link: "/weaviate/configuration/rbac", icon: "fas fa-user-shield", }, { title: "Replication and scaling", description: "Set up data replication and scaling across nodes for high availability.", link: "/deploy/configuration/replication", icon: "fas fa-copy", }, { title: "Storage and backups", description: "Configure backup, restore, and persistence for your Weaviate instance.", link: "/deploy/configuration/backups", icon: "fas fa-save", }, ];

## Questions and feedback import DocsFeedback from "/_includes/docs-feedback.mdx"; --- ### Weaviate/Configuration/Mcp Server (docs/weaviate/configuration/mcp-server.mdx) --- title: Weaviate MCP server description: Enable and configure the Weaviate Model Context Protocol (MCP) server so LLMs and IDE assistants like Claude Desktop, Claude Code, Cursor, and VS Code can inspect schemas, run vector and hybrid searches, and modify objects in your Weaviate instance over HTTP. image: og/docs/configuration.jpg faq: - question: Does Weaviate offer an MCP server? answer: >- Yes. Weaviate ships a built-in Model Context Protocol (MCP) server (generally available as of v1.38; introduced in v1.37.1). Enable it with the MCP_SERVER_ENABLED=true environment variable; it runs on the same port as the REST API at /v1/mcp. - question: Is the MCP server an external library? answer: >- No. It's built into the Weaviate Server binary, not a separate package you install. Setting MCP_SERVER_ENABLED=true exposes the MCP endpoint on the same port as the REST API; nothing extra to run or deploy. The separate "Weaviate Docs MCP server" (Kapa-powered, serves documentation to LLMs) is a distinct product. - question: Which tools does the Weaviate MCP server expose? answer: >- Four tools, gated by RBAC permissions. They are weaviate-collections-get-config (inspect collection schemas), weaviate-tenants-list (list tenants in a multi-tenant collection), weaviate-query-hybrid (run hybrid vector + keyword searches), and weaviate-objects-upsert (create or update objects, requires write access). - question: How do I request a new MCP tool or feature? answer: >- Open a feature request on the Weaviate GitHub repo at https://github.com/weaviate/weaviate/issues/new/choose. Pick the "Feature request" template and describe the tool, parameter, or capability you'd like the MCP server to expose, with a concrete use case. # tags: ["mcp", "configuration"] --- import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem"; import MCPPreview from "/_includes/feature-notes/mcp.mdx"; The Weaviate [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) server is an implementation of the open standard that enables Large Language Models (LLMs) to interact securely with your Weaviate instance. Instead of pasting context manually, MCP allows compatible clients (like Claude Desktop or IDEs) to directly "see" and interact with your database. Weaviate implements this as a Streamable HTTP server that runs on the same port as the main Weaviate REST API. It exposes tools to inspect schemas, search data (vector/hybrid), and modify objects, governed by Weaviate's authentication and authorization. --- ## Using the Weaviate MCP server The Weaviate MCP server by default runs at `http://localhost:8080/v1/mcp` if enabled and supports authentication via Bearer tokens (API Keys). To get started: 1. [Enable the MCP server](#environment-variables) (and optionally write access) through environment variables. 2. [Ensure your API key has the right permissions](#permissions) if using RBAC. 3. [Connect your MCP client](#mcp-client) using the REST API host and port. You can also optionally [customize tool descriptions](#custom-tool-descriptions) to tailor the LLM's understanding of your workflow. #### Connect your MCP client {#mcp-client} Run the following command in your terminal to add the server ([Claude Code MCP docs](https://docs.anthropic.com/en/docs/claude-code/mcp)): ```bash claude mcp add-json weaviate-local '{"type":"http","url":"http://localhost:8080/v1/mcp","headers":{"Authorization":"Bearer YOUR_API_KEY"}}' ``` _If anonymous access is enabled, you can omit the `headers` field._ [Claude Desktop](https://claude.ai/download) does not natively support Streamable HTTP transport. Use [`mcp-proxy`](https://github.com/sparfenyuk/mcp-proxy) to bridge between Claude Desktop's `stdio` transport and the Weaviate MCP server. **Config Location:** - macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` - Windows: `%APPDATA%\Claude\claude_desktop_config.json` ```json { "mcpServers": { "weaviate-local": { "command": "mcp-proxy", "args": [ "http://localhost:8080/v1/mcp", "--headers", "Authorization", "Bearer YOUR_API_KEY", "--transport", "streamablehttp" ] } } } ``` _Note: Replace `YOUR_API_KEY` with your actual Weaviate API key. If anonymous access is enabled, you can omit the `--headers` arguments._ Add the following to your `.cursor/mcp.json` file ([Cursor MCP docs](https://docs.cursor.com/context/model-context-protocol)). Cursor supports Streamable HTTP connections directly. ```json { "mcpServers": { "weaviate-local": { "type": "streamable-http", "url": "http://localhost:8080/v1/mcp", "headers": { "Authorization": "Bearer YOUR_API_KEY" } } } } ``` **Prerequisites:** VS Code 1.102+ with GitHub Copilot enabled ([VS Code MCP docs](https://code.visualstudio.com/docs/copilot/chat/mcp-servers)). Create or edit the `mcp.json` file in your workspace `.vscode` folder: ```json { "servers": { "weaviate-local": { "type": "streamable-http", "url": "http://localhost:8080/v1/mcp", "headers": { "Authorization": "Bearer YOUR_API_KEY" } } } } ``` Most MCP clients support Streamable HTTP. Use the following connection details: - **URL:** `http://localhost:8080/v1/mcp` - **Transport:** Streamable HTTP - **Auth Header:** `Authorization: Bearer ` Standard JSON configuration format: ```json { "mcpServers": { "weaviate-local": { "url": "http://localhost:8080/v1/mcp", "type": "streamable-http" } } } ``` --- ## Configuration The MCP server is built into Weaviate but is **disabled by default** for security. It is served at the `/v1/mcp` endpoint on the same port as the REST API (default `8080`). ### Environment variables To enable and configure the server, set the following [environment variables](/docs/deploy/configuration/env-vars/index.md) in your Weaviate configuration (e.g., `docker-compose.yml`): | Environment Variable | Default | Runtime-configurable | Description | | --------------------------------------------------------------------------------------------------- | ------- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`MCP_SERVER_ENABLED`](/deploy/configuration/env-vars#MCP_SERVER_ENABLED) | `false` | from `v1.38` | **Required.** Set to `true` to start the MCP server. | | [`MCP_SERVER_WRITE_ACCESS_ENABLED`](/deploy/configuration/env-vars#MCP_SERVER_WRITE_ACCESS_ENABLED) | `false` | from `v1.38` | When `true`, enables write tools (`weaviate-objects-upsert`). Default is read-only. | | [`MCP_SERVER_CONFIG_PATH`](/deploy/configuration/env-vars#MCP_SERVER_CONFIG_PATH) | `""` | No | Path to a YAML file for customizing tool descriptions (useful for prompt engineering the LLM's understanding of your specific data). If not provided or file malformed, the default descriptions from the [source code](https://github.com/weaviate/weaviate/tree/main/adapters/handlers/mcp) will be used. Tool descriptions are baked into the tool schemas at registration, so this flag remains startup-only. | ### Permissions If you use [RBAC](/weaviate/configuration/rbac/index.mdx) with fine-grained permissions instead of root access, the role assigned to your API key must include the appropriate MCP permissions. Without them, tool calls are rejected.

Per-tool permissions

| Tool | MCP permissions required | Additional collection permissions | | --------------------------------- | -------------------------- | --------------------------------- | | `weaviate-collections-get-config` | `read_mcp` | `read_collections` | | `weaviate-tenants-list` | `read_mcp` | `read_data` | | `weaviate-query-hybrid` | `read_mcp` | `read_data` | | `weaviate-objects-upsert` | `create_mcp`, `update_mcp` | `create_data`, `update_data` |

### Custom tool descriptions You can override the default descriptions provided to the LLM by mounting a YAML or JSON file at `MCP_SERVER_CONFIG_PATH`. ```yaml # mcp-config.yaml tools: weaviate-query-hybrid: description: "Perform a vector or keyword search on a collection." arguments: query: "The natural language search query to find relevant objects." alpha: "0.0 = pure keyword (BM25), 1.0 = pure vector. Defaults to 0.75." ``` ```json { "tools": { "weaviate-query-hybrid": { "description": "Perform a vector or keyword search on a collection.", "arguments": { "query": "The natural language search query to find relevant objects.", "alpha": "0.0 = pure keyword (BM25), 1.0 = pure vector. Defaults to 0.75." } } } } ``` --- ## Tools The server exposes different tools depending on your configuration. These are all the available tools: - `weaviate-collections-get-config` - `weaviate-tenants-list` - `weaviate-query-hybrid` - `weaviate-objects-upsert` ### `weaviate-collections-get-config` Retrieves the schema configuration for collections. **Arguments:** - `collection_name` (string, optional): Specific collection to retrieve. If omitted, returns all. **Returns:** JSON object containing class names, properties, and vectorizer settings. ### `weaviate-tenants-list` Lists tenants for multi-tenant collections. **Arguments:** - `collection_name` (string, required): The collection to inspect. **Returns:** List of tenants and their activity status (`ACTIVE` or `INACTIVE`, and `OFFLOADED` for tenants that have been offloaded to cold storage). ### `weaviate-query-hybrid` Performs a hybrid search combining vector similarity and keyword matching (BM25). **Arguments:** - `query` (string, required): The natural language search text. - `collection_name` (string, required): The collection to search. - `tenant_name` (string, optional): Tenant to search within for multi-tenant collections. - `alpha` (float, optional): Weighting. `0.0` = pure keyword search, `1.0` = pure vector search. Default is `0.75`, the same default as a regular [hybrid search](/weaviate/api/graphql/search-operators#hybrid). - `limit` (int, optional): Max results. - `target_vectors` (array, optional): Named vectors to use for vector search. - `target_properties` (array, optional): Properties to search with BM25. If omitted, searches all text properties. - `return_properties` (array, optional): Properties to include in results. - `return_metadata` (array, optional): Metadata fields to return (e.g., `id`, `vector`, `distance`, `score`, `creationTimeUnix`, `lastUpdateTimeUnix`). - `filters` (object, optional): A [where filter](/weaviate/api/graphql/filters.md) applied before scoring. A leaf filter is an object with `path` (an array of property names), `operator`, and a typed value field. The typed value field is one of `valueText`, `valueInt`, `valueNumber`, `valueBoolean`, `valueDate`, the corresponding `value*Array` field for a `Contains*` operator, or `valueGeoRange` for `WithinGeoRange`. Combine leaves with `{"operator": "And" | "Or", "operands": [ ... ]}`, nested to any depth. The supported operators are `And`, `Or`, `Not`, `Equal`, `NotEqual`, `Like`, `GreaterThan`, `GreaterThanEqual`, `LessThan`, `LessThanEqual`, `ContainsAny`, `ContainsAll`, `ContainsNone`, `WithinGeoRange`, and `IsNull`. See [Concepts: Filtering](/weaviate/concepts/filtering.md) for how filters interact with search. **Returns:** Ranked objects with similarity scores and distances. ### `weaviate-objects-upsert` :::info MCP write access This tool is only available if `MCP_SERVER_WRITE_ACCESS_ENABLED=true`. ::: Batch inserts or updates objects. **Arguments:** - `collection_name` (string, required): The collection to upsert into. - `tenant_name` (string, optional): Tenant for multi-tenant collections. - `objects` (array, required): List of objects containing `properties` and optional `uuid` or `vectors`. **Returns:** Array of results containing UUIDs or error messages per object. --- ## Monitoring From `v1.38`, the MCP server emits six Prometheus metrics under the `weaviate_mcp_*` prefix on the existing [Prometheus endpoint](/deploy/configuration/monitoring.md). Use them to track tool traffic, latency, auth failures, and the live state of the write-access flag. See [Monitoring → MCP server](/deploy/configuration/monitoring.md#mcp-server) for the full label catalogue and the rest of Weaviate's Prometheus surface. --- ## Further resources - [Vibe coding - Best practices](../best-practices/code-generation.md) - [Weaviate Docs MCP server](../mcp/docs-mcp-server.mdx) ## Questions and feedback import DocsFeedback from "/_includes/docs-feedback.mdx"; --- ### Weaviate/Configuration/Modules (docs/weaviate/configuration/modules.md) --- title: Modules sidebar_position: 11 image: og/docs/configuration.jpg # tags: ['configuration', 'modules'] --- Weaviate's functionality can be customized by using [modules](/weaviate/concepts/modules.md). This page explains how to enable and configure modules. ## Instance-level configuration At the instance (i.e. Weaviate cluster) level, you can: - Enable modules - Configure the default vectorizer module - Configure module-specific variables (e.g. API keys), where applicable This can be done by setting the appropriate [environment variables](/deploy/configuration/env-vars/index.md) as shown below. :::tip What about WCD? Weaviate Cloud (WCD) instances come with modules pre-configured. See [this page](/cloud/manage-clusters/status#enabled-modules) for details. ::: ### Enable individual modules You can enable modules by specifying the list of modules in the `ENABLE_MODULES` variable. For example, this code enables the `text2vec-transformers` module. ```yaml services: weaviate: environment: ENABLE_MODULES: 'text2vec-transformers' ``` To enable multiple modules, add them in a comma-separated list. This example code enables the `'text2vec-huggingface`, `generative-cohere`, and `qna-openai` modules. ```yaml services: weaviate: environment: ENABLE_MODULES: 'text2vec-huggingface,generative-cohere,qna-openai' ``` ### Enable all API-based modules All API-based model integrations are available by default starting with Weaviate `v1.33`. For older versions, you can enable all API-based modules by setting the `ENABLE_API_BASED_MODULES` variable to `true`. This will enable all API-based [model integrations](../model-providers/index.md), such as those for Anthropic, Cohere, OpenAI and so on by enabling the relevant modules. These modules are lightweight, so enabling them all will not significantly increase resource usage. ```yaml services: weaviate: environment: ENABLE_API_BASED_MODULES: 'true' ``` The list of API-based modules can be found on the [model provider integrations page](../model-providers/index.md#api-based). You can also inspect the [source code](https://github.com/weaviate/weaviate/blob/main/adapters/handlers/rest/configure_api.go) where the list is defined. Enabling individual modules can be combined with the API-based modules. For example, since API-based modules are enabled by default from `v1.33`, the example below enables the Ollama modules and the `backup-s3` module alongside them. ```yaml services: weaviate: environment: ENABLE_MODULES: 'text2vec-ollama,generative-ollama,backup-s3' ``` To opt out of the API-based modules from `v1.33` onwards, set `API_BASED_MODULES_DISABLED` to `true`. The older `ENABLE_API_BASED_MODULES` variable is no longer read. Note that enabling multiple vectorizer (e.g. `text2vec`, `multi2vec`) modules will disable the [`Explore` functionality](../api/graphql/explore.md). If you need to use `Explore`, you should only enable one vectorizer module. ### Module-specific variables You may need to specify additional environment variables to configure each module where applicable. For example, the `backup-s3` module requires the backup S3 bucket to be set via `BACKUP_S3_BUCKET`, and the `text2vec-contextionary` module requires the inference API location via `TRANSFORMERS_INFERENCE_API`. Refer to the individual [module documentation](../modules/index.md) for more details. ## Vectorizer modules The [vectorization integration](../model-providers/index.md) enable Weaviate to vectorize data at import, and to perform [`near`](../search/similarity.md) searches such as `nearText` or `nearImage`. :::info List of available vectorizer integrations Can be found [in this section](../model-providers/index.md). ::: ### Enable vectorizer modules You can enable vectorizer modules by adding them to the `ENABLE_MODULES` environment variable. For example, this code enables the `text2vec-cohere`, `text2vec-huggingface`, and `text2vec-openai` vectorizer modules. ```yaml services: weaviate: environment: ENABLE_MODULES: 'text2vec-cohere,text2vec-huggingface,text2vec-openai' ``` ### Default vectorizer module You can specify a default vectorization module with the `DEFAULT_VECTORIZER_MODULE` variable as below. If a default vectorizer module is not set, you must set a vectorizer in the schema before you can use `near` or vectorization at import time. This code sets `text2vec-huggingface` as the default vectorizer. Thus, `text2vec-huggingface` module will be used unless another vectorizer is specified for that class. ``` yaml services: weaviate: environment: DEFAULT_VECTORIZER_MODULE: text2vec-huggingface ``` ## Generative model integrations The [generative model integrations](../model-providers/index.md) enable [retrieval augmented generation](../search/generative.md) functions. ### Enable a generative module You can enable generative modules by adding the desired module to the `ENABLE_MODULES` environment variable. For example, this code enables the `generative-cohere` module and the `text2vec-huggingface` vectorizer module. ```yaml services: weaviate: environment: ENABLE_MODULES: 'text2vec-huggingface,generative-cohere' ``` :::tip `generative` module selection unrelated to `text2vec` module selection Your choice of the `text2vec` module does not restrict your choice of `generative` module, or vice versa. ::: ## Tenant offload modules Tenants can be offloaded to cold storage to reduce memory and disk usage, and onloaded back when needed. See the [dedicated page on tenant offloading](/deploy/configuration/tenant-offloading.md) for more information on how to configure Weaviate for tenant offloading. For information on how to offload and onload tenants, see [How-to: manage tenant states](../manage-collections/tenant-states.mdx). ## Custom modules See [here](../modules/custom-modules.md) how you can create and use your own modules. ## Usage modules The [usage module](../modules/usage-modules.md) collects and uploads usage analytics to GCS or S3. ## Related pages - [Concepts: Modules](../concepts/modules.md) - [References: Modules](../modules/index.md) ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Configuration/Compression/Bq Compression (docs/weaviate/configuration/compression/bq-compression.md) --- title: Binary Quantization (BQ) image: og/docs/configuration.jpg # tags: ['configuration', 'compression', 'bq'] --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import FilteredTextBlock from '@site/src/components/Documentation/FilteredTextBlock'; import PyCode from '!!raw-loader!/\_includes/code/howto/configure.bq-compression.py'; import TSCode from '!!raw-loader!/\_includes/code/howto/configure.bq-compression.ts'; import TSCodeBQOptions from '!!raw-loader!/\_includes/code/howto/configure.bq-compression.options.ts'; import GoCode from '!!raw-loader!/\_includes/code/howto/go/docs/configure/compression.bq_test.go'; import JavaV6Code from "!!raw-loader!/\_includes/code/java-v6/src/test/java/ConfigureBQTest.java"; import CSharpCode from "!!raw-loader!/\_includes/code/csharp/ConfigureBQTest.cs"; import CompressionByDefault from '/\_includes/compression-by-default.mdx'; [**Binary quantization (BQ)**](/weaviate/concepts/vector-quantization#binary-quantization) is a vector compression technique that can reduce the size of a vector. To use BQ, enable it as shown below and add data to the collection.

Additional information

- How to [set the index type](../../manage-collections/vector-config.mdx#set-vector-index-type)

## Enable compression for new collection BQ can be enabled at collection creation time through the collection definition: ## Enable compression for existing collection import BqPostCreation from '/_includes/feature-notes/bq-post-creation.mdx'; BQ can also be enabled for an existing collection by updating the collection definition: ## BQ parameters The following parameters are available for BQ compression, under `vectorIndexConfig`: import BQParameters from '/\_includes/configuration/bq-compression-parameters.mdx' ; For example: ## Additional considerations ### Multiple vector embeddings (named vectors) import NamedVectorCompress from '/\_includes/named-vector-compress.mdx'; ### Multi-vector embeddings (ColBERT, ColPali, etc.) import MultiVectorCompress from '/\_includes/multi-vector-compress.mdx'; ## Further resources - [Starter guides: Compression](/docs/weaviate/starter-guides/managing-resources/compression.mdx) - [Reference: Vector index](/weaviate/config-refs/indexing/vector-index.mdx) - [Concepts: Vector quantization](/docs/weaviate/concepts/vector-quantization.md) - [Concepts: Vector index](/weaviate/concepts/indexing/vector-index.md) ## Questions and feedback import DocsFeedback from '/\_includes/docs-feedback.mdx'; --- ### Weaviate/Configuration/Compression/Index (docs/weaviate/configuration/compression/index.md) --- title: Compression sidebar_position: 5 image: og/docs/configuration.jpg # tags: ['configuration', 'compression', 'pq'] --- Uncompressed vectors can be large. Compressed vectors lose some information, but they use fewer resources and can be very cost effective. ## Vector quantization To balance resource costs and system performance, consider one of these options: - **[Rotational Quantization (RQ)](rq-compression.md)** (_recommended_) - **[Product Quantization (PQ)](pq-compression.md)** - **[Binary Quantization (BQ)](bq-compression.md)** - **[Scalar Quantization (SQ)](sq-compression.md)** You can also [disable quantization](uncompressed.md) for a collection. import CompressionByDefault from '/_includes/compression-by-default.mdx'; ## Multi-vector encoding Aside from quantization, Weaviate also offers encodings for multi-vector embeddings: - **[MUVERA encoding](./multi-vectors.md)** --- ### Weaviate/Configuration/Compression/Multi Vectors (docs/weaviate/configuration/compression/multi-vectors.md) --- title: Multi-vector encodings image: og/docs/configuration.jpg # tags: ['configuration', 'compression'] --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import FilteredTextBlock from '@site/src/components/Documentation/FilteredTextBlock'; import PyCode from '!!raw-loader!/\_includes/code/howto/manage-data.collections.py'; import TSCode from '!!raw-loader!/\_includes/code/howto/manage-data.collections.ts'; import JavaV6Code from "!!raw-loader!/\_includes/code/java-v6/src/test/java/ManageCollectionsTest.java"; import CSharpCode from "!!raw-loader!/\_includes/code/csharp/ManageCollectionsTest.cs"; Multi-vector embeddings represent a single data object, like a document or image, using a set of multiple vectors rather than a single vector. This approach allows for a more granular capture of semantic information, as each vector can represent different parts of the object. However, this leads to a significant increase in memory consumption, as multiple vectors are stored for each item. Compression techniques become especially crucial for multi-vector systems to manage storage costs and improve query latency. **Encodings** transform the entire set of multi-vectors into a new, more compact single vector representation while aiming to preserve semantic relationships. ## MUVERA encoding **MUVERA**, which stands for _Multi-Vector Retrieval via Fixed Dimensional Encodings_, tackles the higher memory usage and slower processing times of multi-vector embeddings by encoding them into single, fixed-dimensional vectors. This leads to reduced memory usage compared to traditional multi-vector approaches. :::tip Weaviate Embeddings multimodal model The [Weaviate Embeddings multimodal model](/weaviate/model-providers/weaviate/embeddings-multimodal) (`ModernVBERT/colmodernvbert`) produces multi-vector embeddings for visual document retrieval. We recommend enabling MUVERA encoding when using this model to optimize memory usage. ::: ```go // Go support coming soon ``` The final dimensionality of the MUVERA encoded vector will be `repetitions * 2^ksim * dprojections`. Carefully tuning these parameters is crucial to balance memory usage and retrieval accuracy. These parameters can be used to fine-tune MUVERA: - **`ksim`** (`int`, default: `4`): The number of Gaussian vectors sampled for the SimHash partitioning function. This parameter determines the number of bits in the hash, and consequently, the number of buckets created in the space partitioning step. The total number of buckets will be $2^{ksim}$. A higher value of `ksim` leads to a finer-grained partitioning of the embedding space, potentially improving the accuracy of the approximation but also increasing the dimensionality of the intermediate encoded vectors. - **`dprojections`** (`int`, default: `16`): The dimensionality of the sub-vectors after the random linear projection in the dimensionality reduction step. After partitioning the multi-vector embedding into buckets, each bucket's aggregated vector is projected down to `dprojections` dimensions using a random matrix. A smaller value of `dprojections` helps in reducing the overall dimensionality of the final fixed-dimensional encoding, leading to lower memory consumption but potentially at the cost of some information loss and retrieval accuracy. - **`repetitions`** (`int`, default: `10`): The number of times the space partitioning and dimensionality reduction steps are repeated. Each repetition captures a different perspective of the multi-vector embedding and can improve the robustness and accuracy of the final fixed-dimensional encoding. The resulting single vectors from each repetition are concatenated. A higher number of repetitions increases the dimensionality of the final encoding but can lead to better approximation of the original multi-vector similarity. :::note Quantization Quantization is also available as a compression technique for multi-vector embeddings. It reduces the memory footprint of individual vectors by approximating their values with less precision. Just like with single vectors, multi-vectors support [PQ](./pq-compression.md), [BQ](./bq-compression.md), [RQ](./rq-compression.md) and [SQ](./sq-compression.md) quantization. ::: ## Further resources - [How-to: Manage collections](../../manage-collections/vector-config.mdx#define-multi-vector-embeddings-eg-colbert-colpali) - [Concepts: Vector quantization](../../concepts/vector-quantization.md) ## Questions and feedback import DocsFeedback from '/\_includes/docs-feedback.mdx'; --- ### Weaviate/Configuration/Compression/Pq Compression (docs/weaviate/configuration/compression/pq-compression.md) --- title: Product Quantization (PQ) image: og/docs/configuration.jpg # tags: ['configuration', 'compression', 'pq'] --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import FilteredTextBlock from '@site/src/components/Documentation/FilteredTextBlock'; import PyCode from '!!raw-loader!/\_includes/code/howto/configure.pq-compression.py'; import TSCodeAutoPQ from '!!raw-loader!/\_includes/code/howto/configure.pq-compression.autopq.ts'; import TSCodeManualPQ from '!!raw-loader!/\_includes/code/howto/configure.pq-compression.manual.ts'; import GoCode from '!!raw-loader!/\_includes/code/howto/go/docs/configure/compression.pq_test.go'; import JavaV6Code from "!!raw-loader!/\_includes/code/java-v6/src/test/java/ConfigurePQTest.java"; import CSharpCode from "!!raw-loader!/\_includes/code/csharp/ConfigurePQTest.cs"; import CompressionByDefault from '/\_includes/compression-by-default.mdx'; import PQOverview from '/\_includes/configuration/pq-compression/overview-text.mdx' ; import PQTradeoffs from '/\_includes/configuration/pq-compression/tradeoffs.mdx' ; To configure HNSW, see [Configuration: Vector index](/weaviate/config-refs/indexing/vector-index.mdx). ## Enable PQ compression PQ is configured at a collection level. There are two ways to enable PQ compression: - [Use AutoPQ to enable PQ compression](./pq-compression.md#configure-autopq). - [Manually enable PQ compression](./pq-compression.md#manually-configure-pq). ## Configure AutoPQ For new collections, use AutoPQ. AutoPQ automates triggering of the PQ training step based on the size of the collection. ### 1. Set the environment variable AutoPQ requires asynchronous indexing. - **Open-source Weaviate users**: To enable AutoPQ, set the environment variable `ASYNC_INDEXING=true` and restart your Weaviate instance. - [**Weaviate Cloud (WCD)**](/go/console?utm_content=howto/) users: Enable async indexing through the WCD Console and restart your Weaviate instance. ### 2. Configure PQ To configure PQ in a collection, use the [PQ parameters](./pq-compression.md#pq-parameters). ### 3. Load your data Load your data. You do not have to load an initial set of training data. AutoPQ creates the PQ codebook when the object count reaches the training limit. By default, the training limit is 100,000 objects per shard. ## Manually configure PQ You can manually enable PQ on an existing collection. After PQ is enabled, Weaviate trains the PQ codebook. Before you enable PQ, verify that the training set has 100,000 objects per shard. To manually enable PQ, follow these steps: - Phase One: Create a codebook - [Define a collection without PQ](./pq-compression.md#1-define-a-collection-without-pq) - [Load some training data](./pq-compression.md#2-load-training-data) - [Enable and train PQ](./pq-compression.md#3-enable-pq-and-create-the-codebook) - Phase Two: Load the rest of your data - [Load the rest of your data](./pq-compression.md#4-load-the-rest-of-your-data) :::tip How large should the training set be? We suggest 10,000 to 100,000 objects per shard. ::: Weaviate [logs a message](#check-the-system-logs) when PQ is enabled and another message when vector compression is complete. Do not import the rest of your data until the initial training step is complete. Follow these steps to manually enable PQ. ### 1. Define a collection without PQ [Create a collection](../../manage-collections/collection-operations.mdx#create-a-collection) without specifying a quantizer. ### 2. Load training data [Add objects](/weaviate/manage-objects/import.mdx) that will be used to train PQ. Weaviate will use the greater of the training limit, or the collection size, to train PQ. We recommend loading a representative sample such that the trained centroids are representative of the entire dataset. From `v1.27.0`, Weaviate uses a sparse [Fisher-Yates algorithm](https://en.wikipedia.org/wiki/Fisher–Yates_shuffle) to select the training set from the available objects when PQ is enabled manually. Nonetheless, it is still recommended to load a representative sample of the data so that the trained centroids are representative of the entire dataset. ### 3. Enable PQ and create the codebook Update your collection definition to enable PQ. Once PQ is enabled, Weaviate trains the codebook using the training data. import PQMakesCodebook from '/\_includes/configuration/pq-compression/makes-a-codebook.mdx' ; To enable PQ, update your collection definition as shown below. For additional configuration options, see the [PQ parameter table](./pq-compression.md#pq-parameters). ### 4. Load the rest of your data Once the [codebook has been trained](#3-enable-pq-and-create-the-codebook), you may continue to add data as per normal. Weaviate compresses the new data when it adds it to the database. If you already have data in your Weaviate instance when you create the codebook, Weaviate automatically compresses the remaining objects (the ones after the initial training set). ## PQ parameters You can configure PQ compression by setting the following parameters at the collection level. import PQParameters from '/\_includes/configuration/pq-compression/parameters.mdx' ; ## Additional tools and considerations ### Change the codebook training limit For most use cases, 100,000 objects is an optimal training size. There is little benefit to increasing `trainingLimit`. If you do increase `trainingLimit`, the training period will take longer. You could also have memory problems if you set a high `trainingLimit`. If you have a small dataset and wish to enable compression, consider using [binary quantization (BQ)](./bq-compression.md). BQ is a simpler compression method that does not require training. ### Check the system logs When compression is enabled, Weaviate logs diagnostic messages like these. ```bash pq-conf-demo-1 | {"action":"compress","level":"info","msg":"switching to compressed vectors","time":"2023-11-13T21:10:52Z"} pq-conf-demo-1 | {"action":"compress","level":"info","msg":"vector compression complete","time":"2023-11-13T21:10:53Z"} ``` If you use `docker-compose` to run Weaviate, you can get the logs on the system console. ```bash docker compose logs -f --tail 10 weaviate ``` You can also view the log file directly. Check `docker` to get the file location. ```bash docker inspect --format='{{.LogPath}}' ``` ### Review the current `pq` configuration To review the current `pq` configuration, you can retrieve it as shown below. ### Multiple vector embeddings (named vectors) import NamedVectorCompress from '/\_includes/named-vector-compress.mdx'; ### Multi-vector embeddings (ColBERT, ColPali, etc.) import MultiVectorCompress from '/\_includes/multi-vector-compress.mdx'; ## Further resources - [Starter guides: Compression](/docs/weaviate/starter-guides/managing-resources/compression.mdx) - [Reference: Vector index](/weaviate/config-refs/indexing/vector-index.mdx) - [Concepts: Vector quantization](/docs/weaviate/concepts/vector-quantization.md) - [Concepts: Vector index](/weaviate/concepts/indexing/vector-index.md) ## Questions and feedback import DocsFeedback from '/\_includes/docs-feedback.mdx'; --- ### Weaviate/Configuration/Compression/Rq Compression (docs/weaviate/configuration/compression/rq-compression.md) --- title: Rotational Quantization (RQ) image: og/docs/configuration.jpg # tags: ['configuration', 'compression', 'rq'] --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import Rq8bit from '/_includes/feature-notes/rq-8bit.mdx'; import Rq1bit from '/_includes/feature-notes/rq-1bit.mdx'; import FilteredTextBlock from '@site/src/components/Documentation/FilteredTextBlock'; import PyCode from '!!raw-loader!/\_includes/code/howto/configure-rq/rq-compression-v4.py'; import GoCode from '!!raw-loader!/\_includes/code/howto/go/docs/configure/compression.rq_test.go'; import TSCode from '!!raw-loader!/\_includes/code/howto/configure-rq/rq-compression-v3.ts'; import JavaCode from '!!raw-loader!/\_includes/code/java-v6/src/test/java/ConfigureRQTest.java'; import CSharpCode from "!!raw-loader!/\_includes/code/csharp/ConfigureRQTest.cs"; import CompressionByDefault from '/\_includes/compression-by-default.mdx'; [**Rotational quantization (RQ)**](../../concepts/vector-quantization.md#rotational-quantization) is a fast vector compression technique that offers significant performance benefits. Two RQ variants are available in Weaviate: - **8-bit RQ**: Up to 4x compression while retaining almost perfect recall (98-99% on most datasets). **Recommended** for most use cases. - **1-bit RQ**: Close to 32x compression as dimensionality increases with moderate recall across various datasets. ## 8-bit RQ [8-bit RQ](../../concepts/vector-quantization.md#8-bit-rq) provides up-to 4x compression while maintaining 98-99% recall in internal testing. It is generally recommended for most use cases as the default quantization techniques. ### Enable compression for new collection RQ can be enabled at collection creation time through the collection definition: ### Enable compression for existing collection RQ can also be enabled for an existing collection by updating the collection definition: ## 1-bit RQ [1-bit RQ](../../concepts/vector-quantization.md#1-bit-rq) is an quantization technique that provides close to 32x compression as dimensionality increases. 1-bit RQ serves as a more robust and accurate alternative to [BQ](./bq-compression.md) with only a slight performance trade-off. While more performant than PQ in terms of encoding time and distance calculations, 1-bit RQ typically offers slightly lower recall than well-tuned [PQ](./pq-compression.md). ### Enable compression for new collection RQ can be enabled at collection creation time through the collection definition: ### Enable compression for existing collection RQ can also be enabled for an existing collection by updating the collection definition: ## RQ parameters To tune RQ, use these quantization and vector index parameters: import RQParameters from '/\_includes/configuration/rq-compression-parameters.mdx' ; ## Additional considerations ### Multiple vector embeddings (named vectors) import NamedVectorCompress from '/\_includes/named-vector-compress.mdx'; ### Multi-vector embeddings (ColBERT, ColPali, etc.) import MultiVectorCompress from '/\_includes/multi-vector-compress.mdx'; :::note Multi-vector performance RQ supports multi-vector embeddings. Each token vector is rounded up to a multiple of 64 dimensions, which may result in less than 4x compression for very short vectors. This is a technical limitation that may be addressed in future versions. ::: ## Further resources - [Starter guides: Compression](/docs/weaviate/starter-guides/managing-resources/compression.mdx) - [Reference: Vector index](/weaviate/config-refs/indexing/vector-index.mdx) - [Concepts: Vector quantization](/docs/weaviate/concepts/vector-quantization.md) - [Concepts: Vector index](/weaviate/concepts/indexing/vector-index.md) ## Questions and feedback import DocsFeedback from '/\_includes/docs-feedback.mdx'; --- ### Weaviate/Configuration/Compression/Sq Compression (docs/weaviate/configuration/compression/sq-compression.md) --- title: Scalar Quantization (SQ) image: og/docs/configuration.jpg # tags: ['configuration', 'compression', 'sq'] --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import FilteredTextBlock from '@site/src/components/Documentation/FilteredTextBlock'; import PyCode from '!!raw-loader!/\_includes/code/howto/configure-sq/sq-compression-v4.py'; import TSCode from '!!raw-loader!/\_includes/code/howto/configure-sq/sq-compression-v3.ts'; import TSCodeSQOptions from '!!raw-loader!/\_includes/code/howto/configure-sq/sq-compression.options-v3.ts'; import GoCode from '!!raw-loader!/\_includes/code/howto/go/docs/configure/compression.sq_test.go'; import JavaV6Code from "!!raw-loader!/\_includes/code/java-v6/src/test/java/ConfigureSQTest.java"; import CSharpCode from "!!raw-loader!/\_includes/code/csharp/ConfigureSQTest.cs"; import CompressionByDefault from '/\_includes/compression-by-default.mdx'; [**Scalar quantization (SQ)**](/weaviate/concepts/vector-quantization#scalar-quantization) is a vector compression technique that can reduce the size of a vector. To use SQ, enable it in the collection definition, then add data to the collection. ## Enable compression for new collection SQ can be enabled at collection creation time through the collection definition: ## Enable compression for existing collection import SqPostCreation from '/_includes/feature-notes/sq-post-creation.mdx'; SQ can also be enabled for an existing collection by updating the collection definition: ## SQ parameters To tune SQ, set these `vectorIndexConfig` parameters. import SQParameters from '/\_includes/configuration/sq-compression-parameters.mdx' ; ## Additional considerations ### Multiple vector embeddings (named vectors) import NamedVectorCompress from '/\_includes/named-vector-compress.mdx'; ### Multi-vector embeddings (ColBERT, ColPali, etc.) import MultiVectorCompress from '/\_includes/multi-vector-compress.mdx'; ## Further resources - [Starter guides: Compression](/docs/weaviate/starter-guides/managing-resources/compression.mdx) - [Reference: Vector index](/weaviate/config-refs/indexing/vector-index.mdx) - [Concepts: Vector quantization](/docs/weaviate/concepts/vector-quantization.md) - [Concepts: Vector index](/weaviate/concepts/indexing/vector-index.md) ## Questions and feedback import DocsFeedback from '/\_includes/docs-feedback.mdx'; --- ### Weaviate/Configuration/Compression/Uncompressed (docs/weaviate/configuration/compression/uncompressed.md) --- title: Uncompressed vector embeddings sidebar_label: No quantization image: og/docs/configuration.jpg # tags: ['configuration', 'compression', 'rq'] --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import FilteredTextBlock from '@site/src/components/Documentation/FilteredTextBlock'; import PyCode from '!!raw-loader!/\_includes/code/howto/configure-rq/rq-compression-v4.py'; import GoCode from '!!raw-loader!/\_includes/code/howto/go/docs/configure/compression.rq_test.go'; import TSCode from '!!raw-loader!/\_includes/code/howto/configure-rq/rq-compression-v3.ts'; import JavaV6Code from "!!raw-loader!/\_includes/code/java-v6/src/test/java/ConfigureRQTest.java"; import CSharpCode from "!!raw-loader!/\_includes/code/csharp/ConfigureRQTest.cs"; import CompressionByDefault from '/\_includes/compression-by-default.mdx'; You can opt-out of using vector quantization to compress your vector data. ## Disable compression for new collection When creating the collection, you can choose not to use quantization through the collection definition: ## Additional considerations ### Multiple vector embeddings (named vectors) import NamedVectorCompress from '/\_includes/named-vector-compress.mdx'; ### Multi-vector embeddings (ColBERT, ColPali, etc.) import MultiVectorCompress from '/\_includes/multi-vector-compress.mdx'; :::note Multi-vector performance RQ supports multi-vector embeddings. Each token vector is rounded up to a multiple of 64 dimensions, which may result in less than 4x compression for very short vectors. This is a technical limitation that may be addressed in future versions. ::: ## Further resources - [Starter guides: Compression](/docs/weaviate/starter-guides/managing-resources/compression.mdx) - [Reference: Vector index](/weaviate/config-refs/indexing/vector-index.mdx) - [Concepts: Vector quantization](/docs/weaviate/concepts/vector-quantization.md) - [Concepts: Vector index](/weaviate/concepts/indexing/vector-index.md) ## Questions and feedback import DocsFeedback from '/\_includes/docs-feedback.mdx'; --- ### Weaviate/Configuration/Rbac/Index (docs/weaviate/configuration/rbac/index.mdx) --- title: RBAC Overview sidebar_label: RBAC sidebar_position: 0 image: og/docs/configuration.jpg # tags: ['rbac', 'roles', 'configuration', 'authorization'] --- import Link from '@docusaurus/Link'; import SkipLink from '/src/components/SkipValidationLink' Weaviate provides differentiated access through [authorization](/deploy/configuration/authorization.md) levels based on the [authenticated](/deploy/configuration/authentication.md) user identity. If **role-based access control (RBAC)** is enabled, access can be further restricted based on the roles of users. The diagram below illustrates the RBAC model in Weaviate, where access is governed by defining roles and assigning them specific permissions. These permissions determine which actions users can perform on designated resource types. The main components are: - **Users**: Individual users (e.g., `user-a` and `user-b`) are assigned to specific roles. - **Roles**: Each role encapsulates a set of permissions. This abstraction lets you manage what actions a group of users can perform. - **Permissions**: Permissions are comprised of three parts: - **Actions**: Operations like create, read, update, delete, and manage. - **Resources**: Specific targets for these actions, such as collections and backups. - **Optional constraints**: Resource-specific constraints such as filtering by collection names. ```mermaid graph LR %% Users Subgraph subgraph Users UA["user-a"] UB["user-b"] end %% Roles Subgraph subgraph Roles RWR["readWriteRole"] BM["backupManager"] end %% Permissions Group (contains Actions and Resources) with extra newline for spacing subgraph Permissions["Permissions"] %% Actions Subgraph subgraph Actions C["create"] R["read"] U["update"] D["delete"] M["manage"] end %% Resources Subgraph subgraph Resources COL["collections"] BAC["backups"] end end %% Connections for user-a UA --> RWR RWR --> C RWR --> R RWR --> U RWR --> D %% Connections for user-b UB --> BM BM --> M %% Connections from actions to resources C --> COL R --> COL U --> COL D --> COL M --> BAC %% Styling with soothing, Weaviate-compatible colors and slightly darker borders style Users fill:#AEDFF7,stroke:#90C7E5,stroke-width:1px style Roles fill:#C8E6C9,stroke:#A5D6A7,stroke-width:1px style Actions fill:#ECEFF1,stroke:#B0BEC5,stroke-width:1px style Resources fill:#CFD8DC,stroke:#AAB4BA,stroke-width:1px style Permissions fill:#E0F7FA,stroke:#B2EBF2,stroke-width:1px ``` This RBAC system ensures that users only have the access necessary for their roles, enhancing both security and manageability within Weaviate. Roles and permissions can be managed through the Weaviate REST API directly or through a **[client library](/weaviate/configuration/rbac/manage-roles)** programmatically. ## Roles ### Predefined roles Weaviate comes with a set of predefined roles. These roles are: - `root`: The root role has **full access** to all resources in Weaviate. - `viewer`: The viewer role has **read-only access** to all resources in Weaviate. The `root` role can be assigned to a user through the Weaviate configuration file using the [`AUTHORIZATION_RBAC_ROOT_USERS`](/deploy/configuration/env-vars/index.md#rbac-authorization) environment variable. A predefined role cannot be modified. The user can, however, be assigned additional roles through the Weaviate API. All roles can also be assigned through the Weaviate API, including the predefined role. The predefined roles cannot be modified, but they can be assigned to or revoked from users. Refer to the [RBAC: Configuration](/deploy/configuration/configuring-rbac.md) page for more information on how to assign predefined roles to users. ### Custom roles Any authenticated user that is not assigned a predefined role has no roles or permissions by default. These users' permissions can be modified through Weaviate by those with the appropriate permissions for **managing roles**. This allows for the creation of custom roles, which can be assigned to users as needed. Role management can be performed with a [predefined `root` role](/deploy/configuration/configuring-rbac.md) or a custom role with [`manage_roles` permissions](/weaviate/configuration/rbac/manage-roles#role-management-permissions). :::caution Role Management Permissions Be careful when assigning permissions to roles that manage roles. These permissions can be used to escalate privileges by assigning additional roles to users. Only assign these permissions to trusted users. ::: ## Permissions Permissions in Weaviate define what actions users can perform on specific resources. Each permission consists of: - A resource type (e.g., collections, objects) - Access levels (read, write, update, delete, manage) - Optional resource-specific constraints ### Available permissions Permissions can be defined with the following resources, access levels and optional constraints:

Resource type	Access levels	Optional resource‑specific constraints
Role Management	Create roles Read role info Update role permissions Delete roles	Role name filter: `string or regex`: specifies which roles can be managed Role scope: `all`: Allow role management with all permissions `match`: Only allow role management with the current user's permission level
User Management	Create users Read user info Update/rotate user API key Delete users Assign and revoke roles to and from users	User name filter: `string or regex`: specifies which users can be managed
Collections (collection definitions only, data object permissions are separate)	Create collections Read collection definitions Update collection definitions Delete collections	Collection name filter: `string or regex`: specifies which collections can be managed
Tenants	Create tenants Read tenant info Update tenants Delete tenants	Collection name filter: `string or regex`: specifies which collections' tenants can be managed Tenant name filter: `string or regex`: specifies which tenants can be managed
Data Objects	Create objects Read objects Update objects Delete objects	Collection name filter: `string or regex`: specifies which collections' objects can be managed Tenant name filter: `string or regex`: specifies which tenants' objects can be managed
Backups	Manage backups	Collection name filter: `string or regex`: specifies which collections' backups can be managed
Cluster Data Access	Read cluster metadata
Node Data Access	Read node metadata at a specified verbosity level	Verbosity level: `minimal`: Minimal read permission for all collections. `verbose`: Verbose read permission for specified collections. Collection name filter (only for `verbose`): `string or regex`: specifies which collections can be managed
Collection aliases	Create aliases Read aliases Update aliases Delete aliases	Alias name filter: `string or regex`: specifies which aliases can be managed
Replications	Create replications Read replications Update replications Delete replications	Collection name filter: `string or regex`: specifies which collections' objects can be managed Shard name filter: `string or regex`: specifies which shards can be managed
Groups	Read groups Assign and revoke group membership	Group name filter: `string or regex`: specifies which groups can be managed Group type filter: `oidc` (only OIDC user groups are supported at the moment)

### Permission behavior When defining permissions, setting a permission to `False` indicates that the permission is _not set_, rather than explicitly denying access. This means that if a user has multiple roles, and one role grants a permission while another sets it to `False`, the user will still have that permission through the role that grants it. For example, if a user has two roles: - Role A sets `read` to `False` for Collection X - Role B sets `read` to `True` for Collection X The user will have read access to Collection X because Role B grants the permission, while Role A's `False` value simply indicates no permission is set rather than blocking access. ### Name filters in permissions Some permissions require a collection name filter to specify which collections the permission applies to. In this case, `"*"` acts as a multi-character wildcard. As an example, setting a permission with `"Test*"` as the collection name filter would apply that permission to all collections that start with `Test`. Or, setting a permission with `"*"` as the collection filter would apply that permission to all available collections. ### Collection and tenant permissions A collection permission is independent of tenant permissions. To have permissions to operate on a tenant that belongs to a collection, the user must have the appropriate tenant-level permissions for that collection. Collection-level permissions, such as that to create collections, do not grant the equivalent tenant-level permissions, such as that to create tenants for that collection. For example, to create a tenant in a collection called `TestCollection`, the user must have permission to "create" tenants in that collection. This is separate from the permission to create a collection called `TestCollection`. ## Users The [user management](./manage-users.mdx) API can be used to create, delete and list users, rotate their API keys and manage their roles. ## Further resources - [RBAC: Configuration](/deploy/configuration/configuring-rbac.md) - [RBAC: Manage roles](./manage-roles.mdx) - [RBAC: Manage users](./manage-users.mdx) - [RBAC: Tutorial](/weaviate/tutorials/rbac.mdx) ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Configuration/Rbac/Manage Groups (docs/weaviate/configuration/rbac/manage-groups.mdx) --- title: Manage groups sidebar_label: Manage groups image: og/docs/configuration.jpg # tags: ['rbac', 'groups', 'configuration', 'authorization', 'oidc'] --- import Link from "@docusaurus/Link"; import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem"; import FilteredTextBlock from "@site/src/components/Documentation/FilteredTextBlock"; import OidcGroupPyCode from "!!raw-loader!/_includes/code/python/howto.configure.rbac.oidc.groups.py"; import OidcGroupTSCode from "!!raw-loader!/_includes/code/typescript/howto.configure.rbac.oidc.groups.ts"; import JavaV6Code from "!!raw-loader!/_includes/code/java-v6/src/test/java/RBACTest.java"; import CSharpCode from "!!raw-loader!/_includes/code/csharp/RBACTest.cs"; import RbacGroups from '/_includes/feature-notes/rbac-groups.mdx'; When using [OIDC](/deploy/configuration/oidc.md) for authentication, you can leverage user groups defined in your identity provider (like Keycloak, Okta, or Auth0) to manage permissions in Weaviate. The user's group memberships are passed to Weaviate in the OIDC token. You can then assign Weaviate roles directly to these **OIDC groups**. Any user who is a member of that group will automatically inherit the permissions of the assigned roles. This is a powerful way to manage access for large teams without assigning roles to each user individually. On this page, you will find examples of how to programmatically **manage OIDC groups** and their associated roles. ## Group management {#group-management} ### Assign roles to an OIDC group You can assign one or more Weaviate roles to an OIDC group. Any user belonging to this group will inherit the roles' permissions. This example assigns the `testRole` and `viewer` roles to the `/admin-group`. ```go // Go support coming soon ``` ### Revoke roles from an OIDC group You can revoke one or more roles from a specific OIDC group. This example removes the `testRole` and `viewer` roles from the `/admin-group`. ```go // Go support coming soon ``` ### List roles assigned to an OIDC group Retrieve a list of all roles that have been assigned to a specific OIDC group. ```go // Go support coming soon ```

Example results

```text Roles assigned to '/admin-group': ['testRole', 'viewer'] ```

### List all known OIDC groups This example shows how to get a list of all OIDC groups that Weaviate is aware of. Weaviate learns about a group when a role is first assigned to it. ```go // Go support coming soon ```

Example results

```text Known OIDC groups (3): ['/viewer-group', '/admin-group', '/my-test-group'] ```

### List groups assigned to a role Retrieve a list of all groups that have been assigned a specific role. This example shows which groups have the `testRole` assigned to them. ```go // Go support coming soon ```

Example results

```text Groups assigned to role 'testRole': - Group ID: /admin-group, Type: oidc ```

## Further resources - [RBAC: Overview](./index.mdx) - [RBAC: Configuration](/deploy/configuration/configuring-rbac.md) - [RBAC: Manage roles](./manage-roles.mdx) - [RBAC: Manage users](./manage-users.mdx) ## Questions and feedback import DocsFeedback from "/_includes/docs-feedback.mdx"; --- ### Weaviate/Configuration/Rbac/Manage Roles (docs/weaviate/configuration/rbac/manage-roles.mdx) --- title: Manage roles sidebar_label: Manage roles sidebar_position: 1 image: og/docs/configuration.jpg # tags: ['rbac', 'roles', 'configuration', 'authorization'] --- import Link from "@docusaurus/Link"; import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem"; import FilteredTextBlock from "@site/src/components/Documentation/FilteredTextBlock"; import PyCode from "!!raw-loader!/_includes/code/python/howto.configure.rbac.permissions.py"; import TSCode from "!!raw-loader!/_includes/code/typescript/howto.configure.rbac.permissions.ts"; import RolePyCode from "!!raw-loader!/_includes/code/python/howto.configure.rbac.roles.py"; import RoleTSCode from "!!raw-loader!/_includes/code/typescript/howto.configure.rbac.roles.ts"; import JavaV6Code from "!!raw-loader!/_includes/code/java-v6/src/test/java/RBACTest.java"; import GoCode from "!!raw-loader!/_includes/code/howto/go/docs/configure/rbac.roles_test.go"; import CSharpCode from "!!raw-loader!/_includes/code/csharp/RBACTest.cs"; In Weaviate, Role-based access control (RBAC) allows you to define roles and assign permissions to those roles. Users can then be assigned to roles and inherit the permissions associated with those roles. On this page, you will find examples of how to **manage roles and permissions** with Weaviate client libraries. import ConfigureRbac from "/_includes/configuration/configure-rbac.mdx"; ## Requirements for managing roles {#requirements} Role management requires appropriate `role` resource permissions that can be obtained through: - A predefined `root` role when [configuring RBAC](/deploy/configuration/configuring-rbac.md). - A role with [`Role Management`](#role-management-permissions) permissions granted. ## Role management {#role-management} ### Create new roles with permissions Permissions for these resource types can be assigned to roles: 1. [**Role Management**](#role-management-permissions) 2. [**User Management**](#user-management-permissions) 3. [**Collections**](#collections-permissions) (collection definitions only, data object permissions are separate) 4. [**Tenants**](#tenants-permissions) 5. [**Data Objects**](#data-permissions) 6. [**Backup**](#backups-permissions) 7. [**Cluster Data Access**](#clusters-permissions) 8. [**Node Data Access**](#nodes-permissions) 9. [**Collection alias**](#aliases-permissions) 10. [**Replications**](#replications-permissions) 11. [**Groups**](#groups-permissions) 12. [**MCP**](#mcp-permissions) #### Create a role with `Role Management` permissions {#role-management-permissions} This example creates a role called `testRole` with permissions to: - Create, read, update and delete all roles starting with `testRole*`. #### Create a role with `User Management` permissions {#user-management-permissions} This example creates a role called `testRole` with permissions to: - Create, read, update and delete all users starting with `testUser*`. - Assign and revoke roles to and from users starting with `testUser*`. #### Create a role with `Collections` permissions {#collections-permissions} This example creates a role called `testRole` with permissions to: - Create, read, update and delete all collections starting with `TargetCollection`. #### Create a role with `Tenant` permissions {#tenants-permissions} This example creates a role called `testRole` with permissions to: - Create and delete tenants starting with `TargetTenant` in collections starting with `TargetCollection`. - Read metadata (like tenant names and status) for tenants starting with `TargetTenant` in collections starting with `TargetCollection`. - Update the status of tenants starting with `TargetTenant` in collections starting with `TargetCollection`. #### Create a role with `Data Objects` permissions {#data-permissions} This example creates a role called `testRole` with permissions to: - Create, read, update and delete data from collections starting with `TargetCollection`. - If multi-tenancy is enabled and the `tenant` filter is set, the permission only applies to tenants starting with `TargetTenant`. #### Create a role with `Backups` permissions {#backups-permissions} This example creates a role called `testRole` with permissions to: - Manage backups for collections starting with `TargetCollection`. #### Create a role with `Cluster Data Access` permissions {#clusters-permissions} This example creates a role called `testRole` with permissions to: - Read cluster metadata. #### Create a role with `Node Data Access` permissions {#nodes-permissions} This example creates a role called `testRole` with permissions to: - Read node metadata at the specified verbosity level for collections starting with `TargetCollection`. #### Create a role with `Collection Alias` permissions {#aliases-permissions} This example creates a role called `testRole` with permissions to: - Create, read, update and delete collection aliases starting with `TargetAlias`. #### Create a role with `Replications` permissions {#replications-permissions} This example creates a role called `testRole` with permissions to: - Create, read, update and delete replica movement operations for collections starting with `TargetCollection` and shards starting with `TargetShard`. ```typescript // TS/JS support coming soon ``` #### Create a role with `Groups` permissions {#groups-permissions} This example creates a role called `testRole` with permissions to: - Read information about and assign/revoke group membership for OIDC groups starting with `TargetGroup`. ```typescript // TS/JS support coming soon ``` #### Create a role with `MCP` permissions {#mcp-permissions} The [Weaviate MCP server](/weaviate/configuration/mcp-server.mdx) uses three granular permissions: | Permission | Tools | | :--------- | :---- | | `read_mcp` | `weaviate-collections-get-config`, `weaviate-tenants-list`, `weaviate-query-hybrid` | | `create_mcp` + `update_mcp` | `weaviate-objects-upsert` | MCP tools also require standard collection-level permissions (e.g., `read_data` for search, `create_data` + `update_data` for upsert). See the [MCP server permissions](/weaviate/configuration/mcp-server.mdx#permissions) for the full per-tool breakdown. ### Grant additional permissions Additional permissions can be granted to a role at any time. The role must already exist. This example grants additional permissions to the role `testRole` to: - **Create new data** in collections that start with `TargetCollection` ### Remove permissions from a role Permissions can be revoked from a role at any time. Removing all permissions from a role will delete the role itself. This example removes the following permissions from the role `testRole`: - Read the data from collections that start with `TargetCollection` - Create and delete collections that start with `TargetCollection` ### Check if a role exists Check if the role `testRole` exists: ### Inspect a role View the permissions assigned to a role. ### List all roles View all roles in the system and their permissions. ### List users with a role List all users who have the role `testRole`. ### Delete a role Deleting a role will remove it from the system, and revoke the associated permissions from all users who had this role. ## User management {#user-management} Visit the [Manage users](./manage-users.mdx) page to learn more about assigning roles to users as well as creating, updating and deleting users. ## Further resources - [RBAC: Overview](./index.mdx) - [RBAC: Configuration](/deploy/configuration/configuring-rbac.md) - [RBAC: Manage users](./manage-users.mdx) ## Questions and feedback import DocsFeedback from "/_includes/docs-feedback.mdx"; --- ### Weaviate/Configuration/Rbac/Manage Users (docs/weaviate/configuration/rbac/manage-users.mdx) --- title: Manage users sidebar_label: Manage users sidebar_position: 1 image: og/docs/configuration.jpg # tags: ['rbac', 'roles', 'configuration', 'authorization'] --- import Link from "@docusaurus/Link"; import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem"; import FilteredTextBlock from "@site/src/components/Documentation/FilteredTextBlock"; import PyCode from "!!raw-loader!/_includes/code/python/howto.configure.rbac.permissions.py"; import TSCode from "!!raw-loader!/_includes/code/typescript/howto.configure.rbac.permissions.ts"; import UserPyCode from "!!raw-loader!/_includes/code/python/howto.configure.rbac.users.py"; import UserTSCode from "!!raw-loader!/_includes/code/typescript/howto.configure.rbac.users.ts"; import OidcUserPyCode from "!!raw-loader!/_includes/code/python/howto.configure.rbac.oidc.users.py"; import OidcUserTSCode from "!!raw-loader!/_includes/code/typescript/howto.configure.rbac.oidc.users.ts"; import JavaV6Code from "!!raw-loader!/_includes/code/java-v6/src/test/java/RBACTest.java"; import GoCode from "!!raw-loader!/_includes/code/howto/go/docs/configure/rbac.users_test.go"; import OidcUserGoCode from "!!raw-loader!/_includes/code/howto/go/docs/configure/rbac.oidc.users_test.go"; import CSharpCode from "!!raw-loader!/_includes/code/csharp/RBACTest.cs"; import RbacUserManagement from '/_includes/feature-notes/rbac-user-management.mdx'; In Weaviate, Role-based access control (RBAC) allows you to define roles and assign permissions to those roles. Users can then be assigned to roles and inherit the permissions associated with those roles. Weaviate differentiates multiple types of users. **Database users** are fully managed by the Weaviate instance, while **OIDC** users are managed by an external identity provider. Both types can be used together with RBAC. On this page, you will find examples of how to programmatically **manage users** and their associated roles with Weaviate client libraries. :::note User types in Weaviate Under the hood, Weaviate differentiates three types of users: - `db_user`: Database users that can be fully managed through the API. - `db_env_user`: Database users that are defined through the `AUTHENTICATION_APIKEY_USERS` environment variable and can only be updated through this variable and by restarting the Weaviate instance. - `oidc`: Users that can only be created/deleted through the external OIDC service. ::: ## User management {#user-management} ### List all users This example shows how to get a list of all the users (`db_user`, `db_env_user` and `oidc`) in Weaviate.

Example results

```text [ UserDB(user_id='custom-user', role_names=['viewer', 'testRole'], user_type=, active=True), UserDB(user_id='root-user', role_names=['root'], user_type=, active=True) ] ```

### Create a database user {#create-a-user} This example creates a user called `custom-user`.

Example results

```text RXF1dU1VcWM1Q3hvVndYT0F1OTBOTDZLZWx0ME5kbWVJRVdPL25EVW12QT1fMXlDUEhUNjhSMlNtazdHcV92MjAw ```

### Delete a database user This example deletes a user called `custom-user`. ### Rotate database user API key {#rotate-user-api-key} This example updates (rotates) the API key for `custom-user`.

Example results

```text SSs3WGVFbUxMVFhlOEsxVVMrQVBzM1VhQTJIM2xXWngwY01HaXFYVnM1az1fMXlDUEhUNjhSMlNtazdHcV92MjAw ```

## Database users: Permissions management {#user-permissions-management} ### Assign a role to a database user A custom user can have any number of roles assigned to them (including none). The role can be a predefined role (e.g. `viewer`) or a custom role. This example assigns the custom `testRole` role and predefined `viewer` role to `custom-user`. ### Remove a role from a database user You can revoke one or more roles from a specific user. This example removes the role `testRole` from the user `custom-user`. ### Get a database user's roles Retrieve the role information for any user.

Example results

```text testRole viewer ```

## OIDC users: Permissions management {#oidc-user-permissions-management} When using [OIDC](/deploy/configuration/oidc.md), an identity provider authenticates the user and issues tokens, which are then validated by Weaviate. These users can be assigned roles with custom permissions using RBAC. ### Assign a role to an OIDC user An OIDC user can have any number of roles assigned to them (including none). The role can be a predefined role (e.g. `viewer`) or a custom role. This example assigns the custom `testRole` role and predefined `viewer` role to `custom-user`. ### Remove a role from an OIDC user You can revoke one or more roles from a specific OIDC user. This example removes the role `testRole` from the user `custom-user`. ### Get an OIDC user's roles Retrieve the role information for an OIDC user.

Example results

```text testRole viewer ```

## Further resources - [RBAC: Overview](./index.mdx) - [RBAC: Configuration](/deploy/configuration/configuring-rbac.md) - [RBAC: Manage roles](./manage-roles.mdx) ## Questions and feedback import DocsFeedback from "/_includes/docs-feedback.mdx"; --- ### Weaviate/Connections/Connect Cloud (docs/weaviate/connections/connect-cloud.mdx) --- title: Weaviate Cloud sidebar_position: 10 image: og/docs/connect.jpg description: "Connect to Weaviate Cloud instances with code examples in Python, TypeScript, Go, Java, and C#." # tags: ['getting started', 'connect'] --- import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem"; import FilteredTextBlock from "@site/src/components/Documentation/FilteredTextBlock"; import PyCodeV4 from "!!raw-loader!/_includes/code/connections/connect-python-v4.py"; import TsCodeV3 from "!!raw-loader!/_includes/code/connections/connect-ts-v3.ts"; import JavaV6Code from "!!raw-loader!/_includes/code/java-v6/src/test/java/ConnectionTest.java"; import ShellCode from "!!raw-loader!/_includes/code/connections/connect.sh"; import GoCode from "!!raw-loader!/_includes/code/connections/connect.go"; import CSharpCode from "!!raw-loader!/_includes/code/csharp/ConnectionTest.cs"; Follow these steps to connect to a [Weaviate Cloud (WCD)](/go/console?utm_content=howto/) instance. ## Retrieve your API key and REST endpoint Open the [Weaviate Cloud console](/go/console?utm_content=howto/) and follow the steps below:

_Recorded during the Weaviate meetup – custom modules section starts @ 13:30min_ ## Background: Module architecture in Weaviate To understand how to create a new module, you'll need to understand how the module system of Weaviate works in general. Weaviate is entirely agnostic on how a module obtains the values it needs for the specific lifecycle hooks. For example, for a vectorizer module, the contract between Weaviate and the module is as follows: At import time, each object is passed to the (configured) vectorizer module and the module must extend it with a vector (embedding). Weaviate is agnostic to how the module does that. For example, if the module's purpose is to use a pre-existing ML model for inference, the module may decide to provide a second inference service and contact that inference service as part of the "vectorize" lifecycle hook. Weaviate is agnostic on how that communication occurs. For example, the `text2vec-contextionary` module uses a gRPC API on its inference service, whereas the the `text2vec-transformers` module uses a REST API for the same purpose. Typically a (vectorizer) module consists of two parts: 1. **Module code for Weaviate, written in Go**, which hooks into specific lifecycles and provides various capabilities (like controlling the API function) to integrate the module into the regular flow in Weaviate. 2. **Inference service**, typically a containerized application that wraps an ML model with a module-specific API which is consumed by the module code executed in Weaviate (part 1). The visualization below shows how modules are part of and connected to Weaviate. The black border indicates Weaviate Database, with the grey boxes as internals. Everything in red involves how Weaviate uses the modules that are connected, with the general Module System API. The red Module API spans two internal 'layers', because it can influence the Weaviate APIs (e.g. by extending GraphQL or providing additional properties), and it can influence the business logic (e.g. by taking the properties of an object and setting a vector). Everything that is blue belongs to a specific module (more than one module can be attached, but here we show one module). Here we have the example of Weaviate using the `text2vec-transformers` module `bert-base-uncased`. Everything that belongs to the `text2vec-transformers` module is thus drawn in blue. The blue box inside Weaviate Database is the part 1 of the module: the module code for Weaviate. The blue box outside Weaviate Database is the separate inference service, part 2. The picture shows three APIs: * The first grey box inside Weaviate Database, which is the user-facing RESTful and GraphQL API. * The red box is the Module System API, which are interfaces written in Go. * The third API is completely owned by the module, which is used to communicate with the separate module container. In this case, this is a Python container, shown on the left. To use a custom ML model with Weaviate, you have two options: ([further explained below](#how-to-build-and-use-a-custom-module)) * A: Replace parts of an existing module, where you only replace the inference service (part 2). You don't have to touch Weaviate Database here. * B: Build a completely new module and replace all existing (blue) module parts (both 1 and 2). You can configure custom behavior like extending the GraphQL API, as long as the module can hook into the 'red' Module System API. Keep in mind that you'll need to write some module code in Go to achieve this. Let's take a more detailed example of how you configure Weaviate to use a specific module: if we look at the [`text2vec-transformers`](/weaviate/model-providers/transformers/embeddings.md) module, you set `ENABLE_MODULES=text2vec-transformers` in the Docker Compose file, which instructs Weaviate to load the respective Go code (part 1). Additionally, you include another service in `docker-compose.yml` which contains the actual model for inference (part 2). In more detail, let's look at how a specific (GraphQL) function is implemented in the [`text2vec-transformers`](/weaviate/model-providers/transformers/embeddings.md) module: 1. **Module code for Weaviate, written in Go:** * Tells the Weaviate GraphQL API that the module provides a specific `nearText` method. * Validates specific configuration and schema settings and makes them available to the APIs. * Tells Weaviate how to obtain a vector (e.g. a word or image embedding) when one is necessary (by sending an HTTP request to a third-party service, in this case the Python application around the inference model) 2. **Inference service:** * Provides a service that can do model inference. * Implements an API that is in contract with A (not with Weaviate itself). Note that this is just one example, and variations are possible as long as both part 1 and 2 are present where 1 contains the connection to Weaviate in Go and 2 contains that inference model that part 1 uses. It would also be possible to amend, for example, the Weaviate `text2vec-transformers` module (part 1) to use the Hugging Face API or some other third-party hosted inference service, instead of its own container (now in part 2) that it brings. A module completely controls the communication with any container or service it depends on. So, for example, in the `text2vec-transformers` module, the API of the inference container is a REST API. But for the `text2vec-contextionary` module has a gRPC, rather than a REST API or another protocol. ### Module characteristics A module is a custom code that can extend Weaviate by hooking into specific lifecycle hooks. As Weaviate is written in Go, so module code must also be written in Go. However, some existing modules make use of independent services which can be written in any language, as is often the case with vectorizer modules which bring along model inference containers often written in Python. Modules can be "vectorizers" (defines how the numbers in the vectors are chosen from the data) or other modules providing additional functions like question answering, custom classification, etc. Modules have the following characteristics: - Naming convention: - Vectorizer: `2vec--`, for example `text2vec-contextionary`, `img2vec-neural` or `text2vec-transformers`. - Other modules: `--`. - A module name must be url-safe, meaning it must not contain any characters which would require url-encoding. - A module name is not case-sensitive. `text2vec-bert` would be the same module as `text2vec-BERT`. - Module information is accessible through the `v1/modules//` RESTful endpoint. - General module information (which modules are attached, version, etc.) is accessible through Weaviate's [`v1/meta` endpoint](/weaviate/api). - Modules can add `additional` properties in the RESTful API and [`_additional` properties in the GraphQL API](/weaviate/api/graphql/additional-properties.md). - A module can add [filters](/weaviate/api/graphql/filters.md) in GraphQL queries. - Which vectorizer and other modules are applied to which data classes is configured in the [collection configuration](../manage-collections/vector-config.mdx). ## How to build and use a custom module There are two different ways to extend Weaviate with custom vectorization capabilities: You can either build a completely custom module (parts 1 + 2) or only replace the inference service of an existing module (only replace part 2, Option A). The latter is a good option for fast prototyping and proofs of concepts. In this case, you simply replace the inference model (part 2), but keep the interface with Weaviate in Go. This is a quick way to integrate completely different model types. You can also choose to build a complete new module (Option B). This is the most flexible option, but it means you'll have to write a Weaviate interface in Go. We recommend to only go for option B if you are happy with the prototype results. With option B you can turn the PoC into a full module, because you can control all configuration and naming when you go for option B. ### A. Replace parts of an existing module The quickest way to integrate a completely different inference model is replacing parts of an existing module. You reuse part 1 (the interface with Weaviate) and thus adhere to part 1's API contract, and only implement changes to or replace part 2. Because you are not touching the Go Weaviate interface code, you don't have the possibility to introduce a new configuration that is specific to your module inference into Weaviate's APIs provided and consumed by existing modules that are not existing in part 1 (i.e. all the configuration parameters, e.g. those of `text2vec-transformers`). This also implies that you cannot change or introduce new (GraphQL) API functions or filters. _Note that Weaviate APIs are not guaranteed to be stable. Even on a non-breaking Weaviate release, 'internal' APIS could always change._ To use a new inference model (part 2) with an existing Weaviate interface (part 1), you could reuse all the Go-code from the existing module and simply point it to a different inference container. As an example, here's how to use a custom inference module using the `text2vec-transformers` Go-code: 1. In a valid `docker-compose.yml` that's configured to use transformers (e.g. for example configure one via the [configuration configurator](/deploy/installation-guides/docker-installation.md#configurator)), you will find an env var like this: `TRANSFORMERS_INFERENCE_API: 'http://text2vec-transformers:8080'`, you can point that to any app you like. You should keep the variable name `TRANSFORMERS_INFERENCE_API`. 2. Build a small HTTP API wrapper around your model, it should at the minimum have the endpoints listed below (which is in this example entirely specific to the `text2vec-transformers` module and fully in its control): 1. `GET /.well-known/live` -> respond `204` when the app is alive 2. `GET /.well-known/ready` -> respond `204` when the app is ready to serve traffic 3. `GET /meta` -> respond meta information about the inference model 4. `POST /vectors` -> see example request and response payloads below. (Note that the app is exposed locally on port `8090` on my machine by adding `ports: ["8090:8080"]` in the Docker Compose file). Request: ```bash curl localhost:8090/vectors/ -H "Content-Type: application/json" -d '{"text":"hello world"}' ``` Response: ```bash {"text":"hello world","vector":[-0.08469954133033752,0.4564870595932007, ..., 0.14153483510017395],"dim":384} ``` ### B. Build a completely new module Implementing a fully new module with both part 1 and 2 is a lot more flexible, because you can control naming, APIs, behavior, etc. To achieve this, you are essentially contributing to Weaviate. Note that for this option, you need to understand at least parts of Weaviate's architecture, and what a module can and can not control (what is "fixed"). You can fork [Weaviate's repository](https://github.com/weaviate/weaviate) and create a completely new [module](https://github.com/weaviate/weaviate/tree/master/modules) inside it. This new module can also depend on any number of other containers (which you will have to supply), and could use any API for communication with its dependencies (it could also have not any dependencies). Detailed instructions are described in the [contributor guide](/contributor-guide/weaviate-modules/how-to-build-a-new-module) If you choose to build a completely new module including a Weaviate Go interface, you can contact us via [the forum](https://forum.weaviate.io) or through an [issue on GitHub](https://github.com/weaviate/weaviate/issues), so we can help you get started. ## Important notes - The length of the vectors your vectorizer has influences later usage, for example if you're exploring your data by vector with the GraphQL explore filter, the length of this vector should match with the vector length of the data points. - Weaviate APIs internal to a module are not guaranteed to be stable. Even on a non-breaking Weaviate release, 'internal' APIS could always change. ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Modules/Img2vec Neural (docs/weaviate/modules/img2vec-neural.md) --- title: ResNet Image Vectorizer sidebar_position: 20 image: og/docs/modules/img2vec-neural.jpg # tags: ['img2vec', 'img2vec-neural'] --- :::caution CLIP recommended for new projects For new projects, we recommend using the [Transformers multi-modal integration](../model-providers/transformers/embeddings-multimodal.md) module instead of `img2vec-neural`. This uses CLIP models, which uses a more modern model architecture than `resnet` models used in `img2vec-neural`. CLIP models are also multi-modal, meaning they can handle both images and text and therefore applicable to a wider range of use cases. ::: The `img2vec-neural` module enables Weaviate to obtain vectors locally images using a [`resnet50`](https://arxiv.org/abs/1512.03385) model. `img2vec-neural` encapsulates the model in a Docker container, which allows independent scaling on GPU-enabled hardware while keeping Weaviate on CPU-only hardware, as Weaviate is CPU-optimized. Key notes: - This module is not available on Weaviate Cloud (WCD). - Enabling this module will enable the [`nearImage` search operator](#additional-search-operator). - Model encapsulated in a Docker container. - This module is not compatible with Auto-schema. You must define your classes manually as [shown below](#class-configuration). ## Weaviate instance configuration :::info Not applicable to WCD This module is not available on Weaviate Cloud. ::: ### Docker Compose file To use `img2vec-neural`, you must enable it in your Docker Compose file (e.g. `docker-compose.yml`). :::tip Use the configuration tool While you can do so manually, we recommend using the [Weaviate configuration tool](/deploy/installation-guides/docker-installation.md#configurator) to generate the `Docker Compose` file. ::: #### Parameters Weaviate: - `ENABLE_MODULES` (Required): The modules to enable. Include `img2vec-neural` to enable the module. - `DEFAULT_VECTORIZER_MODULE` (Optional): The default vectorizer module. You can set this to `img2vec-neural` to make it the default for all classes. - `IMAGE_INFERENCE_API` (Required): The URL of the inference container. Inference container: - `image` (Required): The image name of the inference container. (e.g. `semitechnologies/img2vec-pytorch:resnet50` or `semitechnologies/img2vec-keras:resnet50`) #### Example This configuration enables `img2vec-neural`, sets it as the default vectorizer, and sets the parameters for the Docker container, including setting it to use `img2vec-pytorch:resnet50` image. ```yaml services: weaviate: image: cr.weaviate.io/semitechnologies/weaviate:||site.weaviate_version|| restart: on-failure:0 ports: - 8080:8080 - 50051:50051 environment: QUERY_DEFAULTS_LIMIT: 20 AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true' PERSISTENCE_DATA_PATH: "./data" # highlight-start ENABLE_MODULES: 'img2vec-neural' IMAGE_INFERENCE_API: "http://i2v-neural:8080" # highlight-end CLUSTER_HOSTNAME: 'node1' # highlight-start i2v-neural: image: cr.weaviate.io/semitechnologies/img2vec-pytorch:resnet50 # highlight-end ... ``` ### Alternative: Run a separate container As an alternative, you can run the inference container independently from Weaviate. To do so, you can: - Enable `img2vec-neural` in your Docker Compose file, - Omit `img2vec-neural` parameters, - Run the inference container separately, e.g. using Docker, and - Set `IMAGE_INFERENCE_API` to the URL of the inference container. Then, for example if Weaviate is running outside of Docker, set `IMAGE_INFERENCE_API="http://localhost:8000"`. Alternatively if Weaviate is part of the same Docker network, e.g. because they are part of the same `docker-compose.yml` file, you can use Docker networking/DNS, such as `IMAGE_INFERENCE_API=http://i2v-neural:8080`. For example, can spin up an inference container with the following command: ```shell docker run -itp "8000:8080" semitechnologies/img2vec-neural:resnet50-61dcbf8 ``` ## Class configuration You can configure how the module will behave in each class through the [collection configuration](../manage-collections/vector-config.mdx). ### Vectorization settings You can set vectorizer behavior using the `moduleConfig` section under each class and property: #### Class-level - `vectorizer` - what module to use to vectorize the data. - `imageFields` - property names for images to be vectorized #### Property-level - `dataType` - the data type of the property. For use in `imageFields`, must be set to `blob`. #### Example The following example class definition sets the `img2vec-neural` module as the `vectorizer` for the class `FashionItem`. It also sets: - `image` property as a `blob` datatype and as the image field, ```json { "classes": [ { "class": "FashionItem", "description": "Each example is a 28x28 grayscale image, associated with a label from 10 classes.", // highlight-start "vectorizer": "img2vec-neural", "moduleConfig": { "img2vec-neural": { "imageFields": [ "image" ] } }, // highlight-end "properties": [ // highlight-start { "dataType": [ "blob" ], "description": "Grayscale image", "name": "image" }, // highlight-end { "dataType": [ "number" ], "description": "Label number for the given image.", "name": "labelNumber" }, { "dataType": [ "text" ], "description": "label name (description) of the given image.", "name": "labelName" } ], } ] } ``` :::note All `blob` properties must be in base64-encoded data. ::: ### Adding `blob` data objects Any `blob` property type data must be base64 encoded. To obtain the base64-encoded value of an image for example, you can use the helper methods in the Weaviate clients or run the following command: ```bash cat my_image.png | base64 ``` ## Additional search operator The `img2vec-neural` vectorizer module will enable the `nearImage` search operator. ## Usage example ### NearImage import CodeNearImage from '/_includes/code/img2vec-neural.nearimage.mdx'; ## About the model [`resnet50`](https://arxiv.org/abs/1512.03385) is a residual convolutional neural network with 25.5 million parameters trained on more than a million images from the [ImageNet database](https://www.image-net.org/). As the name suggests, it has a total of 50 layers: 48 convolution layers, 1 MaxPool layer and 1 Average Pool layer. ### Available img2vec-neural models There are two different inference models you can choose from. Depending on your machine (`arm64` or other) and whether you prefer to use multi-threading to extract feature vectors or not, you can choose between `keras` and `pytorch`. There are no other differences between the two models. - `resnet50` (`keras`): - Supports `amd64`, but not `arm64`. - Does not currently support `CUDA` - Supports multi-threaded inference - `resnet50` (`pytorch`): - Supports both `amd64` and `arm64`. - Supports `CUDA` - Does not support multi-threaded inference ## Model license(s) The `img2vec-neural` module uses the `resnet50` model. It is your responsibility to evaluate whether the terms of its license(s), if any, are appropriate for your intended use. ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Modules/Index (docs/weaviate/modules/index.md) --- title: Reference - Modules description: Learn about Weaviate modules to extend its functionality. sidebar_position: 0 image: og/docs/modules/_title.jpg # tags: ['modules'] --- This section describes Weaviate's individual modules, including their capabilities and how to use them. :::tip Looking for vectorizer, generative AI, or reranker integration docs? They have moved to our [model provider integrations](../model-providers/index.md) section, for a more focused, user-centric look at these integrations. ::: ## General Weaviate's modules are built into the codebase, and [enabled through environment variables](../configuration/modules.md) to provide additional functionalities. ### Module types Weaviate modules can be divided into the following categories: - [Vectorizers](#vectorizer-reranker-and-generative-ai-integrations): Convert data into vector embeddings for import and vector search. - [Rerankers](#vectorizer-reranker-and-generative-ai-integrations): Improve search results by reordering initial search results. - [Generative AI](#vectorizer-reranker-and-generative-ai-integrations): Integrate generative AI models for retrieval augmented generation (RAG). - [Backup](#backup-modules): Facilitate backup and restore operations in Weaviate. - [Offloading](#offloading-modules): Facilitate offloading of tenant data to external storage. - [Others](#other-modules): Modules that provide additional functionalities. #### Vectorizer, reranker, and generative AI integrations For these modules, see the [model provider integrations](../model-providers/index.md) documentation. These pages are organized by the model provider (e.g. Hugging Face, OpenAI) and then the model type (e.g. vectorizer, reranker, generative AI). For example: - [The OpenAI embedding integration page](../model-providers/openai/embeddings.md) shows how to use OpenAI's embedding models in Weaviate. Embedding integration illustration

- [The Cohere reranker integration page](../model-providers/cohere/reranker.md) shows how to use Cohere's reranker models in Weaviate. Reranker integration illustration

- [The Anthropic generative AI integration page](../model-providers/anthropic/generative.md) shows how to use Anthropic's generative AI models in Weaviate. Generative integration illustration

### Module characteristics - Naming convention: - Vectorizer (Retriever module): `2vec--`, for example `text2vec-contextionary`, `img2vec-neural` or `text2vec-transformers`. - Other modules: `--`, for example `qna-transformers`. - A module name must be url-safe, meaning it must not contain any characters which would require url-encoding. - A module name is not case-sensitive. `text2vec-bert` would be the same module as `text2vec-BERT`. - Module information is accessible through the `v1/modules//` RESTful endpoint. - General module information (which modules are attached, version, etc.) is accessible through Weaviate's [`v1/meta` endpoint](/deploy/configuration/status.md#cluster-metadata). - Modules can add `additional` properties in the RESTful API and [`_additional` properties in the GraphQL API](../api/graphql/additional-properties.md). - A module can add [filters](../api/graphql/filters.md) in GraphQL queries. - Which vectorizer and other modules are applied to which data collection is configured in the [schema](../manage-collections/vector-config.mdx#specify-a-vectorizer). ## Backup Modules Backup and restore operations in Weaviate are facilitated by the use of backup provider modules. These are interchangeable storage backends which exist either internally or externally. ### External provider External backup providers coordinate the storage and retrieval of backed-up Weaviate data with external storage services. This type of provider is ideal for production environments. This is because storing the backup data outside of a Weaviate instance decouples the availability of the backup from the Weaviate instance itself. In the event of an unreachable node, the backup is still available. Additionally, multi-node Weaviate clusters _require_ the use of an external provider. Storing a multi-node backup on internally on a single node presents several issues, like significantly reducing the durability and availability of the backup, and is not supported. The supported external backup providers are: - [S3](/deploy/configuration/backups.md#s3-aws-or-s3-compatible) - [GCS](/deploy/configuration/backups.md#gcs-google-cloud-storage) - [Azure](/deploy/configuration/backups.md#azure-storage) Thanks to the extensibility of the module system, new providers can be readily added. If you are interested in an external provider other than the ones listed above, feel free to reach out via our [forum](https://forum.weaviate.io/), or open an issue on [GitHub](https://github.com/weaviate/weaviate). ### Internal provider Internal providers coordinate the storage and retrieval of backed-up Weaviate data within a Weaviate instance. This type of provider is intended for developmental or experimental use, and is not recommended for production. Internal Providers are not compatible for multi-node backups, which require the use of an external provider. As of Weaviate `v1.16`, the only supported internal backup provider is the [filesystem](/deploy/configuration/backups.md#filesystem) provider. ## Offloading Modules Offloading modules facilitate the offloading of tenant data to external storage. This is useful for managing resources and costs. See [how to configure: offloading](/deploy/configuration/tenant-offloading.md) for more information on how to configure and use offloading modules. ## Other modules In addition to the above, there are other modules such as: - [qna-transformers](./qna-transformers.md): Question-answering (answer extraction) capability using transformers models. - [qna-openai](./qna-openai.md): Question-answering (answer extraction) capability using OpenAI models. - [ner-transformers](./ner-transformers.md): Named entity recognition capability using transformers models. - [text-spellcheck](./spellcheck.md): Spell checking capability for GraphQL queries. - [sum-transformers](./sum-transformers.md): Summarize text using transformer models. - [usage-modules](./usage-modules.md): Collect and upload usage analytics to GCS or S3 for the purposes of billing. - [custom-modules](./custom-modules.md): Attach your own machine learning model to Weaviate as a module. ### Other vectorizer modules The following vectorizer modules are not covered by the [model provider integration](../model-providers/index.md) pages: - [text2vec-contextionary](./text2vec-contextionary.md) (deprecated): Vectorize text locally with the lightweight Contextionary model. - [img2vec-neural](./img2vec-neural.md): Vectorize images locally with a `resnet50` model. - [ref2vec-centroid](./ref2vec-centroid.md): Calculate an object's vector from the centroid of its referenced objects' vectors. ## Related pages - [Configuration: Modules](../configuration/modules.md) - [Concepts: Modules](../concepts/modules.md) ## Other third party integrations import IntegrationLinkBack from '/_includes/integrations/link-back.mdx'; ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Modules/Ner Transformers (docs/weaviate/modules/ner-transformers.md) --- title: Named Entity Recognition description: Integrate NER Transformers in Weaviate to identify and categorize entities in text. sidebar_position: 60 image: og/docs/modules/ner-transformers.jpg # tags: ['ner-transformers', 'transformers', 'token classification'] --- ## In short * The Named Entity Recognition (NER) module is a Weaviate module for token classification. * The module depends on a NER Transformers model that should be running with Weaviate. There are pre-built models available, but you can also attach another HuggingFace Transformer or custom NER model. * The module adds a `tokens {}` filter to the GraphQL `_additional {}` field. * The module returns data objects as usual, with recognized tokens in the GraphQL `_additional { tokens {} }` field. ## Introduction Named Entity Recognition (NER) module is a Weaviate module to extract entities from your existing Weaviate (text) objects on the fly. Entity Extraction happens at query time. Note that for maximum performance, transformer-based models should run with GPUs. CPUs can be used, but the throughput will be lower. There are currently three different NER modules available (taken from [Hugging Face](https://huggingface.co/)): [`dbmdz-bert-large-cased-finetuned-conll03-english`](https://huggingface.co/dbmdz/bert-large-cased-finetuned-conll03-english), [`dslim-bert-base-NER`](https://huggingface.co/dslim/bert-base-NER), [`davlan-bert-base-multilingual-cased-ner-hrl`](https://huggingface.co/Davlan/bert-base-multilingual-cased-ner-hrl?text=%D8%A5%D8%B3%D9%85%D9%8A+%D8%B3%D8%A7%D9%85%D9%8A+%D9%88%D8%A3%D8%B3%D9%83%D9%86+%D9%81%D9%8A+%D8%A7%D9%84%D9%82%D8%AF%D8%B3+%D9%81%D9%8A+%D9%81%D9%84%D8%B3%D8%B7%D9%8A%D9%86.). ## How to enable (module configuration) ### Docker Compose The NER module can be added as a service to the Docker Compose file. You must have a text vectorizer like `text2vec-contextionary` or `text2vec-transformers` running. An example Docker Compose file for using the `ner-transformers` module (`dbmdz-bert-large-cased-finetuned-conll03-english`) in combination with the `text2vec-contextionary`: ```yaml --- services: weaviate: command: - --host - 0.0.0.0 - --port - '8080' - --scheme - http image: cr.weaviate.io/semitechnologies/weaviate:||site.weaviate_version|| ports: - 8080:8080 - 50051:50051 restart: on-failure:0 environment: CONTEXTIONARY_URL: contextionary:9999 NER_INFERENCE_API: "http://ner-transformers:8080" QUERY_DEFAULTS_LIMIT: 25 AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true' PERSISTENCE_DATA_PATH: '/var/lib/weaviate' ENABLE_MODULES: 'text2vec-contextionary,ner-transformers' CLUSTER_HOSTNAME: 'node1' contextionary: environment: OCCURRENCE_WEIGHT_LINEAR_FACTOR: 0.75 EXTENSIONS_STORAGE_MODE: weaviate EXTENSIONS_STORAGE_ORIGIN: http://weaviate:8080 NEIGHBOR_OCCURRENCE_IGNORE_PERCENTILE: 5 ENABLE_COMPOUND_SPLITTING: 'false' image: cr.weaviate.io/semitechnologies/contextionary:en0.16.0-v1.2.1 ports: - 9999:9999 ner-transformers: image: cr.weaviate.io/semitechnologies/ner-transformers:dbmdz-bert-large-cased-finetuned-conll03-english ... ``` Variable explanations: * `NER_INFERENCE_API`: where the qna module is running ## How to use (GraphQL) To make use of the modules capabilities, simply extend your query with the following new `_additional` property: ### GraphQL Token This module adds a search filter to the GraphQL `_additional` field in queries: `token{}`. This new filter takes the following arguments: | Field | Data Type | Required | Example value | Description | |- |- |- |- |- | | `properties` | list of strings | yes | `["summary"]` | The properties of the queries Class which contains text (`text` or `string` Datatype). You must provide at least one property | | `certainty` | float | no | `0.75` | Desired minimal certainty or confidence that the recognized token must have. The higher the value, the stricter the token classification. If no certainty is set, all tokens that are found by the model will be returned. | | `limit` | int | no | `1` | The maximum amount of tokens returned per data object in total. | ### Example query import CodeNerTransformer from '/_includes/code/ner-transformers-module.mdx'; ### GraphQL response The answer is contained in a new GraphQL `_additional` property called `tokens`, which returns a list of tokens. It contains the following fields: * `entity` (`string`): The Entity group (classified token) * `word` (`string`): The word that is recognized as entity * `property` (`string`): The property in which the token is found * `certainty` (`float`): 0.0-1.0 of how certain the model is that the token is correctly classified * `startPosition` (`int`): The position of the first character of the word in the property value * `endPosition` (`int`): The position of the last character of the word in the property value ### Example response ```json { "data": { "Get": { "Article": [ { "_additional": { "tokens": [ { "property": "title", "entity": "PER", "certainty": 0.9894614815711975, "word": "Sarah", "startPosition": 11, "endPosition": 16 }, { "property": "title", "entity": "LOC", "certainty": 0.7529033422470093, "word": "London", "startPosition": 31, "endPosition": 37 } ] }, "title": "My name is Sarah and I live in London" } ] } }, "errors": null } ``` ## Use another NER Transformer module from HuggingFace You can build a Docker image which supports any model from the [Hugging Face model hub](https://huggingface.co/models) with a two-line Dockerfile. In the following example, we are going to build a custom image for the [`Davlan/bert-base-multilingual-cased-ner-hrl` model](https://huggingface.co/Davlan/bert-base-multilingual-cased-ner-hrl). #### Step 1: Create a `Dockerfile` Create a new `Dockerfile`. We will name it `my-model.Dockerfile`. Add the following lines to it: ``` FROM semitechnologies/ner-transformers:custom RUN chmod +x ./download.py RUN MODEL_NAME=Davlan/bert-base-multilingual-cased-ner-hrl ./download.py ``` #### Step 2: Build and tag your Dockerfile. We will tag our Dockerfile as `davlan-bert-base-multilingual-cased-ner-hrl`: ``` docker build -f my-model.Dockerfile -t davlan-bert-base-multilingual-cased-ner-hrl . ``` #### Step 3: That's it! You can now push your image to your favorite registry or reference it locally in your Weaviate `docker-compose.yml` using the Docker tag `davlan-bert-base-multilingual-cased-ner-hrl`. ## How it works (under the hood) The code for the application in this repo works well with models that take in a text input like `My name is Sarah and I live in London` and return information in JSON format like this: ```json [ { "entity_group": "PER", "score": 0.9985478520393372, "word": "Sarah", "start": 11, "end": 16 }, { "entity_group": "LOC", "score": 0.999621570110321, "word": "London", "start": 31, "end": 37 } ] ``` The Weaviate NER Module then takes this output and processes this to GraphQL output. ## Model license(s) The `ner-transformers` module is compatible with various models, each with their own license. For detailed information, see the license of the model you are using in the [Hugging Face Model Hub](https://huggingface.co/models). It is your responsibility to evaluate whether the terms of its license(s), if any, are appropriate for your intended use. ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Modules/Qna Openai (docs/weaviate/modules/qna-openai.md) --- title: Question Answering - OpenAI sidebar_position: 41 image: og/docs/modules/qna-openai.jpg # tags: ['qna', 'qna-openai', 'transformers', 'openai'] --- :::caution OpenAI generative integration recommended for new projects Currently, `qna-openai` is not maintained and uses older models such as `gpt-3.5-turbo-instruct`. For new projects, we recommend using the [OpenAI generative integration](../model-providers/openai/generative.md) instead of `qna-openai`. Additionally, the generative integration is more versatile and can be used for a wider range of use cases, not limited to question answering. Since `gpt-3.5-turbo-instruct` is not necessarily strictly trained for question answering, there are limited use cases where `qna-openai` is the best choice. ::: ## In short * The OpenAI Question and Answer (Q&A) module is a Weaviate module for answer extraction from data through the OpenAI [completions endpoint](https://platform.openai.com/docs/api-reference/completions) or the Azure OpenAI equivalent. * The module depends on a text vectorization module that should be running with Weaviate. * The module adds an `ask {}` operator to the GraphQL `Get {}` queries * The module returns a max. of 1 answer in the GraphQL `_additional {}` field. * The answer with the highest `certainty` (confidence level) will be returned. import OpenAIOrAzureOpenAI from '/_includes/openai.or.azure.openai.mdx'; ## Introduction The Question and Answer (Q&A) OpenAI module is a Weaviate module for answer extraction from data. It uses an OpenAI completions endpoint to try and extract an answer from the most relevant docs. This module can be used in GraphQL `Get{...}` queries, as a search operator. The `qna-openai` module tries to find an answer in the data objects of the specified class. If an answer is found within the given `certainty` range, it will be returned in the GraphQL `_additional { answer { ... } }` field. There will be a maximum of 1 answer returned, if this is above the optionally set `certainty`. The answer with the highest `certainty` (confidence level) will be returned. ## Inference API key `qna-openai` requires an API key from OpenAI or Azure OpenAI. :::tip You only need to provide one of the two keys, depending on which service (OpenAI or Azure OpenAI) you are using. ::: ## Organization name For requests that require the OpenAI organization name, you can provide it at query time by adding it to the HTTP header: - `"X-OpenAI-Organization": "YOUR-OPENAI-ORGANIZATION"` for OpenAI ### Providing the key to Weaviate You can provide your API key in two ways: 1. During the **configuration** of your Docker instance, by adding `OPENAI_APIKEY` or `AZURE_APIKEY` as appropriate under `environment` to your `Docker Compose` file, like this: ```yaml environment: OPENAI_APIKEY: 'your-key-goes-here' # For use with OpenAI. Setting this parameter is optional; you can also provide the key at runtime. AZURE_APIKEY: 'your-key-goes-here' # For use with Azure OpenAI. Setting this parameter is optional; you can also provide the key at runtime. ... ``` 2. At **run-time** (recommended), by providing `"X-OpenAI-Api-Key"` or `"X-Azure-Api-Key"` through the request header. You can provide it using the Weaviate client, like this: import ClientKey from '/_includes/code/core.client.openai.apikey.mdx'; ## Module configuration :::tip If you use Weaviate Cloud (WCD), this module is already enabled and pre-configured. You cannot edit the configuration in WCD. ::: ### Docker Compose file (Weaviate Database only) You can enable the OpenAI Q&A module in your Docker Compose file (e.g. `docker-compose.yml`). Add the `qna-openai` module (alongside any other module you may need) to the `ENABLE_MODULES` property, like this: ``` ENABLE_MODULES: 'text2vec-openai,qna-openai' ``` Here is a full example of a Docker configuration, which uses the `qna-openai` module in combination with `text2vec-openai`: ```yaml --- services: weaviate: command: - --host - 0.0.0.0 - --port - '8080' - --scheme - http image: cr.weaviate.io/semitechnologies/weaviate:||site.weaviate_version|| ports: - 8080:8080 - 50051:50051 restart: on-failure:0 environment: QUERY_DEFAULTS_LIMIT: 25 AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true' PERSISTENCE_DATA_PATH: '/var/lib/weaviate' ENABLE_MODULES: 'text2vec-openai,qna-openai' OPENAI_APIKEY: sk-foobar # For use with OpenAI. Setting this parameter is optional; you can also provide the key at runtime. OPENAI_ORGANIZATION: your-orgname # For use with OpenAI. Setting this parameter is optional; you can also provide the key at runtime. AZURE_APIKEY: sk-foobar # For use with Azure OpenAI. Setting this parameter is optional; you can also provide the key at runtime. CLUSTER_HOSTNAME: 'node1' ``` ## Schema configuration You can define settings for this module in the schema. ### OpenAI vs Azure OpenAI - **OpenAI** users can optionally set the `model` parameter. - **Azure OpenAI** users must set the parameters `resourceName` and `deploymentId`. ### Model parameters You can also configure additional parameters for the model through the parameters shown below. ### Example schema For example, the following schema configuration will set Weaviate to use the `qna-openai` model with the `Document` class. The following schema configuration uses the `gpt-3.5-turbo-instruct` model. ```json { "classes": [ { "class": "Document", "description": "A class called document", "vectorizer": "text2vec-openai", "moduleConfig": { "qna-openai": { "model": "gpt-3.5-turbo-instruct", // For OpenAI "resourceName": "", // For Azure OpenAI "deploymentId": "", // For Azure OpenAI "maxTokens": 16, // Applicable to both OpenAI and Azure OpenAI "temperature": 0.0, // Applicable to both OpenAI and Azure OpenAI "topP": 1, // Applicable to both OpenAI and Azure OpenAI "frequencyPenalty": 0.0, // Applicable to both OpenAI and Azure OpenAI "presencePenalty": 0.0 // Applicable to both OpenAI and Azure OpenAI } }, "properties": [ { "dataType": [ "text" ], "description": "Content that will be vectorized", "name": "content" } ] } ] } ``` For information on how to use the individual parameters you [can check here](https://platform.openai.com/docs/api-reference/completions) ## How to use This module adds a search operator to GraphQL `Get{...}` queries: `ask{}`. This operator takes the following arguments: | Field | Data Type | Required | Example value | Description | |- |- |- |- |- | | `question` | string | yes | `"What is the name of the Dutch king?"` | The question to be answered. | | `properties` | list of strings | no | `["summary"]` | The properties of the queries Class which contains text. If no properties are set, all are considered. | Notes: * The GraphQL `Explore { }` function does support the `ask` searcher, but the result is only a beacon to the object containing the answer. It is thus not any different from performing a nearText semantic search with the question. No extraction is happening. * You cannot use the `'ask'` operator along with a `'neaXXX'` operator! ### Example query import CodeQNAOpenAIAsk from '/_includes/code/qna-openai.ask.mdx'; ### GraphQL response The answer is contained in a new GraphQL `_additional` property called `answer`. It contains the following fields: * `hasAnswer` (`boolean`): could an answer be found? * `result` (nullable `string`): An answer if one could be found. `null` if `hasAnswer==false` * `property` (nullable `string`): The property which contains the answer. `null` if `hasAnswer==false` * `startPosition` (`int`): The character offset where the answer starts. `0` if `hasAnswer==false` * `endPosition` (`int`): The character offset where the answer ends `0` if `hasAnswer==false` Note: `startPosition`, `endPosition` and `property` in the response are not guaranteed to be present. They are calculated by a case-insensitive string matching function against the input text. If the transformer model formats the output differently (e.g. by introducing spaces between tokens which were not present in the original input), the calculation of the position and determining the property fails. ### Example response ```json { "data": { "Get": { "Document": [ { "_additional": { "answer": { "hasAnswer": true, "result": " Stanley Kubrick is an American filmmaker who is best known for his films, including \"A Clockwork Orange,\" \"Eyes Wide Shut,\" and \"The Shining.\"" } } } ] } } } ``` ### Token limits If the number of input tokens exceed the limit of the model, the module will return the OpenAI API's error. ## How it works (under the hood) Under the hood, the model uses a two-step approach. First it performs a semantic search to find the documents (e.g. a Sentence, Paragraph, Article, etc.) most likely to contain the answer. In a second step, Weaviate creates the required prompt as an input to an external call made to the OpenAI Completions endpoint. Weaviate uses the most relevant documents to establish a prompt for which OpenAI extracts the answer. There are three possible outcomes: 1. No answer was found because the question can not be answered, 2. An answer was found, but did not meet the user-specified minimum certainty, so it was discarded (typically the case when the document is on topic, but does not contain an actual answer to the question), and 3. An answer was found that matches the desired certainty. It is returned to the user. The module performs a semantic search under the hood, so a `text2vec-...` module is required. It does not need to be of the same type as the `qna-...` module. For example, you can use a `text2vec-contextionary` module to perform the semantic search, and a `qna-openai` module to extract the answer. ## Additional information ### Available models We recommend using: - `gpt-3.5-turbo-instruct` The following models are now deprecated: - `text-ada-001` - `text-babbage-001` - `text-curie-001` - `text-davinci-002` - `text-davinci-003` ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Modules/Qna Transformers (docs/weaviate/modules/qna-transformers.md) --- title: Question Answering - transformers description: Add QnA Transformers to Weaviate for accurate question answering and insights. sidebar_position: 40 image: og/docs/modules/qna-transformers.jpg # tags: ['qna', 'qna-transformers', 'transformers'] --- ## In short * The Question and Answer (Q&A) module is a Weaviate module for answer extraction from data. * The module depends on a text vectorization module that should be running with Weaviate. * The module adds an `ask {}` operator to the GraphQL `Get {}` queries * The module returns a max. of 1 answer in the GraphQL `_additional {}` field. * The answer with the highest `certainty` (confidence level) will be returned. ## Introduction The Question and Answer (Q&A) module is a Weaviate module for answer extraction from data. It uses BERT-related models for finding and extracting answers. This module can be used in GraphQL `Get{...}` queries, as a search operator. The `qna-transformers` module tries to find an answer in the data objects of the specified class. If an answer is found within the given `certainty` range, it will be returned in the GraphQL `_additional { answer { ... } }` field. There will be a maximum of 1 answer returned, if this is above the optionally set `certainty`. The answer with the highest `certainty` (confidence level) will be returned. There are currently five different Question Answering models available (source: [Hugging Face Model Hub](https://huggingface.co/models)): [`distilbert-base-uncased-distilled-squad (uncased)`](https://huggingface.co/distilbert-base-uncased-distilled-squad), [`bert-large-uncased-whole-word-masking-finetuned-squad (uncased)`](https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad), [`distilbert-base-cased-distilled-squad (cased)`](https://huggingface.co/distilbert-base-cased-distilled-squad), [`deepset/roberta-base-squad2`](https://huggingface.co/deepset/roberta-base-squad2), and [`deepset/bert-large-uncased-whole-word-masking-squad2 (uncased)`](https://huggingface.co/deepset/bert-large-uncased-whole-word-masking-squad2). Note that not all models perform well on every dataset and use case. We recommend to use `bert-large-uncased-whole-word-masking-finetuned-squad (uncased)`, which performs best on most datasets (although it's quite heavyweighted). Starting with `v1.10.0`, the answer score can be used as a reranking factor for the search results. ## How to enable (module configuration) ### Docker Compose The Q&A module can be added as a service to the Docker Compose file. You must have a text vectorizer like `text2vec-contextionary` or `text2vec-transformers` running. An example Docker Compose file for using the `qna-transformers` module (`bert-large-uncased-whole-word-masking-finetuned-squad (uncased)`) in combination with the `text2vec-transformers`is as follows: ```yaml --- services: weaviate: command: - --host - 0.0.0.0 - --port - '8080' - --scheme - http image: cr.weaviate.io/semitechnologies/weaviate:||site.weaviate_version|| ports: - 8080:8080 - 50051:50051 restart: on-failure:0 environment: TRANSFORMERS_INFERENCE_API: 'http://text2vec-transformers:8080' QNA_INFERENCE_API: "http://qna-transformers:8080" QUERY_DEFAULTS_LIMIT: 25 AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true' PERSISTENCE_DATA_PATH: '/var/lib/weaviate' ENABLE_MODULES: 'text2vec-transformers,qna-transformers' CLUSTER_HOSTNAME: 'node1' text2vec-transformers: image: cr.weaviate.io/semitechnologies/transformers-inference:sentence-transformers-msmarco-distilbert-base-v2 environment: ENABLE_CUDA: '1' NVIDIA_VISIBLE_DEVICES: all deploy: resources: reservations: devices: - capabilities: [gpu] qna-transformers: image: cr.weaviate.io/semitechnologies/qna-transformers:bert-large-uncased-whole-word-masking-finetuned-squad environment: ENABLE_CUDA: '1' NVIDIA_VISIBLE_DEVICES: all deploy: resources: reservations: devices: - capabilities: [gpu] ... ``` Variable explanations: * `QNA_INFERENCE_API`: where the qna module is running * `ENABLE_CUDA`: if set to 1 it uses GPU (if available on the host machine) _Note: at the moment, text vectorization modules cannot be combined in a single setup. This means that you can either enable the `text2vec-contextionary`, the `text2vec-transformers` or no text vectorization module._ ## How to use (GraphQL) ### GraphQL Ask search This module adds a search operator to GraphQL `Get{...}` queries: `ask{}`. This new operator takes the following arguments: | Field | Data Type | Required | Example value | Description | |- |- |- |- |- | | `question` | string | yes | `"What is the name of the Dutch king?"` | The question to be answered. | | `certainty` | float | no | `0.75` | Desired minimal certainty or confidence of answer to the question. The higher the value, the stricter the search becomes. The lower the value, the fuzzier the search becomes. If no certainty is set, any answer that could be extracted will be returned| | `properties` | list of strings | no | `["summary"]` | The properties of the queries Class which contains text. If no properties are set, all are considered. | | `rerank` | bool | no | `true` | If enabled, the qna module will rerank the result based on the answer score. For example, if the 3rd result - as determined by the previous (semantic) search contained the most likely answer, result 3 will be pushed to position 1, etc. *Not supported prior to v1.10.0* | Notes: * The GraphQL `Explore { }` function does support the `ask` searcher, but the result is only a beacon to the object containing the answer. It is thus not any different from performing a nearText semantic search with the question. No extraction is happening. * You cannot use the `'ask'` operator along with a `'nearXXX'` operator! ### Example query import CodeQnaTransformer from '/_includes/code/qna-transformers.ask.mdx'; ### GraphQL response The answer is contained in a new GraphQL `_additional` property called `answer`. It contains the following fields: * `hasAnswer` (`boolean`): could an answer be found? * `result` (nullable `string`): An answer if one could be found. `null` if `hasAnswer==false` * `certainty` (nullable `float`): The certainty of the answer returned. `null` if `hasAnswer==false` * `property` (nullable `string`): The property which contains the answer. `null` if `hasAnswer==false` * `startPosition` (`int`): The character offset where the answer starts. `0` if `hasAnswer==false` * `endPosition` (`int`): The character offset where the answer ends `0` if `hasAnswer==false` Note: `startPosition`, `endPosition` and `property` in the response are not guaranteed to be present. They are calculated by a case-insensitive string matching function against the input text. If the transformer model formats the output differently (e.g. by introducing spaces between tokens which were not present in the original input), the calculation of the position and determining the property fails. ### Example response ```json { "data": { "Get": { "Article": [ { "_additional": { "answer": { "certainty": 0.73, "endPosition": 26, "hasAnswer": true, "property": "summary", "result": "king willem - alexander", "startPosition": 48 } }, "title": "Bruised Oranges - The Dutch royals are botching covid-19 etiquette" } ] } }, "errors": null } ``` ## Custom Q&A Transformer module You can use the same approach as for `text2vec-transformers`, see [here](/weaviate/model-providers/transformers/embeddings-custom-image.md), i.e. either pick one of the pre-built containers or build your own container from your own model using the `semitechnologies/qna-transformers:custom` base image. Make sure that your model is compatible with Hugging Face's `transformers.AutoModelForQuestionAnswering`. ## How it works (under the hood) Under the hood, the model uses a two-step approach. First it performs a semantic search to find the documents (e.g. a Sentence, Paragraph, Article, etc.) most likely to contain the answer. In a second step, a BERT-style answer extraction is performed on all `text` and `string` properties of the document. There are now three possible outcomes: 1. No answer was found because the question can not be answered, 2. An answer was found, but did not meet the user-specified minimum certainty, so it was discarded (typically the case when the document is on topic, but does not contain an actual answer to the question), and 3. An answer was found that matches the desired certainty. It is returned to the user. The module performs a semantic search under the hood, so a `text2vec-...` module is required. It does not need to be of the same type as the `qna-...` module. For example, you can use a `text2vec-contextionary` module to perform the semantic search, and a `qna-transformers` module to extract the answer. ### Automatic sliding window for long documents If a text value in a data object is longer than 512 tokens, the Q&A Transformer module automatically splits the text into smaller texts. The module uses a sliding window, i.e. overlapping pieces of text, to avoid a scenario that an answer cannot be found if it lies on a boundary. If an answer lies on the boundary, the Q&A module returns the result (answer) with the highest score (as the sliding mechanism could lead to duplicates). ## Model license(s) The `qna-transformers` module is compatible with various models, each with their own license. For detailed information, see the license of the model you are using in the [Hugging Face Hub](https://huggingface.co/models). It is your responsibility to evaluate whether the terms of its license(s), if any, are appropriate for your intended use. ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Modules/Ref2vec Centroid (docs/weaviate/modules/ref2vec-centroid.md) --- title: Ref2Vec Centroid Vectorizer description: Enhance vector search with Ref2Vec Centroid for stronger data representations. sidebar_position: 25 image: og/docs/modules/ref2vec-centroid.jpg # tags: ['ref2vec', 'ref2vec-centroid', 'centroid'] --- ## Introduction The `ref2Vec-centroid` module is used to calculate object vectors based on the centroid of referenced vectors. The idea is that this centroid vector would be calculated from the vectors of an object's references, enabling associations between clusters of objects. This is useful in applications such as making suggestions based on the aggregation of a user's actions or preferences. ## How to enable ### Weaviate Cloud This module is enabled by default on the WCD. ### Weaviate Database Which modules to use in a Weaviate instance can be specified in the `Docker Compose` file. Ref2Vec-centroid can be added like this: ```yaml --- services: weaviate: command: - --host - 0.0.0.0 - --port - '8080' - --scheme - http image: cr.weaviate.io/semitechnologies/weaviate:||site.weaviate_version|| ports: - 8080:8080 - 50051:50051 restart: on-failure:0 environment: QUERY_DEFAULTS_LIMIT: 25 AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true' PERSISTENCE_DATA_PATH: '/var/lib/weaviate' ENABLE_MODULES: 'ref2vec-centroid' CLUSTER_HOSTNAME: 'node1' ... ``` ## How to configure In your Weaviate schema, you must define how you want this module to vectorize your data. If you are new to Weaviate schemas, you might want to check out the [tutorial on the Weaviate schema](../starter-guides/managing-collections/index.mdx) first. For example, here is an `Article` class which is configured to use ref2vec-centroid. Doing so requires only a class-level `moduleConfig`, containing two fields: 1. `referenceProperties`: a list of the class' reference properties which should be used during the calculation of the centroid. 2. `method`: the method by which the centroid is calculated. Currently only `mean` is supported. The `Article` class specifies its `hasParagraphs` property as the only reference property to be used in the calculation of an `Article` object's vector. It is important to note that unlike the other vectorizer modules (e.g. text2vec/multi2vec/img2vec), ref2vec-centroid does not generate embeddings based on the contents of an object. Rather, the point of this module is to calculate an object's vector based on vectors of its *references*. In this case, the `Paragraph` class is configured to generate vectors using the text2vec-contextionary module. Thus, the vector embedding of the `Article` class is an average of text2vec-contextionary vectors sourced from referenced `Paragraph` instances. Although this example uses text2vec-contextionary to generate vectors for the `Paragraph` class, ref2vec-centroid's behavior remains identical for user-provided vectors. In such a case, ref2vec-centroid's output will still be calculated as an average of the reference vectors; the only difference being the provenance of the reference vectors. ```json { "classes": [ { "class": "Article", "description": "A class representing a published article", "moduleConfig": { "ref2vec-centroid": { "referenceProperties": ["hasParagraphs"], "method": "mean" } }, "properties": [ { "dataType": [ "text" ], "description": "Title of the article", "name": "title" } , { "dataType": [ "Paragraph" ], "description": "Paragraphs belonging to this article", "name": "hasParagraphs" } ], "vectorizer": "ref2vec-centroid" }, { "class": "Paragraph", "description": "Paragraphs belonging to an Article", "properties": [ { "dataType": [ "text" ], "description": "Content that will be vectorized", "moduleConfig": { "text2vec-contextionary": { "skip": false, "vectorizePropertyName": false } }, "name": "content" } ], "vectorizer": "text2vec-contextionary" } ] } ``` ## How to use Now that the `Article` class is properly configured to use the ref2vec-centroid module, we can begin to create some objects. If there are not yet any `Paragraph` objects to reference, or if we simply don't want to reference a `Paragraph` object yet, any newly created `Article` object will have its vector set to `nil`. Once we are ready to reference one or more existing `Paragraph` objects (with non-nil vectors), our `Article` object will automatically be assigned a centroid vector, calculated using the vectors from all the `Paragraph` objects which are referenced by our `Article` object. ### Updating the centroid An object whose class is configured to use ref2vec-centroid will have its vector calculated (or recalculated) as a result of these events: - Creating the object with references already assigned as properties - Object `POST`: create a single new object with references - Batch object `POST`: create multiple objects at once, each with references - Updating an existing object's list of references. Note that this can happen several ways: - Object `PUT`: update all of the object's properties with a new set of references. This totally replaces the object's existing reference list with the newly provided one - Object `PATCH`: update an existing object by adding any newly provided reference(s) to the object's existing reference list - Reference `POST`: create a new reference to an existing object - Reference `PUT`: update all of the object's references - Deleting references from the object. Note that this can happen several ways: - Object `PUT`: update all of the object's properties, removing all references - Reference `DELETE`: delete an existing reference from the object's list of references **Note:** Adding references in batches is not currently supported. This is because the batch reference feature is specifically built to avoid the cost of updating the vector index. If this is an important use case for you, open a [feature request](https://github.com/weaviate/weaviate/issues/new) on GitHub. ### Making queries This module can be used with the existing [nearVector](/weaviate/api/graphql/search-operators.md#nearvector) and [`nearObject`](/weaviate/api/graphql/search-operators.md#nearobject) filters. It does not add any additional GraphQL extensions like `nearText`. ## Additional information :::caution It is important to note that updating a _referenced_ object will not automatically trigger an update to the _referencing_ object's vector. ::: In other words, using our `Article`/`Paragraph` example: Let's say an `Article` object, `"On the Philosophy of Modern Ant Colonies"`, references three `Paragraph` objects: `"intro"`, `"body"`, and `"conclusion"`. Over time, `"body"` may be updated as more research has been conducted on the dynamic between worker ants and soldier ants. In this case, the existing vector for the article will not be updated with a new vector based on the refactored `"body"`. If we want `"On the Philosophy of Modern Ant Colonies"`'s centroid vector to be recalculated, we would need to otherwise trigger an update. For example, we could either remove the reference to `"body"` and add it back, or simply `PUT` the `Article` object with an identical object. ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Modules/Spellcheck (docs/weaviate/modules/spellcheck.md) --- title: Spell Check description: Integrate spellcheck in Weaviate to improve text data quality and search accuracy. sidebar_position: 70 image: og/docs/modules/text-spellcheck.jpg # tags: ['modules', 'other modules', 'spellcheck'] --- ## In short * The Spell Check module is a Weaviate module for spell checking of raw text in GraphQL queries. * The module depends on a Python spellchecking library. * The module adds a `spellCheck {}` filter to the GraphQL `nearText {}` search arguments. * The module returns the spelling check result in the GraphQL `_additional { spellCheck {} }` field. ## Introduction The Spell Check module is a Weaviate module for checking spelling in raw texts in GraphQL query inputs. Using the [Python spellchecker](https://pypi.org/project/pyspellchecker/) library, the module analyzes text, gives a suggestion and can force an autocorrection. ## How to enable (module configuration) ### Docker Compose The Spell Check module can be added as a service to the Docker Compose file. You must have a text vectorizer like `text2vec-contextionary` or `text2vec-transformers` running. An example Docker Compose file for using the `text-spellcheck` module with the `text2vec-contextionary` is here: ```yaml --- services: weaviate: command: - --host - 0.0.0.0 - --port - '8080' - --scheme - http image: cr.weaviate.io/semitechnologies/weaviate:||site.weaviate_version|| ports: - 8080:8080 - 50051:50051 restart: on-failure:0 environment: CONTEXTIONARY_URL: contextionary:9999 SPELLCHECK_INFERENCE_API: "http://text-spellcheck:8080" QUERY_DEFAULTS_LIMIT: 25 AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true' PERSISTENCE_DATA_PATH: '/var/lib/weaviate' ENABLE_MODULES: 'text2vec-contextionary,text-spellcheck' CLUSTER_HOSTNAME: 'node1' contextionary: environment: OCCURRENCE_WEIGHT_LINEAR_FACTOR: 0.75 EXTENSIONS_STORAGE_MODE: weaviate EXTENSIONS_STORAGE_ORIGIN: http://weaviate:8080 NEIGHBOR_OCCURRENCE_IGNORE_PERCENTILE: 5 ENABLE_COMPOUND_SPLITTING: 'false' image: cr.weaviate.io/semitechnologies/contextionary:en0.16.0-v1.2.1 ports: - 9999:9999 text-spellcheck: image: cr.weaviate.io/semitechnologies/text-spellcheck-model:pyspellchecker-d933122 ... ``` Variable explanations: * `SPELLCHECK_INFERENCE_API`: where the spellcheck module is running ## How to use (GraphQL) Use the spellchecker module to verify at query time that user-provided search queries are spelled correctly and even suggest alternative, correct spellings. Filters that accept query text include: * [`nearText`](/weaviate/api/graphql/search-operators.md#neartext), if a `text2vec-*` module is used * `ask`, if the [`qna-transformers`](./qna-transformers.md) module is enabled There are two ways to use this module: spell checking, and autocorrection. ### Spell checking The module provides a new GraphQL `_additional` property which can be used to check (but not alter) the provided queries. #### Example query import SpellCheckModule from '/_includes/code/spellcheck-module.mdx'; #### GraphQL response The result is contained in a new GraphQL `_additional` property called `spellCheck`. It contains the following fields: * `changes`: a list with the following fields: * `corrected` (`string`): the corrected spelling if a correction is found * `original` (`string`): the original word in the query * `didYouMean`: the corrected full text in the query * `originalText`: the original full text in the query * `location`: the location of the misspelled string in the query #### Example response ```json { "data": { "Get": { "Article": [ { "_additional": { "spellCheck": [ { "changes": [ { "corrected": "housing", "original": "houssing" } ], "didYouMean": "housing prices", "location": "nearText.concepts[0]", "originalText": "houssing prices" } ] }, "title": "..." } ] } }, "errors": null } ``` ### Autocorrect The module extends existing `text2vec-*` modules with an `autoCorrect` flag, which can be used to automatically correct the query if it was misspelled: #### Example query ```graphql { Get { Article(nearText: { concepts: ["houssing prices"], autocorrect: true }) { title _additional { spellCheck { changes { corrected original } didYouMean location originalText } } } } } ``` ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Modules/Sum Transformers (docs/weaviate/modules/sum-transformers.md) --- title: Summarization description: Summarize data efficiently using the SUM Transformers module in Weaviate. sidebar_position: 80 image: og/docs/modules/sum-transformers.jpg # tags: ['transformers'] --- ## In short * The Summarization (`sum-transformers`) module is a Weaviate module that summarizes whole paragraphs into a short text. * The module containerizes a summarization-focussed transformers model for Weaviate to connect to. We make pre-built models available here, but you can also attach another transformer model from Hugging Face or even a custom model. * The module adds a `summary {}` filter to the GraphQL `_additional {}` field. * The module returns the results in the GraphQL `_additional { summary {} }` field. ## Introduction As the name indicates, the summarization module can produce a summary of Weaviate text objects at query time. **For example**, it allows us to run a query on our data in Weaviate, which can take a text like this: > "The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct." and transform it to a short sentence like this: > "The Eiffel Tower is a landmark in Paris, France." :::note GPUs preferred For maximum performance of your queries, transformer-based models should run with GPUs. CPUs can be used, however, this will significantly slow down your queries. ::: ### Available modules Here is the current list of available `SUM` modules - sourced from [Hugging Face Model Hub](https://huggingface.co/models): * [`bart-large-cnn`](https://huggingface.co/facebook/bart-large-cnn) * [`pegasus-xsum`](https://huggingface.co/google/pegasus-xsum) ## How to enable (module configuration) ### Docker Compose The `sum-transformers` module can be added as a service to the Docker Compose file. You must have a text vectorizer like `text2vec-contextionary` or `text2vec-transformers` running. An example Docker Compose file for using the `sum-transformers` module (with the `facebook-bart-large-cnn` model) in combination with the `text2vec-contextionary` vectorizer module is below: ```yaml --- services: weaviate: command: - --host - 0.0.0.0 - --port - '8080' - --scheme - http image: cr.weaviate.io/semitechnologies/weaviate:||site.weaviate_version|| ports: - 8080:8080 - 50051:50051 restart: on-failure:0 environment: CONTEXTIONARY_URL: contextionary:9999 SUM_INFERENCE_API: "http://sum-transformers:8080" QUERY_DEFAULTS_LIMIT: 25 AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true' PERSISTENCE_DATA_PATH: '/var/lib/weaviate' ENABLE_MODULES: 'text2vec-contextionary,sum-transformers' CLUSTER_HOSTNAME: 'node1' contextionary: environment: OCCURRENCE_WEIGHT_LINEAR_FACTOR: 0.75 EXTENSIONS_STORAGE_MODE: weaviate EXTENSIONS_STORAGE_ORIGIN: http://weaviate:8080 NEIGHBOR_OCCURRENCE_IGNORE_PERCENTILE: 5 ENABLE_COMPOUND_SPLITTING: 'false' image: cr.weaviate.io/semitechnologies/contextionary:en0.16.0-v1.2.1 ports: - 9999:9999 sum-transformers: image: cr.weaviate.io/semitechnologies/sum-transformers:facebook-bart-large-cnn-1.2.0 # image: cr.weaviate.io/semitechnologies/sum-transformers:google-pegasus-xsum-1.2.0 # Could be used instead ... ``` Variable explanations: * `SUM_INFERENCE_API`: where the summarization module is running ## How to use (GraphQL) To make use of the modules capabilities, extend your query with the following new `_additional` property: ### GraphQL Token This module adds a search filter to the GraphQL `_additional` field in queries: `summary{}`. This new filter takes the following arguments: | Field | Data Type | Required | Example value | Description | |- |- |- |- |- | | `properties` | list of strings | yes | `["description"]` | The properties of the queries Class which contains text (`text` or `string` Datatype). You must provide at least one property | ### Example query import CodeSumTransformer from '/_includes/code/sum-transformers-module.mdx'; ### GraphQL response The answer is contained in a new GraphQL `_additional` property called `summary`, which returns a list of tokens. It contains the following fields: * `property` (`string`): The property that was summarized – this is useful when you summarize more than one property * `result` (`string`): The output summary ### Example response ```json { "data": { "Get": { "Article": [ { "_additional": { "summary": [ { "property": "summary", "result": "Finding the perfect pair of jeans can be a challenge." } ] }, "title": "The Most Comfortable Gap Jeans to Shop Now" } ] } }, "errors": null } ``` ## Use another Summarization module from Hugging Face You can build a Docker image which supports any summarization model from the [Hugging Face Model Hub](https://huggingface.co/models?pipeline_tag=summarization) with a two-line Dockerfile. In the following example, we are going to build a custom image for the [`google/pegasus-pubmed` model](https://huggingface.co/google/pegasus-pubmed). #### Step 1: Create a `Dockerfile` Create a new `Dockerfile`. We will name it `my-model.Dockerfile`. Add the following lines to it: ``` FROM semitechnologies/sum-transformers:custom RUN chmod +x ./download.py RUN MODEL_NAME=google/pegasus-pubmed ./download.py ``` #### Step 2: Build and tag your Dockerfile. We will tag our Dockerfile as `google-pegasus-pubmed`: ``` docker build -f my-model.Dockerfile -t google-pegasus-pubmed . ``` #### Step 3: Use the image with Weaviate You can now push your image to your favorite registry or reference it locally in your Weaviate `docker-compose.yml` using the Docker tag `google-pegasus-pubmed`. ## How it works (under the hood) The `sum-transformers` module uses transformer-based summarizer models. They are abstractive, in that they generate new text from the input text, rather than to extract particular sentences. For example, a model may take text like this:

See original text

> *The Loch Ness Monster (Scottish Gaelic: Uilebheist Loch Nis), affectionately known as Nessie, is a creature in Scottish folklore that is said to inhabit Loch Ness in the Scottish Highlands. It is often described as large, long-necked, and with one or more humps protruding from the water. Popular interest and belief in the creature has varied since it was brought to worldwide attention in 1933. Evidence of its existence is anecdotal, with a number of disputed photographs and sonar readings.* > *The scientific community explains alleged sightings of the Loch Ness Monster as hoaxes, wishful thinking, and the misidentification of mundane objects. The pseudoscience and subculture of cryptozoology has placed particular emphasis on the creature.*

And summarize it to produce a text like: > *The Loch Ness Monster is said to be a large, long-necked creature. Popular belief in the creature has varied since it was brought to worldwide attention in 1933. Evidence of its existence is disputed, with a number of disputed photographs and sonar readings. The pseudoscience and subculture of cryptozoology has placed particular emphasis on the creature.* Note that much of output does not copy the input verbatim, but is *based on* it. The `sum-transformers` module then delivers this output in the response. :::note Input length Note that like many other language models, summarizer models can only process a limited amount of text. The `sum-transformers` module will be limited to the maximum length of the model it is using. For example, the `facebook/bart-large-cnn` model can only process 1024 tokens. On the other hand, be aware that providing an input of insufficient length and detail may cause the transformer model to [hallucinate](https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)). ::: ## Model license(s) The `sum-transformers` module is compatible with various models, each with their own license. For detailed information, see the license of the model you are using in the [Hugging Face Model Hub](https://huggingface.co/models). It is your responsibility to evaluate whether the terms of its license(s), if any, are appropriate for your intended use. ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Modules/Text2vec Contextionary (docs/weaviate/modules/text2vec-contextionary.md) --- title: "Contextionary Vectorizer" description: Use Text2Vec Contextionary in Weaviate for improved context-based text vectorization. sidebar_position: 10 image: og/docs/modules/text2vec-contextionary.jpg # tags: ['text2vec', 'text2vec-contextionary', 'contextionary'] --- The `text2vec-contextionary` module enables Weaviate to obtain vectors locally using a lightweight model. :::caution Deprecated module The `Contextionary` model is old, and not recommended for any use cases. Instead, we recommend using other modules. If you are looking for a local, lightweight model for testing or development purposes, try [the `text2vec-model2vec` module](../model-providers/model2vec/embeddings.md). For cases where the vector quality is important, such as in production, we recommend using [other model integrations](../model-providers/index.md) that use a more modern, transformer-based architecture. ::: Key notes: - This module is not available on Weaviate Cloud (WCD). - Enabling this module will enable the [`nearText` search operator](/weaviate/api/graphql/search-operators.md#neartext). - This module is based on FastText and uses a weighted mean of word embeddings (WMOWE) to produce the vector. - Available for multiple languages ## Weaviate instance configuration :::info Not applicable to WCD This module is not available on Weaviate Cloud. ::: ### Docker Compose file To use `text2vec-contextionary`, you must enable it in your Docker Compose file (e.g. `docker-compose.yml`). :::tip Use the configuration tool While you can do so manually, we recommend using the [Weaviate configuration tool](/deploy/installation-guides/docker-installation.md#configurator) to generate the `Docker Compose` file. ::: #### Parameters Weaviate: - `ENABLE_MODULES` (Required): The modules to enable. Include `text2vec-contextionary` to enable the module. - `DEFAULT_VECTORIZER_MODULE` (Optional): The default vectorizer module. You can set this to `text2vec-contextionary` to make it the default for all classes. Contextionary: * `EXTENSIONS_STORAGE_MODE`: Location of storage for extensions to the Contextionary * `EXTENSIONS_STORAGE_ORIGIN`: The host of the custom extension storage * `NEIGHBOR_OCCURRENCE_IGNORE_PERCENTILE`: this can be used to hide very rare words. If you set it to '5', this means the 5th percentile of words by occurrence are removed in the nearestNeighbor search (for example used in the GraphQL `_additional { nearestNeighbors }` feature). * `ENABLE_COMPOUND_SPLITTING`: see [here](#compound-splitting). #### Example This configuration enables `text2vec-contextionary`, sets it as the default vectorizer, and sets the parameters for the Contextionary Docker container. ```yaml --- services: weaviate: command: - --host - 0.0.0.0 - --port - '8080' - --scheme - http image: cr.weaviate.io/semitechnologies/weaviate:||site.weaviate_version|| ports: - 8080:8080 - 50051:50051 restart: on-failure:0 environment: CONTEXTIONARY_URL: contextionary:9999 QUERY_DEFAULTS_LIMIT: 25 AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true' PERSISTENCE_DATA_PATH: '/var/lib/weaviate' # highlight-start ENABLE_MODULES: 'text2vec-contextionary' # highlight-end CLUSTER_HOSTNAME: 'node1' # highlight-start contextionary: environment: OCCURRENCE_WEIGHT_LINEAR_FACTOR: 0.75 EXTENSIONS_STORAGE_MODE: weaviate EXTENSIONS_STORAGE_ORIGIN: http://weaviate:8080 NEIGHBOR_OCCURRENCE_IGNORE_PERCENTILE: 5 ENABLE_COMPOUND_SPLITTING: 'false' image: cr.weaviate.io/semitechnologies/contextionary:en0.16.0-v1.2.1 ports: - 9999:9999 # highlight-end ... ``` ## Collection configuration You can configure how the module will behave in each class through the [collection configuration](../manage-collections/vector-config.mdx). ### Vectorization settings You can set vectorizer behavior using the `moduleConfig` section under each class and property: #### Class-level - `vectorizer` - what module to use to vectorize the data. - `vectorizeClassName` – whether to vectorize the class name. Default: `true`. #### Property-level - `skip` – whether to skip vectorizing the property altogether. Default: `false` - `vectorizePropertyName` – whether to vectorize the property name. Default: `false` #### Example ```json { "classes": [ { "class": "Document", "description": "A class called document", "vectorizer": "text2vec-contextionary", "moduleConfig": { // highlight-start "text2vec-contextionary": { "vectorizeClassName": false } // highlight-end }, "properties": [ { "name": "content", "dataType": [ "text" ], "description": "Content that will be vectorized", // highlight-start "moduleConfig": { "text2vec-contextionary": { "skip": false, "vectorizePropertyName": false } } // highlight-end } ], } ] } ``` ### Class/property names If you are using this module and are vectorizing the class or property name, the name(s) must be a part of the `text2vec-contextionary`. To use multiple words as a class or property definition, concatenate them as: - camel case (e.g. `bornIn`) for class or property names, or - snake case (e.g. `born_in`) for property names. For example, the following are acceptable: ```yaml Publication name hasArticles Article title summary wordCount url hasAuthors inPublication # CamelCase (all versions) publication_date # snake_case (from v1.7.2 on) Author name wroteArticles writesFor ``` ## Usage example This is an example of a `nearText` query with `text2vec-contextionary`. import CodeNearText from '/_includes/code/graphql.filters.nearText.mdx'; ## Additional information ### Find concepts To find concepts or words or to check if a concept is part of the Contextionary, use the `v1/modules/text2vec-contextionary/concepts/` endpoint. ```js GET /v1/modules/text2vec-contextionary/concepts/ ``` #### Parameters The only parameter `concept` is a string that should be camelCased in case of compound words or a list of words. #### Response The result contains the following fields: - `"individualWords"`: a list of the results of individual words or concepts in the query, which contains: - `"word"`: a string of requested concept or single word from the concept. - `"present"`: a boolean value which is `true` if the word exists in the Contextionary. - `"info"`: an object with the following fields: - `""nearestNeighbors"`: a list with the nearest neighbors, containing `"word"` and `"distance"` (between the two words in the high dimensional space). Note that `"word"` can also be a data object. - `"vector"`: the raw 300-long vector value. - `"concatenatedWord"`: an object of the concatenated concept. - `"concatenatedWord"`: the concatenated word if the concept given is a camelCased word. - `"singleWords"`: a list of the single words in the concatenated concept. - `"concatenatedVector"`: a list of vector values of the concatenated concept. - `"concatenatedNearestNeighbors"`: a list with the nearest neighbors, containing `"word"` and `"distance"` (between the two words in the high dimensional space). Note that `"word"` can also be a data object. #### Example ```bash curl http://localhost:8080/v1/modules/text2vec-contextionary/concepts/magazine ``` or (note the camelCased compound concept) import CodeContextionary from '/_includes/code/contextionary.get.mdx'; with a result similar to: ```json { "individualWords": [ { "inC11y": true, "info": { "nearestNeighbors": [ { "word": "magazine" }, { "distance": 6.186641, "word": "editorial" }, { "distance": 6.372504, "word": "featured" }, { "distance": 6.5695524, "word": "editor" }, { "distance": 7.0328364, "word": "titled" }, ... ], "vector": [ 0.136228, 0.706469, -0.073645, -0.099225, 0.830348, ... ] }, "word": "magazine" } ] } ``` ### Model details `text2vec-contextionary` (Contextionary) is Weaviate's own language vectorizer that is trained using [fastText](https://fasttext.cc/) on Wiki and CommonCrawl data. The `text2vec-contextionary` model outputs a 300-dimensional vector. This vector is computed by using a Weighted Mean of Word Embeddings (WMOWE) technique. The vector is calculated based on the centroid of the words weighted by the occurrences of the individual words in the original training text-corpus (e.g., the word `"has"` is seen as less important than the word `"apples"`). ### Available languages Contextionary models are available for the following languages: * Trained with on CommonCrawl and Wiki, using GloVe * English * Dutch * German * Czech * Italian * Trained on Wiki * English * Dutch ### Extending the Contextionary Custom words or abbreviations (i.e., "concepts") can be added to `text2vec-contextionary` through the `v1/modules/text2vec-contextionary/extensions/` endpoint. Using this endpoint will enrich the Contextionary with your own words, abbreviations or concepts in context by [transfer learning](https://en.wikipedia.org/wiki/Transfer_learning). Using the `v1/modules/text2vec-contextionary/extensions/` endpoint adds or updates the concepts in real-time. Note that you need to introduce the new concepts in to Weaviate before adding the data, as this will note cause Weaviate to automatically update the vectors. #### Parameters A body (in JSON or YAML) with the extension word or abbreviation you want to add to the Contextionary with the following fields includes a: - `"concept"`: a string with the word, compound word or abbreviation - `"definition"`: a clear description of the concept, which will be used to create the context of the concept and place it in the high dimensional Contextionary space. - `"weight"`: a float with the relative weight of the concept (default concepts in the Contextionary have a weight of 1.0) #### Response The same fields as the input parameters will be in the response body if the extension was successful. #### Example Let's add the concept `"weaviate"` to the Contextionary. import CodeContextionaryExtensions from '/_includes/code/contextionary.extensions.mdx'; You can always check if the new concept exists in the Contextionary: ```bash curl http://localhost:8080/v1/modules/text2vec-contextionary/concepts/weaviate ``` Note that it is not (yet) possible to extend the Contextionary with concatenated words or concepts consisting of more than one word. You can also overwrite current concepts with this endpoint. Let's say you are using the abbreviation `API` for `Academic Performance Index` instead of `Application Programming Interface`, and you want to reposition this concept in the Contextionary: ```bash curl \ -X POST \ -H 'Content-Type: application/json' \ -d '{ "concept": "api", "definition": "Academic Performance Index a measurement of academic performance and progress of individual schools in California", "weight": 1 }' \ http://localhost:8080/v1/modules/text2vec-contextionary/extensions ``` The meaning of the concept `API` has now changed in your Weaviate setting. ### Stopwords Note that stopwords are automatically removed from camelCased and CamelCased names. #### Vectorization behavior Stopwords can be useful, so we don't want to encourage you to leave them out completely. Instead Weaviate will remove them during vectorization. In most cases you won't even notice that this happens in the background, however, there are a few edge cases that might cause a validation error: * If your camelCased class or property name consists **only** of stopwords, validation will fail. Example: `TheInA` is not a valid class name, however, `TheCarInAField` is (and would internally be represented as `CarField`). * If your keyword list contains stop words, they will be removed. However, if every single keyword is a stop word, validation will fail. #### How does Weaviate decide whether a word is a stop word or not? The list of stopwords is derived from the Contextionary version used and is published alongside the Contextionary files. ### Compound splitting Sometimes Weaviate's Contextionary does not understand words which are compounded out of words it would otherwise understand. This impact is far greater in languages that allow for arbitrary compounding (such as Dutch or German) than in languages where compounding is not very common (such as English). #### Effect Imagine you import an object of class `Post` with content `This is a thunderstormcloud`. The arbitrarily compounded word `thunderstormcloud` is not present in the Contextionary. So your object's position will be made up of the only words it recognizes: `"post", "this"` (`"is"` and `"a"` are removed as stopwords). If you check how this content was vectorized using the `_interpretation` feature, you will see something like the following: ```json "_interpretation": { "source": [ { "concept": "post", "occurrence": 62064610, "weight": 0.3623903691768646 }, { "concept": "this", "occurrence": 932425699, "weight": 0.10000000149011612 } ] } ``` To overcome this limitation the optional **Compound Splitting Feature** can be enabled in the Contextionary. It will understand the arbitrary compounded word and interpret your object as follows: ```json "_interpretation": { "source": [ { "concept": "post", "occurrence": 62064610, "weight": 0.3623903691768646 }, { "concept": "this", "occurrence": 932425699, "weight": 0.10000000149011612 }, { "concept": "thunderstormcloud (thunderstorm, cloud)", "occurrence": 5756775, "weight": 0.5926488041877747 } ] } ``` Note that the newly found word (made up of the parts `thunderstorm` and `cloud` has the highest weight in the vectorization. So this meaning, which would have been lost without Compound Splitting, can now be recognized. #### How to enable You can enable Compound Splitting in the Docker Compose file of the `text2vec-contextionary`. See how this is done [here](#compound-splitting). #### Trade-Off Import speed vs Word recognition Compound Splitting runs an any word that is otherwise not recognized. Depending on your dataset, this can lead to a significantly longer import time (up to 100% longer). Therefore, you should carefully evaluate whether the higher precision in recognition or the faster import times are more important to your use case. As the benefit is larger in some languages (e.g. Dutch, German) than in others (e.g. English) this feature is turned off by default. ### Noise filtering So called "noise words" are concatenated words of random words with no easily recognizable meaning. These words are present in the Contextionary training space, but are extremely rare and therefore distributed seemingly randomly. As a consequence, an "ordinary" result of querying features relying on nearest neighbors (additional properties `nearestNeighbors` or `semanticPath`) might contain such noise words as immediate neighbors. To combat this noise, a neighbor filtering feature was introduced in the contextionary, which ignores words of the configured bottom percentile - ranked by occurrence in the respective training set. By default this value is set to the bottom 5th percentile. This setting can be overridden. To set another value, e.g. to ignore the bottom 10th percentile, provide the environment variable `NEIGHBOR_OCCURRENCE_IGNORE_PERCENTILE=10` to the `text2vec-contextionary` container, in the Docker Compose file. ## Model license(s) The `text2vec-contextionary` module is based on the [`fastText`](https://github.com/facebookresearch/fastText/tree/main) library, which is released under the MIT license. See the [license file](https://github.com/facebookresearch/fastText/blob/main/LICENSE) for more information. It is your responsibility to evaluate whether the terms of its license(s), if any, are appropriate for your intended use. ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/Modules/Usage Modules (docs/weaviate/modules/usage-modules.md) --- title: Usage Module description: Add the usage module to collect and upload usage analytics data to Google Cloud Storage (GCS) or AWS S3. --- import UsageModules from '/_includes/feature-notes/usage-modules.mdx'; :::danger This module is in development and breaking changes can and will happen. ::: ### What it does: - Periodically collecting usage data. - Uploading JSON reports to either S3 or GCS. - Supports runtime configuration overrides. - Includes metrics and logging. - Verifies storage permissions before uploading. ## Configuration ### Example configuration with backup and usage ```yaml # environment variables ENABLE_MODULES="backup-gcs,usage-gcs" BACKUP_GCS_BUCKET=weaviate-backups USAGE_GCS_BUCKET=weaviate-usage USAGE_GCS_PREFIX=billing-usage TRACK_VECTOR_DIMENSIONS=true # won't be needed from 1.32.1 RUNTIME_OVERRIDES_ENABLED=true RUNTIME_OVERRIDES_PATH="${PWD}/tools/dev/config.runtime-overrides.yaml" RUNTIME_OVERRIDES_LOAD_INTERVAL=30s # in tools/dev/config.runtime-overrides.yaml usage_scrape_interval: 1h usage_shard_jitter_interval: 100ms #(optional) usage_gcs_bucket: weaviate-usage usage_gcs_prefix: billing ``` ### Environment variables :::tip The usage module must be enabled for any configuration to take effect. While this module is not related to backups, if backups are not enabled it won't collect metrics for backups. ::: ```bash # Enable the usage modules (required) ENABLE_MODULES=usage-gcs # if you want gcs ENABLE_MODULES=usage-s3 # if you want s3 ENABLE_MODULES=usage-s3,usage-gcs # or both ``` #### Runtime overrides :::tip `TRACK_VECTOR_DIMENSIONS=true` is required to collect vector dimension metrics in your usage reports, from `v1.32.1` this will no longer be required. ::: ```shell RUNTIME_OVERRIDES_ENABLED=true RUNTIME_OVERRIDES_PATH="${PWD}/tools/dev/config.runtime-overrides.yaml" RUNTIME_OVERRIDES_LOAD_INTERVAL=30s # Required: Enable vector dimension tracking metrics TRACK_VECTOR_DIMENSIONS=true # won't be needed from 1.32.1 # Collection interval (default: 1h) USAGE_SCRAPE_INTERVAL=2h # (optional) Shard loop jitter (default: 100ms) USAGE_SHARD_JITTER_INTERVAL=50ms # (optional) Policy version (default: 2025-06-01) USAGE_POLICY_VERSION=2025-06-01 # (optional) verify the bucket permission on start (default:false) USAGE_VERIFY_PERMISSIONS=true ``` :::info Enable runtime overrides to avoid needing to restart Weaviate when updating usage module configurations. ::: #### Example `runtime-overrides.yaml` ```yaml usage_scrape_interval: 1s usage_shard_jitter_interval: 100ms # (optional) usage_verify_permissions: true/false # (optional) # usage-gcs config usage_gcs_bucket: weaviate-usage usage_gcs_prefix: billing # usage-s3 config usage_s3_bucket: weaviate-usage usage_s3_prefix: billing ``` #### AWS S3 variables ```bash # Required: S3 bucket name USAGE_S3_BUCKET=my-weaviate-usage-bucket # Optional: Object prefix (default: empty) USAGE_S3_PREFIX=usage-reports ``` #### GCP GCS variables ```bash # Required: GCS bucket name USAGE_GCS_BUCKET=my-weaviate-usage-bucket # Optional: Object prefix (default: empty) USAGE_GCS_PREFIX=usage-reports ``` ### Monitoring The modules provide Prometheus metrics: - `weaviate_usage_{gcs|s3}_operations_total`: Total number of operations for module labels (`operation`:collect/upload, status: success, error). - `weaviate_usage_{gcs|s3}_operation_latency_seconds`: Latency of usage operations in seconds labels (`operation` :collect/upload). - `weaviate_usage_{gcs|s3}_resource_count`: Number of resources tracked by module, labels (`resource_type` :collections/shards/backups). - `weaviate_usage_{gcs|s3}_uploaded_file_size_bytes`: Size of the uploaded usage file in bytes. ### Debug logs See detailed module activity by enabling debugging. ```bash LOG_LEVEL=debug ``` ### Testing If using Minio, set the environment variable `AWS_ENDPOINT`, `AWS_REGION` #### Local ```bash AWS_REGION=us-east-1 AWS_ENDPOINT=http://localhost:9000 ``` #### Cloud ``` AWS_REGION=us-east-1 AWS_ENDPOINT=minio.weaviate.svc.cluster.local:9000 ``` ## Further resources - [Monitoring](/docs/deploy/configuration/monitoring.md) - [Environment variables](/docs/deploy/configuration/env-vars/index.md) ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/More Resources/Example Datasets (docs/weaviate/more-resources/example-datasets.md) --- title: Example datasets sidebar_position: 5 image: og/docs/more-resources.jpg # tags: ['example datasets'] --- ## Multi-Modal Text/Image search using CLIP This example application spins up a Weaviate instance using the [multi2vec-clip](/weaviate/model-providers/transformers/embeddings-multimodal.md) integration, imports a few sample images (you can add your own images, too!) and provides a very simple search frontend in [React](https://reactjs.org/) using the [TypeScript/JavaScript](../client-libraries/typescript/index.mdx) client. [Get started here](https://github.com/weaviate/weaviate-examples/blob/main/clip-multi-modal-text-image-search/README.md) ## Semantic Search through Wikipedia We imported the complete English language Wikipedia article dataset into a single Weaviate instance to conduct semantic search queries through the Wikipedia articles, besides this, we've made all the graph relations between the articles too. We have made the import scripts, pre-processed articles, and backup available so that you can run the complete setup yourself. [Get started here](https://github.com/weaviate/semantic-search-through-Wikipedia-with-Weaviate) ## Meta AI Research - Biggraph on Wikidata We have imported the complete Wikidata PBG model into a Weaviate to search through the entire dataset in < 50 milliseconds (excluding internet latency). The demo GraphQL queries contain both pure vector search and scalar and vector searched mixed queries. [Get started here](https://github.com/weaviate/biggraph-wikidata-search-with-weaviate) ## News publications This dataset contains +/- 1000 random news articles from; Financial Times, New York Times, Guardian, Wallstreet Journal, CNN, Fox News, The Economist, New Yorker, Wired, Vogue, Game Informer. It includes a [schema](../starter-guides/managing-collections/index.mdx) with classes for `Article`, `Publication`, `Category` and `Author`. ### Run with Docker Compose If you want to run this dataset locally, you can run it in one go with Docker Compose. You can run this demo dataset with any `text2vec` module. Examples: #### Text2vec-contextionary The Docker Compose file contains both Weaviate with the `text2vec-contextionary` module and the dataset. Download the Docker Compose file ```bash curl -o docker-compose.yml https://raw.githubusercontent.com/weaviate/weaviate-examples/main/weaviate-contextionary-newspublications/docker-compose.yaml ``` Run Docker (optional: run with `-d` to run Docker in the background) ```bash docker compose up ``` To work with the News Articles demo dataset, connect to `http://localhost:8080/`. #### Text2vec-transformers (without GPU) The Docker Compose file contains both Weaviate with the `text2vec-contextionary` module, `NER` module, `Q&A` module and `spellcheck` module, and the dataset. Download the Docker Compose file ```bash curl -o docker-compose.yml https://raw.githubusercontent.com/weaviate/weaviate-examples/main/weaviate-transformers-newspublications/docker-compose.yml ``` Run Docker (optional: run with `-d` to run Docker in the background) ```bash docker compose up ``` To work with the News Articles demo dataset, connect to `http://localhost:8080/`. #### Text2vec-transformers (with GPU enabled) The Docker Compose file contains both Weaviate with the `text2vec-contextionary` module, `NER` module, `Q&A` module and `spellcheck` module, and the dataset. GPU should be available on your machine when running this configuration. Download the Docker Compose file ```bash curl -o docker-compose.yml https://raw.githubusercontent.com/weaviate/weaviate-examples/main/weaviate-transformers-newspublications/docker-compose-gpu.yaml ``` Run Docker (optional: run with `-d` to run Docker in the background) ```bash docker compose up ``` To work with the News Articles demo dataset, connect to `http://localhost:8080/`. ### Run manually If you have your own version of Weaviate running on an **external** host or localhost **without** Docker Compose; ```bash # WEAVIATE ORIGIN (e.g., https://foobar.weaviate.network), note paragraph basics for setting the local IP export WEAVIATE_ORIGIN=WEAVIATE_ORIGIN # Optionally you can specify which newspaper language you want (only two options `cache-en` or `cache-nl`, if not specified by default it is `cache-en` ) export CACHE_DIR= # Optionally you can set the batch size (if not specified by default 200) export BATCH_SIZE= # Make sure to replace WEAVIATE_ORIGIN with the Weaviate origin as mentioned in the basics above docker run -it -e weaviate_host=$WEAVIATE_ORIGIN -e cache_dir=$CACHE_DIR -e batch_size=$BATCH_SIZE semitechnologies/weaviate-demo-newspublications:latest ``` Usage with Docker on **local with** Docker Compose; _Note: run this from the same directory where the Docker Compose files are located_ ```bash # This gets the Weaviate container name and because the docker uses only lowercase we need to do it too (Can be found manually if 'tr' does not work for you) export WEAVIATE_ID=$(echo ${PWD##*/}_weaviate_1 | tr "[:upper:]" "[:lower:]") # WEAVIATE ORIGIN (e.g., http://localhost:8080), note the paragraph "basics" for setting the local IP export WEAVIATE_ORIGIN="http://$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $WEAVIATE_ID):8080" # WEAVIATE NETWORK (see paragraph: Running on the localhost) export WEAVIATE_NETWORK=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.NetworkID}}{{end}}' $WEAVIATE_ID) # Optionally you can specify which newspaper language you want (only two options `cache-en` or `cache-nl`, if not specified by default it is `cache-en` ) export CACHE_DIR= # Optionally you can set the batch size (if not specified by default 200) export BATCH_SIZE= # Run docker docker run -it --network=$WEAVIATE_NETWORK -e weaviate_host=$WEAVIATE_ORIGIN -e cache_dir=$CACHE_DIR -e batch_size=$BATCH_SIZE semitechnologies/weaviate-demo-newspublications:latest ``` ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/More Resources/Example Use Cases (docs/weaviate/more-resources/example-use-cases.md) --- title: Example use cases and demos sidebar_position: 6 image: og/docs/more-resources.jpg # tags: ['use cases'] --- This page illustrates various use cases for [vector databases](https://weaviate.io/blog/what-is-a-vector-database) by way of open-source demo projects. You can fork and modify any of them. If you would like to contribute your own project to this page, create an issue on [GitHub](https://github.com/weaviate/docs/issues). ## Similarity search A vector databases enables fast, efficient similarity searches on and across any modalities, such as text or images, as well as their combinations. Vector database' similarity search capabilities can be used for other complex use cases, such as recommendation systems in classical machine learning applications. |Title | Description | Modality | Code | | --- | --- | --- | --- | | Plant search | Semantic search over plants. | Text | [JavaScript](https://github.com/weaviate-tutorials/DEMO-text-search-plants) | | Wine search | Semantic search over wines. | Text | [Python](https://github.com/weaviate-tutorials/DEMO-text-search-wines) | | Book recommender system ([Video](https://www.youtube.com/watch?v=SF1ZlRjVsxw)) | Find book recommendations based on search query. | Text | [TypeScript](https://github.com/weaviate/BookRecs) | | Movie recommender system ([Blog](https://medium.com/towards-data-science/recreating-andrej-karpathys-weekend-project-a-movie-search-engine-9b270d7a92e4)) | Find similar movies. | Text | [JavaScript](https://github.com/weaviate-tutorials/awesome-moviate) | | Multilingual Wikipedia Search | Search through Wikipedia in multiple languages. | Text | [TypeScript](https://github.com/weaviate/weaviate-examples/tree/main/cohere-multilingual-wikipedia-search/frontend) | | Podcast search | Semantic search over podcast episodes. | Text | [Python](https://github.com/weaviate-tutorials/DEMO-semantic-search-podcast) | | Video Caption Search| Find the timestamp of the answer to your question in a video. | Text | [Python](https://github.com/weaviate-tutorials/DEMO-text-search-video-captions) | | Facial Recognition | Identify people in images | Image | [Python](https://github.com/weaviate-tutorials/DEMO-face-recognition) | | Image Search over dogs ([Blog](https://weaviate.io/blog/how-to-build-an-image-search-application-with-weaviate)) | Find images of similar dog breeds based on uploaded image. | Image | [Python](https://github.com/weaviate-tutorials/DEMO-image-search-dogs) | | Text to image search | Find images most similar to a text query. | Multimodal | [JavaScript](https://github.com/weaviate-tutorials/DEMO-multimodal-text-to-image-search) | | Text to image and image to image search | Find images most similar to a text or image query. | Multimodal | [Python](https://github.com/weaviate-tutorials/DEMO-multimodal-search) | ## LLMs and search Vector databases and LLMs go together like cookies and milk! Vector databases help to address some of large language models (LLMs) limitations, such as hallucinations, by helping to retrieve the relevant information to provide to the LLM as a part of its input. |Title | Description | Modality | Code | | --- | --- | --- | --- | | Verba, the golden RAGtriever ([Video](https://www.youtube.com/watch?v=OSt3sFT1i18)) | Retrieval-Augmented Generation (RAG) system to chat with Weaviate documentation and blog posts. | Text | [Python](https://github.com/weaviate/Verba) | | HealthSearch ([Blog](https://weaviate.io/blog/healthsearch-demo)) | Recommendation system of health products based on symptoms. | Text | [Python](https://github.com/weaviate/healthsearch-demo) | | Magic Chat | Search through Magic The Gathering cards | Text | [Python](https://github.com/weaviate/st-weaviate-connection/tree/main) | | AirBnB Listings ([Blog](https://weaviate.io/blog/generative-feedback-loops-with-llms)) | Generation of customized advertisements for AirBnB listings with Generative Feedback Loops | Text | [Python](https://github.com/weaviate/Generative-Feedback-Loops/) | | Distyll | Summarize text or video content. | Text | [Python](https://github.com/databyjp/distyll) | Learn more in our [LLMs and Search](https://weaviate.io/blog/llms-and-search) blog post. ## Classification Weaviate can leverage its vectorization capabilities to enable automatic, real-time classification of unseen, new concepts based on its semantic understanding. |Title | Description | Modality | Code | | --- | --- | --- | --- | | Toxic Comment Classification | Classify whether a comment is toxic or non-toxic. | Text | [Python](https://github.com/weaviate-tutorials/DEMO-classification-toxic-comment) | | Audio Genre Classification | Classify the music genre of an audio file. | Audio | [Python](https://github.com/weaviate-tutorials/DEMO-classification-audio-genre/) | ## Other use cases Weaviate's [modular ecosystem](../modules/index.md) unlocks many other use cases of the Weaviate vector database, such as [Named Entity Recognition](../modules/ner-transformers.md) or [spell checking](../modules/spellcheck.md). |Title | Description | Code | | --- | --- | --- | | Named Entity Recognition (NER)| Extract named entities, such as people, organizations and locations, from text stored in Weaviate. | [Python](https://github.com/weaviate/weaviate-examples/tree/main/example-with-NER-module) | ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/More Resources/Faq (docs/weaviate/more-resources/faq.md) --- title: FAQ sidebar_position: 3 image: og/docs/more-resources.jpg # tags: ['FAQ'] --- ## General #### Q: Why would I use Weaviate as my vector database?

Answer

> Our goal is three-folded. Firstly, we want to make it as easy as possible for others to create their own semantic systems or vector search engines (hence, our APIs are GraphQL based). Secondly, we have a strong focus on the semantic element (the "knowledge" in "vector databases," if you will). Our ultimate goal is to have Weaviate help you manage, index, and "understand" your data so that you can build newer, better, and faster applications. And thirdly, we want you to be able to run it everywhere. This is the reason why Weaviate comes containerized.

#### Q: Can I use Weaviate to store memory for AI agents?

Answer

> Yes. Weaviate works well as a vector store for agent memory. If you'd rather not build and operate the memory layer yourself, we also offer [Engram](/engram/), a dedicated managed memory service built on Weaviate that automatically extracts, stores, and retrieves memories for your agents and applications.

#### Q: What is the difference between Weaviate and for example Elasticsearch?

Answer

> Other database systems like Elasticsearch rely on inverted indexes, which makes search super fast. Weaviate also uses inverted indexes to store data and values. But additionally, Weaviate is also a vector-native search database, which means that data is stored as vectors, which enables semantic search. This combination of data storage is unique, and enables fast, filtered and semantic search from end-to-end.

#### Q: Do you offer Weaviate as a managed service?

Answer

> Yes, we do - check out [Weaviate Cloud](https://weaviate.io/pricing).

## Configuration and setup #### Q: How should I configure the size of my instance?

Answer

> You can find this in the [architecture section](/weaviate/concepts/resources.md#an-example-calculation) of the docs.

#### Q: Do I need to know about Docker (Compose) to use Weaviate?

Answer

> Weaviate uses Docker images as a means to distribute releases and uses Docker Compose to tie a module-rich runtime together. If you are new to those technologies, we recommend reading the [Docker Introduction for Weaviate Users](https://medium.com/semi-technologies/what-weaviate-users-should-know-about-docker-containers-1601c6afa079).

#### Q: What happens when the Weaviate Docker container restarts? Is my data in the Weaviate database lost?

Answer

> There are three levels: > 1. You have no volume configured (the default in our `Docker Compose` files), if the container restarts (e.g. due to a crash, or because of `docker stop/start`) your data is kept > 2. You have no volume configured (the default in our `Docker Compose` files), if the container is removed (e.g. from `docker compose down` or `docker rm`) your data is gone > 3. If a volume is configured, your data is persisted regardless of what happens to the container. They can be completely removed or replaced, next time they start up with a volume, all your data will be there

#### Q: How to enable RBAC in Weaviate?

Answer

> Role-based access control (RBAC) can be enabled when configuring Weaviate via the `AUTHORIZATION_RBAC_ENABLED` environment variable. > For more info visit the [RBAC: Configuration](/deploy/configuration/configuring-rbac.md) guide.

## Schema and data structure #### Q: Are there any 'best practices' or guidelines to consider when designing a schema? *(E.g. if I was looking to perform a semantic search over a the content of a Book would I look to have Chapter and Paragraph represented in the schema etc, would this be preferred over including the entire content of the novel in a single property?)*

Answer

> As a rule of thumb, the smaller the units, the more accurate the search will be. Two objects of e.g. a sentence would most likely contain more information in their vector embedding than a common vector (which is essentially just the mean of sentences). At the same time more objects leads to a higher import time and (since each vector also makes up some data) more space. (E.g. when using transformers, a single vector is 768xfloat32 = 3KB. This can easily make a difference if you have millions, etc.) of vectors. As a rule of thumb, the more vectors you have the more memory you're going to need. > > So, basically, it's a set of tradeoffs. Personally we've had great success with using paragraphs as individual units, as there's little benefit in going even more granular, but it's still much more precise than whole chapters, etc. > > You can use cross-references to link e.g. chapters to paragraphs. Note that resolving a cross-references takes a performance penalty. Essentially resolving A1->B1 is the same cost as looking up both A1 and B1 indvidually. But at scale, this can add up. > > So, consider denormalizing your data, i.e. storing the data in a way that you can resolve the cross-references without actually looking them up. This is a common pattern in databases, and it's also a common pattern in Weaviate.

#### Q: Should I use references in my schema?

Answer

> In short: for convenience you can add relations to your data schema, because you need less code and queries to get data. But resolving references in queries takes some of the performance. > > 1. If your ultimate goal is performance, references probably don't add any value, as resolving them adds a cost. > 2. If your goal is represent complex relationships between your data items, they can help a lot. You can resolve references in a single query, so if you have collections with multiple links, it could definitely be helpful to resolve some of those connections in a single query. On the other hand, if you have a single (bi-directional) reference in your data, you could also just denormalize the links (e.g. with an ID field) and resolve them during search.

#### Q: Is it possible to create one-to-many relationships in the schema?

Answer

> Yes, it is possible to reference to one or more objects (Class -> one or more Classes) through cross-references. Referring to lists or arrays of primitives, this will be available [soon](https://github.com/weaviate/weaviate/issues/1611).

#### Q: What is the difference between `text` and `string` and `valueText` and `valueString`?

Answer

> The `text` and `string` datatypes differ in tokenization behavior. Note that `string` is now deprecated. Read more in [this section](../config-refs/collections.mdx#tokenization) on the differences.

#### Q: Do Weaviate collections have namespaces?

Answer

Yes. Each collection itself acts like namespaces. Additionally, you can use the [multi-tenancy](../concepts/data.md#multi-tenancy) feature to create isolated storage for each tenant. This is especially useful for use cases where one cluster might be used to store data for multiple customers or users.

#### Q: Are there restrictions on UUID formatting? Do I have to adhere to any standards?

Answer

> The UUID must be presented as a string matching the [Canonical Textual representation](https://en.wikipedia.org/wiki/Universally_unique_identifier#Format). If you don't specify a UUID, Weaviate will generate a `v4` i.e. a random UUID. If you generate them yourself you could either use random ones or deterministically determine them based on some fields that you have. For this you'll need to use [`v3` or `v5`](https://en.wikipedia.org/wiki/Universally_unique_identifier#Versions_3_and_5_(namespace_name-based)).

#### Q: If I do not specify a UUID during adding data objects, will Weaviate create one automatically?

Answer

> Yes, Weaviate creates a UUID if one is not specified.

#### Q: Why does Weaviate have a schema and not an ontology?

Answer

> We use a schema because it focusses on the representation of your data (in our case in the GraphQL API) but you can use a Weaviate schema to express an ontology. One of Weaviate's core features is that it semantically interprets your schema (and with that your ontology) so that you can search for concepts rather than formally defined entities.

#### Q: What is the difference between a Weaviate data schema, ontologies and taxonomies?

Answer

> Read about how taxonomies, ontologies and schemas are related to Weaviate in [this blog post](https://medium.com/semi-technologies/taxonomies-ontologies-and-schemas-how-do-they-relate-to-weaviate-9f76739fc695).

## Text and language processing #### Q: How to deal with custom terminology?

Answer

> Sometimes, users work with custom terminology, which often comes in the form of abbreviations or jargon. You can find more information on how to use the endpoint [here](/weaviate/modules/text2vec-contextionary.md#extending-the-contextionary)

#### Q: How can you index data near-realtime without losing semantic meaning?

Answer

> Every data object gets its vector embedding based on its semantic meaning. In a nutshell, we calculate the vector position of the data object based on the words and concepts used in the data object. The existing model in the contextionary gives already enough context. If you want to get in the nitty-gritty, you can [browse the code here](https://github.com/weaviate/contextionary/tree/master/server), but you can also ask a [specific question on Stackoverflow](https://stackoverflow.com/tags/weaviate/) and tag it with Weaviate.

#### Q: Why isn't there a text2vec-contextionary in my language?

Answer

> Because you are probably one of the first that needs one! Ping us [here on GitHub](https://github.com/weaviate/weaviate/issues), and we will make sure in the next iteration it will become available (unless you want it in [Silbo Gomero](https://en.wikipedia.org/wiki/Silbo_Gomero) or another language which is whistled).

#### Q: How do you deal with words that have multiple meanings?

Answer

> How can Weaviate interpret that you mean a company, as in business, and not as the division of the army? We do this based on the structure of the schema and the data you add. A schema in Weaviate might contain a company collection with the property name and the value Apple. This simple representation (company, name, apple) is already enough to gravitate the vector position of the data object towards businesses or the iPhone. You can read [here](../) how we do this, or you can ask a specific question on [Stackoverflow](https://stackoverflow.com/tags/weaviate/) and tag it with Weaviate.

#### Q: Is there support to multiple versions of the query/document embedding models to co-exist at a given time? (helps with live experiments of new model versions)

Answer

> You can create multiple collections in the Weaviate schema, where one collection will act like a namespace in Kubernetes or an index in Elasticsearch. So the spaces will be completely independent, this allows space 1 to use completely different embeddings from space 2. The configured vectorizer is always scoped only to a single collection. You can also use Weaviate's Cross-Reference features to make a graph-like connection between an object of Class 1 to the corresponding object of Class 2 to make it easy to see the equivalent in the other space.

## Queries #### Q: How can I retrieve the total object count in a collection?

Answer

import HowToGetObjectCount from '/_includes/how.to.get.object.count.mdx'; > This `Aggregate` query returns the total object count in a collection.

#### Q: How do I get the cosine similarity from Weaviate's certainty?

Answer

> To obtain the [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity) from weaviate's `certainty`, you can do `cosine_sim = 2*certainty - 1`

#### Q: The quality of my search results change depending on the specified limit. Why? How can I fix this?

Answer

Weaviate makes use of ANN indexes to serve vector searches. An ANN index is an approximate nearest neighbor index. The "approximate" part refers to an explicit recall-query-speed tradeoff. This trade-off is presented in detail in the [ANN benchmarks section](/weaviate/benchmarks/ann.md#benchmark-results). For example, a 98% recall for a given set of HNSW parameters means that 2% of results will not match the true nearest neighbors. What build parameters lead to what recall depends on the dataset used. The benchmark pages shows 4 different example datasets. Based on the characteristic of each dataset you can pick the one closest to your production load and draw conclusions about the expected recall for the respective build and query-time parameters. Generally if you need a higher recall than the default parameters provide you with, you can use stronger parameters. This can either be done at build time (`efConstruction`, `maxConnections`) or at query time (`ef`). Roughly speaking, a higher `ef` value at query time means a more thorough search. It will have a slightly higher latency, but also lead to a slightly better recall. By changing the specified limit, you are implicitly changing the `ef` parameter. This is because the default `ef` value is set to `-1`, indicating that Weaviate should pick the parameter based on the limit. The dynamic `ef` value is controlled using the configuration fields `dynamicEfMin` which acts as a lower boundary, `dynamicEfMax` which acts as an upper boundary and `dynamicEfFactor` which is the factor to derive the target `ef` based on the limit within the lower and upper boundary. Example: Using the default parameters `ef=-1`, `dynamicEfMin=100`, `dynamicEfMax=500`, `dynamicEfFactor=8`, you will end up with the following `ef` values based on the limit: * `limit=1`, dynamically calculated: `ef=1*8=8`. This value is below the lower boundary, so `ef` is set to `100`. * `limit=20`, dynamically calculated: `ef=20*8=160`. This value is within the boundaries, so `ef` is `160`. * `limit=100`, dynamically calculated: `ef=100*8=800`. This value is above the upper boundary, so `ef` is set to `500`. If you need a higher search quality for a given limit you can consider the following options: 1. Instead of using a dynamic `ef` value, use a fixed one that provides the desired recall. 1. If your search quality varies a lot depending on the query-time `ef` values, you should also consider choosing stronger build parameters. The [ANN benchmarks section](/weaviate/benchmarks/ann.md#benchmark-results) present a combination of many different parameter combination for various datasets.

#### Q: Why did you use GraphQL instead of SPARQL?

Answer

> For user experience. We want to make it as simple as possible to integrate Weaviate into your stack, and we believe that GraphQL is the answer to this. The community and client libraries around GraphQL are enormous, and you can use almost all of them with Weaviate.

## Data management #### Q: What is the best way to iterate through objects? Can I do paginated API calls?

Answer

> Yes, Weaviate supports cursor-based iteration as well as pagination through a result set. > > To iterate through all objects, you can use the [`after` operator](../manage-objects/read-all-objects.mdx). > > For pagination through a result set, you can use the `offset` and `limit` operators for GraphQL API calls. Take a look at [this page](../api/graphql/filters.md) which describes how to use these operators, including tips on performance and limitations.

#### Q: What is best practice for updating data?

Answer

> Here are top 3 best practices for updating data: > 1. Use the [batch API](../manage-objects/import.mdx) > 2. Start with a small-ish batch size e.g. 100 per batch. Adjust up if it is very fast, adjust down if you run into timeouts > 3. If you have unidirectional relationships (e.g. `Foo -> Bar`.) it's easiest to first import all `Bar` objects, then import all `Foo` objects with the refs already set. If you have more complex relationships, you can also import the objects without references, then [add references](../manage-objects/import.mdx#import-with-references) to set links between collections in arbitrary directions.

## Modules #### Q: Can I connect my own module?

Answer

> [Yes!](/weaviate/modules/custom-modules.md)

#### Q: Can I train my own text2vec-contextionary vectorizer module?

Answer

> Not at the moment. You can currently use the [available contextionaries](/weaviate/modules/text2vec-contextionary.md) in a variety of languages and use the transfer learning feature to add custom concepts if needed.

## Indexes in Weaviate #### Q: Does Weaviate use Hnswlib?

Answer

> No > > Weaviate uses a custom implementation of HNSW that overcomes certain limitations of [hnswlib](https://github.com/nmslib/hnswlib), such as durability requirements, CRUD support, pre-filtering, etc. > > Custom HNSW implementation in Weaviate references: > > - [HNSW plugin (GitHub)](https://github.com/weaviate/weaviate/tree/master/adapters/repos/db/vector/hnsw) > - [vector dot product ASM](https://github.com/weaviate/weaviate/blob/master/adapters/repos/db/vector/hnsw/distancer/asm/dot_amd64.s) > > More information: > > - [Weaviate, an ANN Database with CRUD support – DB-Engines.com](https://db-engines.com/en/blog_post/87) ⬅️ best resource on the topic > - [Weaviate's HNSW implementation in the docs](/weaviate/concepts/indexing/vector-index.md#hierarchical-navigable-small-world-hnsw-index) > > _Note I: HNSW is just one implementation in Weaviate, but Weaviate can support multiple indexing algoritmns as outlined [here](/weaviate/concepts/indexing/vector-index.md)_

#### Q: Are all ANN algorithms potential candidates to become an indexation plugin in Weaviate?

Answer

> No > > Some algorithms (e.g., Annoy or ScaNN) are entirely immutable once built, they can neither be changed nor built up incrementally. Instead, they require you to have all of your vectors present, then you build the algorithm once. After a build, you can only query them, but cannot add more elements or change existing elements. Thus, they aren't capable of the CRUD operations we want to support in Weaviate.

#### Q: Does Weaviate use pre- or post-filtering ANN index search?

Answer

> Weaviate currently uses pre-filtering exclusively on filtered ANN search. > See "How does Weaviate's vector and scalar filtering work" for more details.

#### Q: How does Weaviate's vector and scalar filtering work?

Answer

> It's a 2-step process: > > 1. The inverted index (which is [built at import time](#q-does-weaviate-use-hnswlib)) queries to produce an allowed list of the specified document ids. Then the ANN index is queried with this allow list (the list being one of the reasons for our custom implementation). > 2. If we encounter a document id which would be a close match, but isn't on the allow list the id is treated as a candidate (i.e. we add it to our list of links to evaluate), but is never added to the result set. Since we only add allowed IDs to the set, we don't exit early, i.e. before the top `k` elements are reached. > > For more information on the technical implementations, see [this video](https://www.youtube.com/watch?v=6hdEJdHWXRE).

#### What is the maximum number of vector dimensions for embeddings?

Answer

> As the embedding is currently stored using `uint16`, the maximum possible length is currently 65535.

## Performance #### Q: What would you say is more important for query speed in Weaviate: More CPU power, or more RAM? More concretely: If you had to pick between a machine that has 16 GB of RAM and 2 CPUs, or a machine that has 8 GB of RAM and 4 CPUs, which would you pick?

Answer

> This is a very difficult to answer 100% correctly, because there are several factors in play: > * **The vector search itself**. This part is CPU-bound, however only with regards to throughput: A single search is single-threaded. Multiple parallel searches can use multiple threads. So if you measure the time of a single request (otherwise idle), it will be the same whether the machine has 1 core or 100. However, if your QPS approach the throughput of a CPU, you'll see massive benefits by adding more Cores > * **The retrieval of the objects**. Once the vector search part is done, we are essentially left with a list of n IDs which need to be resolved to actual objects. This is IO-bound in general. However, all disk files are memory-mapped. So generally, more mem will allow you to hold more of the disk state in memory. In real life however, it's not that simple. Searches are rarely evenly distributed. So let's pretend that 90% of searches will return just 10% of objects (because these are more popular search results). Then if those 10% of the disk objects are already cached in mem, there's no benefit in adding more memory. > > Taking the above in mind: we can carefully say: If throughput is the problem, increase CPU, if response time is the problem increase mem. However, note that the latter only adds value if there are more things that can be cached. If you have enough mem to cache your entire disk state (or at least the parts that are relevant for most queries), additional memory won't add any additional benefit. > If we are talking about imports on the other hand, they are almost always CPU-bound because of the cost of creating the HNSW index. So, if you can resize between import and query, my recommendation would be roughly prefer CPUs while importing and then gradually replace CPU with memory at query time - until you see no more benefits. (This assumes that there is a separation between importing and querying which might not always be the case in real life).

#### Q: Data import takes long / is slow, what is causing this and what can I do?

Answer

> HNSW is super fast at query time, but slower on vectorization. This means that adding and updating data objects costs relatively more time. You could try [asynchronous indexing](../config-refs/indexing/vector-index.mdx#asynchronous-indexing), which separates data ingestion from vectorization.

#### Q: How can slow queries be optimized?

Answer

> Queries containing deeply nested references that need to be filtered or resolved can take some time. Read on optimization strategies [here](./performance.md#costs-of-queries-and-operations).

#### Q: When scalar and vector search are combined, will the scalar filter happen before or after the nearest neighbor (vector) search?

Answer

> The mixed structured vector searches in Weaviate are pre-filter. There is an inverted index which is queried first to basically form an allow-list, in the HNSW search the allow list is then used to treat non-allowed doc ids only as nodes to follow connections, but not to add to the result set.

#### Q: Regarding "filtered vector search": Since this is a two-phase pipeline, how big can that list of IDs get? Do you know how that size might affect query performance?

Answer

> Essentially the list ids uses the internal doc id which is a `uint64` or 8 bytes per ID. The list can grow as long as you have memory available. So for example with 2GB of free memory, it could hold 250M ids, with 20GB it could hold 2.5B ids, etc. > > Performance wise there are two things to consider: > 1. Building the lookup list > 2. Filtering the results when vector searching > > Building the list is a typical inverted index look up, so depending on the operator this is just a single read on == (or a set of range reads, e.g. for >7, we'd read the value rows from 7 to infinity). This process is pretty efficient, similar to how the same thing would happen in a traditional search engine, such as elasticsearch > > Performing the filtering during the vector search depends on whether the filter is very restrictive or very loose. In the case you mentioned where a lot of IDs are included, it will be very efficient. Because the equivalent of an unfiltered search would be the one where your ID list contains all possible IDs. So the HNSW index would behave normally. There is however, a small penalty whenever a list is present: We need to check if the current ID is contained an the allow-list. This is essentially a hashmap lookup, so it should be O(1) per object. Nevertheless, there is a slight performance penalty. > > Now the other extreme, a very restrictive list, i.e few IDs on the list, actually takes considerably more time. Because the HNSW index will find neighboring IDs, but since they're not contained, they cannot be added as result candidates, meaning that all we can do with them is evaluating their connections, but not the points themselves. In the extreme case of a list that is very, very restrictive, say just 10 objects out of 1B in the worst case the search would become exhaustive if you the filtered ids are very far from the query. In this extreme case, it would actually be much more efficient to just skip the index and do a brute-force indexless vector search on the 10 ids. So, there is a cut-off when a brute-force search becomes more efficient than a heavily-restricted vector search with HNSW. We do not yet have any optimization to discovery such a cut-off point and skip the index, but this should be fairly simple to implement if this ever becomes an actual problem.

#### Q: My Weaviate instance uses more memory than I think is reasonable. How can I debug this?

Answer

> Check that your import uses the latest version of Weaviate. `v1.12.0` and `v1.12.1` fix an [issue](https://github.com/weaviate/weaviate/issues/1868) where excessive amounts of data are written to disk, resulting in unreasonable memory consumption after restarts. If upgrading does not fix the issue, see this post on [how to profile memory use](https://stackoverflow.com/a/71793178/5322199).

## Troubleshooting / debugging #### Q: How can I print a stack trace of Weaviate?

Answer

You can do this by sending a `SIGQUIT` signal to the process. This will print a stack trace to the console. The logging level and debugging variables can be set with `LOG_LEVEL` and `DEBUG` [environment variables](/deploy/configuration/env-vars/index.md). Read more on SIGQUIT [here](https://en.wikipedia.org/wiki/Signal_(IPC)#SIGQUIT) and this [StackOverflow answer](https://stackoverflow.com/questions/19094099/how-to-dump-goroutine-stacktraces/35290196#35290196).

#### Q: 'invalid properties' error when creating a collection (Python client versions 4.16.0 to 4.16.3)

Answer

In Weaviate Python client versions `4.16.0` to `4.16.3`, the following pattern when creating a collection with a text2vec_xxx vectorizer will result in an error: ```python client.collections.create( "CollectionName", vector_config=Configure.Vectorizer.text2vec_cohere(), # also applies to other vectorizers ) ``` The error message will look like this: ```text UnexpectedStatusCodeError: Collection may not have been created properly.! Unexpected status code: 422, with response body: {'error': [{'message': "module 'text2vec-cohere': invalid properties: didn't find a single property which is of type string or text and is not excluded from indexing.... ``` This is a known issue, which will occur when setting a vectorizer definition without defining any `TEXT` or `TEXT_ARRAY` properties in the collection, in order to rely on AutoSchema to create the data schema for you. **This issue is addressed in Weaviate Python client patch release `4.16.4`. So, we recommend updating to the version `4.16.4` of the Weaviate Python client, or later.** If you are unable to change your Weaviate Python client version from the affected ones, you can work around this issue in one of two ways: 1. By explicitly defining at least one `TEXT` or `TEXT_ARRAY` property in the collection schema, like this: ```python client.collections.create( "CollectionName", properties=[ Property(name="", data_type=DataType.TEXT), ], vector_config=Configure.Vectorizer.text2vec_cohere(), # Additional configuration not shown ) ``` 2. By setting `vectorize_collection_name` to `True` in the vectorizer definition, like this: ```python client.collections.create( "CollectionName", vector_config=Configure.Vectorizer.text2vec_cohere( vectorize_collection_name=True ), # Additional configuration not shown ) ```

#### Q: Why does `insert_many` fail with a "message larger than max" (`RESOURCE_EXHAUSTED`) error?

Answer

`insert_many` (Python) and `insertMany` (TypeScript, Java) send all objects in a **single gRPC request**. Requests larger than the server's [`GRPC_MAX_MESSAGE_SIZE`](/deploy/configuration/env-vars/index.md#GRPC_MAX_MESSAGE_SIZE) limit are rejected with the gRPC status `RESOURCE_EXHAUSTED`, so a sufficiently large list fails as a whole with an error similar to: ```text WeaviateBatchError: Query call with protocol GRPC batch failed with message CLIENT: Sent message larger than max (3002340 vs. 1000000). ``` The two numbers are the size of your request and the server's limit (the values shown here are from a test with a 1 MB limit). Recent Python clients read the server's limit at connect time and reject oversized requests before sending them. With older clients, the server-side variant `grpc: received message larger than max` may appear instead. For large lists, use [server-side batching](../manage-objects/import.mdx#server-side-batching) instead, which splits the data into server-paced batches: - **Python**: Use `collection.data.ingest(objects)`, a drop-in replacement for `insert_many` that returns the same return object. Alternatively, use the `collection.batch.stream()` context manager. - **TypeScript**: `collection.data.ingest(objects)`. - **Java** (`v6`): the `collection.batch.start()` streaming context. - **C#**: `collection.Batch.InsertMany(items)` (already uses server-side batching). Alternatively, you can raise the `GRPC_MAX_MESSAGE_SIZE` [environment variable](/deploy/configuration/env-vars/index.md#GRPC_MAX_MESSAGE_SIZE) on the server, but batching is the recommended solution.

## Miscellaneous #### Q: Can I request a feature in Weaviate?

Answer

> Sure (also, feel free to [issue a pull request](https://github.com/weaviate/weaviate/pulls) 😉) you can [add those requests here](https://github.com/weaviate/weaviate/issues). The only thing you need is a GitHub account, and while you're there, make sure to give us a star 😇.

#### Q: What is Weaviate's consistency model in a distributed setup?

Answer

> Weaviate is generally modeled to prefer Availability over Consistency (AP over CP). It is designed to deliver low search latencies under high throughput in situations where availability is more business-critical than consistency. If strict serializability is required on your data, we generally recommend storing your data in a different primary data store, use Weaviate as an auxiliary data store, and set up replication between the two. If you do not need serializability and eventual consistency is enough for your use case, Weaviate can be used as a primary datastore. > > Weaviate has no notion of transactions, operations always affect exactly a single key, therefore Serializability is not applicable. In a distributed setup (under development) Weaviate's consistency model is eventual consistency. When a cluster is healthy, all changes are replicated to all affected nodes by the time the write is acknowledged by the user. Objects will immediately be present in search results on all nodes after the import request completes. If a search query occurs concurrently with an import operation nodes may not be in sync yet. This means some nodes might already include the newly added or updated objects, while others don't yet. In a healthy cluster, all nodes will have converged by the time the import request has been completed successfully. If a node is temporarily unavailable and rejoins a cluster it may temporarily be out of sync. It will then sync the missed changes from other replica nodes and eventually serve the same data again.

#### Q: With your aggregations I could not see how to do time buckets, is this possible?

Answer

> At the moment, we cannot aggregate over timeseries into time buckets yet, but architecturally there's nothing in the way. If there is demand, this seems like a nice feature request, you can submit an [issue here](https://github.com/weaviate/weaviate/issues). (We're a very small company though and the priority is on Horizontal Scaling at the moment.)

#### Q: How can I run the latest master branch with Docker Compose?

Answer

> You can run Weaviate with `Docker Compose`, you can build your own container off the [`master`](https://github.com/weaviate/weaviate) branch. Note that this is not an officially released Weaviate version, so this might contain bugs. > > ```sh > git clone https://github.com/weaviate/weaviate.git > cd weaviate > docker build --target weaviate -t name-of-your-weaviate-image . > ``` > > Then, make a `docker-compose.yml` file with this new image. For example: > > ```yml > > services: > weaviate: > image: name-of-your-weaviate-image > ports: > - 8080:8080 > environment: > CONTEXTIONARY_URL: contextionary:9999 > QUERY_DEFAULTS_LIMIT: 25 > AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true' > PERSISTENCE_DATA_PATH: './data' > ENABLE_MODULES: 'text2vec-contextionary' > AUTOSCHEMA_ENABLED: 'false' > contextionary: > environment: > OCCURRENCE_WEIGHT_LINEAR_FACTOR: 0.75 > EXTENSIONS_STORAGE_MODE: weaviate > EXTENSIONS_STORAGE_ORIGIN: http://weaviate:8080 > NEIGHBOR_OCCURRENCE_IGNORE_PERCENTILE: 5 > ENABLE_COMPOUND_SPLITTING: 'false' > image: cr.weaviate.io/semitechnologies/contextionary:en0.16.0-v1.0.2 > ``` > > After the build is complete, you can run this Weaviate build with docker compose: ```bash docker compose up ```

#### Q: Can I run Weaviate on Windows?

Answer

Weaviate can be used on Windows via containerized environments like [Docker](/deploy/installation-guides/docker-installation.md) or [WSL](https://learn.microsoft.com/en-us/windows/wsl/), Keep in mind that we don't offer native Windows support at this time and deployment options like [Weaviate Embedded](/docs/deploy/installation-guides/embedded.md) should be avoided.

#### Q: What is Weaviate Academy?

Answer

Weaviate Academy is a full-fledged learning platform available at [academy.weaviate.io](https://academy.weaviate.io). :::note If you need resources from the previous version of Weaviate Academy, check out the [documentation archive](https://archive.docs.weaviate.io/academy) :::

#### Q: What happened to the Weaviate Community Slack?

Answer

> The Weaviate Community Slack has been decommissioned. We've moved community discussions to the [Weaviate Community Forum](https://forum.weaviate.io/), which offers better long-term discoverability: conversations are indexed and searchable, so valuable answers don't get lost over time. > > Join us at [forum.weaviate.io](https://forum.weaviate.io/) to ask questions, share ideas, and connect with the community. For private support inquiries, you can reach us at [support@weaviate.io](mailto:support@weaviate.io).

#### Q: Does Weaviate have MCP server support?

Answer

> Yes, Weaviate provides two MCP (Model Context Protocol) servers: > > - **[Weaviate MCP server](/weaviate/configuration/mcp-server.mdx)**: Built into Weaviate itself. Exposes tools for inspecting schemas, searching data (vector/hybrid), and modifying objects. Runs on the same port as the REST API at `/v1/mcp`. Disabled by default. Enable it with `MCP_SERVER_ENABLED=true`. > - **[Weaviate Docs MCP server](/weaviate/mcp/docs-mcp-server.mdx)**: A standalone server that gives LLMs access to Weaviate's documentation. Useful for AI-assisted development with Weaviate. > > Both servers use the Streamable HTTP transport and work with MCP clients like Claude Code, Claude Desktop, Cursor, and VS Code.

## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/More Resources/Glossary (docs/weaviate/more-resources/glossary.md) --- title: Glossary sidebar_position: 4 description: Access a glossary of terms for better understanding Weaviate concepts. image: og/docs/more-resources.jpg # tags: ['glossary', 'terminology'] --- import APITable from '@site/src/components/APITable'; ```mdx-code-block ``` | Term | Description | | :------------------------------------------------ | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Aggregation** | The process of computing summaries or statistics over a set of query results (e.g., counting objects, finding min/max/mean of properties). Typically performed using the GraphQL `Aggregate{}` function. | | **API key** | A secret token used for authenticating requests to a Weaviate instance (especially for Weaviate Cloud) or to integrated third-party services like model providers (e.g., OpenAI, Cohere). | | **Beacon** | A beacon is a reference to a particular data object in Weaviate or inside the knowledge network. This data object in turn has a position in the vector space. Often defined as follows: `weaviate://{peerName}/{className}/{UUID}`. (For Weaviate version \< `v1.14.0`, it is defined as follows: `weaviate://{peerName}/{UUID}`.) | | **Class** | A class is the former name for a [collection](../starter-guides/managing-collections/index.mdx), a container where data objects sharing the same structure (properties, vectorizer settings, etc.) are stored. | | **Concept** | Concepts are related to entities. Often you will use concepts to search in your datasets. If your dataset has data about _An Actor with the name Arnold Schwarzenegger_ and _an Actor with the name Al Pacino_, the concepts _Movie_ and _Terminator_ will find a closer relation to the first actor rather than the latter. | | **Contextionary** | Derived from _dictionary_ with _context_. Pre-trained vector space which contains vectors for nearly all words used in a specific language. The Contextionary (text2vec-contextionary) gives context to the language used in the dataset, inspired by the [_Global Vectors for Word Representation_](https://github.com/stanfordnlp/GloVe) concept. Read more about the Contextionary [here](../modules/text2vec-contextionary.md). | | **Embedding model** | A machine learning model that transforms data (text, images, audio, etc.) into numerical vector representations (embeddings). This is the core component within a Vectorizer module. | | **Entity** | An entity refers to something -often- in the world around us. E.g., _a Company with the name Apple_ refers to an entity with a relation to _a Product with the name iPhone_. Weaviate's Contextionary tries to find as many entities in your data as possible. | | **Fuzzy** | Opposed to most other data solutions, Weaviate uses [fuzzy logic](https://en.wikipedia.org/wiki/Fuzzy_logic) to interpret a query. The upside of this is that it might find answers to queries where a traditional data solution might not. | | **Generative model** | An AI model integrated with Weaviate (often via a module like `generative-openai`) that can generate new content (e.g., summaries, answers) based on the context provided by search results. | | **HNSW** | Hierarchical Navigable Small World - a multilayered graph vector index type. | | **Inverted index** | An index storing a mapping from data property values, to its locations of data objects in a database (named in contrast to a forward index, which maps from data objects to property data values). | | **Model provider integrations** | Weaviate's ability to connect with external services (like OpenAI, Cohere, Hugging Face, Google Vertex AI) that host and serve machine learning models (Embedding models, Generative models, Rerankers) used within Weaviate modules. | | **Multimodal** | The capability to process and understand information from multiple types (modalities) of data simultaneously, such as text, images, audio, etc. Multimodal vectorizer modules (e.g., `multi2vec-clip`) create embeddings that represent combined concepts. | | **Multiple vector embeddings** | The capability to store multiple, distinct named vectors for a single data object. This allows representing different aspects or using embeddings from different models for the same object (e.g., one vector for content, one for title). | | **Multi-vectors** | Multi-vector embeddings, also known as multi-vectors, represent a single object with multiple vectors, i.e. a 2-dimensional matrix. | | **NearText** | A search operator that takes text input, uses the configured Vectorizer module to dynamically generate a query vector, and then performs a vector similarity search based on that vector. | | **Property** | All classes have properties. E.g., the class Company might have the property _name_. In Weaviate, properties can be recognized because they always have a lowercase first character. | | **Quantization** | Vector compression techniques (like Product Quantization 'PQ' or Binary Quantization 'BQ') used to reduce the memory footprint of vector embeddings, potentially trading some precision for significant storage and performance gains. | | **Reranker** | An AI model integrated with Weaviate (often via a module like `reranker-cohere`) that takes the initial list of search results and re-orders them based on a secondary relevance calculation, often improving the quality of the top results. | | **Replication** | The process of creating copies (replicas) of data shards across different nodes in a Weaviate cluster to ensure data durability and high availability in case of node failures. (Part of Clustering). | | **Schema** | In Weaviate, a schema is used to define the types of data you will be adding and querying. You can learn more about it [here](../starter-guides/managing-collections/index.mdx). | | **Sharding** | The process of splitting a Class's data and index horizontally across multiple nodes (shards) in a Weaviate cluster. This allows the dataset size and workload to scale beyond the capacity of a single node. (Part of Clustering). | | **Vector index** | A data storage mechanism where data is stored as vectors (long arrays of numbers, also seen as coordinates in a high dimensional space), allowing for context-based search. | | **Vectorizer** | A module within Weaviate (e.g., `text2vec-openai`, `multi2vec-clip`) responsible for automatically converting specified data properties into vector embeddings using an underlying Embedding model, either during data import or at query time (e.g., for `nearText`). | | **WCS** | Weaviate Cloud Service. The former name for Weaviate's managed cloud offering. Now known as Weaviate Cloud (WCD). | | **[Weaviate Cloud (WCD)](../../cloud/index.mdx)** | WCD is our SaaS for providing cloud instances of Weaviate. | | **Weaviate Cluster** | A managed Weaviate cluster. | ```mdx-code-block ``` ## Questions and feedback import DocsFeedback from '/\_includes/docs-feedback.mdx'; --- ### Weaviate/More Resources/Index (docs/weaviate/more-resources/index.md) --- title: More resources sidebar_position: 0 image: og/docs/more-resources.jpg # tags: ['More resources'] --- import Badges from '/_includes/badges.mdx'; These pages can help with common questions: - [FAQ](./faq.md) - [Glossary](./glossary.md) - [Example datasets](./example-datasets.md) - [Example use cases](./example-use-cases.md) - [Index types and performance](./performance.md) - [Migration Guide](/deploy/migration/index.md) ## (Even) more resources For additional information, try these sources. - [Weaviate Community Forum](https://forum.weaviate.io/) - [Knowledge base of old issues](https://github.com/weaviate/weaviate/issues?utf8=%E2%9C%93&q=label%3Abug) ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/More Resources/Performance (docs/weaviate/more-resources/performance.md) --- title: Index types and performance sidebar_position: 7 image: og/docs/more-resources.jpg # tags: ['performance'] --- Weaviate uses different types of indexes to achieve performance goals. This page focuses on [HNSW](https://arxiv.org/abs/1603.09320) and [inverted indexes](https://en.wikipedia.org/wiki/Inverted_index). These indexes are available: Vector Indexes: - [HNSW](/weaviate/config-refs/indexing/vector-index.mdx#hnsw-index) - [Flat](/weaviate/config-refs/indexing/vector-index.mdx#flat-index) Traditional Indexes: - [Inverted index](#inverted-index) ## Inverted index The inverted index is essentially what powers all the [GraphQL `where` filters](../api/graphql/filters.md), where vectors or semantics are needed to find results. With inverted indexes, contents or data object properties such as words and numbers are mapped to its location in the database. This is the opposite of the more traditional forward index, which maps from documents to its content. Inverted indexes are used often in document retrieval systems and search engines, because it allows fast full-text search and fast key-based search instead of brute-force. This fast data retrieval comes with the only cost of slight increase of processing time when a new data object is added, since the data object will indexed and stored in an inverted way, rather than only storing the index of the data object. In the database (Weaviate), there is a big lookup table which contains all the inverted indexes. If you want to retrieve objects with a specific property or content, then the database starts looking for only one row with this property which points to the relevant data objects (the row contains pointers to the data object IDs). This makes data object retrieval with these kind of queries very fast. Even if there are more than a billion entries, if you only care about the entries that contain the specific words or properties you're looking for, only one row will be read with the document pointers. The inverted index currently does not do any weighing (e.g. tf-idf) for sorting, since the vector index is used for these features like sorting. The inverted index is thus, at the moment, rather a binary operation: including or excluding data objects from the query result list, which results in an 'allow list'. ## Vector index Everything that has a vector, thus every data object in Weaviate, is also indexed in the vector index. Weaviate supports several vector index types, which trade off search speed, recall and resource use in different ways. The default is [HNSW](https://arxiv.org/abs/1603.09320). See [Concepts: vector index](../concepts/indexing/vector-index.md) for more information about the vector index. ## Costs of queries and operations This section discusses the cost of some common operations. ### Cost of data import At the moment, data import is relatively slow compared to the query times, because of the HNSW indexing. The cheapest data import operation is a simple 'write' operation of a data object that was not seen before. It will get a completely new index. If you update a data object, the update itself is also really cheap, because in the backend an entirely new object will be created and indexed as if it was new. The 'old' object will be cleaned up, which happens asynchronous and will thus add up to the operation time. ### Cost of queries Simple `Get` queries that only have a `where` filter are very cheap, because the [inverted index](#inverted-index) is used. A simple `Get` query that uses only the `explore` filter (vector search) is also very cheap, since the very efficient vector index HNSW is used. Sub-50ms 20NN-vector queries on datasets of over 1-100M objects are possible. Weaviate relies on a number of caches, but does not require keeping all vectors in memory. Thus it is also possible to run Weaviate on machines where the available memory is smaller than the size of all vectors. Combining the `explore` (vector) and `where` filters in one search query (which is what makes Weaviate unique), is slightly more expensive. The inverted index is called first which returns all data items that match the `where` filter. This list is then passed on to the vector index search with HNSW. The cost of this combined operation depends on the dataset size and the amount of data returned by the inverted index search. The less items that are returned from the `where` filter search, the more items the vector search needs to skip, thus the longer it will take. These differences are however very small, perhaps not even noticeable. ### Cost of resolving referencing Weaviate is a database with a graph-like data model, not a pure graph database. Graph databases are built in a fashion where following links and references are very cheap, where querying links is cheaper than querying multiple items. Weaviate, on the other hand, is a [vector database](https://weaviate.io/blog/what-is-a-vector-database). This means that one of the cheapest operations you can do with Weaviate is listing data. In a traditional graph database that is quite expensive. Weaviate does however have graph functionalities on top of the vector-search focus. So although its primary focus is on searching through data objects with the inverted index and/or vector index, we offer graph references between data objects. Searching, following and retrieving graph references between data objects is therefore less optimized than pure search. This means that using the graph-like features like resolving object references need more query time than pure data object search as described above. Important to know is that the more connections between data objects and the deeper you try to query in a single query, the more costly the query operation gets. The best you can deal with these kind of queries is to not do *wide* and *deep* searches at the same time. If you have to resolve a lot of nested references, try to set a low limit (a low number of data objects to be returned). A second tip is to not try to resolve all the references. This could perhaps be split into separate queries, depending on your query. In practice, this could mean that you do a first search to retrieve the top 100 data objects of your query, and only get the deeper references of the top 5 results you're actually interested in. ### Cost of filtering by reference If you have a nested reference filter, Weaviate starts by resolving the deepest reference and from there go upwards to the inner layers resolving other references. It thus finds the beacons of the deepest references first. This allows to use inverted index lookups for the other layers, which makes matching of results relatively cheap. However, queries that have nested references in the filter are still relatively costly because multiple search queries (for each nested layer) are performed and the results need to be combined into one result. The cost increases when a lot of results are returned on an inner layer, which needs to be searched through by the one layer deeper, and so on. Thus, this cost could in theory go up exponentially. A tip is to avoid deeply nested filters in the queries. Additionally, try to make your queries as restrictive as possible, because a ten-level deep query would for example not be so expensive if all levels return only a single ID. In that case only ten one ID searches need to be performed, which is a lot of searches in one query, but each search is very cheap. ## Profiling query performance To diagnose slow queries, use [query profiling](/weaviate/search/query-profile.md) to get per-shard timing breakdowns. This shows exactly how long each phase takes (vector search, keyword scoring, filter evaluation, and object retrieval), broken down by shard and cluster node. ## Questions and feedback import DocsFeedback from '/_includes/docs-feedback.mdx'; --- ### Weaviate/More Resources/Write Great Bug Reports (docs/weaviate/more-resources/write-great-bug-reports.md) --- title: How to write great bug reports sidebar_position: 99 image: og/docs/more-resources.jpg # tags: ['how to', 'reporting a bug', 'bugfix', 'reproducing example'] --- ## Write great bug reports! This page outlines what an ideal bug report would look like. We know that it is not always possible to write a perfect bug report, and we don't want to discourage you from reporting a bug just because you might not be able to provide all the info needed to make the report great. At the same time we want to provide you with the information to make the lives of our engineers a bit easier. Sometimes we also need to prioritize and decide about which bug ticket to pick up first. If a bug report is well-prepared, it has a greater chance of being picked up first. ### What makes a great bug report stand out? Here are some points that make a bug report great: - **Providing Context** When you have been working on a specific use-case or fighting against a specific bug for ages there is probably a lot of context in your or your teams' head(s). Sometimes this context gets lost when handing over a bug report to one of our engineers. Since they have probably been working on something completely different before, it may be difficult for them to understand all your goals and assumptions that are like second-nature to you. A great bug reports sets that context and makes sure that any engineer - whether an inside our outside contributor - can get started on this ticket easily. - **Right level of information** Depending on the kind of bug, there is a different need for information. Let's consider two different possible bugs: For the first one an image when using the `img2vec` module was not vectorized correctly and your results are off because of it. For the second one, consider a scenario where you a performing a lookup by id and it is much slower than you think it should be. Those are both valid bugs, but we need different kind of information for each. For example, for the image-vectorization bug we need to know a lot of the details about the `img2vec` module: What versions were used? Which inference container was running? Was there a GPU involved? What file format did the image have? But looking at the performance bug, we probably need more info regarding your hardware. How was the machine sized? What kind of disks were used? What were the vitals (CPU usage, Memory usage, Disk pressure) during the slow query, etc.? We do not expect you to know all the internals of Weaviate, but we ask you to think about what details may be helpful in reproducing the bug and which are most likely superfluous. - **Quick to reproduce** Every bug is important and we are happy about every single report. However, we must still prioritize. A bug report that is easier for us to reproduce is a bug report we might prefer. A great bug report contains a reproducing example that makes no assumptions about prior state and reproduces be bug in its entirety. Below are some examples for a great reproducing example in a bug report. - **Narrowed down to a particular area** Weaviate is more than just the Weaviate server, it's an entire ecosystem that often contains the Weaviate Server, a language-specific Weaviate client and any number of optional modules. Those modules may bring their own inference containers if they make use of a Machine-Learning model. A great bug report tries to narrow down where the problem goes wrong. There are some helpful tips below to see how you can find out where the bug occurs. Now that we have established *what* makes a great bug report, let's look at some of the individual areas and see *how* we can write better reports. ## What is the minimal information and context that should always be provided? - Make sure that all the versions used are explicitly listed. This includes at the minimum the version of the Weaviate server and the client. - Is there a chance that the bug was introduced in a recent version? In this case, report the last version that does not have the specified issue. - Is there module involved that is vital to reproducing the bug? If so, specify the module. If the module uses different models, specify the model names too. ## How do I provide a good reproducing example? - A great reproducing example makes zero assumptions about state. This means that the example always starts with an empty Weaviate instance and imports any object that is required to reproduce the bug. Our engineers cannot predict what kind of objects should be imported based on a read/search query. - Anything that is required to reproduce the error is part of the reproducing example. Our engineers should be able to copy/paste the example and immediately see that something is wrong. - The reproducing example is expressed as code. This could be one of Weaviate's language clients or a series of `curl` commands. - The reproducing example tells us what you expected to happen. In some cases it might not be obvious why the actual behavior is not the desired behavior. Let us know what you expected to happen instead. This can be either in the form of code, a code comment, or text accompanying your example. ## How do I know if a problem occurs in Weaviate, a client or somewhere else? - If you have a suspicion that a problem doesn't actually occur in the Weaviate server, but possibly in one of the clients, you can verify sending a similar request using a different language client or no language client. The latter case is the best as it rules out client problems altogether. If you can still reproduce the error by sending a request using pure HTTP (e.g. via `curl`, Postman, etc.), you can be sure that the error occurs on the Weaviate server-side. - If you see a stack trace from your language-client you can make an educated guess about where the error occurred. If the stack trace contains a network request, a non-2xx HTTP status code or an error message containing information about shards and indexes, there is a good chance the bug occurred inside the Weaviate server. If you see something that is very specific to the client's language however, it may be an indication that the error occurred in the client. - If you are using any other tools from the Weaviate eco-system, for example the `weaviate-helm` repository to run on Kubernetes, there is also a chance that something goes wrong there. If you think that the bug might be specific to the runtime and its manifests, it might make sense to also try the setup on a different runtime. Let us know what you have already tried. ## What if it's not feasible to provide the information mentioned above? Don't worry about it. We know that sometimes bugs are tricky and not so easy to reproduce. If it is simply not feasible to write a perfect bug report, write one anyway. We are very happy when we see that you made an effort to write a good report. ## Thank you A bug report is a contribution to Weaviate. We are really thankful for you taking the time to report the issue and helping us improve Weaviate. Thank you! --- ### Weaviate/Quickstart/Index (docs/weaviate/quickstart/index.md) --- title: "Quickstart: With Cloud resources" image: og/docs/quickstart-tutorial.jpg # tags: ['getting started'] --- import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem"; import SkipLink from "/src/components/SkipValidationLink"; import CardsSection from "/src/components/CardsSection"; import Tooltip from "/src/components/Tooltip"; import styles from "/src/components/CardsSection/styles.module.scss"; export const quickstartOptions = [ { title: ( <> Vectorize objects during import
(recommended) ), description: "Import objects and vectorize them with the Weaviate Embeddings service.", link: "?import=vectorization#create-a-collection", icon: "fas fa-arrows-spin", groupId: "import", activeTab: "vectorization", }, { title: "Import vectors", description: "Import pre-computed vector embeddings along with your data.", link: "?import=custom-embeddings#create-a-collection", icon: "fas fa-circle-nodes", groupId: "import", activeTab: "custom-embeddings", }, ]; Weaviate is an open-source vector database built to power AI applications. This quickstart guide will show you how to: 1. **Set up a collection** - Create a collection and import data into it. 2. **Search** - Perform a similarity (vector) search on your data. 3. **RAG** - Perform Retrieval Augmented Generation (RAG) with a generative model. 4. **Query Agent** - Get answers from your data by using a natural language prompt/question. import KapaAI from "/src/components/KapaAI"; If you encounter any issues along the way or have additional questions, use the Ask AI feature. import PromptStarter from "/src/components/PromptStarter"; ## Prerequisites A **[Weaviate Cloud](https://console.weaviate.cloud/)** free cluster - you will need an admin **API key** and a **REST endpoint URL** to connect to your instance. See the instructions below for more info. If you don't want to use Weaviate Cloud, check out the [Local Quickstart](local.md) with Docker.

How to set up a Weaviate Cloud free cluster

Go to the [Weaviate Cloud console](https://console.weaviate.cloud) and create a free cluster as shown in the interactive example below.