## File: README.md --- **Ten million 768-dimensional embeddings take 31 GB of RAM as float32. turbovec holds them in 4 GB and searches them 4.4× faster than FAISS, at matching recall.** turbovec is an in-process vector index for Python and Rust. It stores each vector at 2 or 4 bits per dimension using Google Research's [TurboQuant](https://arxiv.org/abs/2504.19874) and searches that compressed form directly with agent-optimised SIMD kernels. - **Fast.** 4.4× FAISS FastScan at 4-bit and 2.2× at 2-bit, on ARM and x86. A staged search scans one bit plane and rescores the shortlist exactly. - **Low memory.** 7.8× smaller than float32 at 4-bit, 15.5× at 2-bit, at the same recall as FAISS. - **Online.** No training step, no rebuilds: the codebook comes from the maths, not your data. Add a vector and it is searchable; remove one in microseconds. - **Incremental.** `sync(path)` writes only what changed since the last call — one fsync, crash-safe at any byte, milliseconds however large the index. - **Local.** A library, not a service: nothing leaves your process. Pair it with an open-source embedding model for a fully offline stack. - **Filtered.** Pass an allowlist of ids or a bitmask; the kernel skips what the filter excludes and returns up to `k` results from the allowed set. ## Python ```bash pip install turbovec ``` ```python import numpy as np from turbovec import TurboQuantIndex vectors = np.load("embeddings.npy").astype(np.float32) # shape (n, 1536) index = TurboQuantIndex(dim=1536, bit_width=4) index.add(vectors) scores, indices = index.search(vectors[:1], k=5) print(indices) # [[ 0 86544 3385 84492 24967]] int64 slot positions print(scores.shape) # (1, 5) float32 inner products index.sync("index.tv") # first sync writes the file index.add(more_vectors) index.sync("index.tv") # later syncs write just the change, durably index = TurboQuantIndex.load("index.tv") # loads either a sync'd or a write() file ``` Inputs are 2-D `float32` arrays of shape `(n, dim)`; other dtypes raise rather than silently convert. Scores are inner products, so normalise your vectors if you want cosine similarity. Need ids that survive deletes? `IdMapIndex` keys everything by your own `uint64` ids: ```python from turbovec import IdMapIndex index = IdMapIndex(dim=1536, bit_width=4) index.add_with_ids(vectors, ids) # ids: uint64 array, one per row scores, ids = index.search(query, k=10) # your ids, not slot positions index.remove(1002) # O(1) # Filter: only ids another system approved — a SQL predicate, an ACL, a time window. allowed = np.array([1001, 1003, 1042], dtype=np.uint64) scores, ids = index.search(query, k=10, allowlist=allowed) # shape (1, min(k, 3)) ``` Framework integrations are drop-in replacements for each framework's in-memory store — same public surface, same persistence, same retriever wiring: [LangChain](https://github.com/RyanCodrai/turbovec/blob/main/docs/integrations/langchain.md) (`pip install turbovec[langchain]`) · [LlamaIndex](https://github.com/RyanCodrai/turbovec/blob/main/docs/integrations/llama_index.md) (`turbovec[llama-index]`) · [Haystack](https://github.com/RyanCodrai/turbovec/blob/main/docs/integrations/haystack.md) (`turbovec[haystack]`) · [Agno](https://github.com/RyanCodrai/turbovec/blob/main/docs/integrations/agno.md) (`turbovec[agno]`). ## Rust ```bash cargo add turbovec ``` ```rust use turbovec::{IdMapIndex, TurboQuantIndex}; let mut index = TurboQuantIndex::new(1536, 4)?; index.add(&vectors); // &[f32], row-major, n × 1536 let results = index.search(&queries, 10); // scores and slot indices, nq × 10 index.write("index.tv")?; let index = TurboQuantIndex::load("index.tv")?; let mut ids = IdMapIndex::new(1536, 4)?; ids.add_with_ids(&vectors, &[1001, 1002, 1003])?; let (scores, found) = ids.search(&queries, 10); ids.remove(1002); ``` The Rust API mirrors the Python one; [`docs/api.md`](https://github.com/RyanCodrai/turbovec/blob/main/docs/api.md) covers both, including the file formats, `sync()`, calibration and the staged search. ## How it compares All figures: 100K OpenAI embeddings (d=1536 and d=3072) and GloVe (d=200), k=64, 1K queries, median of 5 runs, on a GCP c4a-standard-8 (ARM) and c3-standard-8 (x86), against FAISS at a matched bit rate. Scripts, raw JSON and every chart are in [`docs/benchmarks.md`](https://github.com/RyanCodrai/turbovec/blob/main/docs/benchmarks.md). | | turbovec | FAISS | | |---|---|---|---| | Search, 4-bit | 3.8–5.1× faster across the eight cells (ARM 4.0×, x86 4.9×) | `IndexPQFastScan` | every cell | | Search, 2-bit | 1.8–2.5× faster (ARM 2.05×, x86 2.35×) | `IndexPQFastScan` | every cell | | Recall@1, OpenAI | 0.959 / 0.981 at 4-bit, 0.901 / 0.931 at 2-bit (d=1536 / d=3072) | 0.966 / 0.971, 0.876 / 0.911 | `IndexPQ`, m matched to bit rate | | Recall@1, GloVe d=200 | 0.860 at 4-bit, 0.572 at 2-bit | 0.842, 0.565 | `IndexPQ`, m matched to bit rate | | Memory, d=1536 | 75 MB at 4-bit, 38 MB at 2-bit | 586 MB float32 | 7.8× / 15.5× | | Single `add()` | 7.9–21.7 µs | 6.3–12.6× slower | into a trained, populated index | | `remove(id)` | 1.4–5.8 µs | 0.19–1.03 **s** | FAISS repacks all codes per call | | Whole-file save | 1.0–1.4× slower than FAISS | `write_index` | turbovec fsyncs and renames atomically | | Load → first search | 1.0–1.5× slower than FAISS | `read_index` | 40 vs 31 ms at d=1536 4-bit on x86 | Both always return the true nearest neighbour within the first 8 results on the OpenAI corpora. The full suite — every chart, the raw JSON and the scripts to re-run it — is in [`docs/benchmarks.md`](https://github.com/RyanCodrai/turbovec/blob/main/docs/benchmarks.md). ## When to use something else - **You need a graph index.** turbovec scans every vector. At 100K vectors a query costs 0.13–0.25 ms on all eight cores of the benchmark boxes and 0.5–2.0 ms on one thread (k=64, 1,000-query batches, d=1536 to d=3072), and the cost grows linearly with the corpus. For hundreds of millions of vectors behind strict latency, an HNSW or IVF index in front of it is the right shape, and turbovec is not one. - **You need exact float results.** Quantization is lossy. On the OpenAI corpora the top result matches the float ground truth 96–98% of the time at 4-bit (90–93% at 2-bit) and is always within the first 8 returned; low-dimensional embeddings such as GloVe d=200 lose more. Check recall on a sample of your own data. - **You need a server.** turbovec is a library with a file format. There is no network API, replication or multi-tenant service; the integrations above are how it slots into a stack that has those. - **Your vectors are not multiples of 8 wide, or wider than 16,384.** Those are the dimension limits. Bit widths are 2 and 4. ## How it works Normalise each vector and store its length. Rotate every vector by one shared random orthogonal matrix, after which each coordinate follows a known distribution whatever the input. Quantise each coordinate with a Lloyd-Max codebook computed for that distribution — 4 levels at 2-bit, 16 at 4-bit — and bit-pack. At search time the query is rotated once and scored against the codebook with SIMD lookup tables; a per-vector scalar fixed at encode time removes the inner-product bias quantization introduces. An optional one-shot `calibrate(sample)` fits two scalars per coordinate for data that drifts from the asymptotic distribution (low-dimensional and word-vector embeddings gain most). The full walk-through, with the maths and the references, is in [`docs/how-it-works.md`](https://github.com/RyanCodrai/turbovec/blob/main/docs/how-it-works.md). ## Building from source Python wheel and Rust crate ```bash pip install maturin cd turbovec-python && maturin build --release && pip install target/wheels/*.whl ``` ```bash cargo build --release ``` x86_64 builds target `x86-64-v2` (SSE4.2, Nehalem 2008+) via `.cargo/config.toml`; the AVX-512 and AVX2 kernels are `#[target_feature]`-gated and chosen at runtime with `is_x86_feature_detected!`, so one binary runs everywhere and uses what the CPU has. Running the benchmark suite is described in [`docs/benchmarks.md`](https://github.com/RyanCodrai/turbovec/blob/main/docs/benchmarks.md#running-the-suite-yourself). ## References - [TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate](https://arxiv.org/abs/2504.19874) (ICLR 2026) — the paper this implements - [RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search](https://arxiv.org/abs/2405.12497) (SIGMOD 2024) — the per-vector length-renormalization correction - [FAISS Fast accumulation of PQ and AQ codes](https://github.com/facebookresearch/faiss/wiki/Fast-accumulation-of-PQ-and-AQ-codes-(FastScan)) — turbovec's x86 kernel adapts FastScan's pack layout, nibble-LUT scoring and u16 accumulator strategy --- ## File: docs/integrations/agno.md # Agno integration `turbovec.agno.TurboQuantVectorDb` is an [Agno](https://github.com/agno-agi/agno) `VectorDb` backed by an `IdMapIndex`. It implements the same public surface as `agno.vectordb.lancedb.LanceDb` (the closest in-tree single-machine backend) so this can be swapped in wherever LanceDb is used. ## Install ```bash pip install turbovec[agno] ``` ## Basic usage ```python from agno.agent import Agent from agno.knowledge import Knowledge from agno.knowledge.embedder.openai import OpenAIEmbedder from turbovec.agno import TurboQuantVectorDb vector_db = TurboQuantVectorDb(embedder=OpenAIEmbedder()) knowledge = Knowledge(vector_db=vector_db) knowledge.add_content(text_content="Turbovec compresses vectors to 4 bits per dimension.") agent = Agent(knowledge=knowledge) agent.print_response("What does turbovec do?") ``` ## Constructor ```python TurboQuantVectorDb( *, id: Optional[str] = None, name: Optional[str] = None, similarity_threshold: Optional[float] = None, embedder: Embedder, # required bit_width: int = 4, search_type: SearchType = SearchType.vector, distance: Distance = Distance.cosine, reranker: Optional[Reranker] = None, path: Optional[str] = None, ) ``` | Parameter | Notes | |---|---| | `embedder` | **Required.** Source of truth for the embedding dimension — `embedder.dimensions` sizes the underlying quantized index. | | `bit_width` | Quantization width per coordinate; 2 or 4. | | `search_type` | Only `SearchType.vector` is supported. Constructing with `keyword` or `hybrid` raises `ValueError` (keyword/hybrid would require an external BM25/lexical index that turbovec doesn't ship). | | `distance` | `Distance.cosine` (default) or `Distance.max_inner_product` — see [Similarity modes](#similarity-modes). `Distance.l2` raises `ValueError`. | | `similarity_threshold` | Optional. Results scoring below the threshold are dropped. Under `Distance.cosine` the score is the raw cosine, clamped to `[0, 1]` — the same definition agno's `normalize_cosine` and the pgvector backend use. Under `Distance.max_inner_product` the raw inner product is mapped via `(ip + 1) / 2` and thresholds are dataset-relative (see below). | | `reranker` | Optional Agno reranker applied to the result set after vector retrieval. | | `path` | Optional directory for save/load persistence. When given, `create()` loads existing data from this path if present. | ## Similarity modes The `distance` parameter selects how scores are computed. It is fixed for the lifetime of the store: - **`Distance.cosine` (default).** Document embeddings are L2-normalized at insert time and query embeddings at search time, so the kernel's raw score is cosine similarity in `[-1, 1]` for embeddings of any magnitude, and `similarity_threshold` compares against that cosine directly, so it behaves as a true `[0, 1]` relevance cutoff. A negative cosine clamps to 0, so any positive threshold drops it. Zero vectors are kept as-is and score `0` against everything. - **`Distance.max_inner_product`.** Vectors are stored and queried raw: ranking is by raw inner product (magnitude-aware). The `(ip + 1) / 2` mapping agno defines for this mode saturates at 0/1 once raw scores leave `[-1, 1]`, so `similarity_threshold` values are dataset-relative here — calibrate against your embedder, or leave the threshold unset. ## Insert / upsert `insert` and `upsert` follow the same `(content_hash, documents, filters)` signature as `LanceDb`. The internal `doc_id` is derived as `md5(f"{base_id}_{content_hash}")` where `base_id` is `doc.id` (or `md5(content)` when missing). The contract: the same `(base_id, content_hash)` pair always produces the same internal id, and the same `base_id` with a *different* `content_hash` is treated as a new entry — letting you keep content versions side-by-side. Because `doc_id` is derived from `base_id` + `content_hash` (not from `name`, `content_id`, or metadata), two documents can collide on the same `doc_id` — a repeated explicit `doc.id`, or two documents with identical content and no id. When that happens **both are stored and both remain individually deletable** — keep-all, matching `LanceDb`'s append-only behavior. (This differs from the LangChain store, which keeps the last write per id.) At query time, duplicate-content hits collapse to a single search result — see [Filtered search](#filtered-search). ```python from agno.knowledge.document import Document docs = [Document(id="doc-1", name="paper.pdf", content="...", meta_data={"source": "arxiv"})] vector_db.insert(content_hash="v1", documents=docs) # Same doc with a new content_hash → new stored entry. vector_db.insert(content_hash="v2", documents=docs) ``` Documents without embeddings are embedded via `self.embedder` before insertion. If embedding fails (`get_embedding` returns `None`) the call raises `ValueError` rather than silently dropping the document. ## Filtered search Filters are resolved to an allowlist **before** scoring — the kernel only ever inserts allowed candidates into the per-query heap. You always get up to `limit` results from the filtered set; no over-fetching, no recall hit on selective filters. ```python results = vector_db.search( "quantum computing applications", limit=5, filters={"source": "arxiv", "year": 2024}, # AND of exact equality ) ``` Dict filters use AND-of-exact-equality on `Document.meta_data`. List-style `FilterExpr` filters (Agno's structured filter type) are silently ignored, matching `LanceDb`'s behaviour. Search results are deduplicated by content: after filtering (and reranking, when a reranker is set), hits with identical `content` (keyed by `md5` of the text) collapse to the first occurrence, matching `LanceDb.search`. When duplicate-content documents are stored — e.g. the same text inserted under two `content_hash`es — a search can therefore return fewer than `limit` results; there is no over-fetch to refill the list. ## Existence checks ```python vector_db.name_exists("paper.pdf") # bool — by Document.name vector_db.id_exists("derived-md5-id") # bool — by the internally-derived id vector_db.content_hash_exists("v1") # O(1) — set lookup, not a scan ``` ## Delete ```python vector_db.delete_by_id(derived_id) # by internal id vector_db.delete_by_name("paper.pdf") # by Document.name vector_db.delete_by_metadata({"source": "web"}) # AND-of-equality on meta_data vector_db.delete_by_content_id("cid-42") # by Document.content_id vector_db.drop() # clear all vector_db.delete() # returns False, deletes nothing — use drop() ``` Each `delete_by_*` returns `True` iff at least one document was removed. `delete_by_name` / `delete_by_content_id` / `delete_by_metadata` remove only the documents matching that exact predicate, even when other stored documents share the same derived `doc_id`. `delete_by_id` removes every document under that internal id. ## update_metadata ```python vector_db.update_metadata("cid-42", {"reviewed": True}) ``` Merges the given metadata into `meta_data` of every document with the matching `content_id`. Overrides the base class's no-op warning. ## Save / load ```python vector_db = TurboQuantVectorDb(embedder=embedder, path="./my-store") vector_db.create() # loads from path if existing # ... insert documents ... vector_db.save() # persists to path ``` Writes two files under the given folder path: - `index.tvim` — the `IdMapIndex` payload. - `docstore.json` — JSON-encoded document text, metadata, and id maps. `create()` starts a fresh empty index only when the folder holds neither file. A folder holding just one of them is a partial save, and `create()` raises `FileNotFoundError` rather than starting empty — starting empty there would let the next `save()` overwrite the surviving file. Document metadata must be JSON-serializable — same constraint Agno's `LanceDb` imposes on its payload column. The side-car carries a `schema_version` field; loaders refuse to deserialize unknown versions, and validate that the side-car's id maps are consistent with the loaded `index.tvim` (a mismatched or out-of-sync pair raises at load rather than failing later at query time). The similarity mode is recorded in `docstore.json`. Because the store is constructed (with a `distance`) before `create()` loads the files, a recorded mode that conflicts with the constructor's raises `ValueError` — construct with the matching `distance` to load. A save written before the mode field existed holds raw, unnormalized vectors: it loads as `Distance.max_inner_product` — exactly the scoring it was written under — and `self.distance` is updated to reflect that. `save` is atomic with respect to the destination: both files are written to sibling temp files and moved into place, so a failed save (e.g. non-JSON-serializable metadata) leaves a store previously saved at the same path intact. The store also supports `pickle` (e.g. for `multiprocessing` workers, provided the embedder/reranker are picklable) and `copy.copy` / `copy.deepcopy` — both copies return a fully independent store (there is no shallow copy that shares the underlying index). ## Async The lifecycle, write, and read methods have async counterparts: `async_create`, `async_drop`, `async_exists`, `async_name_exists`, `async_get_count`, `async_insert`, `async_upsert`, `async_search`. The remaining methods (the `delete_by_*` family, `update_metadata`, `save`, `id_exists`, `content_hash_exists`, `optimize`) are sync-only. The async paths call the embedder's `async_get_embedding` / `async_get_embeddings_batch_and_usage` for genuine async embedding generation. Agno's `Embedder` base class always defines both, so an embedder that inherits them without implementing them raises `NotImplementedError` on the async paths — use the sync methods with such an embedder. `async_create`, `async_drop`, `async_insert`, `async_upsert`, and `async_search` run the index work on a worker thread (`asyncio.to_thread`) so the event loop stays responsive while a large insert or search is in flight. That is the shape Agno's own sync-backed vector DBs use (`chromadb`, `pgvector`, `cassandra`, `pineconedb` all wrap their sync bodies in `asyncio.to_thread`). `async_exists`, `async_name_exists`, and `async_get_count` answer inline — they are O(1) reads, and a thread hop would cost more than it saves. Cancellation is only partial, and the distinction matters: - `asyncio.wait_for`, `task.cancel()`, or a client disconnect returns control to the awaiting caller promptly — that part now works, where previously the coroutine ran to completion and the timeout never fired. - It does **not** decide what happened to the insert. If the worker thread had already started, it runs the call to completion — work inside the Rust core is not interruptible at all — and the insert commits in full. If the executor was saturated, the call is cancelled before it ever starts and nothing is inserted. **A cancelled `async_insert` is "outcome unknown": it may have fully committed, or may never have begun.** Retry through `async_upsert` on the same `content_hash` if you need the retry to be idempotent. - What *is* guaranteed: the outcome is all-or-nothing. The store is never left in a torn state. - Timing out does not make the work go away: the loop's shutdown (`asyncio.run` on the way out, or `loop.shutdown_default_executor()`) waits for the worker thread, so a process that exits right after a short timeout can still block for the rest of the in-flight call. ## Thread safety The store is safe for concurrent multi-threaded use: - **Reads run concurrently and scale.** `search`, the existence checks, and `get_count` take no lock; the underlying index releases the GIL during scoring, so independent searches from multiple threads overlap and scale. - **Writes serialize.** `insert`, `upsert`, the `delete_by_*` family, `update_metadata`, `drop`, and `save` serialize on a per-store lock. The `async_*` variants delegate to the same locked bodies. - **A read overlapping a write sees pre- or post-write state** — never a torn one. Under heavy concurrent churn a search may transiently return fewer than `limit` results (hits deleted mid-search are skipped). What the contract does *not* cover: - **No cross-call atomicity.** A caller-side check-then-act sequence (`id_exists` then `delete_by_id`) can interleave with other writers. Batch writes are not atomic with respect to readers: a search overlapping an `upsert` can briefly see both the old and new generation of a `content_hash`. - **`save` serializes with writes** (so it always snapshots a consistent store); reads may proceed during a save. - **The embedder and reranker are invoked outside the store's lock** and must be thread-safe themselves. - **Two stores writing to the same path is safe.** Concurrent `save` calls to one destination from several threads each publish atomically and the last writer wins; a caller never sees a torn file, and never an error caused only by the other writer. Which writer wins is not defined. - **Multi-process access is not supported.** ## Known limitations - **Vector search only.** `search_type=SearchType.keyword` and `SearchType.hybrid` are not supported (would require an external BM25 / lexical index). Constructor raises `ValueError` on those. - **No L2 distance.** `Distance.cosine` and `Distance.max_inner_product` are the supported metrics; `Distance.l2` raises `ValueError` (the underlying kernel scores by inner product). - **Embeddings are not retained after quantization.** Stored vectors are the quantized form; the original full-precision embedding can't be recovered. - **JSON-serializable metadata only.** Non-JSON-serializable values fail at `save()` time. --- ## File: docs/integrations/haystack.md # Haystack integration `turbovec.haystack.TurboQuantDocumentStore` is a Haystack [`DocumentStore`](https://docs.haystack.deepset.ai/docs/document-store) backed by an `IdMapIndex`. It implements the same public surface as `haystack.document_stores.in_memory.InMemoryDocumentStore`, so anywhere that store is *written to or read from directly* it can be swapped in. The query half of a RAG pipeline is not part of that surface — see [Using in a Haystack Pipeline](#using-in-a-haystack-pipeline) for the retriever you have to bring yourself. ## Install ```bash pip install turbovec[haystack] ``` ## Basic usage ```python from haystack import Document from turbovec.haystack import TurboQuantDocumentStore store = TurboQuantDocumentStore() store.write_documents([ Document(content="...", embedding=[...], meta={"source": "a"}), Document(content="...", embedding=[...], meta={"source": "b"}), ]) results = store.embedding_retrieval(query_embedding=[...], top_k=5) ``` Documents must have pre-computed embeddings — `TurboQuantDocumentStore` doesn't invoke an embedder. Pipe a Haystack embedder component upstream if your documents arrive without embeddings. ## Constructor ```python TurboQuantDocumentStore( dim: Optional[int] = None, bit_width: int = 4, *, embedding_similarity_function: Literal["dot_product", "cosine"] = "cosine", async_executor: Optional[ThreadPoolExecutor] = None, return_embedding: bool = False, ) ``` | Parameter | Notes | |---|---| | `dim` | Optional. When omitted the vector dimensionality is inferred from the first `write_documents` call. | | `bit_width` | Quantization width per coordinate; 2 or 4. | | `embedding_similarity_function` | The store's similarity mode — see [Similarity modes](#similarity-modes). Selects both how vectors are stored (`"cosine"`, the default, normalizes; `"dot_product"` keeps them raw) and the `scale_score=True` formula on retrieval. Any other value raises `ValueError`. | | `async_executor` | Optional `ThreadPoolExecutor` for the `*_async` methods. If omitted, a single-threaded executor is created and cleaned up with the store. | | `return_embedding` | Accepted for API parity with `InMemoryDocumentStore`. The full-precision embedding is never available (quantized away), so `Document.embedding` on retrieved docs is always `None` regardless of the flag. | ## Similarity modes `embedding_similarity_function` selects how scores are computed. It is fixed for the lifetime of the store: - **`"cosine"` (default).** Document embeddings are L2-normalized at write time and query embeddings at retrieval time, so raw scores are cosine similarity in `[-1, 1]` for embeddings of any magnitude, ranking matches `InMemoryDocumentStore`'s cosine branch, and `scale_score=True` maps scores into `[0, 1]` via `(s + 1) / 2` preserving order. Zero vectors are kept as-is and score `0` against everything (matching the reference, which substitutes a norm of 1 for zero-norm vectors). - **`"dot_product"`.** Vectors are stored and queried raw: scores are raw inner products and ranking is magnitude-aware — matching `InMemoryDocumentStore`'s dot-product branch. `scale_score=True` applies the reference's `expit(s / 100)` sigmoid. ## `DuplicatePolicy` `write_documents` takes a `policy` argument controlling how id collisions are handled: ```python from haystack.document_stores.types import DuplicatePolicy store.write_documents(docs, policy=DuplicatePolicy.FAIL) # raise if any id collides store.write_documents(docs, policy=DuplicatePolicy.SKIP) # silently skip colliding ids store.write_documents(docs, policy=DuplicatePolicy.OVERWRITE) # remove-then-re-add colliding ids # DuplicatePolicy.NONE is treated as FAIL. ``` Returns the number of documents actually written (so `SKIP` may return less than `len(docs)`). Under `FAIL` (and `NONE`), documents are committed one at a time in batch order and the `DuplicateDocumentError` is raised on the first colliding id — every non-duplicate document *before* the collision stays persisted, matching `InMemoryDocumentStore`'s post-exception state exactly. ## Delete ```python store.delete_documents(["id-1", "id-2"]) # by id; missing ids are silently ignored store.delete_by_filter(filters) # by filter; returns count store.delete_all_documents() # clear everything ``` `delete_documents` and `delete_by_filter` are O(1) per matching document via the inner `IdMapIndex`. ## Filters `filter_documents(filters)`, `embedding_retrieval(..., filters=...)`, and the other filter-aware helpers accept the full [Haystack filter DSL](https://docs.haystack.deepset.ai/docs/metadata-filtering): ```python filters = { "operator": "AND", "conditions": [ {"field": "meta.source", "operator": "==", "value": "manual"}, {"field": "meta.version", "operator": ">=", "value": 2}, ], } # All docs matching the filter (no vector search): docs = store.filter_documents(filters=filters) # Top-k nearest to a query, filtered: results = store.embedding_retrieval( query_embedding=[...], top_k=5, filters=filters, ) ``` Filter evaluation is delegated to `haystack.utils.filters.document_matches_filter` — anything Haystack's own stores support, we support. `embedding_retrieval` validates `query_embedding` up front and raises `ValueError("query_embedding should be a non-empty list of floats.")` for an empty or non-numeric vector, matching `InMemoryDocumentStore`. A negative `top_k` also raises, where the reference returns `n - 1` documents. For `embedding_retrieval`, filters are resolved to an allowlist **before** scoring rather than via post-filtering. Selective filters return up to `top_k` matches from the filtered set; you never get fewer than `top_k` results just because the filter happened to exclude the top-scoring candidates. ## Metadata helpers ```python store.count_documents_by_filter(filters) # int store.count_unique_metadata_by_filter(filters, ["source", "tag"]) # dict[str, int] store.update_by_filter(filters, {"reviewed": True}) # bulk metadata update; returns count store.get_metadata_fields_info() # {"source": {"type": "keyword"}, "version": {"type": "int"}, ...} store.get_metadata_field_min_max("version") # {"min": 1, "max": 5} store.get_metadata_field_unique_values("source") # (["a", "b", "c"], 3) ``` `update_by_filter` updates metadata only — embeddings are quantized at write time and not re-encoded. ## Async Every public method has an `*_async` variant: ```python await store.write_documents_async(docs) results = await store.embedding_retrieval_async(query_embedding=q, top_k=5) await store.delete_documents_async(["id-1"]) ``` By default they run on a single-threaded executor owned by the store. Pass an `async_executor=` to the constructor to share an executor across stores (or to use more workers). ## Save / load ```python store.save_to_disk("./my-store") # ... later ... store = TurboQuantDocumentStore.load_from_disk("./my-store") ``` Writes two files under the given folder path: - `index.tvim` — the `IdMapIndex` payload (quantized vectors + id maps). - `docstore.json` — JSON-encoded document text, metadata, and id maps. Document metadata must be JSON-serializable — the same constraint `InMemoryDocumentStore.save_to_disk` imposes. If the `docstore.json` side-car is out of sync with its `index.tvim` (a partial copy, a stale backup, tampering), `load_from_disk` raises a `ValueError` immediately rather than failing later with a `KeyError` at query time. `save_to_disk` is atomic with respect to the destination: both files are written to sibling temp files and moved into place, so a failed save (e.g. non-JSON-serializable metadata) leaves a store previously saved at the same path intact. The store also supports `pickle` (e.g. for `multiprocessing` workers; the restored store owns a fresh async executor) and `copy.copy` / `copy.deepcopy` — both copies return a fully independent store (there is no shallow copy that shares the underlying index). ## Using in a Haystack Pipeline `TurboQuantDocumentStore` implements `to_dict` / `from_dict` so it can be serialized as part of a Haystack `Pipeline`. `to_dict` captures the component *config* (`dim`, `bit_width`, `embedding_similarity_function`, `return_embedding`); persisting the stored documents is the job of `save_to_disk` / `load_from_disk`. Plug into a standard RAG pipeline, with two differences from `InMemoryDocumentStore` worth knowing before you wire it up. **No paired retriever ships.** In Haystack the query half of a pipeline is a store-specific `@component` retriever, and core's `InMemoryEmbeddingRetriever` hard-rejects any store that is not the in-memory one. turbovec does not ship a retriever, so supply your own thin component that calls `store.embedding_retrieval(...)`, or query the store directly outside the pipeline. **Reloading a serialized pipeline needs an allowlist.** `to_dict` / `from_dict` work, but on haystack-ai 3.x — permitted by the declared `haystack-ai>=2.23.0` floor — deserializing a pipeline that references an out-of-tree store raises `DeserializationError` unless the module is trusted. `InMemoryDocumentStore` is exempt because Haystack trusts its own module: ```python Pipeline.loads(pipeline.dumps()) # DeserializationError Pipeline.loads(pipeline.dumps(), allowed_modules=["turbovec.haystack"]) # OK ``` `HAYSTACK_DESERIALIZATION_ALLOWLIST` sets the same thing process-wide. The sentence-transformers embedders live in their own integration package (`pip install sentence-transformers-haystack`, which requires `haystack-ai` 2.24 or newer): ```python from haystack import Pipeline from haystack.components.writers import DocumentWriter from haystack_integrations.components.embedders.sentence_transformers import ( SentenceTransformersDocumentEmbedder, ) store = TurboQuantDocumentStore() # dim inferred from first batch indexing = Pipeline() indexing.add_component("embedder", SentenceTransformersDocumentEmbedder( model="sentence-transformers/all-MiniLM-L6-v2", )) indexing.add_component("writer", DocumentWriter(document_store=store)) indexing.connect("embedder.documents", "writer.documents") indexing.run({"embedder": {"documents": my_docs}}) ``` ## Thread safety The store is safe for concurrent multi-threaded use: - **Reads run concurrently and scale.** `embedding_retrieval`, `filter_documents`, the count and metadata helpers, and `storage` take no lock; the underlying index releases the GIL during scoring, so independent retrievals from multiple threads overlap and scale. - **Writes serialize.** `write_documents`, `delete_documents` / `delete_all_documents` / `delete_by_filter`, `update_by_filter`, and `save_to_disk` serialize on a per-store lock. The `*_async` variants delegate to the same locked bodies. - **A read overlapping a write sees pre- or post-write state** — never a torn one. Under heavy concurrent churn a retrieval may transiently return fewer than `top_k` documents (hits deleted mid-retrieval are skipped). What the contract does *not* cover: - **No cross-call atomicity.** A caller-side check-then-act sequence (`count_documents` then `filter_documents`) can interleave with other writers. Batch writes are not atomic with respect to readers: a retrieval overlapping an `OVERWRITE` write can briefly see a document id under both its old and new entry. - **`save_to_disk` serializes with writes** (so it always snapshots a consistent store); reads may proceed during a save. - **`to_dict` / `from_dict` and the executor lifecycle** are assumed single-threaded. - **Two stores writing to the same path is safe.** Concurrent `save_to_disk` calls to one destination from several threads each publish atomically and the last writer wins; a caller never sees a torn file, and never an error caused only by the other writer. Which writer wins is not defined. - **Multi-process access is not supported.** ## Known limitations - **Embeddings are not retained.** `embedding_retrieval(..., return_embedding=True)` is accepted for signature compatibility but `Document.embedding` is always `None` on retrieved docs — turbovec discards the full-precision vector after quantization. - **JSON-serializable metadata only.** Document metadata is stored as JSON in the side-car. Non-JSON-serializable values (custom objects, sets, etc.) fail at save time — the same constraint `InMemoryDocumentStore.save_to_disk` imposes. - **`dim` is locked on the first add.** Subsequent calls with a different shape raise `ValueError`. If you need to change `dim`, construct a fresh store. --- ## File: docs/integrations/langchain.md # LangChain integration `turbovec.langchain.TurboQuantVectorStore` is a [LangChain `VectorStore`](https://python.langchain.com/docs/integrations/vectorstores/) backed by an `IdMapIndex`. It implements the same public surface as `langchain_core.vectorstores.in_memory.InMemoryVectorStore` and can be used as a drop-in replacement wherever the in-memory store is used. ## Install ```bash pip install turbovec[langchain] ``` ## Basic usage ```python from langchain_huggingface import HuggingFaceEmbeddings from turbovec.langchain import TurboQuantVectorStore embeddings = HuggingFaceEmbeddings(model_name="BAAI/bge-base-en-v1.5") store = TurboQuantVectorStore.from_texts( texts=["Document 1...", "Document 2...", "Document 3..."], embedding=embeddings, bit_width=4, ) retriever = store.as_retriever(search_kwargs={"k": 5}) ``` The dimensionality of the underlying quantized index is inferred from the embedding model on the first `add_*` call — no need to specify it up front. ## Construction ```python # No-arg: lazy. dim is inferred from the first add. store = TurboQuantVectorStore(embeddings) # from_texts: same lazy behaviour, plus immediate ingest. store = TurboQuantVectorStore.from_texts(texts, embeddings, bit_width=4) # Pre-built index: bring your own IdMapIndex (e.g. one loaded from disk). from turbovec import IdMapIndex store = TurboQuantVectorStore(embeddings, index=IdMapIndex(1536, 4)) ``` `bit_width` is 2 or 4 and is fixed once the index is created. ## Similarity modes The `similarity` keyword (on the constructor and `from_texts`/`afrom_texts`) selects how scores are computed. It is fixed for the lifetime of the store: - **`"cosine"` (default).** Document vectors are L2-normalized before they reach the quantized index and query vectors are normalized before search, so scores are true cosine similarity in `[-1, 1]` and ranking matches `InMemoryVectorStore` regardless of embedding magnitude. Zero vectors are kept as-is and score `0` against everything (matching the reference's behavior). - **`"dot_product"`.** Vectors are stored and queried raw: scores are raw inner products and ranking is magnitude-aware. The `(sim + 1) / 2` relevance mapping still applies for continuity, but it is **not** clamped to `[0, 1]` in this mode — an unbounded inner product has no calibrated relevance, so clamping would silently collapse every score `>= 1.0` onto exactly `1.0` and defeat `score_threshold` retrieval. Requesting a relevance-score fn in this mode emits a `UserWarning`, and out-of-range values reach LangChain's own out-of-range warning. `score_threshold` retrieval is only meaningful in this mode if your embeddings are unit-normalized upstream. ```python store = TurboQuantVectorStore(embeddings, similarity="dot_product") ``` The `similarity` keyword is a turbovec extension: `InMemoryVectorStore` computes cosine unconditionally, so code written against the reference behaves identically under the default. ## Adding with explicit ids ```python store.add_texts( texts=["a", "b", "c"], ids=["doc-a", "doc-b", "doc-c"], metadatas=[{"source": "x"}, {"source": "y"}, {"source": "z"}], ) # add_documents honours per-Document.id, falling back to a UUID per # document if .id is missing — partial ids are not dropped wholesale. store.add_documents([ Document(id="explicit", page_content="..."), Document(page_content="..."), # gets a UUID ]) ``` If an id is already present, `add_texts` **upserts** — the existing entry is removed and the new one added with the same id. This matches the typical user expectation that re-indexing a document with the same id should replace it, not duplicate it. Ids must be `str`. A `None` entry in an explicit ids list is replaced with a generated UUID; any other non-`str` id (including `bool`/`int`) raises `TypeError` — naming the offending id, its type, and its position — before anything is stored. This is stricter than `InMemoryVectorStore`, which accepts non-str ids and then corrupts them through JSON persistence (an `int` `2` coexisting with the `str` `"2"` collapses to one document across `dump`/`load`). Async equivalents (`aadd_texts`, `aadd_documents`) use the embedding model's `aembed_documents` so they benefit from concurrent embedding generation when the model supports it. ## Search ```python # By string query (uses the embedding function) docs = store.similarity_search("what is turbovec?", k=5) # With scores docs_and_scores = store.similarity_search_with_score("...", k=5) # By raw vector import numpy as np qvec = np.random.randn(768).astype(np.float32) qvec /= np.linalg.norm(qvec) docs = store.similarity_search_by_vector(qvec.tolist(), k=5) ``` Under the default `similarity="cosine"` mode, scores are cosine similarity — higher is better, range `[-1, 1]` — for embeddings of any magnitude (see [Similarity modes](#similarity-modes)). `similarity_search_with_relevance_scores` and `as_retriever(search_type="similarity_score_threshold")` work: the cosine is mapped to `[0, 1]` via `(sim + 1) / 2` (clamped to absorb the tiny overshoot caused by quantization noise). The clamp is cosine-only — see [Similarity modes](#similarity-modes) for `dot_product`. `similarity_search_with_score_by_vector` returns `(document, score)` pairs for a precomputed query vector. Async equivalents (`asimilarity_search`, `asimilarity_search_with_score`, `asimilarity_search_by_vector`, `asimilarity_search_with_score_by_vector`, `aget_by_ids`) are all implemented. ## Filters `similarity_search`, `similarity_search_with_score`, and `similarity_search_by_vector` all accept a `filter` keyword: ```python # Dict — AND of exact equality on Document.metadata. docs = store.similarity_search( "query", k=5, filter={"source": "manual", "version": 2}, ) # Callable — predicate over the Document. docs = store.similarity_search( "query", k=5, filter=lambda doc: doc.metadata.get("score", 0) > 0.8, ) ``` The callable form matches the `Callable[[Document], bool]` convention used by `InMemoryVectorStore`, so predicates ported from there work unchanged. The dict form is a turbovec convenience on top of it — `InMemoryVectorStore` itself takes callables only. A dict entry requires the key to be **present**: `filter={"source": None}` matches documents that store `source=None`, not documents with no `source` key at all. To match on absence, use the callable form (`lambda doc: "source" not in doc.metadata`). Filters are resolved to an id allowlist **before** scoring; the kernel only ever inserts allowed documents into the per-query heap. You get up to `k` results from the filtered set, never fewer than `k` because the filter happened to exclude the top-scoring candidates. ## Document retrieval by id ```python docs = store.get_by_ids(["doc-a", "doc-c"]) # Missing ids are silently skipped. ``` `aget_by_ids` is also available. ## Delete ```python store.delete(["doc-a", "doc-b"]) # missing ids silently skipped, returns None ``` Delete is O(1) per id. `delete(None)` is a no-op (matches the `InMemoryVectorStore` contract). ## Save / load ```python store.dump("./my-store") # ... later ... store = TurboQuantVectorStore.load("./my-store", embedding=embeddings) ``` Writes two files under the given folder path: - `index.tvim` — the `IdMapIndex` payload (see [api.md](../api.md#tvim--idmapindex)). - `docstore.json` — JSON-encoded document text, metadata, and id maps. The similarity mode is recorded in `docstore.json` and restored by `load`. A store folder written before the mode field existed holds raw, unnormalized vectors, so it loads in `"dot_product"` mode — exactly the scoring it was written under — with no migration needed. Document metadata must be JSON-serializable — the same constraint `InMemoryVectorStore.dump` imposes. If the `docstore.json` side-car is out of sync with its `index.tvim` (a partial copy, a stale backup, tampering), `load` raises a `ValueError` immediately rather than failing later with a `KeyError` at query time. `dump` is atomic with respect to the destination: both files are written to sibling temp files and moved into place, so a failed dump (e.g. non-JSON-serializable metadata) leaves a store previously saved at the same path intact. The store also supports `pickle` (e.g. for `multiprocessing` workers, provided the embedder is picklable) and `copy.copy` / `copy.deepcopy` — both copies return a fully independent store (there is no shallow copy that shares the underlying index). ## Async Every read and write method has an `a*` counterpart: `aadd_texts`, `aadd_documents`, `asimilarity_search*`, `aget_by_ids`, `adelete`, `afrom_texts`. They run the index work on a worker thread (`asyncio.to_thread`), so the event loop stays responsive while a large add or search is in flight — the same contract as `VectorStore`'s own default async implementations, which offload via `run_in_executor`. The embedding step still awaits the embedder's own `aembed_*` coroutine. Cancellation is only partial, and the distinction matters: - `asyncio.wait_for`, `task.cancel()`, or a client disconnect returns control to the awaiting caller promptly — that part now works, where previously the coroutine ran to completion and the timeout never fired. - It does **not** decide what happened to the write. If the worker thread had already started, it runs the call to completion — work inside the Rust core is not interruptible at all — and the write commits in full. If the executor was saturated, the call is cancelled before it ever starts and nothing is written. **A cancelled write is "outcome unknown": it may have fully committed, or may never have begun.** Neither "it happened" nor "it did not happen" is safe to assume; make retries idempotent by passing explicit `ids`, or read back to find out. - What *is* guaranteed: the outcome is all-or-nothing. The store is never left in a torn state. - Timing out does not make the work go away: the loop's shutdown (`asyncio.run` on the way out, or `loop.shutdown_default_executor()`) waits for the worker thread, so a process that exits right after a short timeout can still block for the rest of the in-flight call. ## Thread safety The store is safe for concurrent multi-threaded use: - **Reads run concurrently and scale.** `similarity_search*` and `get_by_ids` take no lock; the underlying index releases the GIL during scoring, so independent searches from multiple threads overlap and scale. - **Writes serialize.** `add_texts` / `add_documents`, `delete`, and `dump` (and their async counterparts) serialize on a per-store lock. - **A read overlapping a write sees pre- or post-write state** — never a torn one. Under heavy concurrent churn a search may transiently return fewer than `k` results (hits deleted mid-search are skipped). What the contract does *not* cover: - **No cross-call atomicity.** A caller-side check-then-act sequence (`get_by_ids` then `delete`, a count then a search) can interleave with other writers. Batch writes are not atomic with respect to readers: a search overlapping an upsert can briefly see a document id under both its old and new entry. - **`dump` serializes with writes** (so it always snapshots a consistent store); reads may proceed during a dump. - **The embedder is invoked outside the store's lock** and must be thread-safe itself. - **Two stores writing to the same path is safe.** Concurrent `dump` calls to one destination from several threads each publish atomically and the last writer wins; a caller never sees a torn file, and never an error caused only by the other writer. Which writer wins is not defined. - **Multi-process access is not supported.** ## Known limitations - **Max-marginal-relevance search is not supported.** `max_marginal_relevance_search` and its variants raise `NotImplementedError` with an explanation. MMR requires the full-precision embedding of each candidate to compute pairwise diversity; turbovec discards full-precision vectors after quantization. If you need MMR, keep a parallel store with the raw embeddings and run MMR over that. - **Embeddings are not retained.** `search` returns `Document` objects with `page_content` and `metadata`, but the original embedding is not recoverable. - **JSON-serializable metadata only.** Non-JSON-serializable values (custom objects, sets, etc.) fail at save time — same constraint as the in-tree reference store. --- ## File: docs/integrations/llama_index.md # LlamaIndex integration `turbovec.llama_index.TurboQuantVectorStore` is a LlamaIndex [`BasePydanticVectorStore`](https://docs.llamaindex.ai/en/stable/module_guides/storing/vector_stores/) backed by an `IdMapIndex`. It implements the same public surface as `llama_index.core.vector_stores.simple.SimpleVectorStore` and can be used as a drop-in replacement wherever the simple in-memory store is used. ## Install ```bash pip install turbovec[llama-index] ``` ## Basic usage ```python from llama_index.core import VectorStoreIndex, StorageContext from turbovec.llama_index import TurboQuantVectorStore vector_store = TurboQuantVectorStore() storage_context = StorageContext.from_defaults(vector_store=vector_store) index = VectorStoreIndex.from_documents(documents, storage_context=storage_context) retriever = index.as_retriever(similarity_top_k=5) ``` The vector dimensionality is inferred from the embedding model on the first `add()` call. ## Construction ```python # No-arg: lazy. dim is inferred from the first add. vector_store = TurboQuantVectorStore() # from_params: same lazy behaviour, plus an explicit bit_width. vector_store = TurboQuantVectorStore.from_params(bit_width=4) # Pre-built index: bring your own IdMapIndex (e.g. one you loaded from disk). from turbovec import IdMapIndex vector_store = TurboQuantVectorStore(index=IdMapIndex(1536, 4)) ``` `bit_width` is 2 or 4 and is fixed once the index is created. ## Similarity modes The `similarity` keyword (on the constructor and `from_params`) selects how the `similarities` returned by `query` are computed. It is fixed for the lifetime of the store: - **`"cosine"` (default).** Node embeddings are L2-normalized at add time and query embeddings at query time, so `result.similarities` are true cosine similarities in `[-1, 1]` and ranking matches `SimpleVectorStore` regardless of embedding magnitude — safe to feed into similarity-cutoff postprocessors. Zero vectors are kept as-is and score `0` against everything. - **`"dot_product"`.** Vectors are stored and queried raw: `result.similarities` are raw inner products and ranking is magnitude-aware. ```python vector_store = TurboQuantVectorStore(similarity="dot_product") ``` The `similarity` keyword is a turbovec extension: `SimpleVectorStore` computes cosine unconditionally, so code written against the reference behaves identically under the default. ## The two `delete` signatures LlamaIndex's vector-store protocol has two distinct delete entry points: ### `delete(ref_doc_id: str)` — remove an entire source document Removes **every node** whose `ref_doc_id` matches. Use this when you want to delete a whole parent document and its chunks in one call. ```python vector_store.delete("my-source-document-123") ``` Missing `ref_doc_id`s are silently ignored. ### `delete_nodes(node_ids, filters)` — remove specific chunks Removes nodes matching either `node_ids`, `filters`, or both (intersected). Missing `node_id`s are silently ignored. ```python # By node_id vector_store.delete_nodes(node_ids=["abc-123", "def-456"]) # By metadata filter from llama_index.core.vector_stores.types import ( MetadataFilter, MetadataFilters, FilterOperator, ) filters = MetadataFilters( filters=[MetadataFilter(key="tier", value="archived", operator=FilterOperator.EQ)], ) vector_store.delete_nodes(filters=filters) # Both: intersect — delete only nodes in this list that ALSO match the filter vector_store.delete_nodes(node_ids=["abc-123"], filters=filters) ``` ### `clear()` — drop everything ```python vector_store.clear() ``` Resets the store while preserving the configured `bit_width`. The cleared store is immediately usable for new adds; `dim` is inferred again from the next batch. ## Query LlamaIndex calls `query(VectorStoreQuery)` internally. If you've gone through `VectorStoreIndex.from_documents(...)`, you won't call this directly — the retriever does. For direct use: ```python from llama_index.core.vector_stores.types import VectorStoreQuery result = vector_store.query(VectorStoreQuery( query_embedding=[...], similarity_top_k=5, )) # result.nodes, result.similarities, result.ids ``` `query_embedding` is **required**. turbovec doesn't embed query text itself; the calling component (retriever / query engine) is responsible for that. ### Filtered query `VectorStoreQuery` accepts `filters`, `node_ids`, and `doc_ids`. All three intersect when more than one is supplied: ```python from llama_index.core.vector_stores.types import ( MetadataFilter, MetadataFilters, FilterCondition, FilterOperator, VectorStoreQuery, ) filters = MetadataFilters( filters=[ MetadataFilter(key="tier", value="pro", operator=FilterOperator.EQ), MetadataFilter(key="year", value=2024, operator=FilterOperator.GTE), ], condition=FilterCondition.AND, ) result = vector_store.query(VectorStoreQuery( query_embedding=[...], similarity_top_k=5, filters=filters, node_ids=["chunk-1", "chunk-2", "chunk-3"], # restrict to these chunks doc_ids=["src-doc-42"], # restrict to chunks of this source doc )) ``` Supported operators on `MetadataFilter`: `EQ`, `NE`, `GT`, `LT`, `GTE`, `LTE`, `IN`, `NIN`, `TEXT_MATCH`, `TEXT_MATCH_INSENSITIVE`, `CONTAINS`, `ANY`, `ALL`, `IS_EMPTY`. Conditions: `AND`, `OR`, `NOT`. Nested `MetadataFilters` work. Filter semantics match `SimpleVectorStore`'s reference implementation — notably, every operator except `IS_EMPTY` returns `False` when the filter key is missing from the document's metadata, and `TEXT_MATCH` is case-insensitive (it lowercases both sides, as `TEXT_MATCH_INSENSITIVE` does). Filters are resolved to a handle allowlist **before** scoring. Selective filters return up to `similarity_top_k` matches from the filtered set; you never get fewer just because the filter happened to exclude the top-scoring candidates. An **empty** `node_ids` (or `doc_ids`) list restricts nothing — it behaves like omitting the argument. This follows the framework's own calling convention: `VectorStoreIndex.as_retriever` passes `node_ids=list(index_struct.nodes_dict.values())`, and for a `stores_text=True` store like this one that list is always empty — it means "unrestricted", and treating it as match-nothing would make every retriever query return zero results. `get_nodes` / `delete_nodes` are different: there `node_ids` *is* the selection, so an explicit empty list selects nothing. ## Get nodes ```python nodes = vector_store.get_nodes(node_ids=["chunk-1", "chunk-2"]) nodes = vector_store.get_nodes(filters=filters) nodes = vector_store.get_nodes(node_ids=["chunk-1", "chunk-2"], filters=filters) # intersect ``` Returns a `List[BaseNode]` reconstructed from the side-car. Missing `node_id`s are silently skipped. `node_ids` is the explicit selection: an empty list selects nothing and returns `[]` (same for `delete_nodes`, where an empty list is a no-op). ## Upsert semantics Calling `add()` with a node whose `node_id` already exists **replaces** the existing entry. Matches LlamaIndex user expectation when re-indexing the same chunks. A `node_id` repeated **within a single `add()` batch** raises `ValueError` — deduplicate before calling. (This differs from the LangChain and Haystack stores, which silently keep the last occurrence; here it's a hard error so an accidental duplicate doesn't quietly drop a node.) ```python node = TextNode(text="v1", embedding=[...]) vector_store.add([node]) # Same node_id, different text/embedding → replaces. updated = TextNode(text="v2", id_=node.node_id, embedding=[...]) vector_store.add([updated]) assert len(vector_store._index) == 1 ``` ## Async Every public method has an async counterpart, suitable for use in LlamaIndex's async retriever / query-engine paths: ```python await vector_store.async_add(nodes) result = await vector_store.aquery(VectorStoreQuery(...)) fetched = await vector_store.aget_nodes(node_ids=[...]) await vector_store.adelete("ref-doc-id") await vector_store.adelete_nodes(node_ids=[...]) await vector_store.aclear() ``` They run the index work on a worker thread (`asyncio.to_thread`), so the event loop stays responsive while a large add or query is in flight. `BasePydanticVectorStore`'s defaults call straight into the sync body, which would block the loop for the operation's full duration. Cancellation is only partial, and the distinction matters: - `asyncio.wait_for`, `task.cancel()`, or a client disconnect returns control to the awaiting caller promptly — that part now works, where previously the coroutine ran to completion and the timeout never fired. - It does **not** decide what happened to the add. If the worker thread had already started, it runs the call to completion — work inside the Rust core is not interruptible at all — and the add commits in full. If the executor was saturated, the call is cancelled before it ever starts and nothing is added. **A cancelled `async_add` is "outcome unknown": it may have fully committed, or may never have begun.** Re-adding the same `node_id` is an overwrite, so retrying is safe either way. - What *is* guaranteed: the outcome is all-or-nothing. The store is never left in a torn state. - Timing out does not make the work go away: the loop's shutdown (`asyncio.run` on the way out, or `loop.shutdown_default_executor()`) waits for the worker thread, so a process that exits right after a short timeout can still block for the rest of the in-flight call. ## Persist / load ### Direct (file-stem) interface ```python vector_store.persist("./store/vectors.json") # ... later ... vector_store = TurboQuantVectorStore.from_persist_path("./store/vectors.json") ``` `persist_path` is treated as a path *stem* — the binary index and JSON side-car are written next to each other as `{stem}.tvim` and `{stem}.nodes.json`. The extension on `persist_path` (e.g. `.json`, as LlamaIndex's StorageContext default uses) is replaced. Node metadata must be JSON-serializable. If the `{stem}.nodes.json` side-car is out of sync with its `{stem}.tvim` index (a partial copy, a stale backup, tampering), `from_persist_path` raises a `ValueError` immediately rather than failing later with a `KeyError` at query time. `persist` is atomic with respect to the destination: both files are written to sibling temp files and moved into place, so a failed persist (e.g. non-JSON-serializable metadata) leaves a store previously persisted at the same stem intact. The similarity mode is recorded in `{stem}.nodes.json` and restored by `from_persist_path`. A store persisted before the mode field existed holds raw, unnormalized vectors, so it loads in `"dot_product"` mode — exactly the scoring it was written under — with no migration needed. ### Via `StorageContext` The store works with `StorageContext.from_defaults(persist_dir=...)` the same way `SimpleVectorStore` does: ```python # Persist storage_context.persist(persist_dir="./store") # Load vector_store = TurboQuantVectorStore.from_persist_dir(persist_dir="./store") storage_context = StorageContext.from_defaults( vector_store=vector_store, persist_dir="./store", ) ``` `from_persist_dir(persist_dir, namespace="default", fs=None)` constructs the namespaced filename (`{persist_dir}/{namespace}__vector_store.json`) and delegates to `from_persist_path`. Multiple namespaced stores can share a persist directory — including dotted namespaces (`v1.2`, `v1.3`), which map to distinct file pairs (`v1.2__vector_store.tvim` / `v1.2__vector_store.nodes.json`, and so on). `namespace` names a store *within* `persist_dir`, so it must be non-empty and must not contain path separators, `..`, or `:` (a Windows drive-relative name like `C:foo` would escape `persist_dir`); such a value raises `ValueError`. Any other string (alphanumerics, dash, underscore, dots) is accepted. A store persisted by an older turbovec under a dotted namespace sits on disk under a truncated filename (`v1.tvim` for namespace `v1.2`). Loading finds it via a legacy-filename fallback — used only when the correct filename is absent — and the next `persist` writes the correct filenames. ### Config-only round-trip ```python config = vector_store.to_dict() # {"bit_width": 4, "dim": 1536, "similarity": "cosine"} fresh = TurboQuantVectorStore.from_dict(config) # empty store with the same config ``` `to_dict` / `from_dict` serialize only the store's configuration. Node data round-trips through `persist` / `from_persist_path`. The store also supports `pickle` with full data fidelity (e.g. for `multiprocessing` workers) and `copy.copy` / `copy.deepcopy` — both copies return a fully independent store (there is no shallow copy that shares the underlying index). ## Thread safety The store is safe for concurrent multi-threaded use: - **Reads run concurrently and scale.** `query` and `get_nodes` take no lock; the underlying index releases the GIL during scoring, so independent queries from multiple threads overlap and scale. - **Writes serialize.** `add`, `delete`, `delete_nodes`, `clear`, and `persist` serialize on a per-store lock. The `async_add` / `a*` variants delegate to the same locked bodies, so concurrent adds issue unique handles — no batch is ever rejected or lost to a handle collision. - **A read overlapping a write sees pre- or post-write state** — never a torn one. Under heavy concurrent churn a query may transiently return fewer than `similarity_top_k` results (hits deleted mid-query are skipped). What the contract does *not* cover: - **No cross-call atomicity.** A caller-side check-then-act sequence (`get_nodes` then `delete_nodes`) can interleave with other writers. Batch writes are not atomic with respect to readers: a query overlapping a re-`add` of an existing `node_id` can briefly see that id under both its old and new entry. - **`persist` serializes with writes** (so it always snapshots a consistent store); reads may proceed during a persist. - **Two stores writing to the same path is safe.** Concurrent `persist` calls to one destination from several threads each publish atomically and the last writer wins; a caller never sees a torn file, and never an error caused only by the other writer. Which writer wins is not defined. - **Multi-process access is not supported.** ## Known limitations - **MMR is not supported.** Max-marginal-relevance retrieval requires the full-precision embedding of each candidate to compute pairwise diversity; turbovec discards full-precision vectors after quantization. - **`get(text_id)` raises** rather than returning a vector — same reason. The full-precision embedding is not recoverable. - **`fsspec` filesystems are not supported.** `persist`, `from_persist_path`, and `from_persist_dir` accept a local path. Pass `fs=None` (the default). - **JSON-serializable metadata only.** Node metadata is stored as JSON in the side-car. Non-JSON-serializable values fail at persist time — same constraint as `SimpleVectorStore.persist`. - **`stores_text = True`.** Unlike `SimpleVectorStore`, we keep node text in the side-car so query results return populated `TextNode`s without depending on a separate docstore. If you're swapping this in for `SimpleVectorStore` and your pipeline expects text to live elsewhere, the difference is harmless — the framework treats `stores_text` as informational. --- ## File: docs/api.md # API Reference turbovec exposes two index types and one serialization format per type. - [`TurboQuantIndex`](#turboquantindex) — positional index, O(1) `swap_remove` delete. - [`IdMapIndex`](#idmapindex) — stable external `u64` ids on top of `TurboQuantIndex`. - [TQ+ calibration](#tq-calibration) — the per-coordinate calibration lifecycle. - [File formats](#file-formats) — `.tv` and `.tvim`, plus [incremental saves](#incremental-saves--sync). All examples below are Python. The Rust API mirrors it closely (exceptions noted below) — see each type's rustdoc for the exact signatures. --- ## `TurboQuantIndex` Positional index. Each vector is identified by its insertion slot (`0..n`). Fast and small, but external references to slots are invalidated by `swap_remove`. If you need stable ids, use [`IdMapIndex`](#idmapindex). ```python from turbovec import TurboQuantIndex idx = TurboQuantIndex(dim=1536, bit_width=4) idx.add(vectors) # np.ndarray of shape (n, dim), float32 scores, indices = idx.search(queries, k=10) idx.swap_remove(5) # O(1); the previously-last vector moves into slot 5 idx.write("index.tv") # .tv format loaded = TurboQuantIndex.load("index.tv") ``` `dim` is optional. Omit it to let the index pick up the dimensionality from the first batch of vectors: ```python idx = TurboQuantIndex(bit_width=4) # dim inferred on first add idx.add(vectors) # locks dim to vectors.shape[1] ``` Before the first add, `idx.dim` is `None`, `len(idx)` is `0`, and `search()` returns empty results. Adding a zero-row batch is a no-op: `dim` is still checked against the batch, but a lazy index stays lazy and its serialized bytes are unchanged. (On the Rust API, `dim_opt()` is the equivalent of `idx.dim` and returns `Option`; `dim()` is deprecated — it returns `usize` with `0` for a lazy index, which is unsafe to do arithmetic with, so use `dim_opt()` on any path that can see one.) ### Methods | Method | Notes | |---|---| | `TurboQuantIndex(dim=None, bit_width=4)` | `bit_width ∈ {2, 3, 4}`. `dim` must be a positive multiple of 8 and `≤ 16384` (`MAX_DIM`). `dim` is optional; when omitted it is inferred from the first `add` call. | | `add(vectors)` | `vectors` is a contiguous float32 array of shape `(n, dim)`. On a lazy index the first call locks `dim`; subsequent calls must match. Raises `ValueError` on dim mismatch, a zero-width (0-column) batch, or any coordinate that is non-finite (NaN/Inf) or `\|value\| ≥ 1e16`. A vector whose L2 norm is at or below `1e-10` has no representable direction and is stored with scale 0, scoring 0 against every query. On the Rust API, a lazy index's first add must use `add_2d(vectors, dim)` — the flat `add(&[f32])` requires an already-committed dim and panics otherwise. (Python arrays carry their shape, so this applies to Rust only.) | | `search(queries, k, *, mask=None)` | Returns `(scores, indices)`, both shape `(nq, effective_k)`. Indices are `int64` slot positions. `mask` is an optional `bool` array of length `len(idx)`; when given, only slots with `mask[i] == True` contribute. `effective_k = min(k, mask.sum())`. Raises `ValueError` on a non-finite or `\|value\| ≥ 1e16` query coordinate. The returned ids are invariant to multiplying a query by any positive constant, for as long as the scaled coordinates stay within float32's normal range (smallest normal `1.18e-38`). Scale a query far enough that its coordinates go subnormal and they lose relative precision before scoring, so the ranking can change — measured at query magnitude `~1e-36` for `dim=256` and `~1e-35` for `dim=768`, well below any realistic embedding. The scores are inner products, so they scale with the query. | | `swap_remove(idx)` | O(1). Moves the last vector into `idx`; returns the previous position of that moved vector (so external refs can be updated if needed). | | `prepare()` | Optional. Eagerly builds the rotation matrix, Lloyd-Max centroids and SIMD-blocked layout so the first `search` call doesn't pay the one-time cost. No-op on a lazy index that hasn't seen its first add. | | `sync(path)` | Incremental save: writes only what changed since the last sync to the same path. See [Incremental saves](#incremental-saves--sync). `load(path)` reads both formats. | | `write(path, *, durable=True)` / `load(path)` | `.tv` format. `durable=False` skips the fsync before the atomic rename — faster, but a power loss can lose the file. A `durable=True` save whose post-rename directory fsync fails still succeeds (the file is committed and visible) and raises a `RuntimeWarning` saying the rename may not survive power loss. On the Rust API this is not a flag: use `write_with_durability(path, io::Durability::Fast \| Durable)`. | | `to_bytes()` / `from_bytes(data)` | In-memory `.tv` serialization — see [In-memory serialization](#in-memory-serialization). | | `pickle` / `copy.copy` / `copy.deepcopy` | Supported on both index types via `__reduce__`; see [In-memory serialization](#in-memory-serialization). Indexes are also weakly referenceable, so one can be cached in a `weakref.WeakValueDictionary`. | | `len(idx)` / `idx.dim` / `idx.bit_width` | Introspection. `idx.dim` returns `int` once committed, or `None` on a lazy index that hasn't seen its first add. | | `idx.calibration_state` | TQ+ calibration state: `"uncalibrated"` or `"calibrated"` — see [TQ+ calibration](#tq-calibration). | ### `swap_remove` semantics `swap_remove(i)` is named to match Rust's [`Vec::swap_remove`](https://doc.rust-lang.org/std/vec/struct.Vec.html#method.swap_remove): the last element moves into slot `i`, and the vector is truncated by one. It is **not** a shift — the slots after `i` do not move down by one. Order is not preserved; slot indices of vectors you didn't delete may now point at different vectors than before. Use [`IdMapIndex`](#idmapindex) if external references have to stay stable across deletes. Search masks are external references too — see [A mask is invalidated by any mutation](#a-mask-is-invalidated-by-any-mutation-not-just-a-length-changing-one). ### Low-level construction from raw parts (Rust) Rust embedders that hold an index payload already in memory — e.g. read out of a database page instead of a `.tv` file — can construct an index directly from its decoded fields with `TurboQuantIndex::from_parts`, skipping the file round-trip: ```rust let index = TurboQuantIndex::from_parts( dim_opt, // Option: Some(dim) committed, or None for lazy bit_width, // 2, 3, or 4 n_vectors, packed_codes, // Vec scales, // Vec tqplus_shift, // Vec (length dim, or empty = identity) tqplus_scale, // Vec (length dim, or empty = identity) )?; ``` It is the single validated entry point for raw-part construction: every structural invariant is checked once and any violation returns a named `FromPartsError` (bit_width range, dim a positive multiple of 8 and `≤ 16384`, `packed_codes` / `scales` / TQ+ lengths with overflow-checked size math, the lazy-state constraints, and the same value-level validation as the file loader — finite non-negative per-vector scales, finite TQ+ shifts, finite positive TQ+ scales) rather than panicking or reading out of bounds. An index accepted by `from_parts` therefore always survives its own `write` → `load` round-trip. The paired accessors `packed_codes()`, `scales()`, `tqplus_shift()`, `tqplus_scale()`, `bit_width()`, `dim_opt()` and `len()` return the fields it consumes, so an index round-trips through your own storage format. The per-coordinate `encode` / `pack` / `search` / `codebook` kernels are crate-internal — `from_parts` is the supported low-level API. (Rust only; the Python binding uses `write` / `load`.) --- ## `IdMapIndex` Stable-id wrapper around `TurboQuantIndex`: a hash-table-backed `u64 id ↔ slot` mapping, with O(1) `remove(id)`. Slot indices still move when a vector is removed, but ids do not. ```python import numpy as np from turbovec import IdMapIndex idx = IdMapIndex(dim=1536, bit_width=4) idx.add_with_ids(vectors, np.array([1001, 1002, 1003], dtype=np.uint64)) scores, ids = idx.search(queries, k=10) # ids are uint64 external ids idx.remove(1002) # O(1) by id assert 1003 in idx # __contains__ sugar idx.write("index.tvim") # .tvim format loaded = IdMapIndex.load("index.tvim") ``` As with [`TurboQuantIndex`](#turboquantindex), `dim` is optional and gets inferred from the first `add_with_ids` call: ```python idx = IdMapIndex(bit_width=4) # dim inferred on first add idx.add_with_ids(vectors, ids) # locks dim to vectors.shape[1] ``` ### Methods | Method | Notes | |---|---| | `IdMapIndex(dim=None, bit_width=4)` | `bit_width ∈ {2, 3, 4}`; `dim` must be a positive multiple of 8 and `≤ 16384`. `dim` is optional; when omitted it is inferred from the first `add_with_ids` call. | | `add_with_ids(vectors, ids)` | `ids` is a `uint64` array with length `vectors.shape[0]`. On a lazy index the first call locks `dim`. Raises `ValueError` on dim mismatch, duplicate ids, `len(ids) != vectors.shape[0]`, a zero-width batch, or a non-finite / `\|value\| ≥ 1e16` coordinate. On the Rust API, a lazy index's first add must use `add_with_ids_2d(vectors, dim, ids)` — the flat `add_with_ids` requires an already-committed dim and panics otherwise. (Rust only; Python arrays carry their shape.) | | `remove(id) -> bool` | `True` if the id was present and removed, `False` otherwise. O(1). | | `search(queries, k, *, allowlist=None)` | Returns `(scores, ids)` — `ids` are `uint64` external ids. `allowlist` is an optional `uint64` array of ids; when given, results are restricted to those ids and `effective_k = min(k, number of unique ids in allowlist)` (the allowlist is deduplicated; repeated ids don't widen the result). Raises `ValueError` on an empty allowlist or a non-finite / `\|value\| ≥ 1e16` query coordinate, and `KeyError` on unknown ids. On the Rust API `search_with_allowlist` returns `Result<(Vec, Vec), SearchError>` and reports every one of those conditions as `Err`: `AllowlistEmpty`, `UnknownId(id)`, and the query-shape pair `QueryBufferNotMultipleOfDim` / `InvalidQueryValue`. The allowlist-free `search` returns the tuple directly and is the panicking form — it re-panics with the same message on the two query-shape conditions. Rust callers who want the row count and the *effective* `k` rather than a bare tuple use `try_search` / `try_search_with_allowlist`, which return `Result` — the id-space counterpart of `SearchResults`, with `scores`, `ids`, `nq`, `k` and `scores_for_query` / `ids_for_query` row accessors. | | `contains(id)` / `id in idx` | Membership. | | `sync(path)` | Incremental save: writes only what changed since the last sync to the same path. See [Incremental saves](#incremental-saves--sync). `load(path)` reads both formats. | | `write(path, *, durable=True)` / `load(path)` | `.tvim` format. `durable=False` skips the fsync before the atomic rename — faster, but a power loss can lose the file. A `durable=True` save whose post-rename directory fsync fails still succeeds (the file is committed and visible) and raises a `RuntimeWarning` saying the rename may not survive power loss. On the Rust API this is not a flag: use `write_with_durability(path, io::Durability::Fast \| Durable)`. | | `to_bytes()` / `from_bytes(data)` | In-memory `.tvim` serialization — see [In-memory serialization](#in-memory-serialization). | | `pickle` / `copy.copy` / `copy.deepcopy` | Same as `TurboQuantIndex`. | | `len(idx)` / `idx.dim` / `idx.bit_width` / `idx.calibration_state` | Same as `TurboQuantIndex`. | | `prepare()` | As `TurboQuantIndex.prepare()`, and additionally warms the lazy `id -> slot` map, so the first `search(..., allowlist=)`, `contains()` or `remove()` after a load doesn't pay the one-time O(n) build either. | ### When to use which - `TurboQuantIndex` — you never delete, or you're fine with positional ids. - `IdMapIndex` — you need stable external ids (e.g. string-id → vector mapping maintained by the caller). All the framework integrations (LangChain, LlamaIndex, Haystack) use `IdMapIndex` internally for exactly this reason. --- ## TQ+ calibration TQ+ fits a per-coordinate `(shift, scale)` pair from the empirical quantiles of a sample of your vectors, and every stored vector is encoded in that one calibrated coordinate system. It is worth roughly +2.5 points of R@10 on average, and up to ~8.7 on the most anisotropic data measured. The calibration comes from exactly one place: an explicit `idx.calibrate(sample)` call (`calibrate` / `calibrate_2d` in Rust). The index never fits one on its own — an index that is never calibrated is plain TurboQuant, with no fitted state anywhere, and its encoded bytes are independent of how adds were batched or ordered. `idx.calibration_state` reports which of the two states an index is in: | state | meaning | |---|---| | `"uncalibrated"` | No calibration committed. Fully functional, just without the TQ+ recall gain. | | `"calibrated"` | A calibration is committed, and every stored row is encoded under it — including rows added *before* the `calibrate` call, which that call re-encoded. | **The sample is your responsibility.** `calibrate` uses every row you give it. Around 1024 rows gets within half a point of R@10 of a fit on the entire corpus on most corpora measured, and 2048 does so everywhere — but it must be a *representative, random* sample of the vectors the index will hold. A sorted or clustered prefix of the same size fits quantiles that are shifted and far too narrow, and actively destroys recall. Passing the whole corpus is always safe. `calibrate` may be called at any time and repeatedly. On a populated index it re-encodes every stored row from its stored codes — no original vectors needed. Know what that re-encode can and cannot do: - Refitting with the same or a nearby pair is free: the codes reach an exact fixed point. - Calibrating **after** a large uncalibrated ingest costs several points of recall versus calibrating first (the re-encode is a second quantization). Calibrate before adding when you can. - A **badly biased** earlier calibration cannot be repaired by refitting: its too-narrow fit clipped coordinates to the outer centroids at encode time, and no later pair recovers what clipping destroyed. Rebuild from the source vectors. The calibration round-trips exactly through `write`/`load`, `to_bytes`/`from_bytes`, pickling and copying, on both index types. Draining an index to zero vectors keeps its committed calibration. --- ## Filtering Both index types support restricting the returned top-`k` to a caller-supplied subset of vectors. Unlike post-filtering (search then drop), the kernel never inserts disallowed vectors into the per-query heap, so you always get up to `k` results from the allowed set rather than fewer. ```python # IdMapIndex — allowlist of external ids (typical use) allowed = np.array([1003, 1010, 1042], dtype=np.uint64) scores, ids = idx.search(queries, k=10, allowlist=allowed) # scores.shape == (nq, min(k, n_allowed)) == (nq, 3) # 3 unique allowed ids # TurboQuantIndex — bool mask over slots mask = np.ones(len(idx), dtype=bool) mask[disabled_slots] = False scores, slots = idx.search(queries, k=10, mask=mask) ``` The output shape is `(nq, min(k, n_allowed))`, where `n_allowed` is the number of *distinct* allowed vectors — unique ids in the allowlist, or `mask.sum()` for a mask — the same shrinking behaviour you already see when `k > len(idx)`. No `-1` / `NaN` padding; pad on the caller side if you need a fixed-width batch. ### A mask is invalidated by any mutation, not just a length-changing one A mask names slots, and [`swap_remove`](#swap_remove-semantics) renumbers slots — so **any** mutation invalidates a mask, including one that leaves `len(idx)` unchanged. Rebuild the mask after every mutation. The length check is not what protects you. It catches only a size difference (`ValueError: mask length 100 does not match index size 99`); a `swap_remove(i)` + `add(...)` pair restores the original length while leaving a *different* vector in slot `i`, so a mask built before that pair passes validation and then silently selects a different set of vectors than you intended. Nothing outside the index leaks and no error is raised — the selected set is simply the wrong one. Allowlists on `IdMapIndex` do not have this failure mode, because they name external ids and the index never renumbers an id. An allowlist entry for an id that has since been removed raises `KeyError` (Rust: `SearchError::UnknownId`) instead of quietly resolving to some other vector. The one way an allowlist entry can come to name a different vector is if you re-add that same id yourself with different data. Common use cases: - Hybrid retrieval where a SQL/BM25 stage produces a candidate id set. - Access control or multi-tenant queries (only return ids the caller can see). - Time-windowed search (e.g. only documents from the last 7 days). --- ## Two-stage 2-bit search A 2-bit index of 32,768 vectors or more searches in two stages. Nothing changes on disk: the file is the same whichever way it is searched. `TURBOVEC_2BIT_PLANES=0` in the environment before the process first searches keeps the whole-index exact scan instead; it is read once per process. A 2-bit code is a sign bit and a magnitude bit per coordinate. The in-memory search cache holds the two bits apart — the same bytes per vector, arranged differently — and a search 1. scans the sign bits alone (half the bytes of a full scan) for a shortlist of `max(384, 16 × k)` candidates — on a calibrated index the scan ranks against the query pulled halfway toward the coordinates' centre, which is what keeps the shortlist honest on corpora whose vectors share a strong common direction (see the MedCPT row below), and each candidate carries its plain score on, 2. ranks the shortlist with an estimate that adds the magnitude bits (a bit-count against the query rounded to 6 bits per coordinate), and 3. rescores the best `max(64, 2 × k)` with the exact scan's own arithmetic. **Scores are exact; the candidate set is approximate.** Every returned score is bit-identical to what the whole-index scan returns for that id. What can differ is *which* ids are returned: a vector whose sign bits alone rank it outside the shortlist is not seen. How often that happens depends on the data: | data | queries returning exactly the whole-index scan's ids | |---|---| | OpenAI `text-embedding-3` d=1536 and d=3072, N=200K, k = 1, 10, 100 | 99.99–100% of 10,000 | | `all-mpnet-base-v2` d=768, N=41K, k = 1, 10, 100 | 99.95–100% of 10,000 | | MedCPT (PubMed) d=768, N=100K, k = 1 / 10 / 100 | 99.9% / 99.0% / 88.7% of 3,000 — recall k@k 0.737 / 0.778 / 0.797, the whole-index scan's 0.737 / 0.779 / 0.798 | | isotropic random unit vectors, d=64–1536, N=50K, k=10 | 4–7% (75% of ids shared) | Recall against float ground truth on the OpenAI corpora is unchanged at every k the benchmark suite reports. On random vectors — where a query has no real neighbours and the top-k is decided by noise-sized margins — the true nearest neighbour is in the top 10 for 74% of queries at d=768, against 86% with the whole-index scan. Check agreement on your own data if it is unlike the corpora above, and set `TURBOVEC_2BIT_PLANES=0` if the whole-index scan's id set is what you need. **When it is faster.** The first stage is cheaper than a full scan, and the later stages cost more as `k` grows, so the gain is largest for small `k`. Milliseconds per query on 100K OpenAI d=1536 vectors, 1,000 queries, whole-index scan → two-stage: | | k=10 | k=64 | k=100 | |---|---|---|---| | ARM (c4a), 1 thread, batch | 1.43 → 0.76 | 1.51 → 0.95 | 1.55 → 1.07 | | ARM (c4a), 1 thread, one query per call | 1.74 → 1.00 | 1.89 → 1.18 | 2.03 → 1.27 | | ARM (c4a), 8 threads, batch | 0.167 → 0.100 | 0.190 → 0.130 | 0.195 → 0.143 | | ARM (c4a), 8 threads, one query per call | 0.259 → 0.205 | 0.305 → 0.312 | 0.351 → 0.366 | | x86 (c3), 1 thread, batch | 0.55 → 0.39 | 0.62 → 0.57 | 0.68 → 0.68 | | x86 (c3), 1 thread, one query per call | 1.29 → 0.77 | 1.35 → 0.93 | 1.42 → 1.04 | | x86 (c3), 8 threads, batch | 0.137 → 0.096 | 0.161 → 0.137 | 0.183 → 0.174 | | x86 (c3), 8 threads, one query per call | 0.397 → 0.286 | 0.450 → 0.439 | 0.538 → 0.507 | Through `k=100` every cell is faster or within 4% of the whole-index scan; the cells at parity are multi-threaded single queries and x86 batches at the largest `k`. **What scans the whole index instead.** Indexes below 32,768 vectors (an index that grows past the threshold switches then, and keeps the layout if it later shrinks); dimensions that are not a multiple of 32; x86 CPUs without AVX-512 VBMI and VNNI. Filtered searches use the two-stage path with a plain top-shortlist heap. --- ## Staged 4-bit search A 4-bit index of 32,768 vectors or more searches in stages. Nothing changes on disk: the file is the same whichever way it is searched. `TURBOVEC_4BIT_PLANES=0` in the environment before the process first searches keeps the whole-index exact scan instead; it is read once per process. A 4-bit code is a sign bit and three lower bits per coordinate. The in-memory search cache holds the four bit planes apart — the same bytes per vector, arranged differently — and a search 1. scans the sign bits alone (a quarter of the bytes of a full scan) for a shortlist of `max(256, 20 × k)` candidates (`16 × k` from `k = 64`) — on a calibrated index the scan ranks against the query pulled halfway toward the coordinates' centre, which is what keeps the shortlist honest on corpora whose vectors share a strong common direction (see the MedCPT row below), and each candidate carries its plain score on, 2. ranks the shortlist with an estimate that adds the next bit plane and keeps the best `max(256, 6 × k)`, 3. ranks those with all three lower planes, and 4. rescores the best `max(64, 1.5 × k)` with the exact scan's own arithmetic. **Scores are exact; the candidate set is approximate.** Every returned score is bit-identical to what the whole-index scan returns for that id. What can differ is *which* ids are returned: a vector whose sign bits alone rank it outside the shortlist is not seen. On the corpora below the first stage keeps a wide margin: | data | queries returning exactly the whole-index scan's ids | |---|---| | OpenAI `text-embedding-3` d=1536 and d=3072, N=200K, k = 1, 10, 100 | 99.98–100% of 10,000 | | `all-mpnet-base-v2` d=768, N=41K, k = 1, 10, 100 | 99.92–100% of 10,000 | | MedCPT (PubMed) d=768, N=100K, k = 1 / 10 / 100 | 100% / 99.1% / 84.4% of 3,000 — recall k@k 0.936 / 0.943 / 0.956, the whole-index scan's 0.936 / 0.944 / 0.957 | The MedCPT corpus (#562) is the bunched case: mean pairwise cosine 0.65, a shared direction of norm 0.80, and a median gap between a query's first and second neighbour four times smaller than OpenAI's. Its ids differ from the whole-index scan's more often, but almost every difference is a swap at the list's edge: recall is within 0.1% of the scan's at every k and at both bit widths, and the true nearest neighbour is in the 4-bit top 10 for every query measured. Recall against float ground truth on the OpenAI corpora is unchanged at every k the benchmark suite reports. Structureless random vectors are a different matter: with no real neighbours, a shortlist of sign bits misses the exact top-k for most queries. Check agreement on your own data if it is unlike the corpora above, and set `TURBOVEC_4BIT_PLANES=0` if the whole-index scan's id set is what you need. **How much faster.** Milliseconds per query on 100K OpenAI d=1536 vectors, whole-index scan → staged; batches are 1,000 queries in one call: | | k=10 | k=32 | k=64 | k=100 | |---|---|---|---|---| | ARM (c4a), 1 thread, batch | 0.985 → 0.741 | 1.012 → 0.800 | 1.052 → 0.872 | 1.105 → 0.956 | | ARM (c4a), 1 thread, one query per call | 3.58 → 0.95 | 3.62 → 1.01 | 3.70 → 1.09 | 3.86 → 1.19 | | ARM (c4a), 8 threads, batch | 0.111 → 0.094 | 0.126 → 0.101 | 0.135 → 0.112 | 0.133 → 0.123 | | ARM (c4a), 8 threads, one query per call | 0.506 → 0.179 | 0.514 → 0.214 | 0.553 → 0.235 | 0.606 → 0.270 | | x86 (c3), 1 thread, batch | 0.639 → 0.348 | 0.659 → 0.419 | 0.701 → 0.509 | 0.765 → 0.609 | | x86 (c3), 1 thread, one query per call | 3.34 → 0.73 | 3.35 → 0.82 | 3.45 → 0.92 | 3.47 → 0.96 | | x86 (c3), 8 threads, batch | 0.155 → 0.087 | 0.162 → 0.099 | 0.175 → 0.117 | 0.199 → 0.142 | | x86 (c3), 8 threads, one query per call | 0.999 → 0.303 | 1.010 → 0.327 | 1.066 → 0.401 | 1.137 → 0.470 | Every cell is faster: one query per call 2.25x–4.57x, batches 1.09x–1.84x (harmonic mean over the 32 cells 1.87x). A single query is where the full 4-bit scan is dearest, so that is where the staged search gains most; a batch already shares each block's bytes across queries, so its gain is the quarter-width first stage alone, and it shrinks as `k` grows. On 100K OpenAI d=3072 the same cells read 1.10x–5.65x (harmonic mean 1.83x on ARM, 2.27x on x86). **Small indexes.** The shortlist is sized by `k`, not by the index, so on an index of 40K vectors a `k=100` shortlist is 4% of it and the ranking stages cost a batch more than the shorter scan saves: measured on 40K vectors (OpenAI d=1536 cut down, and all-mpnet-base-v2 d=768 at 41K), batches at `k=100` run 0.80x–1.04x and at `k=64` 0.88x–1.15x of the whole-index scan, while every other cell is faster (one query per call 1.3x–3.7x; harmonic mean 1.3x–1.5x over the 16 cells). From about 64K vectors every cell is ahead. **What scans the whole index instead.** Indexes below 32,768 vectors (an index that grows past the threshold switches then, and keeps the layout if it later shrinks); dimensions that are not a multiple of 32; x86 CPUs without AVX-512 VBMI and VNNI. Filtered searches use the staged path with a plain top-shortlist heap. --- ## File formats ### `.tv` — `TurboQuantIndex` ``` ┌───────────────────────────────────────────┐ │ magic "TVPI" (4 bytes) │ │ version u8 = 6 │ ├───────────────────────────────────────────┤ │ core header │ │ bit_width (u8) │ │ dim (u32 LE) │ │ n_vectors (u64 LE) │ ├───────────────────────────────────────────┤ │ Lloyd-Max codebook │ │ boundaries ((2^bit_width − 1) × f32 LE)│ │ centroids (2^bit_width × f32 LE) │ ├───────────────────────────────────────────┤ │ codes — sequential blocked layout │ │ n_byte_groups = dim / (8 / bit_width) │ │ ceil(n_vectors / 32) │ │ × n_byte_groups × 32 bytes │ ├───────────────────────────────────────────┤ │ scales (n_vectors × f32 LE) │ │ per-vector length-renormalization │ ├───────────────────────────────────────────┤ │ TQ+ trailer │ │ n_calib (u32 LE) — 0 or dim │ │ shift (n_calib × f32 LE) │ │ scale (n_calib × f32 LE) │ └───────────────────────────────────────────┘ ``` The code payload is grouped into 32-vector blocks and padded up to a whole block, so it is `ceil(n_vectors / 32) * 32` vectors wide on disk. At `bit_width = 3` a byte holds only two codes rather than 8/3, making the payload ~33% larger than `dim * bit_width / 8` per vector would suggest. ### `.tvim` — `IdMapIndex` ``` ┌───────────────────────────────────────────┐ │ magic "TVIM" (4 bytes) │ │ version u8 = 6 │ ├───────────────────────────────────────────┤ │ core payload (same as .tv: header + │ │ codebook + codes + scales + TQ+) │ ├───────────────────────────────────────────┤ │ slot_to_id (n_vectors × u64 LE) │ └───────────────────────────────────────────┘ ``` On load, the reverse `id → slot` map is rebuilt in memory. Duplicate ids in the `slot_to_id` table are rejected as corrupt. ### The v8 container What `write()`, `to_bytes()` and `sync()` produce today is the v8 container (magic `TV8\0`), the v7 container with one more superblock byte: ``` ┌───────────────────────────────────────────┐ │ superblock │ │ magic "TV8\0" │ │ revision u8 = 1 │ │ bit_width u8 · kind u8 (0 .tv, 1 .tvim) │ │ layout u8 — 0 sequential, 1 planes │ │ dim u32 · nonce u64 · ops capacity u32 │ │ codebook, TQ+ calibration, CRC │ ├───────────────────────────────────────────┤ │ header slot A │ header slot B (alternate │ │ commits: generation, n, the partial │ │ tail block's rows, pending redo ops, │ │ delta digest, CRC) │ ├───────────────────────────────────────────┤ │ block units — one per whole 32-row block │ │ codes in `layout`, 32 scales, 32 ids │ │ (.tvim) │ └───────────────────────────────────────────┘ ``` A unit's codes are the block's sequential-blocked rows (`layout` 0, as every v7 unit) or its bit planes (`layout` 1): the sign plane first, one byte per group of eight coordinates for each of the 32 rows, group-major, then the 32 rows' lower planes, least significant first. The two forms are the same number of bytes, so every offset in the file is the same whichever a unit holds. An index of 32,768 vectors or more keeps its search cache as bit planes (see the two-stage and staged searches above), and writes `layout` 1; smaller indexes, 3-bit indexes and dimensions that are not a multiple of 32 write `layout` 0. Either loads anywhere: a planes file on a host that scans the whole index (x86 without AVX-512, or under `TURBOVEC_*BIT_PLANES=0`) converts on load, as a sequential file converts to planes on a host that uses them. The planes form is arch-neutral; x86 permutes the sign bytes in place at load. A `sync()` into a v7 file rewrites it as v8 once, as it would for a calibration change, and continues incrementally from there. ### In-memory serialization Both index types (de)serialize their wire format in memory, without a filesystem round-trip: ```python payload = idx.to_bytes() # bytes, byte-identical to write(path)'s file restored = IdMapIndex.from_bytes(payload) # same validation as load(path) on a write() file ``` `to_bytes()` returns exactly the bytes `write(path)` would put in the file (`.tv` for `TurboQuantIndex`, `.tvim` for `IdMapIndex`). `from_bytes(data)` accepts `bytes` or `bytearray` and applies exactly the same validation `load` applies to a `write()` file — version handling, structural and value-level checks, the embedded codebook check (a v6 file carries the Lloyd-Max codebook, and a file whose codebook is not a valid one for its `(bit_width, dim)` is rejected — the relevant case for anyone hand-writing files through the raw `io::*` writers), and the `.tvim` duplicate-id check — raising `ValueError` on a corrupt payload (there is no file to blame, so it is not an `OSError`). Both release the GIL. This is the path to use for caches and database columns, and it is what both index types' own `pickle` / `copy` support and the integration stores' are built on. `pickle.dumps(idx)`, `copy.copy(idx)` and `copy.deepcopy(idx)` work on both index types — they reduce to `from_bytes(to_bytes())`, so an index can cross a `multiprocessing` `spawn` boundary (the default start method on macOS and Windows) and a container holding one can be deep-copied. The copy is fully independent of the original. Everything true of `to_bytes` is therefore true of a pickle — in particular the calibration state round-trips exactly. Equality and hashing stay identity-based, so `idx == pickle.loads(pickle.dumps(idx))` is `False` even though the two hold the same vectors. Compare `to_bytes()` payloads to check that a saved and a loaded index agree. An index defines no `__bool__`, so truthiness falls back to `__len__`: an index holding no vectors is **falsy**, and `idx = idx or build_index()` therefore discards a perfectly good empty index. Test for an index with `idx is None`, and for its contents with `len(idx)`. An index also takes no user attributes (`idx.tag = "x"` raises `AttributeError`) and is not subclassable (`class Sub(IdMapIndex): ...` raises `TypeError`). Both are deliberate. A per-instance `__dict__` is not traversed by the garbage collector, so a reference cycle running through an attribute would leak the whole index rather than being collected, and the attributes would be silently dropped by `pickle` / `copy`, which reduce through `from_bytes(to_bytes())` and carry the payload only. A subclass instance would likewise pickle and copy back to the base class, silently changing type. Attach per-index state by holding the index in an object of your own instead. Re-invoking `idx.__init__(dim=...)` on a built index does nothing at all — it neither resets nor re-shapes it — so build a new index instead of trying to reconfigure one in place. On the Rust API the same pair exists as `to_bytes()` / `from_bytes(&[u8])`, alongside generic-sink forms `write_to_writer` / `load_from_reader` on both types. All four move the same v7 image the file writers produce: `to_bytes()` is byte-identical to the file `write()` leaves on disk, and `write_to_writer` streams it unit by unit rather than building a second copy in memory. The raw module-level `io::write*` / `io::load*` entry points were removed when v7 became the only format — build an index and use its own methods instead. On the Rust API `TurboQuantIndex::serialized_len()` returns the exact byte count `to_bytes()` will return and `write(path)` will put in the file, computed from the index's geometry without serializing anything — for sizing a buffer, a database column or a quota check ahead of time. It is exact rather than an upper bound, and `to_bytes()` uses it to allocate its buffer once. ### Incremental saves — `sync()` `write(path)` rewrites the whole file. `sync(path)` writes only what changed since the last sync to the same path, in a second container format (`.tv` / `.tvim` magic `TV7\0`) built for repeated small commits: ```python idx.sync("index.tv") # first sync to a fresh path: writes the whole container idx.add(more_vectors) idx.swap_remove(3) idx.sync("index.tv") # writes the delta and commits reloaded = TurboQuantIndex.load("index.tv") # load() recognises both formats ``` A loaded index stays bound to the path it came from, so it keeps syncing forward incrementally rather than rewriting. **What it costs.** An append writes the new 32-row blocks plus a commit header. A removal writes no block at all — it rides the commit header as a redo op, and a later sync folds it into its block. The whole-file events are an explicit `calibrate()` (a refit re-encodes every stored code) and enough accumulated removals to exceed the header's op capacity; both compact the file by rewriting it whole, through the same temp-file-and-rename path `write()` uses. **Durability.** Every sync is durable when it returns — there is no fast mode, and the fsync is `sync_all`, not a data-only variant. A crash at any byte leaves the previous commit intact: the file carries two alternating commit headers, so a torn header fails its checksum and the other one is adopted, and each header names the blocks its own sync wrote along with their digest, so a commit that reached disk ahead of its data is detected rather than served. See [Versioning and limits](#versioning-and-limits) for what this does *not* cover — damage arriving after the write. **One writer per path.** Each full write stamps the file with a random nonce, so if another process replaces the file underneath a bound index, the next `sync` reports it rather than writing over their commits. Two processes syncing one path concurrently is not supported; the check makes the unsupported case loud, not safe. **`sync()` files are path-only.** `from_bytes` and `load_from_reader` read the `write()` format. A v7 container needs random access — two header slots, fixed-stride block units, redo ops — which a byte stream cannot serve, and `to_bytes()` only ever emits the `write()` format. Handing v7 bytes to `from_bytes` raises an error saying so and pointing at `load(path)`. ### Load performance The file stores the codes in the layout the search kernels consume — the arch-neutral sequential blocked rows, or, for an index of 32,768 vectors or more, its bit planes — plus the Lloyd-Max codebook, so a load seeds the search caches directly: there is no O(n·dim) repack and no codebook solve on first search. Non-x86 uses the stored layout as-is; x86 applies one cheap in-block permutation at load (a threaded pass, a few milliseconds for a 77 MB index). A v7 file, which holds the sequential rows only, converts to the planes on load (block-parallel; ~20 ms for 100K × 1536 at 4 bits with the pool, ~80 ms on one thread) and is rewritten as v8 by its next `sync()`. The rotation is deterministic and rebuilt from `dim` in well under a millisecond. A stored index survives cross-platform load → re-save byte-identically; the format itself adds no platform dependence. ### Versioning and limits Both `.tv` and `.tvim` loads validate the header **before allocating**: `bit_width` must be 2/3/4, `dim` a positive multiple of 8 and `≤ 16384` (`MAX_DIM` — the same cap enforced at construction, so any index this build can create it can also load back), and every payload size is computed with checked arithmetic and read through a length-capped reader. A malformed or untrusted file therefore raises a clean error rather than panicking, dividing by zero, or driving an oversized allocation. Codebook, scale, and calibration values are additionally validated at the value level (finite, in-support), so a structurally valid file carrying out-of-range values in those fields is rejected rather than loaded. What that validation does **not** give you is integrity checking of the payload. Neither format checksums its stored codes, and the value-level checks only reject values that leave the valid range — a flipped mantissa bit in a scale is still a finite positive float, and a flipped code byte is indistinguishable from a legitimate one. Damage arriving from outside the writer (a failing disk, a truncated copy, a bad transfer) therefore loads clean and changes search results silently. Measured by flipping every one of the 32,912 bits of a 4114-byte `.tv` file in turn and loading each result: **1460 flips (4%) are rejected and 31,452 (96%) load and return a different index.** By section, rejections are 144 of 144 header bits, 988 of 992 codebook bits, 169 of 3072 scale bits, 159 of 4128 calibration-trailer bits, and **0 of 24,576 code bits**. The scale and trailer figures are the value-level checks doing exactly what they claim and no more: a flip that drives a float non-finite, negative, or out of support is caught, and every mantissa flip is not. The codes carry no validation of any kind, which is why that column is zero and why it is also the largest section. This is a deliberate scope choice, not an oversight. A save is atomic and a crash mid-write leaves the previous file intact, so the writer cannot leave a torn index behind; what is out of scope is damage that arrives afterwards. If you need to detect that, checksum the file yourself or store it on a filesystem that does. `n_calib = 0` in the TQ+ trailer means an uncalibrated index; otherwise it equals `dim`. v7 and v8 are read: a v5 or v6 file is refused with an error naming its version, and `turbovec::convert` moves a file between v5, v6, v7 and v8 in any direction (`cargo run --example convert -- v8`); converting a v8 planes file down re-containers the same codes, never re-quantizes. Versions 1 through 4 predate the v5 rotation change and cannot be decoded at all — their codes were encoded under a rotation this build cannot reproduce — so they must be rebuilt from the source vectors. `dim = 0` in the core header signals a lazy uncommitted index. It is only valid alongside `n_vectors = 0`; on load it produces an index whose `dim` is `None` until the first `add` / `add_with_ids` call. Both formats carry a magic + version byte and are stable across minor versions. Breaking changes bump the version byte. `write()`, `to_bytes()` and `sync()` all produce v8: `write()` and `to_bytes()` an *unclaimed* snapshot, `sync()` a container it claims and then updates incrementally (see [Incremental saves](#incremental-saves--sync)). v8 files are not readable by earlier turbovec releases, whose loaders reject the magic rather than misparse it; `turbovec::convert` writes a v7, v6 or v5 file for one.