zvec Documentation

File: README.md Copy Chapter

English | 中文

Zvec is an open-source, in-process vector database — lightweight, lightning-fast, and designed to embed directly into applications. Battle-tested within Alibaba Group, it delivers production-grade, low-latency and scalable similarity search with minimal setup.

[!Important]
🚀 v0.6.0 (July 20, 2026)

• Group-By Search: Retrieve top-K results per group instead of globally (group-by deduplication) across Flat, HNSW, HNSW-RaBitQ, and sparse indexes.
• Random Rotation Quantization: Optional random rotation for INT8/INT4 quantization distributes variance evenly across dimensions, significantly boosting recall.
• Enhanced Full-Text Search: Upgraded FTS pipeline with a Unicode UAX #29 standard tokenizer, UTF-8 / ASCII folding, and a Snowball-based stemmer supporting 34+ languages.
• Faster & More Robust: Block-max skip speeds up FTS conjunction queries by 22–38%, plus a new DiskANN C API and numerous stability fixes.

👉 Read the Release Notes | View Roadmap 📍

💫 Features Copy Chapter

• Blazing Fast: Searches billions of vectors in milliseconds.
• Simple, Just Works: Install and start searching in seconds. Pure local, no servers, no config, no fuss.
• Dense + Sparse Vectors: Support dense and sparse embeddings, multi-vector queries, and a rich selection of vector index types that scale from memory to disk.
• Full-Text Search (FTS): Native keyword-based full-text search — query string fields with natural-language or structured expressions.
• Hybrid Search: Fuse vector similarity, full-text search, and structured filters in a single query for precise results.
• Durable Storage: Write-ahead logging (WAL) guarantees persistence — data is never lost, even on process crash or power failure.
• Concurrent Access: Multiple processes can read the same collection simultaneously; writes are single-process exclusive.
• Runs Anywhere: As an in-process library, Zvec runs wherever your code runs — notebooks, servers, CLI tools, or even edge devices.

📦 Installation Copy Chapter

Zvec offers official SDKs across multiple languages:

• Python: pip install zvec (requires 64-bit Python 3.10–3.14)
• Node.js: npm install @zvec/zvec
• Go: High-performance Go bindings.
• Rust: cargo add zvec-rust
• Dart/Flutter: flutter pub add zvec

Prefer a visual tool? Try Zvec Studio to browse data and debug queries — no code required.

✅ Supported Platforms Copy Section

• Linux (x86_64, ARM64)
• macOS (ARM64)
• Windows (x86_64)

🛠️ Building from Source Copy Section

If you prefer to build Zvec from source, please check the Building from Source guide.

⚡ One-Minute Example Copy Chapter

import zvec

# Define collection schema
schema = zvec.CollectionSchema(
    name="example",
    vectors=zvec.VectorSchema("embedding", zvec.DataType.VECTOR_FP32, 4),
)

# Create collection
collection = zvec.create_and_open(path="./zvec_example", schema=schema)

# Insert documents
collection.insert([
    zvec.Doc(id="doc_1", vectors={"embedding": [0.1, 0.2, 0.3, 0.4]}),
    zvec.Doc(id="doc_2", vectors={"embedding": [0.2, 0.3, 0.4, 0.1]}),
])

# Search by vector similarity
results = collection.query(
    zvec.Query(field_name="embedding", vector=[0.4, 0.3, 0.3, 0.1]),
    topk=10
)

# Results: list of {'id': str, 'score': float, ...}, sorted by relevance
print(results)

📈 Performance at Scale Copy Chapter

Zvec delivers exceptional speed and efficiency, making it ideal for demanding production workloads.

For detailed benchmark methodology, configurations, and complete results, please see our Benchmarks documentation.

🤝 Join Our Community Copy Chapter

❤️ Contributing Copy Chapter

We welcome and appreciate contributions from the community! Whether you're fixing a bug, adding a feature, or improving documentation, your help makes Zvec better for everyone.

Check out our Contributing Guide to get started!

File: tools/core/README.md Copy Chapter

Benchmarking scripts

This directory contains benchmarking scripts and reproducing steps.

COHERE experiments Copy Chapter

Getting COHERE Data Copy Section

Please download the COHERE 10M dataset to cohere_large_10m as follows:

... ...           
neighbors.parquet    
shuffle_train-00-of-10.parquet     
shuffle_train-01-of-10.parquet          
shuffle_train-02-of-10.parquet  
shuffle_train-03-of-10.parquet 
shuffle_train-04-of-10.parquet  
shuffle_train-05-of-10.parquet 
shuffle_train-06-of-10.parquet
shuffle_train-07-of-10.parquet
shuffle_train-08-of-10.parquet
shuffle_train-09-of-10.parquet
scalar_labels.parquet     
test.parquet

For convenience, we prepared a docker image with cohere bench datasets: registry.cn-hongkong.cr.aliyuncs.com/zvec/cohere-bench-data.

You can run a container as follows:

docker run -it --net=host -d -e DEBUG_MODE=true  --user root --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v /home/zvec/:/home/zvec/  -w /home/zvec --name=cohere_bench zvec-registry.cn-hongkong.cr.aliyuncs.com/zvec/cohere-bench-data:0.0.1 bash

docker exec -it cohere_bench bash

The datasets locate at /tmp/cohere/

Preparing Environment Copy Section

Clone code and init:

$ git clone [email protected]:alibaba/zvec.git
$ cd zvec
$ git submodule update --init

Build source code:

$ cd /home/zvec/workspace/zvec
$ mkdir build
$ cd build  
$ cmake -DENABLE_SKYLAKE=ON -DCMAKE_BUILD_TYPE=Release ..

Converting Dataset Copy Section

Export vector data using python script:

$ mkdir 10m.output
$ python3 convert_cohere_parquet.py

Convert vector data to binary formatted file.

/home/zvec/workspace/zvec/bin/txt2vecs -input=cohere_train_vector_10m.txt --output=cohere_train_vector_10m.zvec.vecs --dimension=768

We've also prepared preprocessed binary formatted files, which can be found in the container below:

root@iZj6caifjouj5yu8xgsiysZ:/home/zvec# ls -al /tmp/cohere/*zvec 
/tmp/cohere/cohere_large_10m_zvec:
total 30204572
drwxr-xr-x 2 root root        4096 Feb  5 13:12 .
drwxr-xr-x 6 root root        4096 Feb  6 03:38 ..
-rw-r--r-- 1 root root     8664837 Feb  5 13:06 cohere_test_vector_10m.1000.new.txt
-rw-r--r-- 1 root root 30920004295 Feb  5 13:04 cohere_train_vector_10m.new.zvec.vecs
-rw-r--r-- 1 root root      792835 Feb  5 13:05 neighbors.txt

/tmp/cohere/cohere_medium_1m_zvec:
total 3028688
drwxr-xr-x 2 root root       4096 Feb  5 13:14 .
drwxr-xr-x 6 root root       4096 Feb  6 03:38 ..
-rw-r--r-- 1 root root    8661108 Feb  5 13:07 cohere_test_vector_1m.1000.new.txt
-rw-r--r-- 1 root root 3092004295 Feb  5 13:08 cohere_train_vector_1m.new.zvec.vecs
-rw-r--r-- 1 root root     692969 Feb  5 13:08 neighbors.txt

Preparing Bench Config Copy Section

Prepare Build Config

BuilderCommon:
    BuilderClass: HnswStreamer
    BuildFile: /tmp/cohere/cohere_large_10m_zvec/cohere_train_vector_10m.zvec.vecs
    NeedTrain: true 
    TrainFile: /tmp/cohere/cohere_large_10m_zvec/cohere_train_vector_10m.zvec.vecs
    DumpPath:  /home/zvec/bench/config/cohere_train_vector_10m.dump.index
    IndexPath: /home/zvec/bench/config/cohere_train_vector_10m.index

    ConverterName: CosineInt8Converter
    MetricName: Cosine

    ThreadCount: 16

BuilderParams: 
    proxima.general.builder.thread_count: !!int 16
    proxima.hnsw.builder.thread_count: !!int 16

Prepare Search Config

SearcherCommon:
    SearcherClass: HnswStreamer
    IndexPath: /home/zvec/bench/config/cohere_train_vector_10m.index
    TopK: 1,10,50,100 
    QueryFile: /tmp/cohere/cohere_large_10m_zvec/cohere_test_vector_1000.new.txt
    QueryType: float 
    QueryFirstSep: ";" 
    QuerySecondSep: " "
    GroundTruthFile: /tmp/cohere/cohere_large_10m_zvec/neighbors.txt
    RecallThreadCount: 1
    BenchThreadCount: 16 
    BenchIterCount: 1000000000 
    CompareById: true

SearcherParams: 
    proxima.hnsw.streamer.ef: !!int 250

Building Index Copy Section

Conduct Build

$ /home/zvec/workspace/zvec/build/bin/local_build_original ./build.yaml

Performing Bench Copy Section

Conduct Recall

$ /home/zvec/workspace/zvec/build/bin/recall_original ./search.yaml

Conduct Bench

$ /home/zvec/workspace/zvec/build/bin/bench_original ./search.yaml