## File: README.md [](https://docs.rs/crate/tantivy/) [](https://github.com/quickwit-oss/tantivy/actions/workflows/test.yml) [](https://codecov.io/gh/quickwit-oss/tantivy) [](https://scorecard.dev/viewer/?uri=github.com/quickwit-oss/tantivy) [](https://discord.gg/MT27AG5EVE) [](https://opensource.org/licenses/MIT) [](https://crates.io/crates/tantivy) ## Fast full-text search engine library written in Rust **If you are looking for an alternative to Elasticsearch or Apache Solr, check out [Quickwit](https://github.com/quickwit-oss/quickwit), our distributed search engine built on top of Tantivy.** Tantivy is closer to [Apache Lucene](https://lucene.apache.org/) than to [Elasticsearch](https://www.elastic.co/products/elasticsearch) or [Apache Solr](https://lucene.apache.org/solr/) in the sense it is not an off-the-shelf search engine server, but rather a crate that can be used to build such a search engine. Tantivy is, in fact, strongly inspired by Lucene's design. ## Benchmark The following [benchmark](https://tantivy-search.github.io/bench/) breaks down the performance for different types of queries/collections. Your mileage WILL vary depending on the nature of queries and their load. Details about the benchmark can be found at this [repository](https://github.com/quickwit-oss/search-benchmark-game). ## Features - Full-text search - Configurable tokenizer (stemming available for 17 Latin languages) with third party support for Chinese ([tantivy-jieba](https://crates.io/crates/tantivy-jieba) and [cang-jie](https://crates.io/crates/cang-jie)), Japanese ([lindera](https://github.com/lindera-morphology/lindera-tantivy), [Vaporetto](https://crates.io/crates/vaporetto_tantivy), and [tantivy-tokenizer-tiny-segmenter](https://crates.io/crates/tantivy-tokenizer-tiny-segmenter)) and Korean ([lindera](https://github.com/lindera-morphology/lindera-tantivy) + [lindera-ko-dic-builder](https://github.com/lindera-morphology/lindera-ko-dic-builder)) - Fast (check out the :racehorse: :sparkles: [benchmark](https://tantivy-search.github.io/bench/) :sparkles: :racehorse:) - Tiny startup time (<10ms), perfect for command-line tools - BM25 scoring (the same as Lucene) - Natural query language (e.g. `(michael AND jackson) OR "king of pop"`) - Phrase queries search (e.g. `"michael jackson"`) - Incremental indexing - Multithreaded indexing (indexing English Wikipedia takes < 3 minutes on my desktop) - Mmap directory - SIMD integer compression when the platform/CPU includes the SSE2 instruction set - Single valued and multivalued u64, i64, and f64 fast fields (equivalent of doc values in Lucene) - `&[u8]` fast fields - Text, i64, u64, f64, dates, ip, bool, and hierarchical facet fields - Compressed document store (LZ4, Zstd, None) - Range queries - Faceted search - Configurable indexing (optional term frequency and position indexing) - JSON Field - Aggregation Collector: histogram, range buckets, average, and stats metrics - LogMergePolicy with deletes - Searcher Warmer API - Cheesy logo with a horse ### Non-features Distributed search is out of the scope of Tantivy, but if you are looking for this feature, check out [Quickwit](https://github.com/quickwit-oss/quickwit/). ## Getting started Tantivy works on stable Rust and supports Linux, macOS, and Windows. - [Tantivy's simple search example](https://tantivy-search.github.io/examples/basic_search.html) - [tantivy-cli and its tutorial](https://github.com/quickwit-oss/tantivy-cli) - `tantivy-cli` is an actual command-line interface that makes it easy for you to create a search engine, index documents, and search via the CLI or a small server with a REST API. It walks you through getting a Wikipedia search engine up and running in a few minutes. - [Reference doc for the last released version](https://docs.rs/tantivy/) ## How can I support this project? There are many ways to support this project. - Use Tantivy and tell us about your experience on [Discord](https://discord.gg/MT27AG5EVE) or by email (paul.masurel@gmail.com) - Report bugs - Write a blog post - Help with documentation by asking questions or submitting PRs - Contribute code (you can join [our Discord server](https://discord.gg/MT27AG5EVE)) - Talk about Tantivy around you ## Contributing code We use the GitHub Pull Request workflow: reference a GitHub ticket and/or include a comprehensive commit message when opening a PR. Feel free to update CHANGELOG.md with your contribution. ### Tokenizer When implementing a tokenizer for tantivy depend on the `tantivy-tokenizer-api` crate. ### Clone and build locally Tantivy compiles on stable Rust. To check out and run tests, you can simply run: ```bash git clone https://github.com/quickwit-oss/tantivy.git cd tantivy cargo test ``` ## Companies Using Tantivy             ## FAQ ### Can I use Tantivy in other languages? - Python → [tantivy-py](https://github.com/quickwit-oss/tantivy-py) - Ruby → [tantiny](https://github.com/baygeldin/tantiny) You can also find other bindings on [GitHub](https://github.com/search?q=tantivy) but they may be less maintained. ### What are some examples of Tantivy use? - [seshat](https://github.com/matrix-org/seshat/): A matrix message database/indexer - [tantiny](https://github.com/baygeldin/tantiny): Tiny full-text search for Ruby - [lnx](https://github.com/lnx-search/lnx): adaptable, typo tolerant search engine with a REST API - [Bichon](https://github.com/rustmailer/bichon): A lightweight, high-performance Rust email archiver with WebUI - and [more](https://github.com/search?q=tantivy)! ### On average, how much faster is Tantivy compared to Lucene? - According to our [search latency benchmark](https://tantivy-search.github.io/bench/), Tantivy is approximately 2x faster than Lucene. ### Does tantivy support incremental indexing? - Yes. ### How can I edit documents? - Data in tantivy is immutable. To edit a document, the document needs to be deleted and reindexed. ### When will my documents be searchable during indexing? - Documents will be searchable after a `commit` is called on an `IndexWriter`. Existing `IndexReader`s will also need to be reloaded in order to reflect the changes. Finally, changes are only visible to newly acquired `Searcher`. --- ## File: columnar/README.md # Columnar format This crate describes columnar format used in tantivy. ## Goals This format is special in the following way. - it needs to be compact - accessing a specific column does not require to load the entire columnar. It can be done in 2 to 3 random access. - columns of several types can be associated with the same column name. - it needs to support columns with different types `(str, u64, i64, f64)` and different cardinality `(required, optional, multivalued)`. - columns, once loaded, offer cheap random access. - it is designed to allow range queries. # Coercion rules Users can create a columnar by inserting rows to a `ColumnarWriter`, and serializing it into a `Write` object. Nothing prevents a user from recording values with different type to the same `column_name`. In that case, `tantivy-columnar`'s behavior is as follows: - JsonValues are grouped into 3 types (String, Number, bool). Values that corresponds to different groups are mapped to different columns. For instance, String values are treated independently from Number or boolean values. `tantivy-columnar` will simply emit several columns associated to a given column_name. - Only one column for a given json value type is emitted. If number values with different number types are recorded (e.g. u64, i64, f64), `tantivy-columnar` will pick the first type that can represents the set of appended value, with the following prioriy order (`i64`, `u64`, `f64`). `i64` is picked over `u64` as it is likely to yield less change of types. Most use cases strictly requiring `u64` show the restriction on 50% of the values (e.g. a 64-bit hash). On the other hand, a lot of use cases can show rare negative value. # Columnar format This columnar format may have more than one column (with different types) associated to the same `column_name` (see [Coercion rules](#coercion-rules) above). The `(column_name, column_type)` couple however uniquely identifies a column. That couple is serialized as a column `column_key`. The format of that key is: `[column_name][ZERO_BYTE][column_type_header: u8]` ``` COLUMNAR:= [COLUMNAR_DATA] [COLUMNAR_KEY_TO_DATA_INDEX] [COLUMNAR_FOOTER]; # Columns are sorted by their column key. COLUMNAR_DATA:= [COLUMN_DATA]+; COLUMNAR_FOOTER := [RANGE_SSTABLE_BYTES_LEN: 8 bytes little endian] ``` The columnar file starts by the actual column data, concatenated one after the other, sorted by column key. A sstable associates `(column name, column_cardinality, column_type) to range of bytes. Column name may not contain the zero byte `\0`. Listing all columns associated to `column_name` can therefore be done by listing all keys prefixed by `[column_name][ZERO_BYTE]` The associated range of bytes refer to a range of bytes This crate exposes a columnar format for tantivy. This format is described in README.md The crate introduces the following concepts. `Columnar` is an equivalent of a dataframe. It maps `column_key` to `Column`. A `Column` associates a `RowId` (u32) to any number of values. This is made possible by wrapping a `ColumnIndex` and a `ColumnValue` object. The `ColumnValue` represents a mapping that associates each `RowId` to exactly one single value. The `ColumnIndex` then maps each RowId to a set of `RowId` in the `ColumnValue`. For optimization, and compression purposes, the `ColumnIndex` has three possible representation, each for different cardinalities. - Full All RowId have exactly one value. The ColumnIndex is the trivial mapping. - Optional All RowIds can have at most one value. The ColumnIndex is the trivial mapping `ColumnRowId -> Option`. - Multivalued All RowIds can have any number of values. The column index is mapping values to a range. All these objects are implemented an unit tested independently in their own module: - columnar - column_index - column_values - column --- ## File: doc/src/avant-propos.md # Foreword, what is the scope of tantivy? > Tantivy is a **search** engine **library** for Rust. If you are familiar with Lucene, it's an excellent approximation to consider tantivy as Lucene for Rust. Tantivy is heavily inspired by Lucene's design and they both have the same scope and targeted use cases. If you are not familiar with Lucene, let's break down our little tagline. - **Search** here means full-text search : fundamentally, tantivy is here to help you identify efficiently what are the documents matching a given query in your corpus. But modern search UI are so much more : text processing, facetting, autocomplete, fuzzy search, good relevancy, collapsing, highlighting, spatial search. While some of these features are not available in tantivy yet, all of these are relevant feature requests. Tantivy's objective is to offer a solid toolbox to create the best search experience. But keep in mind this is just a toolbox. Which bring us to the second keyword... - **Library** means that you will have to write code. Tantivy is not an *all-in-one* server solution like Elasticsearch for instance. Sometimes a functionality will not be available in tantivy because it is too specific to your use case. By design, tantivy should make it possible to extend the available set of features using the existing rock-solid datastructures. Most frequently this will mean writing your own `Collector`, your own `Scorer` or your own `TokenFilter`... Some of your requirements may also be related to something closer to architecture or operations. For instance, you may want to build a large corpus on Hadoop, fine-tune the merge policy to keep your index sharded in a time-wise fashion, or you may want to convert and existing index from a different format. Tantivy exposes a lot of low level API to do all of these things. --- ## File: doc/src/basis.md # Anatomy of an index ## Straight from disk Tantivy accesses its data using an abstracting trait called `Directory`. In theory, one can come and override the data access logic. In practise, the trait somewhat assumes that your data can be mapped to memory, and tantivy seems deeply married to using `mmap` for its io [^1], and the only persisting directory shipped with tantivy is the `MmapDirectory`. While this design has some downsides, this greatly simplifies the source code of tantivy. Caching is also entirely delegated to the OS. Tantivy works entirely (or almost) by directly reading the datastructures as they are laid on disk. As a result, the act of opening an indexing does not involve loading different datastructures from the disk into random access memory : starting a process, opening an index, and performing your first query can typically be done in a matter of milliseconds. This is an interesting property for a command line search engine, or for some multi-tenant log search engine : spawning a new process for each new query can be a perfectly sensible solution in some use case. In later chapters, we will discuss tantivy's inverted index data layout. One key take away is that to achieve great performance, search indexes are extremely compact. Of course this is crucial to reduce IO, and ensure that as much of our index can sit in RAM. Also, whenever possible its data is accessed sequentially. Of course, this is an amazing property when tantivy needs to access the data from your spinning hard disk, but this is also critical for performance, if your data is read from and an `SSD` or even already in your pagecache. ## Segments, and the log method That kind of compact layout comes at one cost: it prevents our datastructures from being dynamic. In fact, the `Directory` trait does not even allow you to modify part of a file. To allow the addition / deletion of documents, and create the illusion that your index is dynamic (i.e.: adding and deleting documents), tantivy uses a common database trick sometimes referred to as the *log method*. Let's forget about deletes for a moment. As you add documents, these documents are processed and stored in a dedicated datastructure, in a `RAM` buffer. This datastructure is not ready for search, but it is useful to receive your data and rearrange it very rapidly. As you add documents, this buffer will reach its capacity and tantivy will transparently stop adding document to it and start converting this datastructure to its final read-only format on disk. Once written, an brand empty buffer is available to resume adding documents. The resulting chunk of index obtained after this serialization is called a `Segment`. > A segment is a self-contained atomic piece of index. It is identified with a UUID, and all of its files are identified using the naming scheme : `.*`. Which brings us to the nature of a tantivy `Index`. > A tantivy `Index` is a collection of `Segments`. Physically, this really just means and index is a bunch of segment files in a given `Directory`, linked together by a `meta.json` file. This transparency can become extremely handy to get tantivy to fit your use case: *Example 1* You could for instance use hadoop to build a very large search index in a timely manner, copy all of the resulting segment files in the same directory and edit the `meta.json` to get a functional index.[^2] *Example 2* You could also disable your merge policy and enforce daily segments. Removing data after one week can then be done very efficiently by just editing the `meta.json` and deleting the files associated with segment `D-7`. ## Merging As you index more and more data, your index will accumulate more and more segments. Having a lot of small segments is not really optimal. There is a bit of redundancy in having all these term dictionary. Also when searching, we will need to do term lookups as many times as we have segments. It can hurt search performance a bit. That's where merging or compacting comes into place. Tantivy will continuously consider merge opportunities and start merging segments in the background. ## Indexing throughput, number of indexing threads [^1]: This may eventually change. [^2]: Be careful however. By default these files will not be considered as *managed* by tantivy. This means they will never be garbage collected by tantivy, regardless of whether they become obsolete or not. --- ## File: doc/src/index_sorting.md - [Index Sorting](#index-sorting) - [Why Sorting](#why-sorting) - [Compression](#compression) - [Top-N Optimization](#top-n-optimization) - [Pruning](#pruning) - [Other](#other) - [Usage](#usage) # Index Sorting Tantivy allows you to sort the index according to a property. ## Why Sorting Presorting an index has several advantages: ### Compression When data is sorted it is easier to compress the data. E.g. the numbers sequence [5, 2, 3, 1, 4] would be sorted to [1, 2, 3, 4, 5]. If we apply delta encoding this list would be unsorted [5, -3, 1, -2, 3] vs. [1, 1, 1, 1, 1]. Compression ratio is mainly affected on the fast field of the sorted property, every thing else is likely unaffected. ### Top-N Optimization When data is presorted by a field and search queries request sorting by the same field, we can leverage the natural order of the documents. E.g. if the data is sorted by timestamp and want the top n newest docs containing a term, we can simply leveraging the order of the docids. Note: tantivy 0.16 does not do this optimization yet. ### Pruning Let's say we want all documents and want to apply the filter `>= 2010-08-11`. When the data is sorted, we could make a lookup in the fast field to find the docid range and use this as the filter. Note: tantivy 0.16 does not do this optimization yet. ### Other? In principle there are many algorithms possible that exploit the monotonically increasing nature. (aggregations maybe?) ## Usage The index sorting can be configured setting [`sort_by_field`](https://github.com/quickwit-oss/tantivy/blob/000d76b11a139a84b16b9b95060a1c93e8b9851c/src/core/index_meta.rs#L238) on `IndexSettings` and passing it to a `IndexBuilder`. As of tantivy 0.16 only fast fields are allowed to be used. ```rust let settings = IndexSettings { sort_by_field: Some(IndexSortByField { field: "intval".to_string(), order: Order::Desc, }), ..Default::default() }; let mut index_builder = Index::builder().schema(schema); index_builder = index_builder.settings(settings); let index = index_builder.create_in_ram().unwrap(); ``` ## Implementation details Sorting an index is applied in the serialization step. In general there are two serialization steps: [Finishing a single segment](https://github.com/quickwit-oss/tantivy/blob/000d76b11a139a84b16b9b95060a1c93e8b9851c/src/indexer/segment_writer.rs#L338) and [merging multiple segments](https://github.com/quickwit-oss/tantivy/blob/000d76b11a139a84b16b9b95060a1c93e8b9851c/src/indexer/merger.rs#L1073). In both cases we generate a docid mapping reflecting the sort. This mapping is used when serializing the different components (doc store, fastfields, posting list, normfield, facets).