{"owner":"quickwit-oss","repo":"tantivy","hasSkills":true,"totalSkillsCount":4,"totalTokensCount":6205,"categories":["anthropic-skill","project-spec"],"hasMcp":false,"mcpConfig":null,"found":[".claude/skills/simple-pr/SKILL.md",".claude/skills/rationalize-deps/SKILL.md",".claude/skills/update-changelog/SKILL.md","ARCHITECTURE.md"],"skills":{".claude/skills/simple-pr/SKILL.md":"---\nname: simple-pr\ndescription: Create a simple PR from staged changes with an auto-generated commit message\ndisable-model-invocation: true\n---\n\n# Simple PR\n\nFollow these steps to create a simple PR from staged changes:\n\n## Step 1: Check workspace state\n\nRun: `git status`\n\nVerify that all changes have been staged (no unstaged changes). If there are unstaged changes, abort and ask the user to stage their changes first with `git add`.\n\nAlso verify that we are on the `main` branch. If not, abort and ask the user to switch to main first.\n\n## Step 2: Ensure main is up to date\n\nRun: `git pull origin main`\n\nThis ensures we're working from the latest code.\n\n## Step 3: Review staged changes\n\nRun: `git diff --cached`\n\nReview the staged changes to understand what the PR will contain.\n\n## Step 4: Generate commit message\n\nBased on the staged changes, generate a concise commit message (1-2 sentences) that describes the \"why\" rather than the \"what\".\n\nDisplay the proposed commit message to the user and ask for confirmation before proceeding.\n\n## Step 5: Create a new branch\n\nGet the git username: `git config user.name | tr ' ' '-' | tr '[:upper:]' '[:lower:]'`\n\nCreate a short, descriptive branch name based on the changes (e.g., `fix-typo-in-readme`, `add-retry-logic`, `update-deps`).\n\nCreate and checkout the branch: `git checkout -b {username}/{short-descriptive-name}`\n\n## Step 6: Commit changes\n\nCommit with the message from step 3:\n```\ngit commit -m \"{commit-message}\"\n```\n\n## Step 7: Push and open a PR\n\nPush the branch and open a PR:\n```\ngit push -u origin {branch-name}\ngh pr create --title \"{commit-message-title}\" --body \"{longer-description-if-needed}\"\n```\n\nReport the PR URL to the user when complete.\n",".claude/skills/rationalize-deps/SKILL.md":"---\nname: rationalize-deps\ndescription: Analyze Cargo.toml dependencies and attempt to remove unused features to reduce compile times and binary size\n---\n\n# Rationalize Dependencies\n\nThis skill analyzes Cargo.toml dependencies to identify and remove unused features.\n\n## Overview\n\nMany crates enable features by default that may not be needed. This skill:\n1. Identifies dependencies with default features enabled\n2. Tests if `default-features = false` works\n3. Identifies which specific features are actually needed\n4. Verifies compilation after changes\n\n## Step 1: Identify the target\n\nAsk the user which crate(s) to analyze:\n- A specific crate name (e.g., \"tokio\", \"serde\")\n- A specific workspace member (e.g., \"quickwit-search\")\n- \"all\" to scan the entire workspace\n\n## Step 2: Analyze current dependencies\n\nFor the workspace Cargo.toml (`quickwit/Cargo.toml`), list dependencies that:\n- Do NOT have `default-features = false`\n- Have default features that might be unnecessary\n\nRun: `cargo tree -p <crate> -f \"{p} {f}\" --edges features` to see what features are actually used.\n\n## Step 3: For each candidate dependency\n\n### 3a: Check the crate's default features\n\nLook up the crate on crates.io or check its Cargo.toml to understand:\n- What features are enabled by default\n- What each feature provides\n\nUse: `cargo metadata --format-version=1 | jq '.packages[] | select(.name == \"<crate>\") | .features'`\n\n### 3b: Try disabling default features\n\nModify the dependency in `quickwit/Cargo.toml`:\n\nFrom:\n```toml\nsome-crate = { version = \"1.0\" }\n```\n\nTo:\n```toml\nsome-crate = { version = \"1.0\", default-features = false }\n```\n\n### 3c: Run cargo check\n\nRun: `cargo check --workspace` (or target specific packages for faster feedback)\n\nIf compilation fails:\n1. Read the error messages to identify which features are needed\n2. Add only the required features explicitly:\n   ```toml\n   some-crate = { version = \"1.0\", default-features = false, features = [\"needed-feature\"] }\n   ```\n3. Re-run cargo check\n\n### 3d: Binary search for minimal features\n\nIf there are many default features, use binary search:\n1. Start with no features\n2. If it fails, add half the default features\n3. Continue until you find the minimal set\n\n## Step 4: Document findings\n\nFor each dependency analyzed, report:\n- Original configuration\n- New configuration (if changed)\n- Features that were removed\n- Any features that are required\n\n## Step 5: Verify full build\n\nAfter all changes, run:\n```bash\ncargo check --workspace --all-targets\ncargo test --workspace --no-run\n```\n\n## Common Patterns\n\n### Serde\nOften only needs `derive`:\n```toml\nserde = { version = \"1.0\", default-features = false, features = [\"derive\", \"std\"] }\n```\n\n### Tokio\nIdentify which runtime features are actually used:\n```toml\ntokio = { version = \"1.0\", default-features = false, features = [\"rt-multi-thread\", \"macros\", \"sync\"] }\n```\n\n### Reqwest\nOften doesn't need all TLS backends:\n```toml\nreqwest = { version = \"0.11\", default-features = false, features = [\"rustls-tls\", \"json\"] }\n```\n\n## Rollback\n\nIf changes cause issues:\n```bash\ngit checkout quickwit/Cargo.toml\ncargo check --workspace\n```\n\n## Tips\n\n- Start with large crates that have many default features (tokio, reqwest, hyper)\n- Use `cargo bloat --crates` to identify large dependencies\n- Check `cargo tree -d` for duplicate dependencies that might indicate feature conflicts\n- Some features are needed only for tests - consider using `[dev-dependencies]` features\n",".claude/skills/update-changelog/SKILL.md":"---\nname: update-changelog\ndescription: Update CHANGELOG.md with merged PRs since the last changelog update, categorized by type\n---\n\n# Update Changelog\n\nThis skill updates CHANGELOG.md with merged PRs that aren't already listed.\n\n## Step 1: Determine the changelog scope\n\nRead `CHANGELOG.md` to identify the current unreleased version section at the top (e.g., `Tantivy 0.26 (Unreleased)`).\n\nCollect all PR numbers already mentioned in the unreleased section by extracting `#NNNN` references.\n\n## Step 2: Find merged PRs not yet in the changelog\n\nUse `gh` to list recently merged PRs from the upstream repo:\n\n```bash\ngh pr list --repo quickwit-oss/tantivy --state merged --limit 100 --json number,title,author,labels,mergedAt\n```\n\nFilter out any PRs whose number already appears in the unreleased section of the changelog.\n\n## Step 3: Consolidate related PRs\n\nBefore categorizing, group PRs that belong to the same logical change. This is critical for producing a clean changelog. Use PR descriptions, titles, cross-references, and the files touched to identify relationships.\n\n**Merge follow-up PRs into the original:**\n- If a PR is a bugfix, refinement, or follow-up to another PR in the same unreleased cycle, combine them into a single changelog entry with multiple `[#N](url)` links.\n- Also consolidate PRs that touch the same feature area even if not explicitly linked — e.g., a PR fixing an edge case in a new API should be folded into the entry for the PR that introduced that API.\n\n**Filter out bugfixes on unreleased features:**\n- If a bugfix PR fixes something introduced by another PR in the **same unreleased version**, it must NOT appear as a separate Bugfixes entry. Instead, silently fold it into the original feature/improvement entry. The changelog should describe the final shipped state, not the development history.\n- To detect this: check if the bugfix PR references or reverts changes from another PR in the same release cycle, or if it touches code that was newly added (not present in the previous release).\n\n## Step 4: Review the actual code diff\n\n**Do not rely on PR titles or descriptions alone.** For every candidate PR, run `gh pr diff <number> --repo quickwit-oss/tantivy` and read the actual changes. PR titles are often misleading — the diff is the source of truth.\n\n**What to look for in the diff:**\n- Does it change observable behavior, public API surface, or performance characteristics?\n- Is the change something a user of the library would notice or need to know about?\n- Could the change break existing code (API changes, removed features)?\n\n**Skip PRs where the diff reveals the change is not meaningful enough for the changelog** — e.g., cosmetic renames, trivial visibility tweaks, test-only changes, etc.\n\n## Step 5: Categorize each PR group\n\nFor each PR (or consolidated group) that survived the diff review, determine its category:\n\n- **Bugfixes** — fixes to behavior that existed in the **previous release**. NOT fixes to features introduced in this release cycle.\n- **Features/Improvements** — new features, API additions, new options, improvements that change user-facing behavior or add new capabilities.\n- **Performance** — optimizations, speed improvements, memory reductions. **If a PR adds new API whose primary purpose is enabling a performance optimization, categorize it as Performance, not Features.** The deciding question is: does a user benefit from this because of new functionality, or because things got faster/leaner? For example, a new trait method that exists solely to enable cheaper intersection ordering is Performance, not a Feature.\n\nIf a PR doesn't clearly fit any category (e.g., CI-only changes, internal refactors with no user-facing impact, dependency bumps with no behavior change), skip it — not everything belongs in the changelog.\n\nWhen unclear, use your best judgment or ask the user.\n\n## Step 6: Format entries\n\nEach entry must follow this exact format:\n\n```\n- Description [#NUMBER](https://github.com/quickwit-oss/tantivy/pull/NUMBER)(@author)\n```\n\nRules:\n- The description should be concise and describe the user-facing change (not the implementation). Describe the final shipped state, not the incremental development steps.\n- Use sub-categories with bold headers when multiple entries relate to the same area (e.g., `- **Aggregation**` with indented entries beneath). Follow the existing grouping style in the changelog.\n- Author is the GitHub username from the PR, prefixed with `@`. For consolidated entries, include all contributing authors.\n- For consolidated PRs, list all PR links in a single entry: `[#100](url) [#110](url)` (see existing entries for examples).\n\n## Step 7: Present changes to the user\n\nShow the user the proposed changelog entries grouped by category **before** editing the file. Ask for confirmation or adjustments.\n\n## Step 8: Update CHANGELOG.md\n\nInsert the new entries into the appropriate sections of the unreleased version block. If a section doesn't exist yet, create it following the order: Bugfixes, Features/Improvements, Performance.\n\nAppend new entries at the end of each section (before the next section header or version header).\n\n## Step 9: Verify\n\nRead back the updated unreleased section and display it to the user for final review.\n","ARCHITECTURE.md":"# Tantivy\n\n## What is tantivy?\n\nTantivy is a library that is meant to build search engines. Although it is by no means a port of Lucene, its architecture is strongly inspired by it. If you are familiar with Lucene, you may be struck by the overlapping vocabulary.\nThis is not fortuitous.\n\nTantivy's bread and butter is to address the problem of full-text search :\n\nGiven a large set of textual documents, and a text query, return the K-most relevant documents in a very efficient way. To execute these queries rapidly, the tantivy needs to build an index beforehand. The relevance score implemented in the tantivy is not configurable. Tantivy uses the same score as the default similarity used in Lucene / Elasticsearch, called [BM25](https://en.wikipedia.org/wiki/Okapi_BM25).\n\nBut tantivy's scope does not stop there. Numerous features are required to power rich-search applications. For instance, one may want to:\n\n- compute the count of documents matching a query in the different section of an e-commerce website,\n- display an average price per meter square for a real estate search engine,\n- take into account historical user data to rank documents in a specific way,\n- or even use tantivy to power an OLAP database.\n\nA more abstract description of the problem space tantivy is trying to address is the following.\n\nIngest a large set of documents, create an index that makes it possible to\nrapidly select all documents matching a given predicate (also known as a query) and\ncollect some information about them ([See collector](#collector-define-what-to-do-with-matched-documents)).\n\nRoughly speaking the design is following these guiding principles:\n\n- Search should be O(1) in memory.\n- Indexing should be O(1) in memory. (In practice it is just sublinear)\n- Search should be as fast as possible\n\nThis comes at the cost of the dynamicity of the index: while it is possible to add, and delete documents from our corpus, the tantivy is designed to handle these updates in large batches.\n\n## [core/](src/core): Index, segments, searchers\n\nCore contains all of the high-level code to make it possible to create an index, add documents, delete documents and commit.\n\nThis is both the most high-level part of tantivy, the least performance-sensitive one, the seemingly most mundane code... And paradoxically the most complicated part.\n\n### Index and Segments\n\nA tantivy index is a collection of smaller independent immutable segments.\nEach segment contains its own independent set of data structures.\n\nA segment is identified by a segment id that is in fact a UUID.\nThe file of a segment has the format\n\n ```segment-id . ext```\n\nThe extension signals which data structure (or [`SegmentComponent`](src/index/segment_component.rs)) is stored in the file.\n\nA small `meta.json` file is in charge of keeping track of the list of segments, as well as the schema.\n\nOn commit, one segment per indexing thread is written to disk, and the `meta.json` is then updated atomically.\n\nFor a better idea of how indexing works, you may read the [following blog post](https://fulmicoton.com/posts/behold-tantivy-part2/).\n\n### Deletes\n\nDeletes happen by deleting a \"term\". Tantivy does not offer any notion of primary id, so it is up to the user to use a field in their schema as if it was a primary id, and delete the associated term if they want to delete only one specific document.\n\nOn commit, tantivy will find all of the segments with documents matching this existing term and remove from [alive bitset file](src/fastfield/alive_bitset.rs) that represents the bitset of the alive document ids.\nLike all segment files, this file is immutable. Because it is possible to have more than one alive bitset file at a given instant, the alive bitset filename has the format ```segment_id . commit_opstamp . del```.\n\nAn opstamp is simply an incremental id that identifies any operation applied to the index. For instance, performing a commit or adding a document.\n\n### DocId\n\nWithin a segment, all documents are identified by a DocId that ranges within `[0, max_doc)`.\nwhere `max_doc` is the number of documents in the segment, (deleted or not). Having such a compact `DocId` space is key to the compression of our data structures.\n\nThe DocIds are simply allocated in the order documents are added to the index.\n\n### Merges\n\nIn separate threads, tantivy's index writer search for opportunities to merge segments.\nThe point of segment merge is to:\n\n- eventually get rid of tombstoned documents\n- reduce the otherwise ever-growing number of segments.\n\nIndeed, while having several segments instead of one does not hurt search too much, having hundreds can have a measurable impact on the search performance.\n\n### Searcher\n\nThe user of the library usually does not need to know about the existence of Segments.\nSearching is done through an object called a [`Searcher`](src/core/searcher.rs), that captures a\nsnapshot of the index at one point of time, by holding a list of [SegmentReader](src/index/segment_reader.rs).\n\nIn other words, regardless of commits, file garbage collection, or segment merge that might happen, as long as the user holds and reuse the same [Searcher](src/core/searcher.rs), search will happen on an immutable snapshot of the index.\n\n## [directory/](src/directory): Where should the data be stored?\n\nTantivy, like Lucene, abstracts the place where the data should be stored in a key-trait\ncalled [`Directory`](src/directory/directory.rs).\nContrary to Lucene however, \"files\" are quite different from some kind of `io::Read` object.\nCheck out [`src/directory/directory.rs`](src/directory/directory.rs) trait for more details.\n\nTantivy ships two main directory implementation: the `MmapDirectory` and the `RamDirectory`,\nbut users can extend tantivy with their own implementation.\n\n## [schema/](src/schema): What are documents?\n\nTantivy's document follows a very strict schema, decided before building any index.\n\nThe schema defines all of the fields that the indexes [`Document`](src/schema/document/mod.rs) may and should contain, their types (`text`, `i64`, `u64`, `Date`, ...) as well as how it should be indexed / represented in tantivy.\n\nDepending on the type of the field, you can decide to\n\n- put it in the docstore\n- store it as a fast field\n- index it\n\nPractically, tantivy will push values associated with this type to up to 3 respective\ndata structures.\n\n*Limitations*\n\nAs of today, tantivy's schema imposes a 1:1 relationship between a field that is being ingested and a field represented in the search index. In sophisticated search application, it is fairly common to want to index a field twice using different tokenizers, or to index the concatenation of several fields together into one field.\n\nThis is not something tantivy supports, and it is up to the user to duplicate field / concatenate fields before feeding them to tantivy.\n\n## General information about these data structures\n\nAll data structures in tantivy, have:\n\n- a writer\n- a serializer\n- a reader\n\nThe writer builds an in-memory representation of a batch of documents. This representation is not searchable. It is just meant as an intermediary mutable representation, to which we can sequentially add\nthe document of a batch. At the end of the batch (or if a memory limit is reached), this representation\nis then converted into an on-disk immutable representation, that is extremely compact.\nThis conversion is done by the serializer.\n\nFinally, the reader is in charge of offering an API to read on this on-disk read-only representation.\nIn tantivy, readers are designed to require very little anonymous memory. The data is read straight from an mmapped file, and loading an index is as fast as mmapping its files.\n\n## [store/](src/store): Here is my DocId, Gimme my document\n\nThe docstore is a row-oriented storage that, for each document, stores a subset of the fields\nthat are marked as stored in the schema. The docstore is compressed using a general-purpose algorithm\nlike LZ4.\n\n**Useful for**\n\nIn search engines, it is often used to display search results.\nOnce the top 10 documents have been identified, we fetch them from the store, and display them or their snippet on the search result page (aka SERP).\n\n**Not useful for**\n\nFetching a document from the store is typically a \"slow\" operation. It usually consists in\n\n- searching into a compact tree-like data structure to find the position of the right block.\n- decompressing a small block\n- returning the document from this block.\n\nIt is NOT meant to be called for every document matching a query.\n\nAs a rule of thumb, if you hit the docstore more than 100 times per search query, you are probably misusing tantivy.\n\n## [fastfield/](src/fastfield): Here is my DocId, Gimme my value\n\nFast fields are stored in a column-oriented storage that allows for random access.\nThe only compression applied is bitpacking. The column comes with two meta data.\nThe minimum value in the column and the number of bits per doc.\n\nFetching a value for a `DocId` is then as simple as computing\n\n```rust\nmin_value + fetch_bits(num_bits * doc_id..num_bits * (doc_id+1))\n```\n\nThis operation just requires one memory fetch.\nBecause, DocSets are scanned through in order (DocId are iterated in a sorted manner) which\nalso help locality.\n\nIn Lucene's jargon, fast fields are called DocValues.\n\n**Useful for**\n\nThey are typically integer values that are useful to either rank or compute aggregate over\nall of the documents matching a query (aka [DocSet](src/docset.rs)).\n\nFor instance, one could define a function to combine upvotes with tantivy's internal relevancy score.\nThis can be done by fetching a fast field during scoring.\nOne could also compute the mean price of the items matching a query in an e-commerce website.\nThis can be done by fetching a fast field in a collector.\nFinally one could decide to post-filter a docset to remove docset with a price within a specific range.\nIf the ratio of filtered out documents is not too low, an efficient way to do this is to fetch the price and apply the filter on the collector side.\n\nAside from integer values, it is also possible to store an actual byte payload.\nFor advanced search engine, it is possible to store all of the features required for learning-to-rank in a byte payload, access it during search, and apply the learning-to-rank model.\n\nFinally facets are a specific kind of fast field, and the associated source code is in [`fastfield/facet_reader.rs`](src/fastfield/facet_reader.rs).\n\n# The inverted search index\n\nThe inverted index is the core part of full-text search.\nWhen presented a new document with the text field \"Hello, happy tax payer!\", tantivy breaks it into a list of so-called tokens. In addition to just splitting these strings into tokens, it might also do different kinds of operations like dropping the punctuation, converting the character to lowercase, apply stemming, etc. Tantivy makes it possible to configure the operations to be applied in the schema (tokenizer/ is the place where these operations are implemented).\n\nFor instance, the default tokenizer of tantivy would break our text into: `[hello, happy, tax, payer]`.\nThe document will therefore be registered in the inverted index as containing the terms\n`[text:hello, text:happy, text:tax, text:payer]`.\n\nThe role of the inverted index is, when given a term, gives us in return a very fast iterator over the sorted doc ids that match the term.\n\nSuch an iterator is called a posting list. In addition to giving us `DocId`, they can also give us optionally the number of occurrence of the term for each document, also called term frequency or TF.\n\nThese iterators being sorted by DocId, one can create an iterator over the document containing `text:tax AND text:payer`, `(text:tax AND text:payer) OR (text:contribuable)` or any boolean expression.\n\nIn order to represent the function\n```Term ⟶ Posting```\n\nThe inverted index actually consists of two data structures chained together.\n\n- [Term](src/schema/term.rs) ⟶ [TermInfo](src/postings/term_info.rs) is addressed by the term dictionary.\n- [TermInfo](src/postings/term_info.rs) ⟶ [Posting](src/postings/postings.rs) is addressed by the posting lists.\n\nWhere [TermInfo](src/postings/term_info.rs) is an object containing some meta data about a term.\n\n## [termdict/](src/termdict): Here is a term, give me the [TermInfo](src/postings/term_info.rs)\n\nTantivy's term dictionary is mainly in charge of supplying the function\n\n[Term](src/schema/term.rs) ⟶ [TermInfo](src/postings/term_info.rs)\n\nIt is itself broken into two parts.\n\n- [Term](src/schema/term.rs) ⟶ [TermOrdinal](src/termdict/mod.rs) is addressed by a finite state transducer, implemented by the fst crate.\n- [TermOrdinal](src/termdict/mod.rs) ⟶ [TermInfo](src/postings/term_info.rs) is addressed by the term info store.\n\n## [postings/](src/postings): Iterate over documents... very fast\n\nA posting list makes it possible to store a sorted list of doc ids and for each doc store\na term frequency as well.\n\nThe posting lists are stored in a separate file. The [TermInfo](src/postings/term_info.rs) contains an offset into that file and a number of documents for the given posting list. Both are required and sufficient to read the posting list.\n\nThe posting list is organized in block of 128 documents.\nOne block of doc ids is followed by one block of term frequencies.\n\nThe doc ids are delta encoded and bitpacked.\nThe term frequencies are bitpacked.\n\nBecause the number of docs is rarely a multiple of 128, the last block may contain an arbitrary number of docs between 1 and 127 documents. We then use variable int encoding instead of bitpacking.\n\n## [positions/](src/positions): Where are my terms within the documents?\n\nPhrase queries make it possible to search for documents containing a specific sequence of terms.\nFor instance, when the phrase query \"the art of war\" does not match \"the war of art\".\nTo make it possible, it is possible to specify in the schema that a field should store positions in addition to being indexed.\n\nThe token positions of all of the terms are then stored in a separate file with the extension `.pos`.\nThe [TermInfo](src/postings/term_info.rs) gives an offset (expressed in position this time) in this file. As we iterate through the docset,\nwe advance the position reader by the number of term frequencies of the current document.\n\n## [fieldnorm/](src/fieldnorm): Here is my doc, how many tokens in this field?\n\nThe [BM25](https://en.wikipedia.org/wiki/Okapi_BM25) formula also requires to know the number of tokens stored in a specific field for a given document. We store this information on one byte per document in the fieldnorm.\nThe fieldnorm is therefore compressed. Values up to 40 are encoded unchanged.\n\n## [tokenizer/](src/tokenizer): How should we process text?\n\nText processing is key to a good search experience.\nSplits or normalize your text too much, and the search results will have a less precision and a higher recall.\nDo not normalize, or under split your text, you will end up with a higher precision and a lesser recall.\n\nText processing can be configured by selecting an off-the-shelf [`Tokenizer`](./src/tokenizer/tokenizer.rs) or implementing your own to first split the text into tokens, and then chain different [`TokenFilter`](src/tokenizer/tokenizer.rs)'s to it.\n\nTantivy's comes with few tokenizers, but external crates are offering advanced tokenizers, such as [Lindera](https://crates.io/crates/lindera) for Japanese.\n\n## [query/](src/query): Define and compose queries\n\nThe [Query](src/query/query.rs) trait defines what a query is.\nDue to the necessity for some queries to compute some statistics over the entire index, and because the\nindex is composed of several `SegmentReader`, the path from transforming a `Query` to an iterator over documents is slightly convoluted, but fundamentally, this is what a Query is.\n\nThe iterator over a document comes with some scoring function. The resulting trait is called a\n[Scorer](src/query/scorer.rs) and is specific to a segment.\n\nDifferent queries can be combined using the [BooleanQuery](src/query/boolean_query/).\nTantivy comes with different types of queries and can be extended by implementing\nthe `Query`, `Weight`, and `Scorer` traits.\n\n## [collector](src/collector): Define what to do with matched documents\n\nCollectors define how to aggregate the documents matching a query, in the broadest sense possible.\nThe search will push matched documents one by one, calling their\n`fn collect(doc: DocId, score: Score);` method.\n\nUsers may implement their own collectors by implementing the [Collector](src/collector/mod.rs) trait.\n\n## [query-grammar](query-grammar): Defines the grammar of the query parser\n\nWhile the [QueryParser](src/query/query_parser/query_parser.rs) struct is located in the `query/` directory, the actual parser combinator used to convert user queries into an AST is in an external crate called `query-grammar`. This part was externalized to lighten the work of the compiler.\n"},"files":{".claude/skills/simple-pr/SKILL.md":"---\nname: simple-pr\ndescription: Create a simple PR from staged changes with an auto-generated commit message\ndisable-model-invocation: true\n---\n\n# Simple PR\n\nFollow these steps to create a simple PR from staged changes:\n\n## Step 1: Check workspace state\n\nRun: `git status`\n\nVerify that all changes have been staged (no unstaged changes). If there are unstaged changes, abort and ask the user to stage their changes first with `git add`.\n\nAlso verify that we are on the `main` branch. If not, abort and ask the user to switch to main first.\n\n## Step 2: Ensure main is up to date\n\nRun: `git pull origin main`\n\nThis ensures we're working from the latest code.\n\n## Step 3: Review staged changes\n\nRun: `git diff --cached`\n\nReview the staged changes to understand what the PR will contain.\n\n## Step 4: Generate commit message\n\nBased on the staged changes, generate a concise commit message (1-2 sentences) that describes the \"why\" rather than the \"what\".\n\nDisplay the proposed commit message to the user and ask for confirmation before proceeding.\n\n## Step 5: Create a new branch\n\nGet the git username: `git config user.name | tr ' ' '-' | tr '[:upper:]' '[:lower:]'`\n\nCreate a short, descriptive branch name based on the changes (e.g., `fix-typo-in-readme`, `add-retry-logic`, `update-deps`).\n\nCreate and checkout the branch: `git checkout -b {username}/{short-descriptive-name}`\n\n## Step 6: Commit changes\n\nCommit with the message from step 3:\n```\ngit commit -m \"{commit-message}\"\n```\n\n## Step 7: Push and open a PR\n\nPush the branch and open a PR:\n```\ngit push -u origin {branch-name}\ngh pr create --title \"{commit-message-title}\" --body \"{longer-description-if-needed}\"\n```\n\nReport the PR URL to the user when complete.\n",".claude/skills/rationalize-deps/SKILL.md":"---\nname: rationalize-deps\ndescription: Analyze Cargo.toml dependencies and attempt to remove unused features to reduce compile times and binary size\n---\n\n# Rationalize Dependencies\n\nThis skill analyzes Cargo.toml dependencies to identify and remove unused features.\n\n## Overview\n\nMany crates enable features by default that may not be needed. This skill:\n1. Identifies dependencies with default features enabled\n2. Tests if `default-features = false` works\n3. Identifies which specific features are actually needed\n4. Verifies compilation after changes\n\n## Step 1: Identify the target\n\nAsk the user which crate(s) to analyze:\n- A specific crate name (e.g., \"tokio\", \"serde\")\n- A specific workspace member (e.g., \"quickwit-search\")\n- \"all\" to scan the entire workspace\n\n## Step 2: Analyze current dependencies\n\nFor the workspace Cargo.toml (`quickwit/Cargo.toml`), list dependencies that:\n- Do NOT have `default-features = false`\n- Have default features that might be unnecessary\n\nRun: `cargo tree -p <crate> -f \"{p} {f}\" --edges features` to see what features are actually used.\n\n## Step 3: For each candidate dependency\n\n### 3a: Check the crate's default features\n\nLook up the crate on crates.io or check its Cargo.toml to understand:\n- What features are enabled by default\n- What each feature provides\n\nUse: `cargo metadata --format-version=1 | jq '.packages[] | select(.name == \"<crate>\") | .features'`\n\n### 3b: Try disabling default features\n\nModify the dependency in `quickwit/Cargo.toml`:\n\nFrom:\n```toml\nsome-crate = { version = \"1.0\" }\n```\n\nTo:\n```toml\nsome-crate = { version = \"1.0\", default-features = false }\n```\n\n### 3c: Run cargo check\n\nRun: `cargo check --workspace` (or target specific packages for faster feedback)\n\nIf compilation fails:\n1. Read the error messages to identify which features are needed\n2. Add only the required features explicitly:\n   ```toml\n   some-crate = { version = \"1.0\", default-features = false, features = [\"needed-feature\"] }\n   ```\n3. Re-run cargo check\n\n### 3d: Binary search for minimal features\n\nIf there are many default features, use binary search:\n1. Start with no features\n2. If it fails, add half the default features\n3. Continue until you find the minimal set\n\n## Step 4: Document findings\n\nFor each dependency analyzed, report:\n- Original configuration\n- New configuration (if changed)\n- Features that were removed\n- Any features that are required\n\n## Step 5: Verify full build\n\nAfter all changes, run:\n```bash\ncargo check --workspace --all-targets\ncargo test --workspace --no-run\n```\n\n## Common Patterns\n\n### Serde\nOften only needs `derive`:\n```toml\nserde = { version = \"1.0\", default-features = false, features = [\"derive\", \"std\"] }\n```\n\n### Tokio\nIdentify which runtime features are actually used:\n```toml\ntokio = { version = \"1.0\", default-features = false, features = [\"rt-multi-thread\", \"macros\", \"sync\"] }\n```\n\n### Reqwest\nOften doesn't need all TLS backends:\n```toml\nreqwest = { version = \"0.11\", default-features = false, features = [\"rustls-tls\", \"json\"] }\n```\n\n## Rollback\n\nIf changes cause issues:\n```bash\ngit checkout quickwit/Cargo.toml\ncargo check --workspace\n```\n\n## Tips\n\n- Start with large crates that have many default features (tokio, reqwest, hyper)\n- Use `cargo bloat --crates` to identify large dependencies\n- Check `cargo tree -d` for duplicate dependencies that might indicate feature conflicts\n- Some features are needed only for tests - consider using `[dev-dependencies]` features\n",".claude/skills/update-changelog/SKILL.md":"---\nname: update-changelog\ndescription: Update CHANGELOG.md with merged PRs since the last changelog update, categorized by type\n---\n\n# Update Changelog\n\nThis skill updates CHANGELOG.md with merged PRs that aren't already listed.\n\n## Step 1: Determine the changelog scope\n\nRead `CHANGELOG.md` to identify the current unreleased version section at the top (e.g., `Tantivy 0.26 (Unreleased)`).\n\nCollect all PR numbers already mentioned in the unreleased section by extracting `#NNNN` references.\n\n## Step 2: Find merged PRs not yet in the changelog\n\nUse `gh` to list recently merged PRs from the upstream repo:\n\n```bash\ngh pr list --repo quickwit-oss/tantivy --state merged --limit 100 --json number,title,author,labels,mergedAt\n```\n\nFilter out any PRs whose number already appears in the unreleased section of the changelog.\n\n## Step 3: Consolidate related PRs\n\nBefore categorizing, group PRs that belong to the same logical change. This is critical for producing a clean changelog. Use PR descriptions, titles, cross-references, and the files touched to identify relationships.\n\n**Merge follow-up PRs into the original:**\n- If a PR is a bugfix, refinement, or follow-up to another PR in the same unreleased cycle, combine them into a single changelog entry with multiple `[#N](url)` links.\n- Also consolidate PRs that touch the same feature area even if not explicitly linked — e.g., a PR fixing an edge case in a new API should be folded into the entry for the PR that introduced that API.\n\n**Filter out bugfixes on unreleased features:**\n- If a bugfix PR fixes something introduced by another PR in the **same unreleased version**, it must NOT appear as a separate Bugfixes entry. Instead, silently fold it into the original feature/improvement entry. The changelog should describe the final shipped state, not the development history.\n- To detect this: check if the bugfix PR references or reverts changes from another PR in the same release cycle, or if it touches code that was newly added (not present in the previous release).\n\n## Step 4: Review the actual code diff\n\n**Do not rely on PR titles or descriptions alone.** For every candidate PR, run `gh pr diff <number> --repo quickwit-oss/tantivy` and read the actual changes. PR titles are often misleading — the diff is the source of truth.\n\n**What to look for in the diff:**\n- Does it change observable behavior, public API surface, or performance characteristics?\n- Is the change something a user of the library would notice or need to know about?\n- Could the change break existing code (API changes, removed features)?\n\n**Skip PRs where the diff reveals the change is not meaningful enough for the changelog** — e.g., cosmetic renames, trivial visibility tweaks, test-only changes, etc.\n\n## Step 5: Categorize each PR group\n\nFor each PR (or consolidated group) that survived the diff review, determine its category:\n\n- **Bugfixes** — fixes to behavior that existed in the **previous release**. NOT fixes to features introduced in this release cycle.\n- **Features/Improvements** — new features, API additions, new options, improvements that change user-facing behavior or add new capabilities.\n- **Performance** — optimizations, speed improvements, memory reductions. **If a PR adds new API whose primary purpose is enabling a performance optimization, categorize it as Performance, not Features.** The deciding question is: does a user benefit from this because of new functionality, or because things got faster/leaner? For example, a new trait method that exists solely to enable cheaper intersection ordering is Performance, not a Feature.\n\nIf a PR doesn't clearly fit any category (e.g., CI-only changes, internal refactors with no user-facing impact, dependency bumps with no behavior change), skip it — not everything belongs in the changelog.\n\nWhen unclear, use your best judgment or ask the user.\n\n## Step 6: Format entries\n\nEach entry must follow this exact format:\n\n```\n- Description [#NUMBER](https://github.com/quickwit-oss/tantivy/pull/NUMBER)(@author)\n```\n\nRules:\n- The description should be concise and describe the user-facing change (not the implementation). Describe the final shipped state, not the incremental development steps.\n- Use sub-categories with bold headers when multiple entries relate to the same area (e.g., `- **Aggregation**` with indented entries beneath). Follow the existing grouping style in the changelog.\n- Author is the GitHub username from the PR, prefixed with `@`. For consolidated entries, include all contributing authors.\n- For consolidated PRs, list all PR links in a single entry: `[#100](url) [#110](url)` (see existing entries for examples).\n\n## Step 7: Present changes to the user\n\nShow the user the proposed changelog entries grouped by category **before** editing the file. Ask for confirmation or adjustments.\n\n## Step 8: Update CHANGELOG.md\n\nInsert the new entries into the appropriate sections of the unreleased version block. If a section doesn't exist yet, create it following the order: Bugfixes, Features/Improvements, Performance.\n\nAppend new entries at the end of each section (before the next section header or version header).\n\n## Step 9: Verify\n\nRead back the updated unreleased section and display it to the user for final review.\n","ARCHITECTURE.md":"# Tantivy\n\n## What is tantivy?\n\nTantivy is a library that is meant to build search engines. Although it is by no means a port of Lucene, its architecture is strongly inspired by it. If you are familiar with Lucene, you may be struck by the overlapping vocabulary.\nThis is not fortuitous.\n\nTantivy's bread and butter is to address the problem of full-text search :\n\nGiven a large set of textual documents, and a text query, return the K-most relevant documents in a very efficient way. To execute these queries rapidly, the tantivy needs to build an index beforehand. The relevance score implemented in the tantivy is not configurable. Tantivy uses the same score as the default similarity used in Lucene / Elasticsearch, called [BM25](https://en.wikipedia.org/wiki/Okapi_BM25).\n\nBut tantivy's scope does not stop there. Numerous features are required to power rich-search applications. For instance, one may want to:\n\n- compute the count of documents matching a query in the different section of an e-commerce website,\n- display an average price per meter square for a real estate search engine,\n- take into account historical user data to rank documents in a specific way,\n- or even use tantivy to power an OLAP database.\n\nA more abstract description of the problem space tantivy is trying to address is the following.\n\nIngest a large set of documents, create an index that makes it possible to\nrapidly select all documents matching a given predicate (also known as a query) and\ncollect some information about them ([See collector](#collector-define-what-to-do-with-matched-documents)).\n\nRoughly speaking the design is following these guiding principles:\n\n- Search should be O(1) in memory.\n- Indexing should be O(1) in memory. (In practice it is just sublinear)\n- Search should be as fast as possible\n\nThis comes at the cost of the dynamicity of the index: while it is possible to add, and delete documents from our corpus, the tantivy is designed to handle these updates in large batches.\n\n## [core/](src/core): Index, segments, searchers\n\nCore contains all of the high-level code to make it possible to create an index, add documents, delete documents and commit.\n\nThis is both the most high-level part of tantivy, the least performance-sensitive one, the seemingly most mundane code... And paradoxically the most complicated part.\n\n### Index and Segments\n\nA tantivy index is a collection of smaller independent immutable segments.\nEach segment contains its own independent set of data structures.\n\nA segment is identified by a segment id that is in fact a UUID.\nThe file of a segment has the format\n\n ```segment-id . ext```\n\nThe extension signals which data structure (or [`SegmentComponent`](src/index/segment_component.rs)) is stored in the file.\n\nA small `meta.json` file is in charge of keeping track of the list of segments, as well as the schema.\n\nOn commit, one segment per indexing thread is written to disk, and the `meta.json` is then updated atomically.\n\nFor a better idea of how indexing works, you may read the [following blog post](https://fulmicoton.com/posts/behold-tantivy-part2/).\n\n### Deletes\n\nDeletes happen by deleting a \"term\". Tantivy does not offer any notion of primary id, so it is up to the user to use a field in their schema as if it was a primary id, and delete the associated term if they want to delete only one specific document.\n\nOn commit, tantivy will find all of the segments with documents matching this existing term and remove from [alive bitset file](src/fastfield/alive_bitset.rs) that represents the bitset of the alive document ids.\nLike all segment files, this file is immutable. Because it is possible to have more than one alive bitset file at a given instant, the alive bitset filename has the format ```segment_id . commit_opstamp . del```.\n\nAn opstamp is simply an incremental id that identifies any operation applied to the index. For instance, performing a commit or adding a document.\n\n### DocId\n\nWithin a segment, all documents are identified by a DocId that ranges within `[0, max_doc)`.\nwhere `max_doc` is the number of documents in the segment, (deleted or not). Having such a compact `DocId` space is key to the compression of our data structures.\n\nThe DocIds are simply allocated in the order documents are added to the index.\n\n### Merges\n\nIn separate threads, tantivy's index writer search for opportunities to merge segments.\nThe point of segment merge is to:\n\n- eventually get rid of tombstoned documents\n- reduce the otherwise ever-growing number of segments.\n\nIndeed, while having several segments instead of one does not hurt search too much, having hundreds can have a measurable impact on the search performance.\n\n### Searcher\n\nThe user of the library usually does not need to know about the existence of Segments.\nSearching is done through an object called a [`Searcher`](src/core/searcher.rs), that captures a\nsnapshot of the index at one point of time, by holding a list of [SegmentReader](src/index/segment_reader.rs).\n\nIn other words, regardless of commits, file garbage collection, or segment merge that might happen, as long as the user holds and reuse the same [Searcher](src/core/searcher.rs), search will happen on an immutable snapshot of the index.\n\n## [directory/](src/directory): Where should the data be stored?\n\nTantivy, like Lucene, abstracts the place where the data should be stored in a key-trait\ncalled [`Directory`](src/directory/directory.rs).\nContrary to Lucene however, \"files\" are quite different from some kind of `io::Read` object.\nCheck out [`src/directory/directory.rs`](src/directory/directory.rs) trait for more details.\n\nTantivy ships two main directory implementation: the `MmapDirectory` and the `RamDirectory`,\nbut users can extend tantivy with their own implementation.\n\n## [schema/](src/schema): What are documents?\n\nTantivy's document follows a very strict schema, decided before building any index.\n\nThe schema defines all of the fields that the indexes [`Document`](src/schema/document/mod.rs) may and should contain, their types (`text`, `i64`, `u64`, `Date`, ...) as well as how it should be indexed / represented in tantivy.\n\nDepending on the type of the field, you can decide to\n\n- put it in the docstore\n- store it as a fast field\n- index it\n\nPractically, tantivy will push values associated with this type to up to 3 respective\ndata structures.\n\n*Limitations*\n\nAs of today, tantivy's schema imposes a 1:1 relationship between a field that is being ingested and a field represented in the search index. In sophisticated search application, it is fairly common to want to index a field twice using different tokenizers, or to index the concatenation of several fields together into one field.\n\nThis is not something tantivy supports, and it is up to the user to duplicate field / concatenate fields before feeding them to tantivy.\n\n## General information about these data structures\n\nAll data structures in tantivy, have:\n\n- a writer\n- a serializer\n- a reader\n\nThe writer builds an in-memory representation of a batch of documents. This representation is not searchable. It is just meant as an intermediary mutable representation, to which we can sequentially add\nthe document of a batch. At the end of the batch (or if a memory limit is reached), this representation\nis then converted into an on-disk immutable representation, that is extremely compact.\nThis conversion is done by the serializer.\n\nFinally, the reader is in charge of offering an API to read on this on-disk read-only representation.\nIn tantivy, readers are designed to require very little anonymous memory. The data is read straight from an mmapped file, and loading an index is as fast as mmapping its files.\n\n## [store/](src/store): Here is my DocId, Gimme my document\n\nThe docstore is a row-oriented storage that, for each document, stores a subset of the fields\nthat are marked as stored in the schema. The docstore is compressed using a general-purpose algorithm\nlike LZ4.\n\n**Useful for**\n\nIn search engines, it is often used to display search results.\nOnce the top 10 documents have been identified, we fetch them from the store, and display them or their snippet on the search result page (aka SERP).\n\n**Not useful for**\n\nFetching a document from the store is typically a \"slow\" operation. It usually consists in\n\n- searching into a compact tree-like data structure to find the position of the right block.\n- decompressing a small block\n- returning the document from this block.\n\nIt is NOT meant to be called for every document matching a query.\n\nAs a rule of thumb, if you hit the docstore more than 100 times per search query, you are probably misusing tantivy.\n\n## [fastfield/](src/fastfield): Here is my DocId, Gimme my value\n\nFast fields are stored in a column-oriented storage that allows for random access.\nThe only compression applied is bitpacking. The column comes with two meta data.\nThe minimum value in the column and the number of bits per doc.\n\nFetching a value for a `DocId` is then as simple as computing\n\n```rust\nmin_value + fetch_bits(num_bits * doc_id..num_bits * (doc_id+1))\n```\n\nThis operation just requires one memory fetch.\nBecause, DocSets are scanned through in order (DocId are iterated in a sorted manner) which\nalso help locality.\n\nIn Lucene's jargon, fast fields are called DocValues.\n\n**Useful for**\n\nThey are typically integer values that are useful to either rank or compute aggregate over\nall of the documents matching a query (aka [DocSet](src/docset.rs)).\n\nFor instance, one could define a function to combine upvotes with tantivy's internal relevancy score.\nThis can be done by fetching a fast field during scoring.\nOne could also compute the mean price of the items matching a query in an e-commerce website.\nThis can be done by fetching a fast field in a collector.\nFinally one could decide to post-filter a docset to remove docset with a price within a specific range.\nIf the ratio of filtered out documents is not too low, an efficient way to do this is to fetch the price and apply the filter on the collector side.\n\nAside from integer values, it is also possible to store an actual byte payload.\nFor advanced search engine, it is possible to store all of the features required for learning-to-rank in a byte payload, access it during search, and apply the learning-to-rank model.\n\nFinally facets are a specific kind of fast field, and the associated source code is in [`fastfield/facet_reader.rs`](src/fastfield/facet_reader.rs).\n\n# The inverted search index\n\nThe inverted index is the core part of full-text search.\nWhen presented a new document with the text field \"Hello, happy tax payer!\", tantivy breaks it into a list of so-called tokens. In addition to just splitting these strings into tokens, it might also do different kinds of operations like dropping the punctuation, converting the character to lowercase, apply stemming, etc. Tantivy makes it possible to configure the operations to be applied in the schema (tokenizer/ is the place where these operations are implemented).\n\nFor instance, the default tokenizer of tantivy would break our text into: `[hello, happy, tax, payer]`.\nThe document will therefore be registered in the inverted index as containing the terms\n`[text:hello, text:happy, text:tax, text:payer]`.\n\nThe role of the inverted index is, when given a term, gives us in return a very fast iterator over the sorted doc ids that match the term.\n\nSuch an iterator is called a posting list. In addition to giving us `DocId`, they can also give us optionally the number of occurrence of the term for each document, also called term frequency or TF.\n\nThese iterators being sorted by DocId, one can create an iterator over the document containing `text:tax AND text:payer`, `(text:tax AND text:payer) OR (text:contribuable)` or any boolean expression.\n\nIn order to represent the function\n```Term ⟶ Posting```\n\nThe inverted index actually consists of two data structures chained together.\n\n- [Term](src/schema/term.rs) ⟶ [TermInfo](src/postings/term_info.rs) is addressed by the term dictionary.\n- [TermInfo](src/postings/term_info.rs) ⟶ [Posting](src/postings/postings.rs) is addressed by the posting lists.\n\nWhere [TermInfo](src/postings/term_info.rs) is an object containing some meta data about a term.\n\n## [termdict/](src/termdict): Here is a term, give me the [TermInfo](src/postings/term_info.rs)\n\nTantivy's term dictionary is mainly in charge of supplying the function\n\n[Term](src/schema/term.rs) ⟶ [TermInfo](src/postings/term_info.rs)\n\nIt is itself broken into two parts.\n\n- [Term](src/schema/term.rs) ⟶ [TermOrdinal](src/termdict/mod.rs) is addressed by a finite state transducer, implemented by the fst crate.\n- [TermOrdinal](src/termdict/mod.rs) ⟶ [TermInfo](src/postings/term_info.rs) is addressed by the term info store.\n\n## [postings/](src/postings): Iterate over documents... very fast\n\nA posting list makes it possible to store a sorted list of doc ids and for each doc store\na term frequency as well.\n\nThe posting lists are stored in a separate file. The [TermInfo](src/postings/term_info.rs) contains an offset into that file and a number of documents for the given posting list. Both are required and sufficient to read the posting list.\n\nThe posting list is organized in block of 128 documents.\nOne block of doc ids is followed by one block of term frequencies.\n\nThe doc ids are delta encoded and bitpacked.\nThe term frequencies are bitpacked.\n\nBecause the number of docs is rarely a multiple of 128, the last block may contain an arbitrary number of docs between 1 and 127 documents. We then use variable int encoding instead of bitpacking.\n\n## [positions/](src/positions): Where are my terms within the documents?\n\nPhrase queries make it possible to search for documents containing a specific sequence of terms.\nFor instance, when the phrase query \"the art of war\" does not match \"the war of art\".\nTo make it possible, it is possible to specify in the schema that a field should store positions in addition to being indexed.\n\nThe token positions of all of the terms are then stored in a separate file with the extension `.pos`.\nThe [TermInfo](src/postings/term_info.rs) gives an offset (expressed in position this time) in this file. As we iterate through the docset,\nwe advance the position reader by the number of term frequencies of the current document.\n\n## [fieldnorm/](src/fieldnorm): Here is my doc, how many tokens in this field?\n\nThe [BM25](https://en.wikipedia.org/wiki/Okapi_BM25) formula also requires to know the number of tokens stored in a specific field for a given document. We store this information on one byte per document in the fieldnorm.\nThe fieldnorm is therefore compressed. Values up to 40 are encoded unchanged.\n\n## [tokenizer/](src/tokenizer): How should we process text?\n\nText processing is key to a good search experience.\nSplits or normalize your text too much, and the search results will have a less precision and a higher recall.\nDo not normalize, or under split your text, you will end up with a higher precision and a lesser recall.\n\nText processing can be configured by selecting an off-the-shelf [`Tokenizer`](./src/tokenizer/tokenizer.rs) or implementing your own to first split the text into tokens, and then chain different [`TokenFilter`](src/tokenizer/tokenizer.rs)'s to it.\n\nTantivy's comes with few tokenizers, but external crates are offering advanced tokenizers, such as [Lindera](https://crates.io/crates/lindera) for Japanese.\n\n## [query/](src/query): Define and compose queries\n\nThe [Query](src/query/query.rs) trait defines what a query is.\nDue to the necessity for some queries to compute some statistics over the entire index, and because the\nindex is composed of several `SegmentReader`, the path from transforming a `Query` to an iterator over documents is slightly convoluted, but fundamentally, this is what a Query is.\n\nThe iterator over a document comes with some scoring function. The resulting trait is called a\n[Scorer](src/query/scorer.rs) and is specific to a segment.\n\nDifferent queries can be combined using the [BooleanQuery](src/query/boolean_query/).\nTantivy comes with different types of queries and can be extended by implementing\nthe `Query`, `Weight`, and `Scorer` traits.\n\n## [collector](src/collector): Define what to do with matched documents\n\nCollectors define how to aggregate the documents matching a query, in the broadest sense possible.\nThe search will push matched documents one by one, calling their\n`fn collect(doc: DocId, score: Score);` method.\n\nUsers may implement their own collectors by implementing the [Collector](src/collector/mod.rs) trait.\n\n## [query-grammar](query-grammar): Defines the grammar of the query parser\n\nWhile the [QueryParser](src/query/query_parser/query_parser.rs) struct is located in the `query/` directory, the actual parser combinator used to convert user queries into an AST is in an external crate called `query-grammar`. This part was externalized to lighten the work of the compiler.\n"},"items":[{"name":"ARCHITECTURE.md","path":"ARCHITECTURE.md","rawUrl":"https://raw.githubusercontent.com/quickwit-oss/tantivy/HEAD/ARCHITECTURE.md","title":"System Architecture & Module Boundaries","category":"project-spec","format":"markdown","content":"# Tantivy\n\n## What is tantivy?\n\nTantivy is a library that is meant to build search engines. Although it is by no means a port of Lucene, its architecture is strongly inspired by it. If you are familiar with Lucene, you may be struck by the overlapping vocabulary.\nThis is not fortuitous.\n\nTantivy's bread and butter is to address the problem of full-text search :\n\nGiven a large set of textual documents, and a text query, return the K-most relevant documents in a very efficient way. To execute these queries rapidly, the tantivy needs to build an index beforehand. The relevance score implemented in the tantivy is not configurable. Tantivy uses the same score as the default similarity used in Lucene / Elasticsearch, called [BM25](https://en.wikipedia.org/wiki/Okapi_BM25).\n\nBut tantivy's scope does not stop there. Numerous features are required to power rich-search applications. For instance, one may want to:\n\n- compute the count of documents matching a query in the different section of an e-commerce website,\n- display an average price per meter square for a real estate search engine,\n- take into account historical user data to rank documents in a specific way,\n- or even use tantivy to power an OLAP database.\n\nA more abstract description of the problem space tantivy is trying to address is the following.\n\nIngest a large set of documents, create an index that makes it possible to\nrapidly select all documents matching a given predicate (also known as a query) and\ncollect some information about them ([See collector](#collector-define-what-to-do-with-matched-documents)).\n\nRoughly speaking the design is following these guiding principles:\n\n- Search should be O(1) in memory.\n- Indexing should be O(1) in memory. (In practice it is just sublinear)\n- Search should be as fast as possible\n\nThis comes at the cost of the dynamicity of the index: while it is possible to add, and delete documents from our corpus, the tantivy is designed to handle these updates in large batches.\n\n## [core/](src/core): Index, segments, searchers\n\nCore contains all of the high-level code to make it possible to create an index, add documents, delete documents and commit.\n\nThis is both the most high-level part of tantivy, the least performance-sensitive one, the seemingly most mundane code... And paradoxically the most complicated part.\n\n### Index and Segments\n\nA tantivy index is a collection of smaller independent immutable segments.\nEach segment contains its own independent set of data structures.\n\nA segment is identified by a segment id that is in fact a UUID.\nThe file of a segment has the format\n\n ```segment-id . ext```\n\nThe extension signals which data structure (or [`SegmentComponent`](src/index/segment_component.rs)) is stored in the file.\n\nA small `meta.json` file is in charge of keeping track of the list of segments, as well as the schema.\n\nOn commit, one segment per indexing thread is written to disk, and the `meta.json` is then updated atomically.\n\nFor a better idea of how indexing works, you may read the [following blog post](https://fulmicoton.com/posts/behold-tantivy-part2/).\n\n### Deletes\n\nDeletes happen by deleting a \"term\". Tantivy does not offer any notion of primary id, so it is up to the user to use a field in their schema as if it was a primary id, and delete the associated term if they want to delete only one specific document.\n\nOn commit, tantivy will find all of the segments with documents matching this existing term and remove from [alive bitset file](src/fastfield/alive_bitset.rs) that represents the bitset of the alive document ids.\nLike all segment files, this file is immutable. Because it is possible to have more than one alive bitset file at a given instant, the alive bitset filename has the format ```segment_id . commit_opstamp . del```.\n\nAn opstamp is simply an incremental id that identifies any operation applied to the index. For instance, performing a commit or adding a document.\n\n### DocId\n\nWithin a segment, all documents are identified by a DocId that ranges within `[0, max_doc)`.\nwhere `max_doc` is the number of documents in the segment, (deleted or not). Having such a compact `DocId` space is key to the compression of our data structures.\n\nThe DocIds are simply allocated in the order documents are added to the index.\n\n### Merges\n\nIn separate threads, tantivy's index writer search for opportunities to merge segments.\nThe point of segment merge is to:\n\n- eventually get rid of tombstoned documents\n- reduce the otherwise ever-growing number of segments.\n\nIndeed, while having several segments instead of one does not hurt search too much, having hundreds can have a measurable impact on the search performance.\n\n### Searcher\n\nThe user of the library usually does not need to know about the existence of Segments.\nSearching is done through an object called a [`Searcher`](src/core/searcher.rs), that captures a\nsnapshot of the index at one point of time, by holding a list of [SegmentReader](src/index/segment_reader.rs).\n\nIn other words, regardless of commits, file garbage collection, or segment merge that might happen, as long as the user holds and reuse the same [Searcher](src/core/searcher.rs), search will happen on an immutable snapshot of the index.\n\n## [directory/](src/directory): Where should the data be stored?\n\nTantivy, like Lucene, abstracts the place where the data should be stored in a key-trait\ncalled [`Directory`](src/directory/directory.rs).\nContrary to Lucene however, \"files\" are quite different from some kind of `io::Read` object.\nCheck out [`src/directory/directory.rs`](src/directory/directory.rs) trait for more details.\n\nTantivy ships two main directory implementation: the `MmapDirectory` and the `RamDirectory`,\nbut users can extend tantivy with their own implementation.\n\n## [schema/](src/schema): What are documents?\n\nTantivy's document follows a very strict schema, decided before building any index.\n\nThe schema defines all of the fields that the indexes [`Document`](src/schema/document/mod.rs) may and should contain, their types (`text`, `i64`, `u64`, `Date`, ...) as well as how it should be indexed / represented in tantivy.\n\nDepending on the type of the field, you can decide to\n\n- put it in the docstore\n- store it as a fast field\n- index it\n\nPractically, tantivy will push values associated with this type to up to 3 respective\ndata structures.\n\n*Limitations*\n\nAs of today, tantivy's schema imposes a 1:1 relationship between a field that is being ingested and a field represented in the search index. In sophisticated search application, it is fairly common to want to index a field twice using different tokenizers, or to index the concatenation of several fields together into one field.\n\nThis is not something tantivy supports, and it is up to the user to duplicate field / concatenate fields before feeding them to tantivy.\n\n## General information about these data structures\n\nAll data structures in tantivy, have:\n\n- a writer\n- a serializer\n- a reader\n\nThe writer builds an in-memory representation of a batch of documents. This representation is not searchable. It is just meant as an intermediary mutable representation, to which we can sequentially add\nthe document of a batch. At the end of the batch (or if a memory limit is reached), this representation\nis then converted into an on-disk immutable representation, that is extremely compact.\nThis conversion is done by the serializer.\n\nFinally, the reader is in charge of offering an API to read on this on-disk read-only representation.\nIn tantivy, readers are designed to require very little anonymous memory. The data is read straight from an mmapped file, and loading an index is as fast as mmapping its files.\n\n## [store/](src/store): Here is my DocId, Gimme my document\n\nThe docstore is a row-oriented storage that, for each document, stores a subset of the fields\nthat are marked as stored in the schema. The docstore is compressed using a general-purpose algorithm\nlike LZ4.\n\n**Useful for**\n\nIn search engines, it is often used to display search results.\nOnce the top 10 documents have been identified, we fetch them from the store, and display them or their snippet on the search result page (aka SERP).\n\n**Not useful for**\n\nFetching a document from the store is typically a \"slow\" operation. It usually consists in\n\n- searching into a compact tree-like data structure to find the position of the right block.\n- decompressing a small block\n- returning the document from this block.\n\nIt is NOT meant to be called for every document matching a query.\n\nAs a rule of thumb, if you hit the docstore more than 100 times per search query, you are probably misusing tantivy.\n\n## [fastfield/](src/fastfield): Here is my DocId, Gimme my value\n\nFast fields are stored in a column-oriented storage that allows for random access.\nThe only compression applied is bitpacking. The column comes with two meta data.\nThe minimum value in the column and the number of bits per doc.\n\nFetching a value for a `DocId` is then as simple as computing\n\n```rust\nmin_value + fetch_bits(num_bits * doc_id..num_bits * (doc_id+1))\n```\n\nThis operation just requires one memory fetch.\nBecause, DocSets are scanned through in order (DocId are iterated in a sorted manner) which\nalso help locality.\n\nIn Lucene's jargon, fast fields are called DocValues.\n\n**Useful for**\n\nThey are typically integer values that are useful to either rank or compute aggregate over\nall of the documents matching a query (aka [DocSet](src/docset.rs)).\n\nFor instance, one could define a function to combine upvotes with tantivy's internal relevancy score.\nThis can be done by fetching a fast field during scoring.\nOne could also compute the mean price of the items matching a query in an e-commerce website.\nThis can be done by fetching a fast field in a collector.\nFinally one could decide to post-filter a docset to remove docset with a price within a specific range.\nIf the ratio of filtered out documents is not too low, an efficient way to do this is to fetch the price and apply the filter on the collector side.\n\nAside from integer values, it is also possible to store an actual byte payload.\nFor advanced search engine, it is possible to store all of the features required for learning-to-rank in a byte payload, access it during search, and apply the learning-to-rank model.\n\nFinally facets are a specific kind of fast field, and the associated source code is in [`fastfield/facet_reader.rs`](src/fastfield/facet_reader.rs).\n\n# The inverted search index\n\nThe inverted index is the core part of full-text search.\nWhen presented a new document with the text field \"Hello, happy tax payer!\", tantivy breaks it into a list of so-called tokens. In addition to just splitting these strings into tokens, it might also do different kinds of operations like dropping the punctuation, converting the character to lowercase, apply stemming, etc. Tantivy makes it possible to configure the operations to be applied in the schema (tokenizer/ is the place where these operations are implemented).\n\nFor instance, the default tokenizer of tantivy would break our text into: `[hello, happy, tax, payer]`.\nThe document will therefore be registered in the inverted index as containing the terms\n`[text:hello, text:happy, text:tax, text:payer]`.\n\nThe role of the inverted index is, when given a term, gives us in return a very fast iterator over the sorted doc ids that match the term.\n\nSuch an iterator is called a posting list. In addition to giving us `DocId`, they can also give us optionally the number of occurrence of the term for each document, also called term frequency or TF.\n\nThese iterators being sorted by DocId, one can create an iterator over the document containing `text:tax AND text:payer`, `(text:tax AND text:payer) OR (text:contribuable)` or any boolean expression.\n\nIn order to represent the function\n```Term ⟶ Posting```\n\nThe inverted index actually consists of two data structures chained together.\n\n- [Term](src/schema/term.rs) ⟶ [TermInfo](src/postings/term_info.rs) is addressed by the term dictionary.\n- [TermInfo](src/postings/term_info.rs) ⟶ [Posting](src/postings/postings.rs) is addressed by the posting lists.\n\nWhere [TermInfo](src/postings/term_info.rs) is an object containing some meta data about a term.\n\n## [termdict/](src/termdict): Here is a term, give me the [TermInfo](src/postings/term_info.rs)\n\nTantivy's term dictionary is mainly in charge of supplying the function\n\n[Term](src/schema/term.rs) ⟶ [TermInfo](src/postings/term_info.rs)\n\nIt is itself broken into two parts.\n\n- [Term](src/schema/term.rs) ⟶ [TermOrdinal](src/termdict/mod.rs) is addressed by a finite state transducer, implemented by the fst crate.\n- [TermOrdinal](src/termdict/mod.rs) ⟶ [TermInfo](src/postings/term_info.rs) is addressed by the term info store.\n\n## [postings/](src/postings): Iterate over documents... very fast\n\nA posting list makes it possible to store a sorted list of doc ids and for each doc store\na term frequency as well.\n\nThe posting lists are stored in a separate file. The [TermInfo](src/postings/term_info.rs) contains an offset into that file and a number of documents for the given posting list. Both are required and sufficient to read the posting list.\n\nThe posting list is organized in block of 128 documents.\nOne block of doc ids is followed by one block of term frequencies.\n\nThe doc ids are delta encoded and bitpacked.\nThe term frequencies are bitpacked.\n\nBecause the number of docs is rarely a multiple of 128, the last block may contain an arbitrary number of docs between 1 and 127 documents. We then use variable int encoding instead of bitpacking.\n\n## [positions/](src/positions): Where are my terms within the documents?\n\nPhrase queries make it possible to search for documents containing a specific sequence of terms.\nFor instance, when the phrase query \"the art of war\" does not match \"the war of art\".\nTo make it possible, it is possible to specify in the schema that a field should store positions in addition to being indexed.\n\nThe token positions of all of the terms are then stored in a separate file with the extension `.pos`.\nThe [TermInfo](src/postings/term_info.rs) gives an offset (expressed in position this time) in this file. As we iterate through the docset,\nwe advance the position reader by the number of term frequencies of the current document.\n\n## [fieldnorm/](src/fieldnorm): Here is my doc, how many tokens in this field?\n\nThe [BM25](https://en.wikipedia.org/wiki/Okapi_BM25) formula also requires to know the number of tokens stored in a specific field for a given document. We store this information on one byte per document in the fieldnorm.\nThe fieldnorm is therefore compressed. Values up to 40 are encoded unchanged.\n\n## [tokenizer/](src/tokenizer): How should we process text?\n\nText processing is key to a good search experience.\nSplits or normalize your text too much, and the search results will have a less precision and a higher recall.\nDo not normalize, or under split your text, you will end up with a higher precision and a lesser recall.\n\nText processing can be configured by selecting an off-the-shelf [`Tokenizer`](./src/tokenizer/tokenizer.rs) or implementing your own to first split the text into tokens, and then chain different [`TokenFilter`](src/tokenizer/tokenizer.rs)'s to it.\n\nTantivy's comes with few tokenizers, but external crates are offering advanced tokenizers, such as [Lindera](https://crates.io/crates/lindera) for Japanese.\n\n## [query/](src/query): Define and compose queries\n\nThe [Query](src/query/query.rs) trait defines what a query is.\nDue to the necessity for some queries to compute some statistics over the entire index, and because the\nindex is composed of several `SegmentReader`, the path from transforming a `Query` to an iterator over documents is slightly convoluted, but fundamentally, this is what a Query is.\n\nThe iterator over a document comes with some scoring function. The resulting trait is called a\n[Scorer](src/query/scorer.rs) and is specific to a segment.\n\nDifferent queries can be combined using the [BooleanQuery](src/query/boolean_query/).\nTantivy comes with different types of queries and can be extended by implementing\nthe `Query`, `Weight`, and `Scorer` traits.\n\n## [collector](src/collector): Define what to do with matched documents\n\nCollectors define how to aggregate the documents matching a query, in the broadest sense possible.\nThe search will push matched documents one by one, calling their\n`fn collect(doc: DocId, score: Score);` method.\n\nUsers may implement their own collectors by implementing the [Collector](src/collector/mod.rs) trait.\n\n## [query-grammar](query-grammar): Defines the grammar of the query parser\n\nWhile the [QueryParser](src/query/query_parser/query_parser.rs) struct is located in the `query/` directory, the actual parser combinator used to convert user queries into an AST is in an external crate called `query-grammar`. This part was externalized to lighten the work of the compiler.\n","isInternal":false,"tokens":3767,"sizeBytes":16994},{"name":"SKILL.md","path":".claude/skills/rationalize-deps/SKILL.md","rawUrl":"https://raw.githubusercontent.com/quickwit-oss/tantivy/HEAD/.claude/skills/rationalize-deps/SKILL.md","title":"rationalize-deps","category":"anthropic-skill","format":"markdown","content":"---\nname: rationalize-deps\ndescription: Analyze Cargo.toml dependencies and attempt to remove unused features to reduce compile times and binary size\n---\n\n# Rationalize Dependencies\n\nThis skill analyzes Cargo.toml dependencies to identify and remove unused features.\n\n## Overview\n\nMany crates enable features by default that may not be needed. This skill:\n1. Identifies dependencies with default features enabled\n2. Tests if `default-features = false` works\n3. Identifies which specific features are actually needed\n4. Verifies compilation after changes\n\n## Step 1: Identify the target\n\nAsk the user which crate(s) to analyze:\n- A specific crate name (e.g., \"tokio\", \"serde\")\n- A specific workspace member (e.g., \"quickwit-search\")\n- \"all\" to scan the entire workspace\n\n## Step 2: Analyze current dependencies\n\nFor the workspace Cargo.toml (`quickwit/Cargo.toml`), list dependencies that:\n- Do NOT have `default-features = false`\n- Have default features that might be unnecessary\n\nRun: `cargo tree -p <crate> -f \"{p} {f}\" --edges features` to see what features are actually used.\n\n## Step 3: For each candidate dependency\n\n### 3a: Check the crate's default features\n\nLook up the crate on crates.io or check its Cargo.toml to understand:\n- What features are enabled by default\n- What each feature provides\n\nUse: `cargo metadata --format-version=1 | jq '.packages[] | select(.name == \"<crate>\") | .features'`\n\n### 3b: Try disabling default features\n\nModify the dependency in `quickwit/Cargo.toml`:\n\nFrom:\n```toml\nsome-crate = { version = \"1.0\" }\n```\n\nTo:\n```toml\nsome-crate = { version = \"1.0\", default-features = false }\n```\n\n### 3c: Run cargo check\n\nRun: `cargo check --workspace` (or target specific packages for faster feedback)\n\nIf compilation fails:\n1. Read the error messages to identify which features are needed\n2. Add only the required features explicitly:\n   ```toml\n   some-crate = { version = \"1.0\", default-features = false, features = [\"needed-feature\"] }\n   ```\n3. Re-run cargo check\n\n### 3d: Binary search for minimal features\n\nIf there are many default features, use binary search:\n1. Start with no features\n2. If it fails, add half the default features\n3. Continue until you find the minimal set\n\n## Step 4: Document findings\n\nFor each dependency analyzed, report:\n- Original configuration\n- New configuration (if changed)\n- Features that were removed\n- Any features that are required\n\n## Step 5: Verify full build\n\nAfter all changes, run:\n```bash\ncargo check --workspace --all-targets\ncargo test --workspace --no-run\n```\n\n## Common Patterns\n\n### Serde\nOften only needs `derive`:\n```toml\nserde = { version = \"1.0\", default-features = false, features = [\"derive\", \"std\"] }\n```\n\n### Tokio\nIdentify which runtime features are actually used:\n```toml\ntokio = { version = \"1.0\", default-features = false, features = [\"rt-multi-thread\", \"macros\", \"sync\"] }\n```\n\n### Reqwest\nOften doesn't need all TLS backends:\n```toml\nreqwest = { version = \"0.11\", default-features = false, features = [\"rustls-tls\", \"json\"] }\n```\n\n## Rollback\n\nIf changes cause issues:\n```bash\ngit checkout quickwit/Cargo.toml\ncargo check --workspace\n```\n\n## Tips\n\n- Start with large crates that have many default features (tokio, reqwest, hyper)\n- Use `cargo bloat --crates` to identify large dependencies\n- Check `cargo tree -d` for duplicate dependencies that might indicate feature conflicts\n- Some features are needed only for tests - consider using `[dev-dependencies]` features\n","frontmatter":{"name":"rationalize-deps","description":"Analyze Cargo.toml dependencies and attempt to remove unused features to reduce compile times and binary size"},"isInternal":false,"tokens":862,"sizeBytes":3463},{"name":"SKILL.md","path":".claude/skills/simple-pr/SKILL.md","rawUrl":"https://raw.githubusercontent.com/quickwit-oss/tantivy/HEAD/.claude/skills/simple-pr/SKILL.md","title":"simple-pr","category":"anthropic-skill","format":"markdown","content":"---\nname: simple-pr\ndescription: Create a simple PR from staged changes with an auto-generated commit message\ndisable-model-invocation: true\n---\n\n# Simple PR\n\nFollow these steps to create a simple PR from staged changes:\n\n## Step 1: Check workspace state\n\nRun: `git status`\n\nVerify that all changes have been staged (no unstaged changes). If there are unstaged changes, abort and ask the user to stage their changes first with `git add`.\n\nAlso verify that we are on the `main` branch. If not, abort and ask the user to switch to main first.\n\n## Step 2: Ensure main is up to date\n\nRun: `git pull origin main`\n\nThis ensures we're working from the latest code.\n\n## Step 3: Review staged changes\n\nRun: `git diff --cached`\n\nReview the staged changes to understand what the PR will contain.\n\n## Step 4: Generate commit message\n\nBased on the staged changes, generate a concise commit message (1-2 sentences) that describes the \"why\" rather than the \"what\".\n\nDisplay the proposed commit message to the user and ask for confirmation before proceeding.\n\n## Step 5: Create a new branch\n\nGet the git username: `git config user.name | tr ' ' '-' | tr '[:upper:]' '[:lower:]'`\n\nCreate a short, descriptive branch name based on the changes (e.g., `fix-typo-in-readme`, `add-retry-logic`, `update-deps`).\n\nCreate and checkout the branch: `git checkout -b {username}/{short-descriptive-name}`\n\n## Step 6: Commit changes\n\nCommit with the message from step 3:\n```\ngit commit -m \"{commit-message}\"\n```\n\n## Step 7: Push and open a PR\n\nPush the branch and open a PR:\n```\ngit push -u origin {branch-name}\ngh pr create --title \"{commit-message-title}\" --body \"{longer-description-if-needed}\"\n```\n\nReport the PR URL to the user when complete.\n","frontmatter":{"name":"simple-pr","description":"Create a simple PR from staged changes with an auto-generated commit message","disable-model-invocation":true,"disableModelInvocation":true},"isInternal":false,"tokens":418,"sizeBytes":1718},{"name":"SKILL.md","path":".claude/skills/update-changelog/SKILL.md","rawUrl":"https://raw.githubusercontent.com/quickwit-oss/tantivy/HEAD/.claude/skills/update-changelog/SKILL.md","title":"update-changelog","category":"anthropic-skill","format":"markdown","content":"---\nname: update-changelog\ndescription: Update CHANGELOG.md with merged PRs since the last changelog update, categorized by type\n---\n\n# Update Changelog\n\nThis skill updates CHANGELOG.md with merged PRs that aren't already listed.\n\n## Step 1: Determine the changelog scope\n\nRead `CHANGELOG.md` to identify the current unreleased version section at the top (e.g., `Tantivy 0.26 (Unreleased)`).\n\nCollect all PR numbers already mentioned in the unreleased section by extracting `#NNNN` references.\n\n## Step 2: Find merged PRs not yet in the changelog\n\nUse `gh` to list recently merged PRs from the upstream repo:\n\n```bash\ngh pr list --repo quickwit-oss/tantivy --state merged --limit 100 --json number,title,author,labels,mergedAt\n```\n\nFilter out any PRs whose number already appears in the unreleased section of the changelog.\n\n## Step 3: Consolidate related PRs\n\nBefore categorizing, group PRs that belong to the same logical change. This is critical for producing a clean changelog. Use PR descriptions, titles, cross-references, and the files touched to identify relationships.\n\n**Merge follow-up PRs into the original:**\n- If a PR is a bugfix, refinement, or follow-up to another PR in the same unreleased cycle, combine them into a single changelog entry with multiple `[#N](url)` links.\n- Also consolidate PRs that touch the same feature area even if not explicitly linked — e.g., a PR fixing an edge case in a new API should be folded into the entry for the PR that introduced that API.\n\n**Filter out bugfixes on unreleased features:**\n- If a bugfix PR fixes something introduced by another PR in the **same unreleased version**, it must NOT appear as a separate Bugfixes entry. Instead, silently fold it into the original feature/improvement entry. The changelog should describe the final shipped state, not the development history.\n- To detect this: check if the bugfix PR references or reverts changes from another PR in the same release cycle, or if it touches code that was newly added (not present in the previous release).\n\n## Step 4: Review the actual code diff\n\n**Do not rely on PR titles or descriptions alone.** For every candidate PR, run `gh pr diff <number> --repo quickwit-oss/tantivy` and read the actual changes. PR titles are often misleading — the diff is the source of truth.\n\n**What to look for in the diff:**\n- Does it change observable behavior, public API surface, or performance characteristics?\n- Is the change something a user of the library would notice or need to know about?\n- Could the change break existing code (API changes, removed features)?\n\n**Skip PRs where the diff reveals the change is not meaningful enough for the changelog** — e.g., cosmetic renames, trivial visibility tweaks, test-only changes, etc.\n\n## Step 5: Categorize each PR group\n\nFor each PR (or consolidated group) that survived the diff review, determine its category:\n\n- **Bugfixes** — fixes to behavior that existed in the **previous release**. NOT fixes to features introduced in this release cycle.\n- **Features/Improvements** — new features, API additions, new options, improvements that change user-facing behavior or add new capabilities.\n- **Performance** — optimizations, speed improvements, memory reductions. **If a PR adds new API whose primary purpose is enabling a performance optimization, categorize it as Performance, not Features.** The deciding question is: does a user benefit from this because of new functionality, or because things got faster/leaner? For example, a new trait method that exists solely to enable cheaper intersection ordering is Performance, not a Feature.\n\nIf a PR doesn't clearly fit any category (e.g., CI-only changes, internal refactors with no user-facing impact, dependency bumps with no behavior change), skip it — not everything belongs in the changelog.\n\nWhen unclear, use your best judgment or ask the user.\n\n## Step 6: Format entries\n\nEach entry must follow this exact format:\n\n```\n- Description [#NUMBER](https://github.com/quickwit-oss/tantivy/pull/NUMBER)(@author)\n```\n\nRules:\n- The description should be concise and describe the user-facing change (not the implementation). Describe the final shipped state, not the incremental development steps.\n- Use sub-categories with bold headers when multiple entries relate to the same area (e.g., `- **Aggregation**` with indented entries beneath). Follow the existing grouping style in the changelog.\n- Author is the GitHub username from the PR, prefixed with `@`. For consolidated entries, include all contributing authors.\n- For consolidated PRs, list all PR links in a single entry: `[#100](url) [#110](url)` (see existing entries for examples).\n\n## Step 7: Present changes to the user\n\nShow the user the proposed changelog entries grouped by category **before** editing the file. Ask for confirmation or adjustments.\n\n## Step 8: Update CHANGELOG.md\n\nInsert the new entries into the appropriate sections of the unreleased version block. If a section doesn't exist yet, create it following the order: Bugfixes, Features/Improvements, Performance.\n\nAppend new entries at the end of each section (before the next section header or version header).\n\n## Step 9: Verify\n\nRead back the updated unreleased section and display it to the user for final review.\n","frontmatter":{"name":"update-changelog","description":"Update CHANGELOG.md with merged PRs since the last changelog update, categorized by type"},"isInternal":false,"tokens":1158,"sizeBytes":5275}],"systemPromptSnippet":"<yakaai_skills repo=\"quickwit-oss/tantivy\">\n<!-- File: ARCHITECTURE.md (Tokens: ~3767 | Category: project-spec) -->\n# Tantivy\n\n## What is tantivy?\n\nTantivy is a library that is meant to build search engines. Although it is by no means a port of Lucene, its architecture is strongly inspired by it. If you are familiar with Lucene, you may be struck by the overlapping vocabulary.\nThis is not fortuitous.\n\nTantivy's bread and butter is to address the problem of full-text search :\n\nGiven a large set of textual documents, and a text query, return the K-most relevant documents in a very efficient way. To execute these queries rapidly, the tantivy needs to build an index beforehand. The relevance score implemented in the tantivy is not configurable. Tantivy uses the same score as the default similarity used in Lucene / Elasticsearch, called [BM25](https://en.wikipedia.org/wiki/Okapi_BM25).\n\nBut tantivy's scope does not stop there. Numerous features are required to power rich-search applications. For instance, one may want to:\n\n- compute the count of documents matching a query in the different section of an e-commerce website,\n- display an average price per meter square for a real estate search engine,\n- take into account historical user data to rank documents in a specific way,\n- or even use tantivy to power an OLAP database.\n\nA more abstract description of the problem space tantivy is trying to address is the following.\n\nIngest a large set of documents, create an index that makes it possible to\nrapidly select all documents matching a given predicate (also known as a query) and\ncollect some information about them ([See collector](#collector-define-what-to-do-with-matched-documents)).\n\nRoughly speaking the design is following these guiding principles:\n\n- Search should be O(1) in memory.\n- Indexing should be O(1) in memory. (In practice it is just sublinear)\n- Search should be as fast as possible\n\nThis comes at the cost of the dynamicity of the index: while it is possible to add, and delete documents from our corpus, the tantivy is designed to handle these updates in large batches.\n\n## [core/](src/core): Index, segments, searchers\n\nCore contains all of the high-level code to make it possible to create an index, add documents, delete documents and commit.\n\nThis is both the most high-level part of tantivy, the least performance-sensitive one, the seemingly most mundane code... And paradoxically the most complicated part.\n\n### Index and Segments\n\nA tantivy index is a collection of smaller independent immutable segments.\nEach segment contains its own independent set of data structures.\n\nA segment is identified by a segment id that is in fact a UUID.\nThe file of a segment has the format\n\n ```segment-id . ext```\n\nThe extension signals which data structure (or [`SegmentComponent`](src/index/segment_component.rs)) is stored in the file.\n\nA small `meta.json` file is in charge of keeping track of the list of segments, as well as the schema.\n\nOn commit, one segment per indexing thread is written to disk, and the `meta.json` is then updated atomically.\n\nFor a better idea of how indexing works, you may read the [following blog post](https://fulmicoton.com/posts/behold-tantivy-part2/).\n\n### Deletes\n\nDeletes happen by deleting a \"term\". Tantivy does not offer any notion of primary id, so it is up to the user to use a field in their schema as if it was a primary id, and delete the associated term if they want to delete only one specific document.\n\nOn commit, tantivy will find all of the segments with documents matching this existing term and remove from [alive bitset file](src/fastfield/alive_bitset.rs) that represents the bitset of the alive document ids.\nLike all segment files, this file is immutable. Because it is possible to have more than one alive bitset file at a given instant, the alive bitset filename has the format ```segment_id . commit_opstamp . del```.\n\nAn opstamp is simply an incremental id that identifies any operation applied to the index. For instance, performing a commit or adding a document.\n\n### DocId\n\nWithin a segment, all documents are identified by a DocId that ranges within `[0, max_doc)`.\nwhere `max_doc` is the number of documents in the segment, (deleted or not). Having such a compact `DocId` space is key to the compression of our data structures.\n\nThe DocIds are simply allocated in the order documents are added to the index.\n\n### Merges\n\nIn separate threads, tantivy's index writer search for opportunities to merge segments.\nThe point of segment merge is to:\n\n- eventually get rid of tombstoned documents\n- reduce the otherwise ever-growing number of segments.\n\nIndeed, while having several segments instead of one does not hurt search too much, having hundreds can have a measurable impact on the search performance.\n\n### Searcher\n\nThe user of the library usually does not need to know about the existence of Segments.\nSearching is done through an object called a [`Searcher`](src/core/searcher.rs), that captures a\nsnapshot of the index at one point of time, by holding a list of [SegmentReader](src/index/segment_reader.rs).\n\nIn other words, regardless of commits, file garbage collection, or segment merge that might happen, as long as the user holds and reuse the same [Searcher](src/core/searcher.rs), search will happen on an immutable snapshot of the index.\n\n## [directory/](src/directory): Where should the data be stored?\n\nTantivy, like Lucene, abstracts the place where the data should be stored in a key-trait\ncalled [`Directory`](src/directory/directory.rs).\nContrary to Lucene however, \"files\" are quite different from some kind of `io::Read` object.\nCheck out [`src/directory/directory.rs`](src/directory/directory.rs) trait for more details.\n\nTantivy ships two main directory implementation: the `MmapDirectory` and the `RamDirectory`,\nbut users can extend tantivy with their own implementation.\n\n## [schema/](src/schema): What are documents?\n\nTantivy's document follows a very strict schema, decided before building any index.\n\nThe schema defines all of the fields that the indexes [`Document`](src/schema/document/mod.rs) may and should contain, their types (`text`, `i64`, `u64`, `Date`, ...) as well as how it should be indexed / represented in tantivy.\n\nDepending on the type of the field, you can decide to\n\n- put it in the docstore\n- store it as a fast field\n- index it\n\nPractically, tantivy will push values associated with this type to up to 3 respective\ndata structures.\n\n*Limitations*\n\nAs of today, tantivy's schema imposes a 1:1 relationship between a field that is being ingested and a field represented in the search index. In sophisticated search application, it is fairly common to want to index a field twice using different tokenizers, or to index the concatenation of several fields together into one field.\n\nThis is not something tantivy supports, and it is up to the user to duplicate field / concatenate fields before feeding them to tantivy.\n\n## General information about these data structures\n\nAll data structures in tantivy, have:\n\n- a writer\n- a serializer\n- a reader\n\nThe writer builds an in-memory representation of a batch of documents. This representation is not searchable. It is just meant as an intermediary mutable representation, to which we can sequentially add\nthe document of a batch. At the end of the batch (or if a memory limit is reached), this representation\nis then converted into an on-disk immutable representation, that is extremely compact.\nThis conversion is done by the serializer.\n\nFinally, the reader is in charge of offering an API to read on this on-disk read-only representation.\nIn tantivy, readers are designed to require very little anonymous memory. The data is read straight from an mmapped file, and loading an index is as fast as mmapping its files.\n\n## [store/](src/store): Here is my DocId, Gimme my document\n\nThe docstore is a row-oriented storage that, for each document, stores a subset of the fields\nthat are marked as stored in the schema. The docstore is compressed using a general-purpose algorithm\nlike LZ4.\n\n**Useful for**\n\nIn search engines, it is often used to display search results.\nOnce the top 10 documents have been identified, we fetch them from the store, and display them or their snippet on the search result page (aka SERP).\n\n**Not useful for**\n\nFetching a document from the store is typically a \"slow\" operation. It usually consists in\n\n- searching into a compact tree-like data structure to find the position of the right block.\n- decompressing a small block\n- returning the document from this block.\n\nIt is NOT meant to be called for every document matching a query.\n\nAs a rule of thumb, if you hit the docstore more than 100 times per search query, you are probably misusing tantivy.\n\n## [fastfield/](src/fastfield): Here is my DocId, Gimme my value\n\nFast fields are stored in a column-oriented storage that allows for random access.\nThe only compression applied is bitpacking. The column comes with two meta data.\nThe minimum value in the column and the number of bits per doc.\n\nFetching a value for a `DocId` is then as simple as computing\n\n```rust\nmin_value + fetch_bits(num_bits * doc_id..num_bits * (doc_id+1))\n```\n\nThis operation just requires one memory fetch.\nBecause, DocSets are scanned through in order (DocId are iterated in a sorted manner) which\nalso help locality.\n\nIn Lucene's jargon, fast fields are called DocValues.\n\n**Useful for**\n\nThey are typically integer values that are useful to either rank or compute aggregate over\nall of the documents matching a query (aka [DocSet](src/docset.rs)).\n\nFor instance, one could define a function to combine upvotes with tantivy's internal relevancy score.\nThis can be done by fetching a fast field during scoring.\nOne could also compute the mean price of the items matching a query in an e-commerce website.\nThis can be done by fetching a fast field in a collector.\nFinally one could decide to post-filter a docset to remove docset with a price within a specific range.\nIf the ratio of filtered out documents is not too low, an efficient way to do this is to fetch the price and apply the filter on the collector side.\n\nAside from integer values, it is also possible to store an actual byte payload.\nFor advanced search engine, it is possible to store all of the features required for learning-to-rank in a byte payload, access it during search, and apply the learning-to-rank model.\n\nFinally facets are a specific kind of fast field, and the associated source code is in [`fastfield/facet_reader.rs`](src/fastfield/facet_reader.rs).\n\n# The inverted search index\n\nThe inverted index is the core part of full-text search.\nWhen presented a new document with the text field \"Hello, happy tax payer!\", tantivy breaks it into a list of so-called tokens. In addition to just splitting these strings into tokens, it might also do different kinds of operations like dropping the punctuation, converting the character to lowercase, apply stemming, etc. Tantivy makes it possible to configure the operations to be applied in the schema (tokenizer/ is the place where these operations are implemented).\n\nFor instance, the default tokenizer of tantivy would break our text into: `[hello, happy, tax, payer]`.\nThe document will therefore be registered in the inverted index as containing the terms\n`[text:hello, text:happy, text:tax, text:payer]`.\n\nThe role of the inverted index is, when given a term, gives us in return a very fast iterator over the sorted doc ids that match the term.\n\nSuch an iterator is called a posting list. In addition to giving us `DocId`, they can also give us optionally the number of occurrence of the term for each document, also called term frequency or TF.\n\nThese iterators being sorted by DocId, one can create an iterator over the document containing `text:tax AND text:payer`, `(text:tax AND text:payer) OR (text:contribuable)` or any boolean expression.\n\nIn order to represent the function\n```Term ⟶ Posting```\n\nThe inverted index actually consists of two data structures chained together.\n\n- [Term](src/schema/term.rs) ⟶ [TermInfo](src/postings/term_info.rs) is addressed by the term dictionary.\n- [TermInfo](src/postings/term_info.rs) ⟶ [Posting](src/postings/postings.rs) is addressed by the posting lists.\n\nWhere [TermInfo](src/postings/term_info.rs) is an object containing some meta data about a term.\n\n## [termdict/](src/termdict): Here is a term, give me the [TermInfo](src/postings/term_info.rs)\n\nTantivy's term dictionary is mainly in charge of supplying the function\n\n[Term](src/schema/term.rs) ⟶ [TermInfo](src/postings/term_info.rs)\n\nIt is itself broken into two parts.\n\n- [Term](src/schema/term.rs) ⟶ [TermOrdinal](src/termdict/mod.rs) is addressed by a finite state transducer, implemented by the fst crate.\n- [TermOrdinal](src/termdict/mod.rs) ⟶ [TermInfo](src/postings/term_info.rs) is addressed by the term info store.\n\n## [postings/](src/postings): Iterate over documents... very fast\n\nA posting list makes it possible to store a sorted list of doc ids and for each doc store\na term frequency as well.\n\nThe posting lists are stored in a separate file. The [TermInfo](src/postings/term_info.rs) contains an offset into that file and a number of documents for the given posting list. Both are required and sufficient to read the posting list.\n\nThe posting list is organized in block of 128 documents.\nOne block of doc ids is followed by one block of term frequencies.\n\nThe doc ids are delta encoded and bitpacked.\nThe term frequencies are bitpacked.\n\nBecause the number of docs is rarely a multiple of 128, the last block may contain an arbitrary number of docs between 1 and 127 documents. We then use variable int encoding instead of bitpacking.\n\n## [positions/](src/positions): Where are my terms within the documents?\n\nPhrase queries make it possible to search for documents containing a specific sequence of terms.\nFor instance, when the phrase query \"the art of war\" does not match \"the war of art\".\nTo make it possible, it is possible to specify in the schema that a field should store positions in addition to being indexed.\n\nThe token positions of all of the terms are then stored in a separate file with the extension `.pos`.\nThe [TermInfo](src/postings/term_info.rs) gives an offset (expressed in position this time) in this file. As we iterate through the docset,\nwe advance the position reader by the number of term frequencies of the current document.\n\n## [fieldnorm/](src/fieldnorm): Here is my doc, how many tokens in this field?\n\nThe [BM25](https://en.wikipedia.org/wiki/Okapi_BM25) formula also requires to know the number of tokens stored in a specific field for a given document. We store this information on one byte per document in the fieldnorm.\nThe fieldnorm is therefore compressed. Values up to 40 are encoded unchanged.\n\n## [tokenizer/](src/tokenizer): How should we process text?\n\nText processing is key to a good search experience.\nSplits or normalize your text too much, and the search results will have a less precision and a higher recall.\nDo not normalize, or under split your text, you will end up with a higher precision and a lesser recall.\n\nText processing can be configured by selecting an off-the-shelf [`Tokenizer`](./src/tokenizer/tokenizer.rs) or implementing your own to first split the text into tokens, and then chain different [`TokenFilter`](src/tokenizer/tokenizer.rs)'s to it.\n\nTantivy's comes with few tokenizers, but external crates are offering advanced tokenizers, such as [Lindera](https://crates.io/crates/lindera) for Japanese.\n\n## [query/](src/query): Define and compose queries\n\nThe [Query](src/query/query.rs) trait defines what a query is.\nDue to the necessity for some queries to compute some statistics over the entire index, and because the\nindex is composed of several `SegmentReader`, the path from transforming a `Query` to an iterator over documents is slightly convoluted, but fundamentally, this is what a Query is.\n\nThe iterator over a document comes with some scoring function. The resulting trait is called a\n[Scorer](src/query/scorer.rs) and is specific to a segment.\n\nDifferent queries can be combined using the [BooleanQuery](src/query/boolean_query/).\nTantivy comes with different types of queries and can be extended by implementing\nthe `Query`, `Weight`, and `Scorer` traits.\n\n## [collector](src/collector): Define what to do with matched documents\n\nCollectors define how to aggregate the documents matching a query, in the broadest sense possible.\nThe search will push matched documents one by one, calling their\n`fn collect(doc: DocId, score: Score);` method.\n\nUsers may implement their own collectors by implementing the [Collector](src/collector/mod.rs) trait.\n\n## [query-grammar](query-grammar): Defines the grammar of the query parser\n\nWhile the [QueryParser](src/query/query_parser/query_parser.rs) struct is located in the `query/` directory, the actual parser combinator used to convert user queries into an AST is in an external crate called `query-grammar`. This part was externalized to lighten the work of the compiler.\n\n<!-- File: .claude/skills/rationalize-deps/SKILL.md (Tokens: ~862 | Category: anthropic-skill) -->\n---\nname: rationalize-deps\ndescription: Analyze Cargo.toml dependencies and attempt to remove unused features to reduce compile times and binary size\n---\n\n# Rationalize Dependencies\n\nThis skill analyzes Cargo.toml dependencies to identify and remove unused features.\n\n## Overview\n\nMany crates enable features by default that may not be needed. This skill:\n1. Identifies dependencies with default features enabled\n2. Tests if `default-features = false` works\n3. Identifies which specific features are actually needed\n4. Verifies compilation after changes\n\n## Step 1: Identify the target\n\nAsk the user which crate(s) to analyze:\n- A specific crate name (e.g., \"tokio\", \"serde\")\n- A specific workspace member (e.g., \"quickwit-search\")\n- \"all\" to scan the entire workspace\n\n## Step 2: Analyze current dependencies\n\nFor the workspace Cargo.toml (`quickwit/Cargo.toml`), list dependencies that:\n- Do NOT have `default-features = false`\n- Have default features that might be unnecessary\n\nRun: `cargo tree -p <crate> -f \"{p} {f}\" --edges features` to see what features are actually used.\n\n## Step 3: For each candidate dependency\n\n### 3a: Check the crate's default features\n\nLook up the crate on crates.io or check its Cargo.toml to understand:\n- What features are enabled by default\n- What each feature provides\n\nUse: `cargo metadata --format-version=1 | jq '.packages[] | select(.name == \"<crate>\") | .features'`\n\n### 3b: Try disabling default features\n\nModify the dependency in `quickwit/Cargo.toml`:\n\nFrom:\n```toml\nsome-crate = { version = \"1.0\" }\n```\n\nTo:\n```toml\nsome-crate = { version = \"1.0\", default-features = false }\n```\n\n### 3c: Run cargo check\n\nRun: `cargo check --workspace` (or target specific packages for faster feedback)\n\nIf compilation fails:\n1. Read the error messages to identify which features are needed\n2. Add only the required features explicitly:\n   ```toml\n   some-crate = { version = \"1.0\", default-features = false, features = [\"needed-feature\"] }\n   ```\n3. Re-run cargo check\n\n### 3d: Binary search for minimal features\n\nIf there are many default features, use binary search:\n1. Start with no features\n2. If it fails, add half the default features\n3. Continue until you find the minimal set\n\n## Step 4: Document findings\n\nFor each dependency analyzed, report:\n- Original configuration\n- New configuration (if changed)\n- Features that were removed\n- Any features that are required\n\n## Step 5: Verify full build\n\nAfter all changes, run:\n```bash\ncargo check --workspace --all-targets\ncargo test --workspace --no-run\n```\n\n## Common Patterns\n\n### Serde\nOften only needs `derive`:\n```toml\nserde = { version = \"1.0\", default-features = false, features = [\"derive\", \"std\"] }\n```\n\n### Tokio\nIdentify which runtime features are actually used:\n```toml\ntokio = { version = \"1.0\", default-features = false, features = [\"rt-multi-thread\", \"macros\", \"sync\"] }\n```\n\n### Reqwest\nOften doesn't need all TLS backends:\n```toml\nreqwest = { version = \"0.11\", default-features = false, features = [\"rustls-tls\", \"json\"] }\n```\n\n## Rollback\n\nIf changes cause issues:\n```bash\ngit checkout quickwit/Cargo.toml\ncargo check --workspace\n```\n\n## Tips\n\n- Start with large crates that have many default features (tokio, reqwest, hyper)\n- Use `cargo bloat --crates` to identify large dependencies\n- Check `cargo tree -d` for duplicate dependencies that might indicate feature conflicts\n- Some features are needed only for tests - consider using `[dev-dependencies]` features\n\n<!-- File: .claude/skills/simple-pr/SKILL.md (Tokens: ~418 | Category: anthropic-skill) -->\n---\nname: simple-pr\ndescription: Create a simple PR from staged changes with an auto-generated commit message\ndisable-model-invocation: true\n---\n\n# Simple PR\n\nFollow these steps to create a simple PR from staged changes:\n\n## Step 1: Check workspace state\n\nRun: `git status`\n\nVerify that all changes have been staged (no unstaged changes). If there are unstaged changes, abort and ask the user to stage their changes first with `git add`.\n\nAlso verify that we are on the `main` branch. If not, abort and ask the user to switch to main first.\n\n## Step 2: Ensure main is up to date\n\nRun: `git pull origin main`\n\nThis ensures we're working from the latest code.\n\n## Step 3: Review staged changes\n\nRun: `git diff --cached`\n\nReview the staged changes to understand what the PR will contain.\n\n## Step 4: Generate commit message\n\nBased on the staged changes, generate a concise commit message (1-2 sentences) that describes the \"why\" rather than the \"what\".\n\nDisplay the proposed commit message to the user and ask for confirmation before proceeding.\n\n## Step 5: Create a new branch\n\nGet the git username: `git config user.name | tr ' ' '-' | tr '[:upper:]' '[:lower:]'`\n\nCreate a short, descriptive branch name based on the changes (e.g., `fix-typo-in-readme`, `add-retry-logic`, `update-deps`).\n\nCreate and checkout the branch: `git checkout -b {username}/{short-descriptive-name}`\n\n## Step 6: Commit changes\n\nCommit with the message from step 3:\n```\ngit commit -m \"{commit-message}\"\n```\n\n## Step 7: Push and open a PR\n\nPush the branch and open a PR:\n```\ngit push -u origin {branch-name}\ngh pr create --title \"{commit-message-title}\" --body \"{longer-description-if-needed}\"\n```\n\nReport the PR URL to the user when complete.\n\n<!-- File: .claude/skills/update-changelog/SKILL.md (Tokens: ~1158 | Category: anthropic-skill) -->\n---\nname: update-changelog\ndescription: Update CHANGELOG.md with merged PRs since the last changelog update, categorized by type\n---\n\n# Update Changelog\n\nThis skill updates CHANGELOG.md with merged PRs that aren't already listed.\n\n## Step 1: Determine the changelog scope\n\nRead `CHANGELOG.md` to identify the current unreleased version section at the top (e.g., `Tantivy 0.26 (Unreleased)`).\n\nCollect all PR numbers already mentioned in the unreleased section by extracting `#NNNN` references.\n\n## Step 2: Find merged PRs not yet in the changelog\n\nUse `gh` to list recently merged PRs from the upstream repo:\n\n```bash\ngh pr list --repo quickwit-oss/tantivy --state merged --limit 100 --json number,title,author,labels,mergedAt\n```\n\nFilter out any PRs whose number already appears in the unreleased section of the changelog.\n\n## Step 3: Consolidate related PRs\n\nBefore categorizing, group PRs that belong to the same logical change. This is critical for producing a clean changelog. Use PR descriptions, titles, cross-references, and the files touched to identify relationships.\n\n**Merge follow-up PRs into the original:**\n- If a PR is a bugfix, refinement, or follow-up to another PR in the same unreleased cycle, combine them into a single changelog entry with multiple `[#N](url)` links.\n- Also consolidate PRs that touch the same feature area even if not explicitly linked — e.g., a PR fixing an edge case in a new API should be folded into the entry for the PR that introduced that API.\n\n**Filter out bugfixes on unreleased features:**\n- If a bugfix PR fixes something introduced by another PR in the **same unreleased version**, it must NOT appear as a separate Bugfixes entry. Instead, silently fold it into the original feature/improvement entry. The changelog should describe the final shipped state, not the development history.\n- To detect this: check if the bugfix PR references or reverts changes from another PR in the same release cycle, or if it touches code that was newly added (not present in the previous release).\n\n## Step 4: Review the actual code diff\n\n**Do not rely on PR titles or descriptions alone.** For every candidate PR, run `gh pr diff <number> --repo quickwit-oss/tantivy` and read the actual changes. PR titles are often misleading — the diff is the source of truth.\n\n**What to look for in the diff:**\n- Does it change observable behavior, public API surface, or performance characteristics?\n- Is the change something a user of the library would notice or need to know about?\n- Could the change break existing code (API changes, removed features)?\n\n**Skip PRs where the diff reveals the change is not meaningful enough for the changelog** — e.g., cosmetic renames, trivial visibility tweaks, test-only changes, etc.\n\n## Step 5: Categorize each PR group\n\nFor each PR (or consolidated group) that survived the diff review, determine its category:\n\n- **Bugfixes** — fixes to behavior that existed in the **previous release**. NOT fixes to features introduced in this release cycle.\n- **Features/Improvements** — new features, API additions, new options, improvements that change user-facing behavior or add new capabilities.\n- **Performance** — optimizations, speed improvements, memory reductions. **If a PR adds new API whose primary purpose is enabling a performance optimization, categorize it as Performance, not Features.** The deciding question is: does a user benefit from this because of new functionality, or because things got faster/leaner? For example, a new trait method that exists solely to enable cheaper intersection ordering is Performance, not a Feature.\n\nIf a PR doesn't clearly fit any category (e.g., CI-only changes, internal refactors with no user-facing impact, dependency bumps with no behavior change), skip it — not everything belongs in the changelog.\n\nWhen unclear, use your best judgment or ask the user.\n\n## Step 6: Format entries\n\nEach entry must follow this exact format:\n\n```\n- Description [#NUMBER](https://github.com/quickwit-oss/tantivy/pull/NUMBER)(@author)\n```\n\nRules:\n- The description should be concise and describe the user-facing change (not the implementation). Describe the final shipped state, not the incremental development steps.\n- Use sub-categories with bold headers when multiple entries relate to the same area (e.g., `- **Aggregation**` with indented entries beneath). Follow the existing grouping style in the changelog.\n- Author is the GitHub username from the PR, prefixed with `@`. For consolidated entries, include all contributing authors.\n- For consolidated PRs, list all PR links in a single entry: `[#100](url) [#110](url)` (see existing entries for examples).\n\n## Step 7: Present changes to the user\n\nShow the user the proposed changelog entries grouped by category **before** editing the file. Ask for confirmation or adjustments.\n\n## Step 8: Update CHANGELOG.md\n\nInsert the new entries into the appropriate sections of the unreleased version block. If a section doesn't exist yet, create it following the order: Bugfixes, Features/Improvements, Performance.\n\nAppend new entries at the end of each section (before the next section header or version header).\n\n## Step 9: Verify\n\nRead back the updated unreleased section and display it to the user for final review.\n</yakaai_skills>"}