# Technical Documentation: valeriansaliou/sonic
> โน๏ธ **Provenance:** Multi-Tier Ingestion ยท [CodeWiki Reference](https://codewiki.google/github.com/valeriansaliou/sonic)
### Overview (`README.md`)
# Sonic
[](https://deps.rs/repo/github/valeriansaliou/sonic) [](https://www.buymeacoffee.com/valeriansaliou)
**Sonic is a fast, lightweight and schema-less search backend. It ingests search texts and identifier tuples that can then be queried against in a microsecond's time.**
Sonic can be used as a simple alternative to super-heavy and full-featured search backends such as Elasticsearch in some use-cases. It is capable of normalizing natural language search queries, auto-completing a search query and providing the most relevant results for a query. Sonic is an identifier index, rather than a document index; when queried, it returns IDs that can then be used to refer to the matched documents in an external database.
A strong attention to performance and code cleanliness has been given when designing Sonic. It aims at being crash-free, super-fast and puts minimum strain on server resources (our measurements have shown that Sonic - when under load - responds to search queries in the ฮผs range, eats ~30MB RAM and has a low CPU footprint; [see our benchmarks](https://github.com/valeriansaliou/sonic#how-fast--lightweight-is-it)).
_Tested at Rust version: `rustc 1.98.0 (88d9e12ae 2026-08-18)`_
**๐ซ๐ท Crafted in Nantes, France.**
**๐ฐ The Sonic project was initially announced in [a post on my personal journal](https://journal.valeriansaliou.name/announcing-sonic-a-super-light-alternative-to-elasticsearch/).**
> _ยซ Sonic ยป is the mascot of the Sonic project. I drew it to look like a psychedelic hipster hedgehog._
## Who uses it?
 |
 |
| Crisp |
Scrumpy |
_๐ You use Sonic and you want to be listed there? [Contact me](https://valeriansaliou.name/)._
## Demo
Sonic is integrated in all Crisp search products on the [Crisp](https://crisp.chat/) platform. It is used to index half a billion objects on a $5/mth 1-vCPU SSD cloud server (as of 2019). Crisp users use it to search in their messages, conversations, contacts, helpdesk articles and more.
**You can test Sonic live on: [Crisp Helpdesk](https://help.crisp.chat/), and get an idea of the speed and relevance of Sonic search results. You can also test search suggestions from there: start typing at least 2 characters for a word, and get suggested a full word (press the tab key to expand suggestion). _Both search and suggestions are powered by Sonic._**
> _Sonic fuzzy search in helpdesk articles at its best. Lookup for any word or group of terms, get results instantly._
## Features
* **Search terms are stored in collections, organized in buckets**; you may use a single bucket, or a bucket per user on your platform if you need to search in separate indexes.
* **Search results return object identifiers**, that can be resolved from an external database if you need to enrich the search results. This makes Sonic a simple word index, that points to identifier results. Sonic doesn't store any direct textual data in its index, but it still holds a word graph for auto-completion and typo corrections.
* **Search query typos are corrected** if there are not enough exact-match results for a given word in a search query, Sonic tries to correct the word and tries against alternate words. You're allowed to make mistakes when searching.
* **Insert and remove items in the index**; index-altering operations are light and can be committed to the server while it is running. A background tasker handles the job of consolidating the index so that the entries you have pushed or popped are quickly made available for search.
* **Auto-complete any word** in real-time via the suggest operation. This helps build a snappy word suggestion feature in your end-user search interface.
* **Full Unicode compatibility** on 80+ most spoken languages in the world. Sonic removes useless stop words from any text (eg. 'the' in English), after guessing the text language. This ensures any searched or ingested text is clean before it hits the index; [see languages](https://github.com/valeriansaliou/sonic#which-text-languages-are-supported).
* **Simple protocol (Sonic Channel)**, that let you search your index, manage data ingestion (push in the index, pop from the index, flush a collection, flush a bucket, etc.) and perform administrative actions. Sonic Channel was designed to be lightweight on resources and simple to integrate with; [read protocol specification](https://github.com/valeriansaliou/sonic/blob/master/PROTOCOL.md).
* **Easy-to-use libraries**, that let you connect to Sonic from your apps; [see libraries](https://github.com/valeriansaliou/sonic#-sonic-channel-libraries).
## How to use it?
### Installation
Sonic is built in Rust. To install it, either download a version from the [Sonic releases](https://github.com/valeriansaliou/sonic/releases) page, use `cargo install` or pull the source code from `master`.
**๐ Install from packages:**
Sonic provides [pre-built packages](https://packagecloud.io/valeriansaliou/sonic) for Debian-based systems (Debian, Ubuntu, etc.).
**Important: Sonic only provides 64 bits packages targeting Debian 12 for now (codename: `bookworm`). You might still be able to use them on other Debian versions, as well as Ubuntu (although they rely on a specific `glibc` version that might not be available on older or newer systems).**
First, add the Sonic APT repository (eg. for Debian `bookworm`):
```bash
echo "deb [signed-by=/usr/share/keyrings/valeriansaliou_sonic.gpg] https://packagecloud.io/valeriansaliou/sonic/debian/ bookworm main" > /etc/apt/sources.list.d/valeriansaliou_sonic.list
```
```bash
curl -fsSL https://packagecloud.io/valeriansaliou/sonic/gpgkey | gpg --dearmor -o /usr/share/keyrings/valeriansaliou_sonic.gpg
```
```bash
apt-get update
```
Then, install the Sonic package:
```bash
apt-get install sonic
```
Then, edit the pre-filled Sonic configuration file:
```bash
nano /etc/sonic.cfg
```
Finally, restart Sonic:
```bash
service sonic restart
```
**๐ Install from source:**
If you pulled the source code from Git, you can build it using `cargo`:
```bash
cargo build --locked --release
```
You can find the built binaries in the `./target/release` directory.
_Install `build-essential`, `clang`, `libclang-dev`, `libc6-dev`, `g++` and `llvm-dev` to be able to compile the required RocksDB dependency._
Note that the following optional features can be enabled upon building Sonic: `allocator-jemalloc`, `tokenizer-chinese` and `tokenizer-japanese` (some might be already enabled by default).
**๐ Install from Cargo:**
You can install Sonic directly with `cargo install`:
```bash
cargo install sonic-server
```
Ensure that your `$PATH` is properly configured to source the Crates binaries, and then run Sonic using the `sonic` command.
_Install `build-essential`, `clang`, `libclang-dev`, `libc6-dev`, `g++` and `llvm-dev` to be able to compile the required RocksDB dependency._
**๐ Install from Docker Hub:**
You might find it convenient to run Sonic via Docker. You can find the pre-built Sonic image on Docker Hub as [valeriansaliou/sonic](https://hub.docker.com/r/valeriansaliou/sonic/).
First, pull the `valeriansaliou/sonic` image:
```bash
docker pull valeriansaliou/sonic:v1.10.2
```
Then, seed it a configuration file and run it (replace `/path/to/your/sonic/config.cfg` with the path to your configuration file):
```bash
docker run -p 1491:1491 -v /path/to/your/sonic/config.cfg:/etc/sonic/config.toml -v /path/to/your/sonic/store/:/var/lib/sonic/store/ valeriansaliou/sonic:v1.4.9
```
In the configuration file, ensure that:
* `channel.inet` is set to `0.0.0.0:1491` (this lets Sonic be reached from outside the container)
* `store.kv.path` is set to `/var/lib/sonic/store/kv/` (this lets the external KV store directory be reached by Sonic)
* `store.fst.path` is set to `/var/lib/sonic/store/fst/` (this lets the external FST store directory be reached by Sonic)
Sonic will be reachable from `tcp://localhost:1491`.
**๐ Install from another source (non-official):**
Other installation sources are available:
* **Homebrew (macOS)**: `brew install sonic` ([see formula](https://formulae.brew.sh/formula/sonic))
_Note that those sources are non-official, meaning that they are not owned nor maintained by the Sonic project owners. The latest Sonic version available on those sources might be outdated, in comparison to the latest version available through the Sonic project._
### Configuration
Use the sample [config.cfg](https://github.com/valeriansaliou/sonic/blob/master/config.cfg) configuration file and adjust it to your own environment.
_If you are looking to fine-tune your configuration, you may read our [detailed configuration documentation](https://github.com/valeriansaliou/sonic/blob/master/CONFIGURATION.md)._
### Run Sonic
Sonic can be run as such:
`./sonic -c /path/to/config.cfg`
## Perform searches and manage objects
Both searches and object management (i.e. data ingestion) is handled via the Sonic Channel protocol only. As we want to keep things simple with Sonic (similarly to how Redis does it), Sonic does not offer a HTTP endpoint or similar; connecting via Sonic Channel is the way to go when you need to interact with the Sonic search database.
Sonic distributes official libraries, that let you integrate Sonic to your apps easily. Click on a library below to see library integration documentation and code.
_If you are looking for details on the raw Sonic Channel TCP-based protocol, you can read our [detailed protocol documentation](https://github.com/valeriansaliou/sonic/blob/master/PROTOCOL.md). It can prove handy if you are looking to code your own Sonic Channel library._
### ๐ฆ Sonic Channel Libraries
#### 1๏ธโฃ Official Libraries
Sonic distributes official Sonic integration libraries for your programming language (official means that those libraries have been reviewed and validated by a core maintainer):
* **NodeJS**:
* **[node-sonic-channel](https://www.npmjs.com/package/sonic-channel)** by [@valeriansaliou](https://github.com/valeriansaliou)
* Battle tested by [Crisp](https://crisp.chat/)
* **Rust**:
* **[sonic_client](https://crates.io/crates/sonic_client)** by [@RemiBardon](https://github.com/RemiBardon)
* Modern and flexible implementation, maintained alongside Sonic
* **PHP**:
* **[psonic](https://github.com/ppshobi/psonic)** by [@ppshobi](https://github.com/ppshobi)
#### 2๏ธโฃ Community Libraries
You can find below a list of Sonic integrations provided by the community (many thanks to them!):
* **Rust**:
* **[sonic-channel](https://github.com/pleshevskiy/sonic-channel)** by [@pleshevskiy](https://github.com/pleshevskiy)
* ~~**[sonic_client](https://github.com/FrontMage/sonic_client)** by [@FrontMage](https://github.com/FrontMage)~~
* Replaced by **[sonic_client](https://crates.io/crates/sonic_client)** by [@RemiBardon](https://github.com/RemiBardon), many thanks again for the ownership transfer!
* **Python**:
* **[asonic](https://github.com/moshe/asonic)** by [@moshe](https://github.com/moshe)
* **[python-sonic-client](https://github.com/xmonader/python-sonic-client)** by [@xmonader](https://github.com/xmonader)
* **[pysonic-channel](https://github.com/AlongWY/pysonic)** by [@AlongWY](https://github.com/AlongWY)
* **Ruby**:
* **[sonic-ruby](https://github.com/atipugin/sonic-ruby)** by [@atipugin](https://github.com/atipugin)
* **Go**:
* **[go-sonic](https://github.com/expectedsh/go-sonic)** by [@alexisvisco](https://github.com/alexisvisco)
* **[go-sonic](https://github.com/OGKevin/go-sonic)** by [@OGKevin](https://github.com/OGKevin)
* **PHP**:
* **[php-sonic](https://github.com/php-sonic/php-sonic)** by [@touhonoob](https://github.com/touhonoob)
* **[laravel-scout-sonic](https://github.com/james2doyle/laravel-scout-sonic)** by [@james2doyle](https://github.com/james2doyle)
* **Java**:
* **[java-sonic](https://github.com/twohou/java-sonic)** by [@touhonoob](https://github.com/touhonoob)
* **[jsonic](https://github.com/alohaking/jsonic)** by [@alohaking](https://github.com/alohaking)
* **Deno**:
* **[deno-sonic](https://github.com/erfanium/deno_sonic)** by [@erfanium](https://github.com/erfanium)
* **Bun**:
* **[sonic-bun](https://github.com/emilianscheel/sonic-bun)** by [@emilianscheel](https://github.com/emilianscheel)
* **Elixir**:
* **[sonix](https://github.com/imerkle/sonix)** by [@imerkle](https://github.com/imerkle)
* **Crystal**:
* **[sonic-crystal](https://github.com/babelian/sonic-crystal)** by [@babelian](https://github.com/babelian)
* **Nim**:
* **[nim-sonic-client](https://github.com/xmonader/nim-sonic-client)** by [@xmonader](https://github.com/xmonader)
* **.NET**:
* **[nsonic](https://github.com/spikensbror-dotnet/nsonic)** by [@spikensbror](https://github.com/spikensbror)
_โน๏ธ Cannot find the library for your programming language? Build your own and be referenced here! ([contact me](https://valeriansaliou.name/))_
## Which text languages are supported?
Sonic supports a wide range of languages in its lexing system. If a language is not in this list, you will still be able to push this language to the search index, but stop-words will not be eluded, which could lead to lower-quality search results.
**The languages supported by the lexing system are:**
* ๐ฟ๐ฆ Afrikaans
* ๐ธ๐ฆ Arabic
* ๐ฆ๐ฒ Armenian
* ๐ฆ๐ฟ Azerbaijani
* ๐ง๐ฉ Bengali
* ๐ง๐ฌ Bulgarian
* ๐ฒ๐ฒ Burmese
* ๐ณ Catalan
* ๐จ๐ณ Chinese (Simplified)
* ๐น๐ผ Chinese (Traditional)
* ๐ญ๐ท Croatian
* ๐จ๐ฟ Czech
* ๐ฉ๐ฐ Danish
* ๐ณ๐ฑ Dutch
* ๐ฌ๐ง English
* ๐ณ Esperanto
* ๐ช๐ช Estonian
* ๐ซ๐ฎ Finnish
* ๐ซ๐ท French
* ๐ฌ๐ช Georgian
* ๐ฉ๐ช German
* ๐ฌ๐ท Greek
* ๐ฎ๐ณ Gujarati
* ๐ฎ๐ฑ Hebrew
* ๐ฎ๐ณ Hindi
* ๐ญ๐บ Hungarian
* ๐ฎ๐ฉ Indonesian
* ๐ฎ๐น Italian
* ๐ฏ๐ต Japanese
* ๐ฎ๐ณ Kannada
* ๐ฐ๐ญ Khmer
* ๐ฐ๐ท Korean
* ๐ณ Latin
* ๐ฑ๐ป Latvian
* ๐ฑ๐น Lithuanian
* ๐ฎ๐ณ Malayalam
* ๐ฎ๐ณ Marathi
* ๐ณ๐ต Nepali
* ๐ฎ๐ท Persian
* ๐ต๐ฑ Polish
* ๐ต๐น Portuguese
* ๐ฎ๐ณ Punjabi
* ๐ท๐บ Russian
* ๐ท๐ธ Serbian
* ๐ธ๐ฐ Slovak
* ๐ธ๐ฎ Slovene
* ๐ช๐ธ Spanish
* ๐ธ๐ช Swedish
* ๐ต๐ญ Tagalog
* ๐ฎ๐ณ Tamil
* ๐น๐ญ Thai
* ๐น๐ท Turkish
* ๐บ๐ฆ Ukrainian
* ๐ต๐ฐ Urdu
* ๐ป๐ณ Vietnamese
* ๐ด๓ ง๓ ข๓ ท๓ ฌ๓ ณ๓ ฟ Welsh
* ๐ฎ๐ฑ Yiddish
* ๐ฟ๐ฆ Zulu
Only a subset of those languages support stemming. Sonic uses [Snowball stemming algorithms] via [the `waken_snowball` crate]. See the crateโs documentation for a list of supported languages.
[Snowball stemming algorithms]: https://snowballstem.org/ "Snowball homepage"
[the `waken_snowball` crate]: https://crates.io/crates/waken_snowball "waken_snowball - crates.io: Rust Package Registry"
## How fast & lightweight is it?
Sonic was built for [Crisp](https://crisp.chat/) from the start. As Crisp was growing and indexing more and more search data into a full-text search SQL database, we decided it was time to switch to a proper search backend system. When reviewing Elasticsearch (ELS) and others, we found those were full-featured heavyweight systems that did not scale well with Crisp's freemium-based cost structure.
At the end, we decided to build our own search backend, designed to be simple and lightweight on resources.
You can run function-level benchmarks with the command: `cargo bench --features benchmark`
### ๐ฉโ๐ฌ Benchmark #1
#### โก๏ธ Scenario
We performed an extract of all messages from the Crisp team used for [Crisp](https://crisp.chat/) own customer support.
We want to import all those messages into a clean Sonic instance, and then perform searches on the index we built. We will measure the time that Sonic spent executing each operation (ie. each `PUSH` and `QUERY` commands over Sonic Channel), and group results per 1,000 operations (this outputs a mean time per 1,000 operations).
#### โก๏ธ Context
**Our benchmark is ran on the following computer:**
* **Device**: MacBook Pro (Retina, 15-inch, Mid 2014)
* **OS**: MacOS 10.14.3
* **Disk**: 512GB SSD (formatted under the AFS file system)
* **CPU**: 2.5 GHz Intel Core i7
* **RAM**: 16 GB 1600 MHz DDR3
**Sonic is compiled as following:**
* **Sonic version**: 1.0.1
* **Rustc version**: `rustc 1.35.0-nightly (719b0d984 2019-03-13)`
* **Compiler flags**: `release` profile (`-03` with `lto`)
**Our dataset is as such:**
* **Number of objects**: ~1,000,000 messages
* **Total size**: ~100MB of raw message text (this does not account for identifiers and other metas)
#### โก๏ธ Scripts
**The scripts we used to perform the benchmark are:**
1. **PUSH script**: [sonic-benchmark_batch-push.js](https://gist.github.com/valeriansaliou/e5ab737b28601ebd70483f904d21aa09)
2. **QUERY script**: [sonic-benchmark_batch-query.js](https://gist.github.com/valeriansaliou/3ef8315d7282bd173c2cb9eba64fa739)
#### โฌ Results
**Our findings:**
* We imported ~1,000,000 messages of dynamic length (some very long, eg. emails);
* Once imported, the search index weights 20MB (KV) + 1.4MB (FST) on disk;
* CPU usage during import averaged 75% of a single CPU core;
* RAM usage for the Sonic process peaked at 28MB during our benchmark;
* We used a single Sonic Channel TCP connection, which limits the import to a single thread (we could have load-balanced this across as many Sonic Channel connections as there are CPUs);
* We get an import RPS approaching 4,000 operations per second (per thread);
* We get a search query RPS approaching 1,000 operations per second (per thread);
* On the hyper-threaded 4-cores CPU used, we could have parallelized operations to 8 virtual cores, thus theoretically increasing the import RPS to 32,000 operations / second, while the search query RPS would be increased to 8,000 operations / second (we may be SSD-bound at some point though);
**Compared results per operation (on a single object):**
We took a sample of 8 results from our batched operations, which produced a total of 1,000 results (1,000,000 items, with 1,000 items batched per measurement report).
_This is not very scientific, but it should give you a clear idea of Sonic performances._
**Time spent per operation:**
Operation | Average | Best | Worst
--------- | ------- | ----- | -----
PUSH | 275ฮผs | 190ฮผs | 363ฮผs
QUERY | 880ฮผs | 852ฮผs | 1ms
**Batch PUSH results as seen from our terminal (from initial index of: 0 objects):**
**Batch QUERY results as seen from our terminal (on index of: 1,000,000 objects):**
## Limitations
* **Indexed data limits**: Sonic is designed for large search indexes split over thousands of search buckets per collection. An IID (ie. Internal-ID) is stored in the index as a 32 bits number, which theoretically allow up to ~4.2 billion objects to be indexed (ie. OID) per bucket. We've observed storage savings of 30% to 40%, which justifies the trade-off on large databases (versus Sonic using 64 bits IIDs). Also, Sonic only keeps the N most recently pushed results for a given word, in a sliding window way (the sliding window width can be configured).
* **Search query limits**: Sonic Natural Language Processing system (NLP) does not work at the sentence-level, for storage compactness reasons (we keep the FST graph shallow as to reduce time and space complexity). It works at the word-level, and is thus able to search per-word and can predict a word based on user input, though it is unable to predict the next word in a sentence.
* **Real-time limits**: the FST needs to be rebuilt every time a word is pushed or popped from the bucket graph. As this is quite heavy, Sonic batches rebuild cycles. If you have just pushed a new word to the index and you are not seeing it in the `SUGGEST` command yet, wait for the next rebuild cycle to kick-in, or force it with `TRIGGER consolidate` in a `control` channel.
* **Interoperability limits**: The Sonic Channel protocol is the only way to read and write search entries to the Sonic search index. Sonic does not expose any HTTP API. Sonic Channel has been designed with performance and minimal network footprint in mind. If you need to access Sonic from an unsupported programming language, you can either [open an issue](https://github.com/valeriansaliou/sonic/issues/new) or look at the reference [node-sonic-channel](https://github.com/valeriansaliou/node-sonic-channel) implementation and build it in your target programming language.
* **Hardware limits**: Sonic performs the search on the file-system directly; ie. it does not fit the index in RAM. A search query results in a lot of random accesses on the disk, which means that it will be quite slow on old-school HDDs and super-fast on newer SSDs. Do store the Sonic database on SSD-backed file systems only.
## ๐ฅ Report A Vulnerability
If you find a vulnerability in Sonic, you are more than welcome to report it directly to [@valeriansaliou](https://github.com/valeriansaliou) by sending an encrypted email to [valerian@valeriansaliou.name](mailto:valerian@valeriansaliou.name). Do not report vulnerabilities in public GitHub issues, as they may be exploited by malicious people to target production servers running an unpatched Sonic instance.
**โ ๏ธ You must encrypt your email using [@valeriansaliou](https://github.com/valeriansaliou) GPG public key: [:key:valeriansaliou.gpg.pub.asc](https://valeriansaliou.name/files/keys/valeriansaliou.gpg.pub.asc).**
---
### Overview (`client/README.md`)
# `sonic_client`, the official Rust client for Sonic
This crate is a work in progress. API is subject to changes until it reaches
`1.0.0`. We wonโt keep it in [ZeroVer](https://0ver.org/), donโt worry, but for
now some pieces still need some polishing.
## Features
- **Lock free**: No `Mutex`, `RwLock`โฆ
- **Performant**: TCP streams are multiplexed using OS event queues
(via [`mio`]) for optimal performance. Parsing is also minimal and aborts
early to avoid needless computations.
- **Lighweight**: The library has only 7 dependencies, which are either
essential or adding good performance benefits.
- **Almost inert when idle**: If no message should be sent and if no message is
expected, `sonic_client` doesnโt execute a single line of code. No CPU time
is wasted, it does the bare minimum.
- **Ergonomic**: All methods accept generic types so you donโt have to convert
types or `clone` data needlessly.
- **Type-safe**: By construction, you canโt do unsound operations.
- **Command buffering**: Long commands are transparently split into smaller
ones, depending on your Sonic server configuration. This way, you can `PUSH`
megabytes of data without having to worry about Sonic buffer limits.
- **Production-ready**: `sonic_client` can be used at any scale. We benchmarked
it by ingesting the English portion of Wikipedia, if thatโs a good enough
proof for you :)
- **Future-proof**: The library exposes a low-level API which you can use to
send any command. If the Sonicย Channel protocol gets updated but you canโt
bump `sonic_client` for some reason, youโll still have a way to do the new
stuff.
- **Resilient**: One unexpected result or bad UTF-8 line does not prevent other
events from being processed.
- **No async runtime assumption**: `sonic_client` can be used fully
synchronously, but it also exposes an `async` API for use with any runtime.
- **Safe**: No `unsafe`, and `unwrap`s/`expect`s for performance in rare cases.
- **Flexible logging**: By default, `sonic_client` compiles with no log at all.
However, you can enable `std`, `log` or `tracing` logs using feature flags.
[`mio`]: https://crates.io/crates/mio "mio on crates.io"
## Why another Rust client for Sonic?
[sonic-channel] by [@pleshevskiy] used to be the recommended Rust client, but
it has been archived on Mar 1, 2023 signifying it wonโt get updated in the
future. While trying to use it in our benchmarks we noticed `PONG` is
unsupported (although `PING` isโฆ causing a failure every time itโs called) but
more importantly ingested text is not escaped. This means any text containing
a `"` breaks request parsing and causes ingestion to fail.
[sonic-channel]: https://github.com/pleshevskiy/sonic-channel
[@pleshevskiy]: https://github.com/pleshevskiy
[sonic_client] by [@FrontMage] was also recommended by Sonicโs README, but while
using it we noticed it logs passwords on `START` (!). In addition, the README
says the crate is under development but last commit was made on Apr 11, 2019 so
itโs safe to assume the crate is abandonned.
[sonic_client]: https://github.com/FrontMage/sonic_client
[@FrontMage]: https://github.com/FrontMage
To address those issues and as part of modernizations we decided to insource
the official Rust client. This will ensure its lifetime aligns with Sonicโs,
and tests will ensure it always works (and at scale).
After reaching out to [@FrontMage], he transferred us the ownership of his
(abandonned) crate so we can reuse the name and avoid confusing situations.
This is why this crate is published as [pkg:cargo/sonic_client].
[pkg:cargo/sonic_client]: https://crates.io/crates/sonic_client "โsonic_clientโ on crates.io"
---
### Overview (`packaging/README.md`)
# Packaging
This file contains quick reminders and notes on how to package Sonic.
We consider here the packaging flow of Sonic for Linux.
## Requirements
It is highly recommended to [install Task](https://taskfile.dev/docs/installation) as it simplifies all maintenance operations thanks to hand-crafted helper tools.
If you really donโt want to install it, we try our best to always provide alternatives. Check [`Taskfile.dist.yaml`](./Taskfile.dist.yaml) to see what `task` calls internally.
## Releasing Sonic core (library)
1. Make sure [`core/CHANGELOG.md`](./core/CHANGELOG.md) is up to date (changes should be in the โUnreleasedโ section).
- If not up-to-date, run `task changelog:prepare -- core`[^changelog-core] then follow the instructions.
1. Run `task release:core -- major|minor|path`[^release-core], following [Semantic Versioning](https://semver.org/).
[^changelog-core]: Alternative: `./scripts/changelog_prepare.sh core`
[^release-core]: Alternative: `./scripts/make_release.sh core major|minor|path`
CD pipelines will then publish the library on [crates.io](https://crates.io).
Check the [โActionsโ tab on GitHub](https://github.com/valeriansaliou/sonic/actions/workflows/release-core.yml) to see the progress.
## Releasing Sonic server (binary)
1. If you made changes to the core, release it first.
CD pipelines will take 6โ10 minutes to run, you have to wait.
If you donโt, `cargo publish -p sonic-server` will fail because it cannot find the version of `sonic-core` the server depends on. This will abort the entire CD pipeline and force you to manually fix your mess. You donโt want that.
1. Make sure [`server/CHANGELOG.md`](./server/CHANGELOG.md) is up to date (changes should be in the โUnreleasedโ section).
- If not up-to-date, run `task changelog:prepare -- server`[^changelog-server] then follow the instructions.
1. Run `task release:server -- major|minor|path`[^release-server], following [Semantic Versioning](https://semver.org/).
[^changelog-server]: Alternative: `./scripts/changelog_prepare.sh server`
[^release-server]: Alternative: `./scripts/make_release.sh server major|minor|path`
CD pipelines will then build and release Sonic (server) on [crates.io](https://crates.io), GitHub, Dockerย Hub and Packagecloud.
Check the [โActionsโ tab on GitHub](https://github.com/valeriansaliou/sonic/actions/workflows/release-server.yml) to see the progress.
---
### Overview (`server/tests/README.md`)
# Sonic Server integration tests
Those tests are not exhaustive, they just ensure Sonic server works end-to-end.
For more exhaustive tests, see Sonic core.
---
### Search Results Ranking (`docs/search-results-ranking.md`)
# Sonic search results ranking
In search engines like Sonic, itโs common for results to be ranked using
[tf-idf] or its โevolutionโ [BM25]. However, to keep a very small index Sonic
doesnโt store some information thatโs necessary to compute [term frequency], a
core element of both [tf-idf] and [BM25].
Out of simplicity, before `v1.7.0` Sonic used to return results that contained
all search terms (with some fuzzy matching), sorted by reverse ingestion order
(i.e. most recent documents first). While this was good enough for some use
cases, discrepancies in ingestion and query languages would silently affect
which words were considered stopwords and prevent some documents[^doc] from
showing up in results (because of the implicit `AND`)[^1].
[^doc]: Since this document is about information retrieval as a general concept, this document will use the term โdocumentโ to refer to Sonic objects as itโs the strandard term.
[^1]: For example, see [issue #262](https://github.com/valeriansaliou/sonic/issues/262).
In `v1.7.0`, we reworked Sonicโs ranking algorithm to yield better results.
## Ranking algorithm
For each search term, Sonic looks for documents containing exactly that search
term, then it performs fuzzy matching until enough results are found. For each
term, Sonic keeps track of the [Levenshtein distance] of the closest word found
in each document. Once that done, a documentsโs score is calculated by summing
up its scores for each term, and documents are finally sorted by score. When
ties occur, results are sorted by reverse ingestion order[^rio].
[^rio]: This happens at no cost and is invisible in the code, but happens because itโs the order in which results are yielded by the index (on purpose).
Here is some more detailed and accurate pseudo-code:
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Algorithm properties
- At most `store.kv.retain_word_objects` are returned.
- Fuzzy matching yields at most `search.query_alternates_try` results.
- Memory usage is linear, proportional to the number of results.
- Although the pseudo-code doesnโt show it, most operations are lazy, meaning
only useful computations are executed.
- Scoring is calculated using integer values only, without a single floating
point computation, division, square root or any other expensive operation.
### Possible future improvements
- Assign a weight to each term, based on [inverse document frequency], to give
more precedence to rare terms.
- This would imply making some floating point operations, but theyโd only be
`O(len(results))`, with idf computation being `O(1)`.
- This could be done in parallel from lookups, as it is independent.
- Parallelize some operations (e.g. each term scoring)?
- Overhead might be higher than gain, weโd have to benchmark.
[tf-idf]: https://en.wikipedia.org/wiki/Tf%E2%80%93idf
[BM25]: https://en.wikipedia.org/wiki/Okapi_BM25 "โOkapi BM25โ on Wikipedia"
[term frequency]: https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Term_frequency "โTerm frequencyโ in โtf-idfโ on Wikipedia"
[inverse document frequency]: https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Inverse_document_frequency "โInverse document frequencyโ in โtf-idfโ on Wikipedia"
[Levenshtein distance]: https://en.wikipedia.org/wiki/Levenshtein_distance "โLevenshtein distanceโ on Wikipedia"
---
### Tokenizer Pattern Matching (`docs/tokenizer-pattern-matching.md`)
---
author: Rรฉmi Bardon
created: 2026-07-09
---
# Sonic tokenizerโs pattern matching
## Context
Since its very beginning, Sonic has always supported prefix and fuzzy matching.
However, at some point a bug creeped in and caused fuzzy matching to correct at
most 1 typo. This bug went unnoticed for years and users became used to it.
Letโs take a simplified example (pseudo-code):
```txt
Ingest (Sonic <= v1.7.3):
PUSH msg:1 "olivia@example.org"
โโโโโโ โโโโโโโโโโโ
PUSH msg:2 "olivio@example.org" # Lev(msg:1, msg:2) = 1
โโโโโโ โโโโโโโโโโโ
PUSH msg:3 "alicia@example.org" # Lev(msg:1, msg:3) = 2
โโโโโโ โโโโโโโโโโโ
PUSH msg:4 "olivia@example.com" # Lev(msg:1, msg:4) = 3
โโโโโโ โโโโโโโโโโโ
PUSH msg:5 "olivia and olivio"
โโโโโโ โโโ โโโโโโ
In the index: {
"olivia", "example.org", "olivio", "alicia", "example.com", "and"
}
```
Before `v1.7.0`, because typo correction was always limited to 1 character,
searching for `olivia@example.org` would yield the following results:
```txt
Search (v1.6.0):
QUERY "olivia@example.org"
โโโฌโโโ โโโโโโฌโโโโโ
1 typo 1 typo
โโโโโดโโโโ โโโโโ
olivia olivio example.org
โโ OR โโโ โ
โโโโโโ AND โโโโโ
RESULT msg:2 msg:1 # Too many results (expected: msg:1)
```
However, because 1 character differences in both the username or domain part of
an email address are very unlikely, no one ever noticed (and complained).
## The v1.7.0 problem
In v1.7.0 we fixed Sonicโs typo correction, which now allows more typos as the
word gets longer (which was supposed to happen since the beginning).
Unfortunately, this fix didnโt go unnoticed, as it only took a few hours after
deployment for [Crisp] users to start complaining about unexpected results when
querying phone numbers or email addresses.
[Crisp]: https://crisp.chat "Crisp homepage"
In `v1.7.0`, searching for `olivia@example.org` would now yield far too many
results:
```txt
Note: In `v1.7.0`, 6-letters words are allowed 1 typo. For exampleโs sake,
weโll pretend `olivia` is 7-letters long so it matches up to 2 typos. Off the
top of my head I couldnโt find better examples.
Search (v1.7.0):
QUERY "olivia@example.org"
โโโฌโโโ โโโโโโฌโโโโโ
2 typos โโโ 3 typos โโโโ
โโโโโโดโโโฌโโโโโโโโ โโโโโดโโโโโโโโโ
olivia olivio alicia example.org example.com
โโโโโโโโโดโ OR โโโ โโโโโ OR โโโโโ
โโโโโโโโ OR โโโโโโโโโ
RESULT msg:1 msg:2 msg:3 msg:4 msg:5 # Too many results (expected: msg:1)
```
Although results were ordered perfectly, users were expecting not to see fuzzy
matches at all. We didnโt want to revert the typo correction fix, as it made
sense for a ton of use cases, and had to come up with a non-breaking workaround.
## Workaround: Forcing some terms to be matched exactly, and at least once
In [Pull Request #365] and [Pull Request #367] we made changes to the `QUERY`
executor so it would consider any term containing non-prose characters (e.g.
digits, `.`, `_`, etc.) to be an identifier. Identifiers would then not be
subject to prefix nor fuzzy matching, and they would be required to be present
in all results (implicit `AND`). This was a huge win, but it still had flaws:
[Pull Request #365]: https://github.com/valeriansaliou/sonic/pull/365
[Pull Request #367]: https://github.com/valeriansaliou/sonic/pull/367
```txt
Search (v1.7.3):
QUERY "olivia@example.org"
โโโฌโโโ โโโโโโฌโโโโโ
2 typos โโ Exact โโโ
โโโโโโดโโโฌโโโโโโโโ โ
olivia olivio alicia example.org
โโโโโโโโโดโ OR โโโ โ
โโโโโ AND โโโโโ
RESULT msg:1 msg:2 msg:3 # Better than v1.7.0, but still too many results
# and arguably worse than v1.6.0 (expected: msg:1)
```
Because the tokenizer would split on `@`, `olivia` couldnโt be detected as an
identifier, and would still be fuzzy matched too broadly. Without making
changes to the tokenizer, working around this issue would have been very hacky,
so we went on and changed it.
## Solution: Making it so the tokenizer doesnโt split specific patterns
In [Pull Request #368], we made changes to the tokenizer so it could detect
some patterns (e.g. emails, phone numbers, UUIDsโฆ). For those portions of the
query, because users expect exact matches, weโd disable fuzzy matching and
force the term to be present in results.
[Pull Request #368]: https://github.com/valeriansaliou/sonic/pull/368
Unfortunately, without rebuilding Sonicโs index, the `QUERY` executor would
look for terms that are not in the index, making this a breaking change:
```txt
Index (<= v1.7.3): {
"olivia", "example.org", "olivio", "alicia", "example.com", "and"
}
Search (f290006):
QUERY "olivia@example.org"
โโโโโโโโฌโโโโโโโโโโ
Exact
โ
รธ
RESULT # Regression: no more result
```
To avoid having to make a major release for Sonic, we hid this new feature
behind an opt-in configuration flag: `tokenization.detect_special_patterns`.
At least indexes wouldnโt become useless on update, but we were back to square
one (or `v1.7.3` to be more precise):
```txt
Search (ddd6848 with `tokenization.detect_special_patterns = false`):
Same as in v1.7.3 (not great)
```
## Improvement: Making the solution non-breaking
At [Crisp] we canโt just rebuild the index because weโre bumping Sonic, because
โeven though Sonic is fastโ ingesting billions of messages still takes _hours_,
so it was time to look for a smarter idea.
If we want to get the results users expect (i.e. just `msg:1` in our example),
the tokenizer _has to_ be aware of patterns. But we also canโt force a rebuild
of the index in a non-breaking release, so we have to act in-between.
The solution we came up with is to enable `tokenization.detect_special_patterns`
by default but add `tokenization.compat_split_special_patterns` to control
whether or not they should be further split. When `true`, terms are split just
like in `v1.6.0`, but they are now marked special and we can force exact
matching. Here is an example:
```txt
Index (<= v1.7.3): {
"olivia", "example.org", "olivio", "alicia", "example.com", "and"
}
Search (v1.7.4 with `tokenization.compat_split_special_patterns = true`):
QUERY "olivia@example.org"
โโโ Special โโโโโโ
โโโฌโโโ โโโโโโฌโโโโโ
Exact Exact
โ โ
olivia example.org
โโโ AND โโโ
RESULT msg:1 # Expected result
```
In this example, the result is exactly what we want, but there is still one
edge case where a document contains both `olivia` and `example.org` (but not
`olivia@example.org`). Unfortunately itโs not possible for Sonic to work around
this case without its index being rebuilt with
`tokenization.compat_split_special_patterns = false` (which will be the default
in Sonic `v2.0.0`). Here is why it would be perfect:
```txt
Ingest (v1.7.4 with `tokenization.compat_split_special_patterns = false`):
PUSH msg:1 "olivia@example.org"
โโโโโโโโโโโโโโโโโโ
PUSH msg:2 "olivio@example.org"
โโโโโโโโโโโโโโโโโโ
PUSH msg:3 "alicia@example.org"
โโโโโโโโโโโโโโโโโโ
PUSH msg:4 "olivia@example.com"
โโโโโโโโโโโโโโโโโโ
PUSH msg:5 "olivia and olivio"
โโโโโโ โโโ โโโโโโ
In the index: {
"olivia@example.org", "olivio@example.org", "alicia@example.org",
"olivia@example.com", "olivia", "and", "olivio"
}
Search (v1.7.4 with `tokenization.compat_split_special_patterns = false`):
QUERY "olivia@example.org"
โโโ Special โโโโโโ
โ
Exact
โ
olivia@example.org
RESULT msg:1 # Perfect!
```
---
--- METRICS ---
- Files Extracted: 6
- Estimated Token Budget: ~9816 tokens
- Canonical Reference: https://codewiki.google/github.com/valeriansaliou/sonic