flexsearch Documentation

File: CHANGELOG.md Copy Chapter

Changelog

Current Version Copy Section

• Calling index.clear() on a persistent Index does not stack to the task queue by default (which executes on commit), instead it will execute immediately and return a Promise
• Added new tokenizer tolerant, inherits from strict but also matches simple typos like missing letters and swapped letters
• Improved Redis Cleanup
• Resolver: Support Result Highlighting

v0.8.2 Copy Section

• Config-Serialized Query Caches, Improved caching strategy for Document indexes and Resolver
• Resolver Async Processing Workflow (including Queuing)
• Extended Resolver Support: Worker, Persistent, Cache
• Extended Result Highlighting: Boundaries, Ellipsis, Alignment
• Improved TypeScript Typings
• Improved Stemmer Handling
• Improved Result Highlighting
• Use multi-language charset normalization as the default Encoder
• Simplified charset support for multi-language content
• Charset renamed LatinExact => Exact, LatinDefault => Default and LatinSimple => Normalize, these are universal charset presets for any languages
• Charset ArabicDefault and CyrillicDefault was removed, they are fully covered by the default universal charset presets
• Charset Charset.CjkDefault was renamed to Charset.CJK

v0.8.1 Copy Section

• Resolver Support for Documents
• Asynchronous Runtime Balancer, new option priority
• Export/Import Worker Indexes + Document Worker, new extern config options export and import
• Improved interoperability of the different build packages, including source folder
• Support custom filter function for encoder (stop-word filter)

v0.8.0 Copy Section

• Persistent indexes support for: IndexedDB (Browser), Redis, SQLite, Postgres, MongoDB, Clickhouse
• Enhanced language customization via the new Encoder class
• Result Highlighting
• Query performance achieve results up to 4.5 times faster compared to the previous generation v0.7.x by also improving the quality of results
• Enhanced support for larger indexes or larger result sets
• Improved offset and limit processing achieve up to 100 times faster traversal performance through large datasets
• Support for larger In-Memory index with extended key size (the defaults maximum keystore limit is: 2^24)
• Greatly enhanced performance of the whole text encoding pipeline
• Improved indexing of numeric content (Triplets)
• Intermediate result sets and Resolver
• Basic Resolver: and, or, xor, not, limit, offset, boost, resolve
• Improved charset collection
• New charset preset soundex which further reduces memory consumption by also increasing "fuzziness"
• Performance gain when polling tasks to the index by using "Event-Loop-Caches"
• Up to 100 times faster deletion/replacement when not using the additional "fastupdate" register
• Regex Pre-Compilation (transforms hundreds of regex rules into just a few)
• Extended support for multiple tags (DocumentIndex)
• Custom Fields ("Virtual Fields")
• Custom Filter
• Custom Score Function
• Added French language preset (stop-word filter, stemmer)
• Enhanced Worker Support
• Export / Import index in chunks
• Improved Build System + Bundler (Supported: CommonJS, ESM, Global Namespace), also the import of language packs are now supported for Node.js
• Full covering index.d.ts type definitions
• Fast-Boot Serialization optimized for Server-Side-Rendering (PHP, Python, Ruby, Rust, Java, Go, Node.js, ...)

v0.7.0 Copy Section

• Bidirectional Context (the order of words can now vary, does not increase memory when using bidirectional context)
• New memory-friendly strategy for indexes (switchable, saves up to 50% of memory for each index, slightly decrease performance)
• Better scoring calculation (one of the biggest concerns of the old implementation was that the order of arrays processed in the intersection has affected the order of relevance in the final result)
• Fix resolution (the resolution in the old implementation was not fully stretched through the whole range in some cases)
• Skip words (optionally, automatically skip words from the context chain which are too short)
• Hugely improves performance of long queries (up to 450x faster!) and also memory allocation (up to 250x less memory)
• New fast-update strategy (optionally, hugely improves performance of all updates and removals of indexed contents up to 2850x)
• Improved auto-balanced cache (keep and expire cache by popularity)
• Append contents to already existing entries (already indexed documents or contents)
• New method "contain" to check if an ID was already indexed
• Access documents directly from internal store (read/write)
• Suggestions are hugely improved, falls back from context search all the way down to single term match
• Document descriptor has now array support (optionally adds array entries via the new append under the hood to provide a unique relevance context for each entry)
• Document storage handler gets improved
• Results from document index now grouped by field (this is one of the few bigger breaking changes which needs migrations of your old code)
• Boolean search has a new concept (use in combination of the new result structure)
• Node.js Worker Threads
• Improved default latin encoders
• New parallelization model and workload distribution
• Improved Export/Import
• Tag Search
• Offset pagination
• Enhanced Field Search
• Improved sorting by relevance (score)
• Added Context Scoring (context index has its own resolution)
• Enhanced charset normalization
• Improved bundler (support for inline WebWorker)

These features have been removed:

• Where-Clause
• Index Information index.info()
• Paging Cursor (was replaced by offset)

Migration Quick Overview

The "async" options was removed, instead you can call each method in its async version, e.g. index.addAsync or index.searchAsync.

Define document fields as object keys is not longer supported due to the unification of all option payloads.

A full configuration example for a context-based index:

var index = new Index({
    tokenize: "strict",
    resolution: 9,
    minlength: 3,
    optimize: true,
    fastupdate: true,
    cache: 100,
    context: {
        depth: 1,
        resolution: 3,
        bidirectional: true
    }
});

The resolution could be set also for the contextual index.

A full configuration example for a document based index:

const index = new Document({
    tokenize: "forward",
    optimize: true,
    resolution: 9,
    cache: 100,
    worker: true,
    document: {
        id: "id",
        tag: "tag",
        store: [
            "title", "content"
        ],
        index: [{
            field: "title",
            tokenize: "forward",
            optimize: true,
            resolution: 9
        },{
            field:  "content",
            tokenize: "strict",
            optimize: true,
            resolution: 9,
            minlength: 3,
            context: {
                depth: 1,
                resolution: 3
            }
        }]
    }
});

A full configuration example for a document search:

index.search({
    enrich: true,
    bool: "and",
    tag: ["cat", "dog"],
    index: [{
        field: "title",
        query: "some query",
        limit: 100,
        suggest: true
    },{
        field: "content",
        query: "same or other query",
        limit: 100,
        suggest: true
    }]
});

Where Clause Replacement

Old Syntax:

const result = index.where({
    cat: "comedy",
    year: "2018"
});

Equivalent Syntax (0.7.x):

const data = Object.values(index.store);

The line above retrieves data from the document store (just useful when not already available in your runtime).

const result = data.filter(function(item){ 
    return item.cat === "comedy" && item.year === "2018";
});

Also considering using the Tag-Search feature, which partially replaces the Where-Clause with a huge performance boost.

v0.6.0 Copy Section

• Pagination

v0.5.3 Copy Section

• Logical Operator

v0.5.2 Copy Section

• Intersect Partial Results

v0.5.1 Copy Section

• Customizable Scoring Resolution

v0.5.0 Copy Section

• Where / Find Documents
• Document Tags
• Custom Sort

v0.4.0 Copy Section

• Index Documents (Field-Search)

v0.3.6 Copy Section

• Right-To-Left Support
• CJK Word Splitting Support

v0.3.5 Copy Section

• Promise Support

v0.3.4 Copy Section

• Export / Import Indexes (Serialize)

v0.3.0 Copy Section

• Profiler Support

File: docker-compose.yml Copy Chapter

version: "3.6"
services:
postgres:
image: postgres:latest
restart: on-failure
volumes:
- pg-data:/var/lib/postgresql/data
environment:
- POSTGRES_DB=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
ports:
- "5432:5432"
container_name: flexsearch_postgres_db
command:
- "postgres"
- "-c"
- "shared_preload_libraries=pg_stat_statements"
clickhouse:
image: yandex/clickhouse-server:latest
restart: on-failure
volumes:
- ch-data:/var/lib/clickhouse
ports:
- "8123:8123"
container_name: flexsearch_clickhouse_db
redis:
image: redislabs/rejson:latest
restart: on-failure
volumes:
- rd-data:/data
ports:
- "6379:6379"
container_name: flexsearch_redis_db
mongo:
image: mongo:latest
restart: on-failure
volumes:
- mg-data:/data
ports:
- "27017:27017"
container_name: flexsearch_mongo_db
volumes:
pg-data:
driver: local
ch-data:
driver: local
rd-data:
driver: local
mg-data:
driver: local

File: package.json Copy Chapter

{
"public": true,
"preferGlobal": false,
"name": "flexsearch",
"version": "0.8.215",
"description": "Next-Generation full-text search library for Browser and Node.js",
"homepage": "https://github.com/nextapps-de/flexsearch/",
"author": "Thomas Wilkerling",
"copyright": "Nextapps GmbH",
"license": "Apache-2.0",
"readme": "README.md",
"keywords": [
"fulltext search",
"elastic search",
"fastest search",
"contextual search",
"document search",
"fuzzy search",
"fuzzy match",
"search engine"
],
"repository": {
"type": "git",
"url": "https://github.com/nextapps-de/flexsearch.git"
},
"bugs": {
"url": "https://github.com/nextapps-de/flexsearch/issues",
"email": "[email protected]"
},
"main": "dist/flexsearch.bundle.min.js",
"module": "dist/flexsearch.bundle.module.min.mjs",
"types": "./index.d.ts",
"exports": {
".": {
"types": "./index.d.ts",
"import": "./dist/flexsearch.bundle.module.min.mjs",
"require": "./dist/flexsearch.bundle.min.js"
},
"./lang/": {
"import": "./dist/module/lang/.js",
"require": "./dist/lang/.min.js"
},
"./db/": {
"import": "./dist/module/db//index.js",
"require": "./dist/db//index.cjs"
},
"./debug": {
"import": "./dist/flexsearch.bundle.module.debug.mjs",
"require": "./dist/flexsearch.bundle.debug.js"
}
},
"browser": {
"flexsearch": "./dist/flexsearch.bundle.module.min.mjs",
"flexsearch/debug": "./dist/flexsearch.bundle.module.debug.mjs",
"dist/flexsearch.bundle.min.js": "./dist/flexsearch.bundle.min.js",
"dist/flexsearch.bundle.module.min.js": "./dist/flexsearch.bundle.module.min.mjs",
"dist/flexsearch.bundle.module.min.mjs": "./dist/flexsearch.bundle.module.min.mjs",
"worker_threads": false,
"path": false,
"clickhouse": false,
"mongodb": false,
"pg-promise": false,
"redis": false,
"sqlite3": false
},
"scripts": {
"build": "npm run build:bundle && npm run build:bundle:debug && exit 0",
"build:bundle": "node task/build RELEASE=bundle DEBUG=false PROFILER=false SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CHARSET=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false SUPPORT_PERSISTENT=true SUPPORT_RESOLVER=true SUPPORT_HIGHLIGHTING=true SUPPORT_KEYSTORE=true SUPPORT_COMPRESSION=false",
"build:bundle:debug": "node task/build RELEASE=bundle DEBUG=true PROFILER=false SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CHARSET=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false SUPPORT_PERSISTENT=true SUPPORT_RESOLVER=true SUPPORT_HIGHLIGHTING=true SUPPORT_KEYSTORE=true SUPPORT_COMPRESSION=false FORMATTING=PRETTY_PRINT",
"build:compact": "node task/build RELEASE=compact DEBUG=false PROFILER=false SUPPORT_WORKER=false SUPPORT_ENCODER=true SUPPORT_CHARSET=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false SUPPORT_PERSISTENT=false SUPPORT_RESOLVER=false SUPPORT_HIGHLIGHTING=true SUPPORT_KEYSTORE=false SUPPORT_COMPRESSION=false",
"build:compact:debug": "node task/build RELEASE=compact DEBUG=true PROFILER=false SUPPORT_WORKER=false SUPPORT_ENCODER=true SUPPORT_CHARSET=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false SUPPORT_PERSISTENT=false SUPPORT_RESOLVER=false SUPPORT_HIGHLIGHTING=true SUPPORT_KEYSTORE=false SUPPORT_COMPRESSION=false FORMATTING=PRETTY_PRINT",
"build:light": "node task/build RELEASE=light DEBUG=false PROFILER=false SUPPORT_WORKER=false SUPPORT_ENCODER=true SUPPORT_CHARSET=false SUPPORT_CACHE=false SUPPORT_ASYNC=false SUPPORT_STORE=false SUPPORT_TAGS=false SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=false SUPPORT_DOCUMENT=false POLYFILL=false SUPPORT_PERSISTENT=false SUPPORT_RESOLVER=false SUPPORT_HIGHLIGHTING=false SUPPORT_KEYSTORE=false SUPPORT_COMPRESSION=false",
"build:light:debug": "node task/build RELEASE=light DEBUG=true PROFILER=false SUPPORT_WORKER=false SUPPORT_ENCODER=true SUPPORT_CHARSET=false SUPPORT_CACHE=false SUPPORT_ASYNC=false SUPPORT_STORE=false SUPPORT_TAGS=false SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=false SUPPORT_DOCUMENT=false POLYFILL=false SUPPORT_PERSISTENT=false SUPPORT_RESOLVER=false SUPPORT_HIGHLIGHTING=false SUPPORT_KEYSTORE=false SUPPORT_COMPRESSION=false FORMATTING=PRETTY_PRINT",
"build:es5": "node task/build RELEASE=es5 DEBUG=false PROFILER=false SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CHARSET=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=true SUPPORT_PERSISTENT=true SUPPORT_RESOLVER=true SUPPORT_HIGHLIGHTING=true SUPPORT_KEYSTORE=true SUPPORT_COMPRESSION=false LANGUAGE_OUT=ECMASCRIPT5_STRICT",
"build:es5:debug": "node task/build RELEASE=es5 DEBUG=true PROFILER=false SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CHARSET=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=true SUPPORT_PERSISTENT=true SUPPORT_RESOLVER=true SUPPORT_HIGHLIGHTING=true SUPPORT_KEYSTORE=true SUPPORT_COMPRESSION=false FORMATTING=PRETTY_PRINT LANGUAGE_OUT=ECMASCRIPT5_STRICT",
"build:lang": "node task/build RELEASE=lang",
"build:db": "npx rollup tmp/db/indexeddb/index.js --file dist/db/indexeddb/index.cjs --format cjs && npx rollup tmp/db/postgres/index.js --file dist/db/postgres/index.cjs --format cjs && npx rollup tmp/db/sqlite/index.js --file dist/db/sqlite/index.cjs --format cjs && npx rollup tmp/db/mongodb/index.js --file dist/db/mongodb/index.cjs --format cjs && npx rollup tmp/db/redis/index.js --file dist/db/redis/index.cjs --format cjs && npx rollup tmp/db/clickhouse/index.js --file dist/db/clickhouse/index.cjs --format cjs",
"build:module": "node task/babel && exit 0",
"build:module:debug": "node task/babel DEBUG=true && exit 0",
"build:module:min": "node task/babel RELEASE=min && exit 0",
"build:module:bundle": "node task/build RELEASE=bundle.module DEBUG=false PROFILER=false SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CHARSET=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false SUPPORT_PERSISTENT=true SUPPORT_RESOLVER=true SUPPORT_HIGHLIGHTING=true SUPPORT_KEYSTORE=true SUPPORT_COMPRESSION=false",
"build:module:bundle:debug": "node task/build RELEASE=bundle.module DEBUG=true PROFILER=false SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CHARSET=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false SUPPORT_PERSISTENT=true SUPPORT_RESOLVER=true SUPPORT_HIGHLIGHTING=true SUPPORT_KEYSTORE=true SUPPORT_COMPRESSION=false FORMATTING=PRETTY_PRINT",
"build:module:compact": "node task/build RELEASE=compact.module DEBUG=false PROFILER=false SUPPORT_WORKER=false SUPPORT_ENCODER=true SUPPORT_CHARSET=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false SUPPORT_PERSISTENT=false SUPPORT_RESOLVER=false SUPPORT_HIGHLIGHTING=true SUPPORT_KEYSTORE=false SUPPORT_COMPRESSION=false",
"build:module:compact:debug": "node task/build RELEASE=compact.module DEBUG=true PROFILER=false SUPPORT_WORKER=false SUPPORT_ENCODER=true SUPPORT_CHARSET=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false SUPPORT_PERSISTENT=false SUPPORT_RESOLVER=false SUPPORT_HIGHLIGHTING=true SUPPORT_KEYSTORE=false SUPPORT_COMPRESSION=false FORMATTING=PRETTY_PRINT",
"build:module:light": "node task/build RELEASE=light.module DEBUG=false PROFILER=false SUPPORT_WORKER=false SUPPORT_ENCODER=true SUPPORT_CHARSET=false SUPPORT_CACHE=false SUPPORT_ASYNC=false SUPPORT_STORE=false SUPPORT_TAGS=false SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=false SUPPORT_DOCUMENT=false POLYFILL=false SUPPORT_PERSISTENT=false SUPPORT_RESOLVER=false SUPPORT_HIGHLIGHTING=false SUPPORT_KEYSTORE=false SUPPORT_COMPRESSION=false",
"build:module:light:debug": "node task/build RELEASE=light.module DEBUG=true PROFILER=false SUPPORT_WORKER=false SUPPORT_ENCODER=true SUPPORT_CHARSET=false SUPPORT_CACHE=false SUPPORT_ASYNC=false SUPPORT_STORE=false SUPPORT_TAGS=false SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=false SUPPORT_DOCUMENT=false POLYFILL=false SUPPORT_PERSISTENT=false SUPPORT_RESOLVER=false SUPPORT_HIGHLIGHTING=false SUPPORT_KEYSTORE=false SUPPORT_COMPRESSION=false FORMATTING=PRETTY_PRINT",
"build:all": "npm run build:bundle && npm run build:bundle:debug && npm run build:light && npm run build:light:debug && npm run build:compact && npm run build:compact:debug && npm run build:module:bundle && npm run build:module:bundle:debug && npm run build:module:light && npm run build:module:light:debug && npm run build:module:compact && npm run build:module:compact:debug && npm run build:es5 && npm run build:es5:debug && npm run build:lang && npm run build:module && npm run build:module:debug && npm run build:module:min && npm run build:db",
"build:custom": "node task/build RELEASE=custom",
"test": "mocha test/.js --exit",
"test:coverage": "c8 -c test/.c8rc.json mocha test/.js --exit",
"test:keystore": "set NODE_OPTIONS=--max-old-space-size=16000 && c8 -c test/.c8rc.json mocha test/keystore.js --exit",
"test:db": "c8 -c test/.c8rc.json mocha test/persistent..js --exit",
"test:all": "npx mocha test/.js --exit module/bundle && npx mocha test/.js --exit module-debug/bundle && npx mocha test/.js --exit module-min/bundle && npx mocha test/.js --exit flexsearch.bundle.debug && npx mocha test/.js --exit flexsearch.bundle.min && npx mocha test/.js --exit flexsearch.bundle.module.debug && npx mocha test/.js --exit flexsearch.bundle.module.min && npx mocha test/.js --exit flexsearch.compact.debug && npx mocha test/.js --exit flexsearch.compact.min && npx mocha test/.js --exit flexsearch.compact.module.debug && npx mocha test/.js --exit flexsearch.compact.module.min && npx mocha test/.js --exit flexsearch.es5.debug && npx mocha test/.js --exit flexsearch.es5.min && npx mocha test/.js --exit flexsearch.light.debug && npx mocha test/.js --exit flexsearch.light.min && npx mocha test/.js --exit flexsearch.light.module.debug && npx mocha test/.js --exit flexsearch.light.module.min",
"release": "npm version --no-git-tag-version patch && npm run build:all && npm run test:all"
},
"files": [
"dist/",
"src/",
"task/**",
"index.d.ts",
"README.md",
"CHANGELOG.md",
"LICENSE"
],
"devDependencies": {
"babel-cli": "^6.26.0",
"babel-plugin-conditional-compile": "^0.0.5",
"babel-plugin-minify-constant-folding": "^0.5.0",
"babel-plugin-minify-dead-code-elimination": "^0.5.2",
"babel-plugin-minify-flip-comparisons": "^0.4.3",
"babel-plugin-minify-guarded-expressions": "^0.4.4",
"babel-plugin-minify-infinity": "^0.4.3",
"babel-plugin-minify-mangle-names": "^0.5.1",
"babel-plugin-minify-replace": "^0.5.0",
"babel-plugin-minify-simplify": "^0.5.1",
"babel-plugin-minify-type-constructors": "^0.4.3",
"babel-plugin-transform-member-expression-literals": "^6.9.4",
"babel-plugin-transform-merge-sibling-variables": "^6.9.5",
"babel-plugin-transform-minify-booleans": "^6.9.4",
"babel-plugin-transform-property-literals": "^6.9.4",
"babel-plugin-transform-simplify-comparison-operators": "^6.9.4",
"babel-plugin-transform-undefined-to-void": "^6.9.4",
"c8": "^10.1.3",
"chai": "^5.2.0",
"clickhouse": "^2.6.0",
"google-closure-compiler": "^20250520.0.0",
"mocha": "^11.1.0",
"mongodb": "^6.13.0",
"pg-promise": "^11.13.0",
"redis": "^5.1.0",
"rollup": "^4.35.0",
"sqlite3": "^5.1.7"
},
"funding": [
{
"type": "github",
"url": "https://github.com/ts-thomas"
},
{
"type": "paypal",
"url": "https://www.paypal.com/donate/?hosted_button_id=GEVR88FC9BWRW"
},
{
"type": "opencollective",
"url": "https://opencollective.com/flexsearch"
},
{
"type": "patreon",
"url": "https://patreon.com/user?u=96245532"
},
{
"type": "liberapay",
"url": "https://liberapay.com/ts-thomas"
}
]
}

File: SECURITY.md Copy Chapter

Security Policy

Supported Versions Copy Chapter

Reporting a Vulnerability Copy Chapter

Just create an issue or pull request on Github.

The vulnerability has to be a part of the basic installment e.g. when install by npm install flexsearch.
Every vulnerability which is included by any of the devDependencies probably won't be fixed.

File: doc/async.md Copy Chapter

Asynchronous Runtime Balancer Copy Chapter

The async processing model is automatically observed by a runtime balancer to prevent any blocking issues, even on page load.

Those methods of each index type provides an async version:

• addAsync()
• appendAsync()
• updateAsync()
• removeAsync()
• searchAsync()

All those async versions always return a Promise, although a callback can be passed additionally as the last parameter.

When calling async methods of the index, a runtime balancer observe the current event loop and will pass to the next event loop automatically.

Task Priority Copy Section

You can control how early the process should move over to the next event loop by passing the option property priority:

const index = new Index({
    // a value between 1 and 9
    priority: 4
});

The lowest valid priority number is 1 and is typically known as idle (event loop cycles by native ~4ms). The default priority is 4 which is optimized for non-blocking user interfaces within a browser (event loop cycles every ~45ms).
When you have some very smooth running animation you should use a priority of 2 to keep the animation running by 60 fps without any stutter. Targeting 120 fps or higher you should use 1.

On Node.js you can slightly increase this priority e.g. to 6, because here there is no UI involved.
A priority value of 9 will cycle the event loop on every ~250ms which is the maximum recommended blocking time. You should not use a value higher than this.

Polling Tasks Copy Section

Do not forget to await on every async task you apply to the index:

for(let i = 0; i < 99999999; i++){
    await index.addAsync(i, "test " + i);
}

You can perform queries to the index during any other async batch is running.

Examples Copy Section

You can assign callbacks to each async function:

index.addAsync(id, content, function(){
    console.log("Task Done");
});

index.searchAsync(query, function(result){
    console.log("Results: ", result);
});

Or do not pass a callback function and getting back a Promise instead:

index.addAsync(id, content).then(function(){
    console.log("Task Done");
});

index.searchAsync(query).then(function(result){
    console.log("Results: ", result);
});

Or use async and await:

async function add(){
    await index.addAsync(id, content);
    console.log("Task Done");
}

async function search(){
    const results = await index.searchAsync(query);
    console.log("Results: ", result);
}

File: doc/custom-builds.md Copy Chapter

Custom Builds Copy Chapter

The /src/ folder of this repository requires some compilation to resolve the build flags.

You can run any of the basic builds located in the /dist/ folder, e.g.:

npm run build:bundle
npm run build:light
npm run build:module

Perform a custom build (UMD bundle) by passing build flags:

npm run build:custom SUPPORT_DOCUMENT=true SUPPORT_TAGS=true LANGUAGE_OUT=ECMASCRIPT5 POLYFILL=true

Perform a custom build in ESM module format:

npm run build:custom RELEASE=custom.module SUPPORT_DOCUMENT=true SUPPORT_TAGS=true

Perform a debug build:

npm run build:custom DEBUG=true SUPPORT_DOCUMENT=true SUPPORT_TAGS=true

On custom builds each build flag will be set to false by default when not passed.

Just build the core library:

npm run build:custom

The custom build will be saved to dist/flexsearch.custom.xxxx.min.js or when format is module to dist/flexsearch.custom.module.xxxx.min.js (the "xxxx" is a hash based on the used build flags).

Supported Build Flags Copy Section

File: doc/customization.md Copy Chapter

Custom Score Function Copy Chapter

const index = new FlexSearchIndex({
    resolution: 10,
    score: function(content, term, term_index, partial, partial_index){
        // you'll need to return a number between 0 and "resolution"
        // score is starting from 0, which is the highest score
        // for a resolution of 10 you can return 0 - 9
        // ... 
        return 3;
    } 
});

A common situation is you have some predefined labels which are related to some kind of order, e.g. the importance or priority. A priority label could be high, moderate, low so you can derive the scoring from those properties. Another example is when you have something already ordered and you would like to keep this order as relevance.

Probably you won't need the parameters passed to the score function. But when needed here are the parameters from the score function explained:

1. content is the whole content as an array of terms (encoded)
2. term is the current term which is actually processed (encoded)
3. term_index is the index of the term in the content array
4. partial is the current partial of a term which is actually processed
5. partial_index is the index position of the partial within the term

Partials params are empty when using tokenizer strict. Let's take an example by using the tokenizer full.

The content: "This is an example of partial encoding"
The highlighting part marks the partial which is actually processed. Then your score function will called by passing these parameters:

function score(content, term, term_index, partial, partial_index){
    content = ["this", "is", "an", "example", "of", "partial", "encoding"]
    term = "example"
    term_index = 3
    partial = "amp"
    partial_index = 2
}

File: doc/document-search.md Copy Chapter

Document Search (Field-Search)

Whereas the simple Index can just consume id-content pairs, the Document-Index is able to process more complex data structures like JSON.
Technically, a Document-Index is a layer on top of several default indexes. You can create multiple independent Document-Indexes in parallel, any of them can use the Worker or Persistent model optionally.

FlexSearch Documents also contain these features:

• Document Store including Enrichment
• Multi-Field-Search
• Multi-Tag-Search
• Resolver (Chain Complex Queries)
• Result Highlighting
• Export/Import
• Worker
• Persistent

Document Options Copy Section

Document options basically inherits from Index Options, so you can apply most of those options either in the top scope of the config (for all fields) or as per field or both of them.

Document Search Options Copy Section

Document search options basically inherit from Index Search Options, so you can apply most of those options either in the top scope of the config (for all fields) or as per field or both of them.

String | Highlight query matches in the result (for Document Indexes only) | `false` |

| merge | Boolean | Merge multiple fields in resultset into one and group results per ID | false |
| pluck | String | Pick and apply search to just one field and return a flat result representation | false |

The Document Descriptor Copy Chapter

When creating a Document-Index you will need to define a document descriptor in the field document. This descriptor is including any specific information about how the document data should be indexed.

Field Options Copy Section

You can use all standard Index Options within field options.

Assuming our document has a simple data structure like this:

{ 
    "id": 0, 
    "content": "some text"
}

An appropriate Document Descriptor has always to define at least 2 things:

1. the property id describes the location of the document ID within a document item
2. the property index (or tag) containing one or multiple fields from the document, which should be indexed for searching

// create a document index
const index = new Document({
    document: {
        id: "id",
        index: "content"
    }
});

// add documents to the index
index.add({ 
    id: 0, 
    content: "some text"
});

As briefly explained above, the field id describes where the ID or unique key lives inside your documents. When not passed it will always take the field id from the top level scope of your data.

The property index takes all fields you would like to have indexed. When just selecting one field, then you can pass a string.

The next example will add 2 fields title and content to the index:

var docs = [{
    id: 0,
    title: "Title A",
    content: "Body A"
},{
    id: 1,
    title: "Title B",
    content: "Body B"
}];

const index = new Document({
    id: "id",
    index: ["title", "content"]
});

Add both fields to the document descriptor and pass individual Index-Options for each field:

const index = new Document({
    id: "id",
    index: [{
        field: "title",
        tokenize: "forward",
        encoder: Charset.LatinAdvanced,
        resolution: 9
    },{
        field:  "content",
        tokenize: "forward",
        encoder: Charset.LatinAdvanced,
        resolution: 3
    }]
});

Field options inherits from top level options when passed, e.g.:

const index = new Document({
    tokenize: "forward",
    encoder: Charset.LatinAdvanced,
    resolution: 9,
    document: {
        id: "id",
        index:[{
            field: "title"
        },{
            field: "content",
            resolution: 3
        }]
    }
});

Assigning the Encoder instance to the top level configuration will share the encoder to all fields. You should avoid this when contents of fields don't have the same type of content (e.g. one field contains terms, another contains numeric IDs).

Nested Data Fields (Complex Objects) Copy Section

Assume the document array looks more complex (has nested branches etc.), e.g.:

{
  "record": {
    "id": 0,
    "title": "some title",
    "content": {
      "header": "some text",
      "footer": "some text"
    }
  }
}

Then use the colon separated notation root:child:child as a name for each field defining the hierarchy which corresponds to the document:

const index = new Document({
    document: {
        id: "record:id",
        index: [
            "record:title",
            "record:content:header",
            "record:content:footer"
        ]
    }
});

[!TIP]
Just add fields you want to query against. Do not add fields to the index, you just need in the result. For this purpose you can store documents independently of its index (read below).

To query against one or multiple specific fields you have to pass the exact key of the field you have defined in the document descriptor as a field name (with colon syntax):

index.search(query, {
    field: [
        "record:title",
        "record:content:header",
        "record:content:footer"
    ]
});

Same as:

index.search(query, [
    "record:title",
    "record:content:header",
    "record:content:footer"
]);

Using field-specific options:

index.search("some query", [{
    field: "record:title",
    limit: 100,
    suggest: true
},{
    field: "record:content:header",
    limit: 100,
    suggest: false
}]);

You can also perform a search through the same field with different queries:

index.search([{
    field: "record:title",
    query: "some query",
    limit: 100,
    suggest: true
},{
    field: "record:title",
    query: "some other query",
    limit: 100,
    suggest: true
}]);

Complex Documents Copy Section

You need to follow 2 rules for your documents:

1. The document cannot start with an Array at the root. This will introduce sequential data and isn't supported yet. See below for a workaround for such data.

[ // <-- not allowed as document start!
  {
    "id": 0,
    "title": "title"
  }
]

2. The document ID can't be nested inside an Array. This will introduce sequential data and isn't supported yet. See below for a workaround for such data.

{
  "records": [ // <-- not allowed when ID or tag lives inside!
    {
      "id": 0,
      "title": "title"
    }
  ]
}

Here an example for a supported complex document:

{
  "meta": {
    "tag": "cat",
    "id": 0
  },
  "contents": [
    {
      "body": {
        "title": "some title",
        "footer": "some text"
      },
      "keywords": ["some", "key", "words"]
    },
    {
      "body": {
        "title": "some title",
        "footer": "some text"
      },
      "keywords": ["some", "key", "words"]
    }
  ]
}

The corresponding document descriptor (when all fields should be indexed) looks like:

const index = new Document({
    document: {
        id: "meta:id",
        index: [
            "contents:body:title",
            "contents:body:footer"
        ],
        tag: [
            "meta:tag",
            "contents:keywords"
        ]
    }
});

Remember when searching you have to use the same colon-separated-string as a key from your field definition.

index.search(query, { 
    index: "contents:body:title"
});

Not Supported Documents (Sequential Data) Copy Section

This example breaks both rules described above:

[ // <-- not allowed as document start!
  {
    "tag": "cat",
    "records": [ // <-- not allowed when ID or tag lives inside!
      {
        "id": 0,
        "body": {
          "title": "some title",
          "footer": "some text"
        },
        "keywords": ["some", "key", "words"]
      },
      {
        "id": 1,
        "body": {
          "title": "some title",
          "footer": "some text"
        },
        "keywords": ["some", "key", "words"]
      }
    ]
  }
]

You need to unroll your data within a simple loop before adding to the index.

A workaround to such a data structure from above could look like:

const index = new Document({
    document: {
        id: "id",
        index: [
            "body:title",
            "body:footer"
        ],
        tag: [
            "tag",
            "keywords"
        ]
    }
});

function add(sequential_data){

    for(let x = 0, item; x < sequential_data.length; x++){

        item = sequential_data[x];

        for(let y = 0, record; y < item.records.length; y++){
            record = item.records[y];
            // append tag to each record
            record.tag = item.tag;
            // add to index
            index.add(record);
        }
    }  
}

// now just use add() helper method as usual:
add([{
    // sequential structured data
    // take the data example above
}]);

Add/Update/Remove Documents Copy Section

Add a document to the index:

index.add({
    id: 0,
    title: "Foo",
    content: "Bar"
});

Update index:

index.update({
    id: 0,
    title: "Foo",
    content: "Foobar"
});

Remove a document and all its contents from an index, by ID:

index.remove(id);

Or by the document data:

index.remove(doc);

Field-Search Copy Chapter

Search through all fields:

index.search(query);

Search through a specific field:

index.search(query, { index: "title" });

Search through a given set of fields:

index.search(query, { index: ["title", "content"] });

Pass custom options and/or queries to each field:

index.search([{
    field: "content",
    query: "some query",
    limit: 100,
    suggest: true
},{
    field: "content",
    query: "some other query",
    limit: 100,
    suggest: true
}]);

Limit & Offset Copy Section

By default, every query is limited to 100 entries. Unbounded queries leads into issues. You need to set the limit as an option to adjust the size.

You can set the limit and the offset for each query:

index.search(query, { limit: 20, offset: 100 });

You cannot pre-count the size of the result-set. That's a limit by the design of FlexSearch. When you really need a count of all results you are able to page through, then just assign a high enough limit and get back all results and apply your paging offset manually (this works also on server-side). FlexSearch is fast enough that this isn't an issue.

See all available field-search options

The Result Set Copy Chapter

Schema of the default result-set:

fields[] => { field, result[] => id }

Schema of an enriched result-set:

fields[] => { field, result[] => { id, doc }}

The top-level scope of the result set is an array of fields on which the query was applied to. Each of this field has a record (object) with 2 properties field and result. The result could be an array of IDs or is getting enriched by the stored document data (when index was created with store: true).

A default non-enriched result set looks like:

[{
    field: "title",
    result: [0, 1, 2]
},{
    field: "content",
    result: [3, 4, 5]
}]

An enriched result set looks like:

[{
    field: "title",
    result: [
        { id: 0, doc: { /* document */ }},
        { id: 1, doc: { /* document */ }},
        { id: 2, doc: { /* document */ }}
    ]
},{
    field: "content",
    result: [
        { id: 3, doc: { /* document */ }},
        { id: 4, doc: { /* document */ }},
        { id: 5, doc: { /* document */ }}
    ]
}]

Merge Document Results Copy Section

Schema of the merged result-set:

result[] => { id, doc, field[] }}

By passing the search option merge: true all fields of the result set will be merged (grouped by ID):

[{
    id: 1001,
    doc: {/* stored document */}
    field: ["fieldname-1", "fieldname-2"]
},{
    id: 1002,
    doc: {/* stored document */}
    field: ["fieldname-3"]
}]

Pluck Single Fields Copy Section

When using pluck instead of field you can explicitly select just one field and get back a flat representation:

index.search(query, { 
    pluck: "title",
    enrich: true
});

[
    { id: 0, doc: { /* document */ }},
    { id: 1, doc: { /* document */ }},
    { id: 2, doc: { /* document */ }}
]

Tags Copy Chapter

Like the property index within a document descriptor just define a property tag:

const index = new Document({
    document: { 
        id: "id",
        tag: "species",
        index: "content"
    }
});

index.add({
    id: 0,
    species: "cat",
    content: "Some content ..."
});

Your data also can include multiple tags as an array:

index.add({
    id: 1,
    species: ["fish", "dog"],
    content: "Some content ..."
});

You can perform a tag-specific search by:

index.search(query, {
    tag: { species: "fish" } 
});

This just gives you results which was tagged with the given tag.

Use multiple tags when searching:

index.search(query, {
    tag: { species: ["cat", "dog"] }
});

This give you results which was tagged with at least one of the given tags.

Get back all tagged results without passing any query:

index.search({
    tag: { species: "cat" }
});

Multi-Tag Search Copy Section

Assume this document schema (a dataset from IMDB):

{
    "tconst": "tt0000001",
    "titleType": "short",
    "primaryTitle": "Carmencita",
    "originalTitle": "Carmencita",
    "isAdult": 0,
    "startYear": "1894",
    "endYear": "",
    "runtimeMinutes": "1",
    "genres": [
        "Documentary",
        "Short"
    ]
}

An appropriate document descriptor could look like:

import Charset from "flexsearch";
const index = new Document({
    encoder: Charset.Normalize,
    resolution: 3,
    document: {
        id: "tconst",
        //store: true, // document store
        index: [{
            field: "primaryTitle",
            tokenize: "forward"
        },{
            field: "originalTitle",
            tokenize: "forward"
        }],
        tag: [
            "startYear",
            "genres"
        ]
    }
});

The field contents of primaryTitle and originalTitle are encoded by the forward tokenizer. The field contents of startYear and genres are added as tags.

Get all entries of a specific tag:

const result = index.search({
    //enrich: true, // enrich documents
    tag: { "genres": "Documentary" },
    limit: 1000,
    offset: 0
});

Get entries of multiple tags (intersection):

const result = index.search({
    //enrich: true, // enrich documents
    tag: { 
        "genres": ["Documentary", "Short"],
        "startYear": "1894"
    }
});

Combine tags with queries (intersection):

const result = index.search({
    query: "Carmen", // forward tokenizer
    tag: { 
        "genres": ["Documentary", "Short"],
        "startYear": "1894"
    }
});

Alternative declaration:

const result = index.search("Carmen", {
    tag: [{
        field: "genres",
        tag: ["Documentary", "Short"]
    },{
        field: "startYear",
        tag: "1894"
    }]
});

Document Store Copy Chapter

Only a document index can have a store. You can use a document index instead of a flat index to get this functionality also when only storing ID-content-pairs.

You can define independently which fields should be indexed and which fields should be stored. This way you can index fields which should not be included in the search result.

Do not use a store when: 1. an array of IDs as the result is good enough, or 2. you already have the contents/documents stored elsewhere (outside the index).

When the store attribute was set, you have to include all fields which should be stored explicitly (acts like a whitelist).

When the store attribute was not set, the original document is stored as a fallback.

This will add the whole original content to the store:

const index = new Document({
    document: { 
        index: "content",
        store: true
    }
});

index.add({ id: 0, content: "some text" });

Access documents from internal store Copy Section

You can get indexed documents from the store:

var data = index.get(1);

You can update/change store contents directly without changing the index by:

index.set(1, data);

To update the store and also update the index then just use index.update, index.add or index.append.

When you perform a query, weather it is a document index or a flat index, then you will always get back an array of IDs.

Optionally you can enrich the query results automatically with stored contents by:

index.search(query, { enrich: true });

Your results look now like:

[{
    id: 0,
    doc: { /* content from store */ }
},{
    id: 1,
    doc: { /* content from store */ }
}]

Configure Document Store (Recommended) Copy Section

When storing documents, you can configure independently what should be indexed and what should be stored. This can reduce required index space significantly. Indexed fields do not require to be included in the stored data (also the ID isn't necessary to keep in store).
It is recommended to just add fields to the store you'll need in the final result to process further on.

A short example of configuring a document store:

const index = new Document({
    document: { 
        index: "content",
        store: ["author", "email"] 
    }
});

index.add({
    id: 0,
    author: "Jon Doe",
    email: "[email protected]",
    content: "Some content for the index ..."
});

You can query through the contents and will get back the stored values instead:

index.search("some content", { enrich: true });

Your results are now looking like:

[{
    field: "content",
    result: [{
        id: 0,
        doc: {
            author: "Jon Doe",
            email: "[email protected]",
        }
    }]
}]

Both field "author" and "email" are not indexed, whereas the indexed field "content" was not included in the stored data.

Filter Fields (Index / Tags / Datastore) Copy Chapter

You can pass a function to the field option property filter. This function just has to return true if the document should be indexed.

const index = new Document({
    document: {
        id: "id",
        index: [{
            // custom field:
            field: "somefield",
            filter: function(data){
                // return false to filter out
                // return anything else to keep
                return true;
            }
        }],
        tag: [{
            field: "city",
            filter: function(data){
                // return false to filter out
                // return anything else to keep
                return true;
            }
        }],
        store: [{
            field: "anotherfield",
            filter: function(data){
                // return false to filter out
                // return anything else to keep
                return true;
            }
        }]
    }
});

Custom Fields (Index / Tags / Datastore) Copy Chapter

You can pass a function to the field option property custom to either:

1. change and/or extend the original input string
2. create a new "virtual" field which is not included in document data

Dataset example:

{
    "id": 10001,
    "firstname": "John",
    "lastname": "Doe",
    "city": "Berlin",
    "street": "Alexanderplatz",
    "number": "1a",
    "postal": "10178"
}

You can apply custom fields derived from document data or by any external data:

const index = new Document({
    document: {
        id: "id",
        index: [{
            // custom field:
            field: "fullname",
            custom: function(data){
                // return custom string
                return data.firstname + " " + 
                       data.lastname;
            }
        },{
            // custom field:
            field: "location",
            custom: function(data){
                return data.street + " " +
                       data.number + ", " +
                       data.postal + " " +
                       data.city;
            }
        }],
        tag: [{
            // existing field
            field: "city"
        },{
            // custom field:
            field: "category",
            custom: function(data){
                let tags = [];
                // push one or multiple tags
                // ....
                return tags;
            }
        }],
        store: [{
            field: "anotherfield",
            custom: function(data){
                // return a falsy value to filter out
                // return anything else as to keep in store
                return data;
            }
        }]
    }
});

Filter is also available in custom functions when returning false.

Perform a query against the custom field as usual:

const result = index.search({
    query: "10178 Berlin Alexanderplatz",
    field: "location"
});

const result = index.search({
    query: "john doe",
    tag: { "city": "Berlin" }
});

Best Practices: TypeScript Copy Section

When using TypeScript, you can type your document data when creating a Document-Index. This will provide enhanced type checks of your syntax.

Create a schema accordingly to your document data, e.g.:

type doctype = {
    id: number,
    title: string,
    description: string,
    tags: string[]
};

Create the document index by assigning the type doctype:

const document = new Document<doctype>({
    id: "id",
    store: true,
    index: [{
        field: "title"
    },{
        field: "description"
    }],
    tag: "tags"
});

Best Practices: Merge Documents Copy Section

Read here

File: doc/encoder.md Copy Chapter

Encoder Copy Chapter

[!IMPORTANT]
You shouldn't miss this part as it is one of the most important aspects of FlexSearch.

Search capabilities highly depends on language processing. The Encoder class is one of the most important core functionalities of FlexSearch.

Encoders are basically responsible for "fuzziness". Read here about Phonetic Search/Fuzzy Search

Default Encoder Copy Section

The default Encoder (when passing no options on creation) uses this configuration:

const encoder = new Encoder({
    normalize: true,
    dedupe: true,
    cache: true,
    include: {
        letter: true,
        number: true,
        symbol: false,
        punctuation: false,
        control: false,
        char: ""
    }
});

The default configuration will:

1. apply charset normalization, e.g. "é" to "e"
2. apply letter deduplication, e.g. "missing" to "mising"
3. just index alphanumeric content and filter everything else out

This is important to keep in mind, because when you need a different configuration you'll have to change those settings accordingly.

Let's assume you want including the symbols "#", "@" and "-", because those are needed to differentiate search results (otherwise it would be useless), and let's say you don't need numeric content indexed you can do this by:

const encoder = new Encoder({
    // default configuration is applied
    // extend or override:
    include: {
        // by default everything is set to false
        letter: true,
        number: false,
        char: ["#", "@", "-"]
    }
});

Built-In Universal Encoders

1. Charset.Exact
2. Charset.Normalize (Charset.Default)

Built-In Latin Encoders

1. Charset.LatinBalance
2. Charset.LatinAdvanced
3. Charset.LatinExtra
4. Charset.LatinSoundex

Built-In CJK Encoder

1. Charset.CJK

Basic Usage Copy Section

const encoder = new Encoder({
    normalize: true,
    dedupe: true,
    cache: true,
    include: {
        letter: true,
        number: true,
        symbol: false,
        punctuation: false,
        control: false,
        char: "@"
    }
});

You can use an include instead of an exclude definition:

const encoder = new Encoder({
    exclude: {
        letter: false,
        number: false,
        symbol: true,
        punctuation: true,
        control: true
    }
});

Instead of using include or exclude you can pass a regular expression or a string to the field split:

const encoder = new Encoder({ 
    split: /\s+/
});

E.g. this split configuration will tokenize every symbol/char from a content:

const encoder = new Encoder({ 
    split: ""
});

The definitions include and exclude is a replacement for split. You can just define one of those 3.

Adding custom functions to the encoder pipeline:

const encoder = new Encoder({
    normalize: function(str){
        return str.toLowerCase();
    },
    prepare: function(str){
        return str.replace(/&/g, " and ");
    },
    finalize: function(arr){
        return arr.filter(term => term.length > 2);
    }
});

Further reading: Encoder Processing Workflow

Assign an encoder to an index:

const index = new Index({ 
    encoder: encoder
});

Define language specific normalizations/transformations:

const encoder = new Encoder({
    stemmer: new Map([
        ["ly", ""]
    ]),
    filter: new Set([
        "and",
    ]),
    mapper: new Map([
        ["é", "e"]
    ]),
    matcher: new Map([
        ["xvi", "16"]
    ]),
    replacer: [
        /[´`’ʼ]/g, "'"
    ],
});

Further reading: Encoder Processing Workflow

Or use built-in helpers alternatively:

const encoder = new Encoder()
    .addStemmer("ly", "")
    .addFilter("and")
    .addMapper("é", "e")
    .addMatcher("xvi", "16")
    .addReplacer(/[´`’ʼ]/g, "'");

Some of the built-in helpers will automatically detect inputs and use the proper helper under the hood. So theoretically you can lazily just write:

const encoder = new Encoder()
    .addStemmer("ly", "")
    .addFilter("and")
    .addReplacer("é", "e")
    .addReplacer("xvi", "16")
    .addReplacer(/[´`’ʼ]/g, "'");

You can also use presets and extend it with custom options:

import EnglishBookPreset from "flexsearch/lang/en";
const encoder = new Encoder(
    EnglishBookPreset,
    // use the preset but don't filter terms
    { filter: false }
);

Equivalent:

const encoder = new Encoder(EnglishBookPreset);
encoder.assign({ filter: false });

Assign multiple extensions to the encoder instance:

import Charset from "flexsearch";
import EnglishBookPreset from "flexsearch/lang/en";
// stack definitions to the encoder instance
const encoder = new Encoder()
    .assign(Charset.LatinSoundex)
    .assign(EnglishBookPreset)
    // extend or override preset options:
    .assign({ minlength: 3 });
    // assign further presets ...

When adding extension to the encoder every previously assigned configuration is still intact, also when assigning custom functions, the previously added function will still execute.

Add custom transformations to an existing index:

const encoder = new Encoder(Charset.Normalize);
// filter terms
encoder.addFilter("and");
// replace single chars
encoder.addMapper("é", "e");
// replace char sequences
encoder.addMatcher("xvi", "16");
// replace single chars or char sequences
// at the end of a term
encoder.addStemmer("ly", "");
// custom regex replace
encoder.addReplacer(/[´`’ʼ]/g, "'");

Using a custom filter:

encoder.addFilter(function(str){
    // return true to keep the content
    return str.length > 1;
});

Shortcut for just assigning one encoder configuration to an index:

const index = new Index({ 
    encoder: Charset.Normalize
});

Encoder Options Copy Chapter

RegExp
        String
        [Encoder Split Options](#encoder-split-options) | The expression used to split the content into terms | → include { letter: true, number: true } |

| Other options: | | | |
| dedupe | Boolean | Deduplicate consecutive letters, e.g. "missing" to "mising" | true |
| numeric | Boolean | By default, the extended numeric support (Triplets) inherits from chosen Encoder Split Options. You probably might want to disable Triplets to get a more exact result (fewer entries) in some cases. | true |
| minlength | Number | Set the minimum term length which should be added to the index. This limit does not apply to the forward tokenizer. You still get results when just typing "f" on a term "flexsearch" when e.g. minlength: 4 was used. | 1 |
| maxlength | Number | Set the maximum term length which should be added to the index. Larger content will drop. | 1 |
| rtl | Boolean | Force Right-To-Left encoding (you should just apply this when the string content was not already encoded as RTL) | false |
| normalize | true enable normalization (default)
false disable normalization
function(str) => str custom function | The normalization stage will apply basic charset normalization e.g. by replacing "é" to "e" | true |
| prepare | function(str) => str custom function | The preparation stage is a custom function direct followed when normalization was done | false |
| finalize | function([str]) => [str] custom function | The finalization stage is a custom function executed at the last task in the encoding pipeline (here it gets an array of tokens and need to return an array of tokens) | false |
| filter | Set(["and", "to", "be"])
function(str) => bool custom function
encoder.addFilter("and") | Stop-word filter is like a blacklist of words to be filtered out from indexing at all (e.g. "and", "to" or "be"). This is also very useful when using Context Search | false |
| stemmer | Map([["ing", ""], ["ies", "y"]])
encoder.addStemmer("ing", "") | Stemmer will normalize several linguistic mutations of the same word (e.g. "run" and "running", or "property" and "properties"). This is also very useful when using Context Search | false |
| mapper | Map([["é", "e"], ["ß", "ss"]])
encoder.addMapper("é", "e") | Mapper will replace a single char (e.g. "é" into "e") | false |
| matcher | Map([["and", "&"], ["usd", "$"]])
encoder.addMatcher("and", "&") | Matcher will do same as Mapper but instead of single chars it will replace char sequences | false |
| replacer | [/[^a-z0-9]/g, "", /([^aeo])h(.)/g, "$1$2"])
encoder.addReplacer(/[^a-z0-9]/g, "") | Replacer takes custom regular expressions and couldn't get optimized in the same way as Mapper or Matcher. You should take this as the last option when no other replacement can do the same. | false |
| cache | Boolean | In some very rare situations (large consecutive content with high cardinality) it might be useful to disable the internal event-loop-cache | true |

[!TIP]
The methods .addMapper(), .addMatcher() and .addReplacer() might be confusing. For this reason they will automatically resolve to the right one when just using the same method for every rule. You can simplify this e.g. by just use .addReplacer() for each of this 3 rules.

Encoder Split Options Copy Section

Array[String] | Toggle inclusion of specific chars on/off | false |

Custom Encoder Copy Chapter

Since it is very simple to create a custom Encoder, you are welcome to create your own.
e.g.

function customEncoder(content){
   const tokens = [];
   // split content into terms/tokens
   // apply your changes to each term/token
   // you will need to return an Array of terms/tokens
   // so just iterate through the input string and
   // push tokens to the array
   // ...
   return tokens;
}

const index = new Index({
   // set to strict when your tokenization was already done
   tokenize: "strict",
   encode: customEncoder
});

You can't extend to the built-in tokenizer "exact", "forward", "bidirectional" or "full".
If nothing of them are applicable for your task you should tokenize everything inside your custom encoder function.

If you get some good results please feel free creating a pull request to share your encoder to the community.

Encoder Processing Workflow Copy Section

1. Charset Normalization
2. Custom Preparation
3. Split Content (into terms, apply includes/excludes)
4. Filter: Pre-Filter
5. Stemmer (substitute term endings)
6. Filter: Post-Filter
7. Replace Chars (Mapper)
8. Letter Deduplication
9. Matcher (substitute partials)
10. Custom Regex (Replacer)
11. Custom Finalize

This workflow schema might help you to understand each step in the iteration:

Right-To-Left Support Copy Chapter

[!NOTE]
When a string is already encoded/interpreted as Right-To-Left you didn't need to use that. This option is just useful, when the source content wasn't encoded as RTL.

Just set the property rtl: true when creating the Encoder:

const encoder = new Encoder({ rtl: true });

CJK Word Break (Chinese, Japanese, Korean) Copy Chapter

const index = new Index({ encoder: Charset.CJK });
index.add(0, "一个单词");
var results = index.search("单词");

Built-In Language Packs Copy Chapter

• English: en
• German: de
• French: fr

Import Language Packs: ES6 Modules Copy Section

The most simple way to assign charset/language specific encoding via modules is:

import EnglishPreset from "flexsearch/lang/en";
const index = Index({
    charset: EnglishPreset
});

You can stack up and combine multiple presets:

import { Charset } from "flexsearch";
import EnglishPreset from "flexsearch/lang/en";

const index = Index({
    charset: new Encoder(
        Charset.LatinAdvanced,
        EnglishPreset,
        { minlength: 3 }
    )
});

You can also assign the encoder preset directly:

const index = Index({
    encoder: Charset.Default
});

Import Language Packs: ES5 Legacy Browser

When loading language packs, make sure that the library was loaded before:

<script src="dist/flexsearch.compact.min.js"></script>
<script src="dist/lang/en.min.js"></script>

The language packs are registered on FlexSearch.Language:

const index = FlexSearch.Index({
    encoder: FlexSearch.Language["en"]
});

You can stack up and combine multiple presets:

const index = FlexSearch.Index({
    charset: new FlexSearch.Encoder(
        FlexSearch.Charset.LatinAdvanced,
        FlexSearch.Language["en"],
        { minlength: 3 }
    )
});

Import Language Packs: Node.js

In Node.js all built-in language packs files are available by its scope:

const EnglishPreset = require("flexsearch/lang/en");
const index = new Index({
    encoder: EnglishPreset
});

Share Encoders Copy Section

Assigning the Encoder instance to the top level configuration will share the encoder to all fields. You should avoid this when contents of fields don't have the same type of content (e.g. one field contains terms, another contains numeric IDs).

Sharing the encoder can improve encoding efficiency and memory allocation, but when not properly used also has negative effect to the performance. You can share encoders to any type of index, also through multiple instances of indexes (also documents).

You should group similar types of contents to one encoder respectively. When you have different content types then define one for each of them.

In this example there are two Document-Indexes for two different documents "orders" and "billings". You can also share encoder to different fields of just one document.

// usual term encoding
const encoder_terms = Encoder(
    Charset.LatinAdvanced,
    // just add letters (no numbers)
    { include: { letter: true } }
);
// numeric encoding
const encoder_numeric = new Encoder(Charset.Default);

const orders = Document({
   document: {
       id: "id",
       index: [{
          field: "product_title",
          encoder: encoder_terms
       },{
          field: "product_details",
          encoder: encoder_terms
       },{
          field: "order_date",
          encoder: encoder_numeric
       },{
          field: "customer_id",
          encoder: encoder_numeric
       }]
   }
});

const billings = Document({
   document: {
      id: "id",
      index: [{
         field: "product_title",
         encoder: encoder_terms
      },{
         field: "product_content",
         encoder: encoder_terms
      },{
         field: "billing_date",
         encoder: encoder_numeric
      },{
         field: "customer_id",
         encoder: encoder_numeric
      }]
   }
});

Merge Documents Copy Section

When you have multiple document types (indexed by multiple indexes) but some of the data has same fields (like in the example above) and you can refer them by any identifier or key, you should consider merging those documents into one. This will hugely improve index size.

E.g. when you merge "orders" and "billings" from example above by ID, then you can use just one index:

const encoder_terms = Encoder(
    Charset.LatinAdvanced,
    // just add letters (no numbers)
    { include: { letter: true } }
);
const encoder_numeric = new Encoder(
    Charset.Default
);

const merged = Document({
   document: {
       id: "id",
       index: [{
          field: "product_title",
          encoder: encoder_terms
       },{
          field: "product_details",
          encoder: encoder_terms
       },{
          field: "order_date",
          encoder: encoder_numeric
       },{
           field: "billing_date",
           encoder: encoder_numeric
       },{
          field: "customer_id",
          encoder: encoder_numeric
       }]
   }
});

File: doc/export-import.md Copy Chapter

Import / Export (In-Memory) Copy Chapter

Persistent-Indexes and Worker-Indexes don't support Import/Export.

Export an Index or Document-Index to the folder /export/:

import { promises as fs } from "fs";

await index.export(async function(key, data){
  await fs.writeFile("./export/" + key, data, "utf8");
});

Import from folder /export/ into an Index or Document-Index:

const index = new Index({/* keep old config and place it here */});

const files = await fs.readdir("./export/");
for(let i = 0; i < files.length; i++){
  const data = await fs.readFile("./export/" + files[i], "utf8");
  await index.import(files[i], data);
}

You'll need to use the same configuration as you used before the export. Any changes on the configuration needs to be re-indexed.

Fast-Boot Serialization for Server-Side-Rendering (PHP, Python, Ruby, Rust, Java, Go, Node.js, ...) Copy Chapter

This is an experimental feature with limited support which probably might drop in future release. You're welcome to give some feedback.

When using Server-Side-Rendering you can create a different export which instantly boot up. Especially when using Server-side rendered content, this could help to restore a static index on page load. Document-Indexes aren't supported yet for this method.

When your index is too large you should use the default export/import mechanism.

You'll need Javascript to create the serialized output. Alternatively just create a small Node.js script to build the output.

As the first step populate the FlexSearch index with your contents.

You have two options:

1. Create a function as string Copy Section

const fn_string = index.serialize();

The contents of fn_string is a valid Javascript-Function declared as inject(index). Store it or place this somewhere in your code.

This function basically looks like:

function inject(index){
    index.reg = new Set([/* ... */]);
    index.map = new Map([/* ... */]);
    index.ctx = new Map([/* ... */]);
}

You can save this function by e.g. fs.writeFileSync("inject.js", fn_string); or place it as string in your SSR-generated markup.

After creating the index on client side just call the inject method like:

const index = new Index({/* use same configuration! */});
inject(index);

That's it.

You'll need to use the same configuration as you used before the export. Any changes on the configuration needs to be re-indexed.

2. Create just a function body as string Copy Section

Alternatively you can use lazy function declaration by passing false to the serialize function:

const fn_body = index.serialize(false);

You will get just the function body which looks like:

index.reg = new Set([/* ... */]);
index.map = new Map([/* ... */]);
index.ctx = new Map([/* ... */]);

Now you can place this in your code directly (name your index as index), or you can also create an inject function from it, e.g.:

const inject = new Function("index", fn_body);

This function is callable like the above example:

const index = new Index();
inject(index);

Export / Import (In-Memory) Copy Chapter

Node.js Copy Section

Persistent-Indexes and Worker-Indexes don't support Import/Export.

Export an Index or Document-Index to the folder /export/:

import { promises as fs } from "fs";

await index.export(async function(key, data){
  await fs.writeFile("./export/" + key, data, "utf8");
});

Import from folder /export/ into an Index or Document-Index:

const index = new Index({/* keep old config and place it here */});

const files = await fs.readdir("./export/");
for(let i = 0; i < files.length; i++){
  const data = await fs.readFile("./export/" + files[i], "utf8");
  await index.import(files[i], data);
}

You'll need to use the same configuration as you used before the export. Any changes on the configuration needs to be re-indexed.

Browser Copy Section

index.export(function(key, data){ 
    
    // you need to store both the key and the data!
    // e.g. use the key for the filename and save your data
    
    localStorage.setItem(key, data);
});

The size of the export corresponds to the memory consumption of the library. To reduce export size you have to use a configuration which has less memory footprint (use the table at the bottom to get information about configs and its memory allocation).

When your save routine runs asynchronously you have to use async/await or return a promise:

index.export(function(key, data){ 
    
    return new Promise(function(resolve){
        
        // do the saving as async

        resolve();
    });
});

Before you can import data, you need to create your index first. For document indexes provide the same document descriptor you used when export the data. This configuration isn't stored in the export.

const index = new Index({/* keep old config and place it here */});

To import the data just pass a key and data:

const data = localStorage.getItem(key);
index.import(key, data);

You need to import every key! Otherwise, your index does not work. You need to store the keys from the export and use this keys for the import (the order of the keys can differ).

The feature "fastupdate" is automatically disabled on import.

This is just for demonstration and is not recommended, because you might have other keys in your localStorage which aren't supported as an import:

var keys = Object.keys(localStorage);

for(let i = 0, key, data; i < keys.length; i++){
    key = keys[i]
    data = localStorage.getItem(key);
    index.import(key, data);
}

File: doc/keystore.md Copy Chapter

Big In-Memory Keystores Copy Chapter

The default maximum keystore limit for the In-Memory index is 2^24 of distinct terms/partials being stored (cardinality).
An additional register could be enabled and is dividing the index into self-balanced partitions.
The extended keystore is supported by any type of index.

const index = new Index({
    // e.g. set keystore range to 8-Bit:
    // 2^8 * 2^24 = 2^32 keys total
    keystore: 8 
});

You can theoretically store up to 2^88 keys (64-Bit address range).

The internal ID arrays scales automatically when limit of 2^31 has reached by using Proxy.

Persistent storages has no keystore limit by default.
You should not enable keystore when using persistent indexes, as long as you do not stress the buffer too hard before calling index.commit().

There is no additional memory cost when using a Keystore.

File: doc/persistent-clickhouse.md Copy Chapter

Clickhouse FlexSearch

Clickhouse is highly optimized for raw I/O and write access. When your index needs to be frequently updated, this storage engine can help to reduce blocking write access.

You'll need to install the npm package clickhouse into your project:

npm install [email protected]

Create an index and assign a Clickhouse storage adapter to it by using index.mount(db):

import { Index } from "flexsearch";
import Database from "flexsearch/db/clickhouse";

// your database configuration
const config = {
    host: "http://localhost",
    port: "8123",
    database: "default",
    basicAuth: null,
    debug: false
};

// create an index
const index = new Index();
// create db instance with optional prefix
const db = new Database("my-store", config);
// mount and await before transfering data
await index.mount(db);

// update the index as usual
index.add(1, "content...");
index.update(2, "content...");
index.remove(3);

// changes are automatically committed by default
// when you need to wait for the task completion, then you
// can use the commit method explicitely:
await index.commit();

Changes are automatically committed by default when you need to wait for the task completion, then you can use the await index.commit() method explicitely. You can disable the auto-commit feature optionally.

Configuration Copy Chapter

Custom DB Instance Copy Section

Pass a valid clickhouse instance on creation:

import { ClickHouse } from "clickhouse";
import Database from "flexsearch/db/clickhouse";
// assume you've created a custom database instance...
const database = new ClickHouse({/* config */});
// pass database instance as option
const db = new Database("my-store", {
    db: database
});

For every instance of DocumentIndex, WorkerIndex or Index (as standalone) you'll need to create a Database instance having its own name.

ID Type Copy Section

The Clickhouse driver does not properly support upgrading a merge key by ALTER TABLE. Therefore, the default type for ID is text.

You will save required disk space and also gain performance when define a numeric ID type explicitly.

import Database from "flexsearch/db/clickhouse";
// force integer type
const db = new Database("my-store", {
    type: "integer"
});
// ....
index.add("15712", "content...");
// IDs will cast to integer
let result = await index.search("content...");
// -> [15712]

BigInt:

const db = new Database("my-store", {
    type: "bigint"
});

To change the type later the index needs to be deleted by db.destroy() and re-created by db.mount().

Table Structure Copy Section

FlexSearch is creating different DATABASE entries for each index name e.g. "my-store". That is how you can use different stores for different indexes at the same time without getting collision of naming inheritance. Document Indexes will map their field names into table names respectively.

DATABASE
   |__TABLE map:field (FlexSearch Data) 
   |__TABLE ctx:field (FlexSearch Data) 
   |__TABLE tag:field (FlexSearch Data) 
   |__TABLE cfg:field (FlexSearch Data) 
   |__TABLE reg       (FlexSearch Data)

The default DATABASE in Clickhouse is commonly named default and is required to make the first connection for creating the new databases. You can force using a specific default DATABASE by option:

const db = new Database({
    database: "public"
});

File: doc/persistent-indexeddb.md Copy Chapter

IndexedDB FlexSearch

IndexedDB is a persistent storage supported by all major browsers.

Create an index and assign a IndexedDB storage adapter to it by using index.mount(db):

import { Index, IndexedDB } from "../dist/flexsearch.bundle.module.min.js";
// create an index
const index = new Index();
// create db instance with optional namespace
const db = new IndexedDB("my-store");
// mount and await before transfering data
await index.mount(db);

// update the index as usual
index.add(1, "content...");
index.update(2, "content...");
index.remove(3);

// changes are automatically committed by default
// when you need to wait for the task completion, then you
// can use the commit method explicitely:
await index.commit();

Changes are automatically committed by default when you need to wait for the task completion, then you can use the await index.commit() method explicitely. You can disable the auto-commit feature optionally.

Table Structure Copy Section

FlexSearch is creating different DATABASE entries for each index name e.g. "my-store". That is how you can use different stores for different indexes at the same time without getting collision of naming inheritance. Document Indexes will map their field names into table names respectively.

DATABASE
   |__OBJECTSTORE map:field (FlexSearch Data) 
   |__OBJECTSTORE ctx:field (FlexSearch Data) 
   |__OBJECTSTORE tag:field (FlexSearch Data) 
   |__OBJECTSTORE cfg:field (FlexSearch Data) 
   |__OBJECTSTORE reg       (FlexSearch Data)

File: doc/persistent-mongodb.md Copy Chapter

MongoDB FlexSearch

MongoDB is a common NoSQL document store providing everything you might want from a solid FlexSearch storage solution.

You'll need to install the npm package mongodb into your project:

npm install [email protected]

Create an index and assign a MongoDB storage adapter to it by using index.mount(db):

import { Index } from "flexsearch";
import Database from "flexsearch/db/mongodb";
// create an index
const index = new Index();
// create db instance with optional namespace
const db = new Database("my-store");
// mount and await before transfering data
await index.mount(db);

// update the index as usual
index.add(1, "content...");
index.update(2, "content...");
index.remove(3);

// changes are automatically committed by default
// when you need to wait for the task completion, then you
// can use the commit method explicitely:
await index.commit();

Changes are automatically committed by default when you need to wait for the task completion, then you can use the await index.commit() method explicitely. You can disable the auto-commit feature optionally.

Configuration Copy Chapter

Custom DB Instance Copy Section

Pass a valid mongodb instance on creation:

import { MongoClient } from "mongodb";
import Database from "flexsearch/db/mongodb";
// assume you've created a custom database instance...
const database = new MongoClient("mongodb://localhost:27017/");
// connect and await
await database.connect();
// pass database instance as option
const db = new Database("my-store", {
    db: database
});

For every instance of DocumentIndex, WorkerIndex or Index (as standalone) you'll need to create a Database instance having its own name.

Table Structure Copy Section

FlexSearch is creating different DATABASE entries for each index name e.g. "my-store". That is how you can use different stores for different indexes at the same time without getting collision of naming inheritance. Document Indexes will map their field names into table names respectively.

DATABASE
   |__COLLECTION map:field (FlexSearch Data) 
   |__COLLECTION ctx:field (FlexSearch Data) 
   |__COLLECTION tag:field (FlexSearch Data) 
   |__COLLECTION cfg:field (FlexSearch Data) 
   |__COLLECTION reg       (FlexSearch Data)

File: doc/persistent-postgres.md Copy Chapter

PostgreSQL FlexSearch

Postgres is widely used as the default database engine. It can hold large amounts of data and also shines in scaling capabilities.

You'll need to install the npm package pg-promise into your project:

npm install [email protected]

Create an index and assign a Postgres storage adapter to it by using index.mount(db):

import { Index } from "flexsearch";
import Database from "flexsearch/db/postgres";

// your database configuration
const config = {
    user: "postgres",
    pass: "postgres",
    host: "localhost",
    port: "5432",
    // database name:
    name: "postgres"
};

// create an index
const index = new Index();
// create db instance with optional prefix
const db = new Database("my-store", config);
// mount and await before transfering data
await index.mount(db);

// update the index as usual
index.add(1, "content...");
index.update(2, "content...");
index.remove(3);

// changes are automatically committed by default
// when you need to wait for the task completion, then you
// can use the commit method explicitely:
await index.commit();

Changes are automatically committed by default when you need to wait for the task completion, then you can use the await index.commit() method explicitely. You can disable the auto-commit feature optionally.

Configuration Copy Chapter

Custom DB Instance Copy Section

Pass a valid pg-promise instance on creation:

import pg_promise from "pg-promise";
import Database from "flexsearch/db/postgres";
const pgp = pg_promise();
// assume you've created a custom database instance...
const database = pgp(`postgres://${user}:${pass}@${host}:${port}/${name}`);
// pass database instance as option
const db = new Database("my-store", {
    db: database
});

ID Type Copy Section

The Postgres driver supports type upgrading by ALTER TABLE. The index will automatically upgrade to bigint or to string, starting from integer.

Once the type was upgraded, you'll need to re-create the index to switch back.

When you data content is including numeric strings (eg. for ID "15712") then defining the type will automatically cast into the right type:

import Database from "flexsearch/db/postgres";
// force integer type
const db = new Database("my-store", {
    type: "integer"
});
// ....
// the string will cast to integer
index.add("15712", "content...");

You will save required disk space and also gain the performance when using a numeric ID over a string type.

Table Structure Copy Section

FlexSearch is creating different SCHEMA for each index name e.g. "my-store". That is how you can use different stores for different indexes at the same time without getting collision of naming inheritance. Document Indexes will map their field names into table names respectively.

DATABASE_NAME
  |__ SCHEMA
        |__TABLE map:field (FlexSearch Data) 
        |__TABLE ctx:field (FlexSearch Data) 
        |__TABLE tag:field (FlexSearch Data) 
        |__TABLE cfg:field (FlexSearch Data) 
        |__TABLE reg       (FlexSearch Data)

You can force using a specific SCHEMA by option without passing a name:

const db = new Database({
    schema: "public"
});

Every Instance you made of DocumentIndex, WorkerIndex or Index (as standalone) you'll need to create a Database instance having its own schema.

File: doc/persistent-redis.md Copy Chapter

Redis FlexSearch

Redis is a standard storage engine in almost every modern application stack. It combines the advantages of both worlds: the performance of an InMemory engine by also keeping data persistent. The downside is that all the data has to keep in RAM.

You'll need to install the npm package redis into your project:

npm install [email protected]

Create an index and assign a Redis storage adapter to it by using index.mount(db):

import { Index } from "flexsearch";
import Database from "flexsearch/db/redis";

// Redis Connection
const config = {
    host: "localhost",
    port: "6379",
    user: null,
    pass: null
};

// create an index
const index = new Index();
// create db instance with optional prefix
const db = new Database("my-store", config);
// mount and await before transfering data
await index.mount(db);

// update the index as usual
index.add(1, "content...");
index.update(2, "content...");
index.remove(3);

// changes are automatically committed by default
// when you need to wait for the task completion, then you
// can use the commit method explicitely:
await index.commit();

Changes are automatically committed by default when you need to wait for the task completion, then you can use the await index.commit() method explicitely. You can disable the auto-commit feature optionally.

Configuration Copy Chapter

Custom DB Instance Copy Section

Pass a valid redis instance on creation:

import { createClient } from "redis";
import Database from "flexsearch/db/redis";
// assume you've created a custom redis instance...
const redis = await createClient("redis://localhost:6379").connect();
// pass instance as option
const db = new Database("my-store", {
    db: redis
});

ID Type Copy Section

Since Redis stores everything as Strings, you'll need to define the type of ID. Otherwise, you'll get back stringify ID results by default.
Also when your data content includes numeric strings (eg. "15712"), defining a type will automatically cast IDs into your choice:

import Database from "flexsearch/db/redis";
// force integer type
const db = new Database("my-store", {
    type: "integer"
});
// ....
index.add("15712", "content...");
// IDs will cast to integer
let result = await index.search("content...");
// -> [15712]

Table Structure Copy Section

FlexSearch is using a prefix style to prevent collision with other existing keys.

Prefix Schema when no name was set:

flexsearch|map:field
flexsearch|ctx:field
flexsearch|tag:field
flexsearch|cfg:field
flexsearch|reg
flexsearch|doc

Prefix Schema when name was set e.g. "my-store":

my-store|map:field
my-store|ctx:field
my-store|tag:field
my-store|cfg:field
my-store|reg
my-store|doc

For every instance of DocumentIndex, WorkerIndex or Index (as standalone) you'll need to create a Database instance having its own name.

File: doc/persistent-sqlite.md Copy Chapter

SQLite FlexSearch

SQLite offers you a compact and resource-friendly database which stores right into a portable db file on your filesystem.

You'll need to install the npm package sqlite3 into your project:

npm install [email protected]

Create an index and assign a SQLite storage adapter to it by using index.mount(db):

import { Index } from "flexsearch";
import Database from "flexsearch/db/sqlite";

// create an index
const index = new Index();
// create db instance with optional prefix
const db = new Database("my-store");
// mount and await before transfering data
await index.mount(db);

// update the index as usual
index.add(1, "content...");
index.update(2, "content...");
index.remove(3);

// changes are automatically committed by default
// when you need to wait for the task completion, then you
// can use the commit method explicitely:
await index.commit();

Changes are automatically committed by default when you need to wait for the task completion, then you can use the await index.commit() method explicitely. You can disable the auto-commit feature optionally.

Configuration Copy Chapter

Custom Filepath Copy Section

// set the filepath without passing a name:
const db = new Database({
    path: "./path-to-db/main.db"
});

// ...
await index.mount(db);

For any instance you made from type DocumentIndex, WorkerIndex or Index (as standalone) you'll need to create a Database instance having its own path.

Custom DB Instance Copy Section

Pass a valid sqlite3 instance on creation:

import sqlite3 from "sqlite3";
import Database from "flexsearch/db/sqlite";

// assume you've created a custom database instance...
const database = new sqlite3.Database("./path-to-db/main.db");

// pass database instance as option
const db = new Database("my-store", {
    db: database
});

// ...
await index.mount(db);

ID Type Copy Section

The SQLite driver does not properly support upgrading a key field type by ALTER TABLE. Therefore, the default type for ID is text.

You can save required disk space and also gain performance when defining a numeric ID type expicitely.

// pass type in options
const db = new Database("my-store", {
    type: "integer"
});

BigInt Range:

const db = new Database("my-store", {
    type: "bigint"
});

To change the ID type later you'll need to delete and re-create the index by calling index.db.destroy().

In-Memory Engine Copy Section

You can switch to SQLite native In-Memory store by passing :memory: as a name:

const db = new Database(":memory:");

Table Structure Copy Chapter

FlexSearch is creating different files for each index name e.g. "my-store". That is how you can use different stores for different indexes at the same time without getting collision of naming inheritance. Document Indexes will map their field names into table names respectively.

FILENAME
  |__ MAIN
        |__TABLES map:field (FlexSearch Data) 
        |__TABLES ctx:field (FlexSearch Data) 
        |__TABLES tag:field (FlexSearch Data) 
        |__TABLES cfg:field (FlexSearch Data) 
        |__TABLES reg       (FlexSearch Data)

File: doc/persistent.md Copy Chapter

Persistent Indexes

FlexSearch provides a new Storage Adapter where indexes are delegated through persistent storages.

Supported:

• IndexedDB (Browser)
• Redis
• SQLite
• Postgres
• MongoDB
• Clickhouse

The .export() and .import() methods are still available for non-persistent In-Memory indexes.

All search capabilities are available on persistent indexes like:

• Context-Search
• Suggestions
• Cursor-based Queries (Limit/Offset)
• Scoring (supports a resolution of up to 32767 slots)
• Document-Search
• Partial Search
• Multi-Tag-Search
• Boost Fields
• Custom Encoder
• Resolver
• Tokenizer
• Document Store (incl. enrich results)
• Worker Threads to run in parallel
• Auto-Balanced Cache (top queries + last queries)

All persistent variants are optimized for larger sized indexes under heavy workload. Almost every task will be streamlined to run in batch/parallel, getting the most out of the selected database engine. Whereas the InMemory index can't share their data between different nodes when running in a cluster, every persistent storage can handle this by default.

Examples Node.js:

• nodejs-commonjs:
  • basic-persistent
  • document-persistent
• nodejs-esm:
  • basic-persistent
  • document-persistent

Examples Browser:

• browser-legacy:
  • basic-persistent
  • document-persistent
• browser-module:
  • basic-persistent
  • document-persistent

Browser (IndexedDB) Copy Chapter

import { Index, IndexedDB } from "../dist/flexsearch.bundle.module.min.js";
// create an index
const index = new Index();
// create db instance with optional prefix
const db = new IndexedDB("my-store");
// mount and await before transfering data
await index.mount(db);

// update the index as usual
index.add(1, "content...");
index.update(2, "content...");
index.remove(3);

// changes are automatically committed by default
// when you need to wait for the task completion, then you
// can use the commit method explicitely:
await index.commit();

Alternatively mount a store by index creation:

const index = new Index({
    db: new IndexedDB("my-store")
});

// await for the db response before access the first time
await index.db;
// apply changes to the index
// ...

Query against a persistent storage just as usual:

const result = await index.search("gulliver");

Auto-Commit is enabled by default and will process changes asynchronously in batch.
You can fully disable the auto-commit feature and perform them manually:

const index = new Index({
    db: new Storage("my-store"),
    commit: false
});
// update the index
index.add(1, "content...");
index.update(2, "content...");
index.remove(3);

// transfer all changes to the db
await index.commit();

You can call the commit method manually also when commit: true option was set.

Node.js Copy Chapter

import { Index } from "flexsearch";
import Database from "flexsearch/db/postgres";
// create an index
const index = new Index();
// create db instance with optional prefix
const db = new Database("my-store");
// mount and await before transfering data
await index.mount(db);

// update the index as usual
index.add(1, "content...");
index.update(2, "content...");
index.remove(3);

// changes are automatically committed by default
// when you need to wait for the task completion, then you
// can use the commit method explicitely:
await index.commit();

Benchmark Copy Chapter

The benchmark was measured in "terms per second".

Search 1: Single term query
Search N: Multi term query (Context-Search)

The benchmark was executed against a single client.

Delete Store + Migration Copy Chapter

Actually there exist no migration tool. You will probably need some kind of migration on future updates or when you need to re-create the index on the database.

[!CAUTION]
Please use the methods index.destroy() and index.clear() carefully. This methods will delete contents (truncate, drop) from the database accordingly to the passed name on initialization.

Just clear all contents (truncate equivalent) from a store which connected to an index:

// always define a unique name when assigning a storage
const db = new Database("my-store", config);
await index.mount(db);
// truncate all contents
await index.clear();

Drop all tables (and its schema):

// always define a unique name when assigning a storage
const db = new Database("my-store", config);
await index.mount(db);
// drop all associated tables
await index.destroy();

A full migration cycle could be combined by:

// always define a unique name when assigning a storage
const db = new Database("my-store", config);
await index.mount(db);
// drop all associated tables
await index.destroy();
// when destroyed you'll need to mount again
// to run table creation
await index.mount(db);
// access index ...

File: doc/resolver.md Copy Chapter

Resolver (Complex Queries) Copy Chapter

Retrieve an unresolved result:

const raw = index.search("a short query", { 
    resolve: false
});

You can apply and chain different resolver methods to the raw result, e.g.:

raw.and( ... )
   .and( ... )
   .boost(2)
   .or( ... ,  ... )
   .limit(100)
   .xor( ... )
   .not( ... )
   // final resolve
   .resolve({
       limit: 10,
       offset: 0,
       enrich: true
   });

The default resolver:

const raw = index.search("a short query", { 
    resolve: false
});
const result = raw.resolve();

Alternatively you can create a Resolver by passing an initial query:

import { Resolver } from "flexsearch";
const raw = new Resolver({
    // pass the index is required when query was set
    index: index,
    query: "a short query"
});
const result = raw.resolve();

Chainable Boolean Operations Copy Section

The basic concept explained:

// 1. get one or multiple unresolved results
const raw1 = index.search("a short query", { 
    resolve: false
});
const raw2 = index.search("another query", {
    resolve: false,
    boost: 2
});

// 2. apply and chain resolver operations
const raw3 = raw1.and(raw2, /* ... */);
// raw1 has changed, raw2 is same, raw3 refers to raw1
// you can access the raw result by
console.log("The aggregated result is:", raw3.result)
// apply further operations ...

// 3. resolve final result
const result = raw3.resolve({
    limit: 100,
    offset: 0
});
console.log("The final result is:", result)

Chain operations and nest inline queries:

const result = index.search("further query", {
    // set resolve to false on the first query
    resolve: false,
    // boost the first query
    boost: 2
})
.or({
    // nested expression
    and: [{
        query: "a query"
    },{
        query: "another query"
    }]
})
.not({
    query: "some query"
})
// resolve the result
.resolve({
    limit: 100,
    offset: 0
});

Alternatively you can create a Resolver by passing an initial query:

import { Resolver } from "flexsearch";
const result = new Resolver({
    // pass the index is required when query was set
    index: index,
    query: "further query",
    boost: 2
})
.or({
    and: [{
        query: "a query"
    },{
        // you can bind a different index for this
        // query when IDs are from same source
        index: index,
        query: "another query"
    }]
})
.not({
    index: index,
    query: "some query"
})
.resolve({
    limit: 100,
    offset: 0
});

When all queries are made against the same index, you can skip the index in every resolver stage followed after initially calling new Resolve({ index: ... }):

import { Resolver } from "flexsearch";
const result = new Resolver({
    index: index,
    query: "a query"
})
.and({ query: "another query" })
.or ({ query: "further query", boost: 2 })
.not({ query: "some query" })
.resolve(100);

Resolver Tasks Copy Chapter

`.or(options,...)`
        `.not(options,...)`
        `.xor(options,...)` | Apply an operation | Returns a `Resolver` when `resolve` was not set to `false` within the options, otherwise it returns the result (or promise in async context). |

| .limit(number)
.offset(number)
.boost(number) | Apply boost, limit and offset to the result | Returns a Resolver |
| .resolve(options) | Resolve results | Returns the final result or promise in async context (can't be executed twice) |

Resolver Options Copy Chapter

`Document` | Assign the index where the query should be applied to |  |

| suggest | Boolean | Enables suggestions in results | false |
| boost | Number | Boost or reduce the score of this query | 0 |
| async | Boolean | Use a parallel processing workflow | false |
| queue | Boolean | Use a queued processing workflow | false |
| and
or
not
xor | Array<ResolverOptions> | Apply nested queries | |
| resolve | Boolean | Resolve the result immediately or not. When set to true all final resolve options are also allowed and there can't exist any further resolver operations. | false |
| Document Resolver Options: | | | |
| field``pluck | String | Select the Document field on which the query should apply to. | |
| Final Resolve Options: | | | |
| enrich | Boolean | Enrich IDs from the results with the corresponding documents (for Document Indexes only) | true |
| highlight | Highlighting Options
String | Highlight query matches in the result (for Document Indexes only) | |
| limit | Number | Sets the limit of results | 100 |
| offset | Boolean | Apply offset (skip items) | 0 |

Using Cached Queries Copy Section

import { Resolver } from "flexsearch";
const result = new Resolver({
    index: index,
    query: "a query",
    cache: true
})
.and({ 
    query: "another query",
    cache: true
})
.resolve(100);

Async Resolver Copy Chapter

Using Async Queries (incl. Runtime Balancer) Copy Section

All async tasks will run in parallel, balanced by the runtime observer:

import { Resolver } from "flexsearch";
const resolver = new Resolver({
    index: index,
    query: "a query",
    async: true
})
.and({ 
    query: "another query",
    async: true
})
.or({
    query: "some query",
    async: true
});
const result = await resolver.resolve(100);

When you need to access the raw result resolver.result you should await for the task completion of all added resolver stages up to this point.

const resolver = new Resolver({
    index: index,
    query: "a query",
    async: true
})
.and({ 
    query: "another query",
    async: true
});

// await for the task completion
await resolver.await;
// get the raw result
const raw = resolver.result;
// continue adding further tasks ...

Queuing Async Queries Copy Section

All queued tasks will run consecutively, also balanced by the runtime observer:

import { Resolver } from "flexsearch";
const resolver = await new Resolver({
    index: index,
    query: "a query",
    async: true
})
.and({ 
    query: "another query",
    queue: true
})
.or({
    query: "some query",
    queue: true
})
.resolve(100);

When tasks are processed consecutively, it will skip specific resolver stages when there is no result expected.

Compare Parallel VS. Consecutive Copy Section

When using the parallel workflow by passing { async: true }, all resolver stages will send their requests (including nested tasks) to the DB immediately and calculate the results in the right order as soon as the request resolves. When the overall workload of your applications has some free resources, a parallel request workflow improves performance compared to the consecutive counterpart.

When using the consecutive workflow by passing { queue: true }, all resolver stages will send their requests (including nested tasks) to the DB only when the previous request resolves. The advantage of this variant is when a stage becomes invalid because of the previous result, it can skip the request completely and continue with the next stage. This can reduce the overall workload.

File: doc/result-highlighting.md Copy Chapter

Result Highlighting Copy Chapter

Demo: Auto-Complete

Result highlighting could be just enabled when using Document-Index with enabled document store by passing option store on creation.

Alternatively you can simply upgrade id-content-pairs to a flat document when calling .add(...).

// 1. create the document index
const index = new Document({
  document: {
    // using store is required  
    store: true,
    index: [{
      field: "title",
      tokenize: "forward",
      encoder: Charset.LatinBalance
    }]
  }
});

// 2. add data
index.add({
  "id": 1,
  "title": "Carmencita"
});
index.add({
  "id": 2,
  "title": "Le clown et ses chiens"
});

// 3. perform a query
const result = index.search({
  query: "karmen or clown or not found",
  // also get results when query has no exact match  
  suggest: true,
  // use highlighting options or pass a template, where $1 is
  // a placeholder for the matched partial
  highlight: "<b>$1</b>",
  // optionally pick and apply search to just
  // one field and get back a flat result
  pluck: "title"
});

The result will look like:

[{
  "id": 1,
  "highlight": "<b>Carmen</b>cita"
},{
  "id": 2,
  "highlight": "Le <b>clown</b> et ses chiens"
}]

There are several options to customize result highlighting.

Highlighting Options Copy Section

Number | Limit the total length of highlighted content (add ellipsis by default). The template markup does not stack to the total length. | `false` |

| ellipsis | Ellipsis Options
Boolean
String | Define a custom ellipsis or disable | "..." |
| merge | Boolean | Wrap consecutive matches by just a single template | false |
| clip | Boolean | Allow to clip terms | true |
| Boundary Options | | | |
| boundary.total | Number | Limit the total length of highlighted content | false |
| boundary.before | Number | Limit the length of content before highlighted parts | (auto) |
| boundary.after | Number | Limit the length of content after highlighted parts | (auto) |
| Ellipsis Options | | | |
| ellipsis.template | String | The template to be applied on ellipsis (e.g. "<i>$1</i>"), where $1 is a placeholder for the ellipsis | (mandatory) |
| ellipsis.pattern | Boolean
String | Define a custom ellipsis or disable | "..." |

Boundaries & Alignment Copy Section

You can limit the length of the highlighted content and also define a custom ellipsis.
By default, all matches are automatically aligned to fit into the total size. You can customize these boundaries when also passing limits for surrounded text.

Add some content to the index:

index.add({
    "id": 1,
    "title": "Lorem ipsum dolor sit amet consetetur sadipscing elitr."
});

Perform a highlighted search (no boundaries):

const result = index.search({
  query: "sit amet",
  highlight: "<b>$1</b>"
});

Result:

"Lorem ipsum dolor <b>sit</b> <b>amet</b> consetetur sadipscing elitr."

Limit total boundary

const result = index.search({
  query: "sit amet",
  highlight: {
    template: "<b>$1</b>",
    boundary: 32
  }
});

The highlight markup does not stack to the total length.

Result:

"...um dolor <b>sit</b> <b>amet</b> consetet..."

Define custom ellipsis (text)

const result = index.search({
  query: "sit amet",
  highlight: {
    template: "<b>$1</b>",
    boundary: 32,
    ellipsis: "[...]"
  }
});

Result:

"[...] dolor <b>sit</b> <b>amet</b> conset[...]"

You can also apply "" or false to remove ellipsis.

Do not clip terms

const result = index.search({
  query: "sit amet",
  highlight: {
    template: "<b>$1</b>",
    boundary: 32,
    clip: false
  }
});

Result:

"... dolor <b>sit</b> <b>amet</b> ..."

Merge consecutive matches

const result = index.search({
  query: "sit amet",
  highlight: {
    template: "<b>$1</b>",
    boundary: 32,
    merge: true
  }
});

Result:

"...um dolor <b>sit amet</b> consetet..."

Limit surrounded text

Each of the boundary limits are optionally. Combine them as needed.

const result = index.search({
  query: "sit amet",
  highlight: {
    template: "<b>$1</b>",
    boundary: {
      // length before match  
      before: 3,
      // length after match  
      after: 15,
      // overall length  
      total: 32  
    }
  }
});

Result:

"...or <b>sit</b> <b>amet</b> consetetur sad..."

Use custom ellipsis (markup)

When using markup within ellipsis, the markup length stack up to the total boundary. You can provide a template also for ellipsis to apply total boundary properly by do not stack up the markup length.

const result = index.search({
  query: "sit amet",
  highlight: {
    template: "<b>$1</b>",
    // limit the total length to 32 chars
    boundary: 32,
    ellipsis: {
        // pass a template, where $1 is
        // a placeholder for the ellipsis
        template: "<i>$1</i>",
        // define custom ellipsis
        pattern: "..."
    }
  }
});

Result:

"<i>...</i> dolor <b>sit</b> <b>amet</b> conset<i>...</i>"

Using Result Highlighting on Resolver Copy Section

When using complex queries by Resolver you can pass a highlight option to any one of the resolver stages that is also including a query. The last resolver stage will then automatically inherit necessary options.

const raw = new Resolver({
  index: index,
  field: "title",
  query: "some query"
})
.or({
    field: "title",
    // highlight requires a query
    query: "highlight this",
    // define on a single resolver stage
    highlight: { /* ... */ }
})
.not({
    field: "title",
    query: "undefined",
})
.resolve();

File: doc/worker.md Copy Chapter

Worker Parallelism (Browser + Node.js)

Worker Index Copy Section

Using a Worker-Index is pretty much the same as using a standard Index.

Worker-Index always return a Promise for all methods called on the index.

When adding/updating/removing large bulks of content to the index, it is recommended to use the async version of each method to prevent blocking issues on the main thread. Read more about Asynchronous Runtime Balancer

Worker Document Copy Section

The internal worker model is distributed by document fields and will solve subtasks in parallel.

Documents will create worker automatically for each field by just apply the option worker: true:

const index = new Document({
    worker: true,
    document: { 
        id: "id",
        index: ["name", "title"],
        tag: ["cat"]
    }
});

index.add({ 
    id: 1, cat: "catA", name: "Tom", title: "some" 
}).add({
    id: 2, cat: "catA", name: "Ben", title: "title"
}).add({ 
    id: 3, cat: "catB", name: "Max", title: "to"
}).add({ 
    id: 4, cat: "catB", name: "Tim", title: "index"" 
});

When you perform a field search through multiple fields then this task is being well-balanced through all involved workers, which can solve their subtasks independently.

Examples Copy Chapter

ES6 Module (Bundle): Copy Section

When using one of the bundles from /dist/ you can create a Worker-Index:

import { Worker } from "./dist/flexsearch.bundle.module.min.js";
const index = new Worker({/* options */ });
await index.add(1, "some");
await index.add(2, "content");
await index.add(3, "to");
await index.add(4, "index");

ES6 Module (Non-Bundle): Copy Section

When not using a bundle you can take the worker file from /dist/ folder as follows:

import Worker from "./dist/module/worker.js";
const index = new Worker({/* options */ });
index.add(1, "some")
     .add(2, "content")
     .add(3, "to")
     .add(4, "index");

Browser Legacy (Bundle): Copy Section

When loading a legacy bundle via script tag (non-modules):

const index = new FlexSearch.Worker({/* options */ });
await index.add(1, "some");
await index.add(2, "content");
await index.add(3, "to");
await index.add(4, "index");

Worker (Node.js) Copy Section

The worker model for Node.js is based on native worker threads and works exactly the same way:

const { Document } = require("flexsearch");

const index = new Document({
    worker: true,
    document: {
        id: "id",
        index: ["name", "title"],
        tag: ["cat"]
    }
});

Or create a single worker instance for a non-document index:

const { Worker } = require("flexsearch");
const index = new Worker({ options });

Worker-Index Options Copy Section

Worker-Index Options extends the default Index Options, you can apply also.

Extern Worker Configuration Copy Chapter

When using Worker by also assign custom functions to the options e.g.:

• Custom Encoder
• Custom Encoder methods (normalize, prepare, finalize)
• Custom Score (function)
• Custom Filter (function)
• Custom Fields (function)

... then you'll need to move your field configuration into a file which exports the configuration as a default export. The field configuration is not the whole Document-Descriptor.

When not using custom functions in combination with Worker you can skip this part.

Since every field resolves into a dedicated Worker, also every field which includes custom functions should have their own configuration file accordingly.

Let's take this document descriptor:

{
    document: {
        index: [{
            // this is the field configuration
            // ---->
            field: "custom_field",
            custom: function(data){
                return "custom field content";
            }
            // <------
        }]
    }
};

The configuration which needs to be available as a default export is:

{
    field: "custom_field",
    custom: function(data){
        return "custom field content";
    }
};

You're welcome to make some suggestions how to improve the handling of extern configuration.

Example Node.js: Copy Section

An extern configuration for one WorkerIndex, let's assume it is located in ./custom_field.js:

const { Charset } = require("flexsearch");
const { LatinSimple } = Charset;
// it requires a default export:
module.exports = {
    encoder: LatinSimple,
    tokenize: "forward",
    // custom function:
    custom: function(data){
        return "custom field content";
    }
};

Create Worker Index along the configuration above:

const { Document } = require("flexsearch");
const flexsearch = new Document({
    worker: true,
    document: {
        index: [{
            // the field name needs to be set here
            field: "custom_field",
            // path to your config from above:
            config: "./custom_field.js",
        }]
    }
});

Browser (ESM) Copy Section

An extern configuration for one WorkerIndex, let's assume it is located in ./custom_field.js:

import { Charset } from "./dist/flexsearch.bundle.module.min.js";
const { LatinSimple } = Charset;
// it requires a default export:
export default {
    encoder: LatinSimple,
    tokenize: "forward",
    // custom function:
    custom: function(data){
        return "custom field content";
    }
};

Create Worker Index with the configuration above:

import { Document } from "./dist/flexsearch.bundle.module.min.js";
// you will need to await for the response!
const flexsearch = await new Document({
    worker: true,
    document: {
        index: [{
            // the field name needs to be set here
            field: "custom_field",
            // Absolute URL to your config from above:
            config: "http://localhost/custom_field.js"
        }]
    }
});

Here it needs the absolute URL, because the WorkerIndex context is from type Blob and you can't use relative URLs starting from this context.

Test Case Copy Section

As a test the whole IMDB data collection was indexed, containing of:

JSON Documents: 9,273,132
Fields: 83,458,188
Tokens: 128,898,832

The used index configuration has 2 fields (using bidirectional context of depth: 1), 1 custom field, 2 tags and a full datastore of all input json documents.

A non-Worker Document index requires 181 seconds to index all contents.
The Worker index just takes 32 seconds to index them all, by processing every field and tag in parallel. For such large content it is a quite impressive result.

Export / Import Worker Indexes (Node.js) Copy Chapter

Worker will save/load their data dedicated and does not need the message channel for the data transfer.

Basic Worker Index Copy Section

This feature follows the strategy of using Extern Worker Configuration in combination with Basic Export Import.

Example (CommonJS): basic-worker-export-import
Example (ESM): basic-worker-export-import

Provide the index configuration and keep it, because it isn't stored. Provide a parameter config which is including the filepath to the extern configuration file:

const dirname = import.meta.dirname;
const config = {
    tokenize: "forward",
    config: dirname + "/config.js"
};

Any changes you made to the configuration will almost require a full re-index.

Provide the extern configuration file e.g. /config.js as a default export including the methods export and import:

import { promises as fs } from "fs";

export default {
    tokenize: "forward",
    export: async function(key, data){
        // like the usual export write files by key + data
        await fs.writeFile("./export/" + key, data, "utf8");
    },
    import: async function(index){
        // get the file contents of the export directory
        let files = await fs.readdir("./export/");
        files = await Promise.all(files);
        // loop through the files and push their contents to the index
        // by also passing the filename as the first parameter
        for(let i = 0; i < files.length; i++){
            const data = await fs.readFile("./export/" + files[i], "utf8");
            index.import(files[i], data);
        }
    }
};

Create your index by assigning the configuration file from above:

import { Worker as WorkerIndex } from "flexsearch";
const index = await new WorkerIndex(config);
// add data to the index
// ...

Export the index:

await index.export();

Import the index:

// create the same type of index you have used by .export()
// along with the same configuration
const index = await new WorkerIndex(config);
await index.import();

Document Worker Index Copy Section

This feature follows the strategy of using Extern Worker Configuration in combination with Document Export Import.

Document Worker exports all their feature including:

• Multi-Tag Indexes
• Context-Search Indexes
• Document-Store

Example (CommonJS): document-worker-export-import
Example (ESM): document-worker-export-import

Provide the index configuration and keep it, because it isn't stored. Provide a parameter config which is including the filepath to the extern configuration file:

const dirname = import.meta.dirname;
const config = {
    worker: true,
    document: {
        id: "tconst",
        store: true,
        index: [{
            field: "primaryTitle",
            config: dirname + "/config.primaryTitle.js"
        },{
            field: "originalTitle",
            config: dirname + "/config.originalTitle.js"
        }],
        tag: [{
            field: "startYear"
        },{
            field: "genres"
        }]
    }
};

Any changes you made to the configuration will almost require a full re-index.

Provide the extern configuration file as a default export including the methods export and import:

import { promises as fs } from "fs";

export default {
    tokenize: "forward",
    export: async function(key, data){
        // like the usual export write files by key + data
        await fs.writeFile("./export/" + key, data, "utf8");
    },
    import: async function(file){
        // instead of looping you will get the filename as 2nd paramter
        // just return the loaded contents as a string
        return await fs.readFile("./export/" + file, "utf8");
    }
};

Create your index by assigning the configuration file from above:

import { Document } from "flexsearch";
const document = await new Document(config);
// add data to the index
// ...

Export the index by providing a key-data handler:

await document.export(async function(key, data){
    await fs.writeFile("./export/" + key, data, "utf8");
});

Import the index:

const files = await fs.readdir("./export/");
// create the same type of index you have used by .export()
// along with the same configuration
const document = await new Document(config);
await Promise.all(files.map(async file => {
    const data = await fs.readFile("./export/" + file, "utf8");
    // call import (async)
    await document.import(file, data);
}));

CSP-friendly Worker (Browser) Copy Chapter

When using worker via one of the bundled versions (e.g. flexearch.bundle.min.js), the worker will be created by code generation under the hood. This might have issues when using strict CSP settings.

You can overcome this issue by using the non-bundled versions e.g. dist/module/ or by passing the filepath to the worker file instead of true like worker: "dist/module/worker/worker.js".

File: example/nodejs-commonjs/basic/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-commonjs/basic/package.json Copy Chapter

{
"name": "nodejs-commonjs-basic",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-commonjs/basic-export-import/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-commonjs/basic-export-import/package.json Copy Chapter

{
"name": "nodejs-commonjs-basic-export-import",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-commonjs/basic-persistent/README.md Copy Chapter

npm install

npm install [email protected]

node index.js

File: example/nodejs-commonjs/basic-persistent/package.json Copy Chapter

{
"name": "nodejs-commonjs-basic-persistent",
"dependencies": {
"flexsearch": "^0.8.200",
"sqlite3": "^5.1.7"
}
}

File: example/nodejs-commonjs/basic-resolver/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-commonjs/basic-resolver/package.json Copy Chapter

{
"name": "nodejs-commonjs-basic-resolver",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-commonjs/basic-suggestion/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-commonjs/basic-suggestion/package.json Copy Chapter

{
"name": "nodejs-commonjs-basic-suggestion",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-commonjs/basic-worker/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-commonjs/basic-worker/package.json Copy Chapter

{
"name": "nodejs-commonjs-basic-worker",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-commonjs/basic-worker-export-import/package.json Copy Chapter

{
"name": "nodejs-commonjs-basic-worker-extern-config",
"dependencies": {
"flexsearch": "github:nextapps-de/flexsearch"
}
}

File: example/nodejs-commonjs/basic-worker-extern-config/package.json Copy Chapter

{
"name": "nodejs-commonjs-basic-worker-extern-config",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-commonjs/document/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-commonjs/document/data.json Copy Chapter

[
{
"tconst": "tt0000001",
"titleType": "short",
"primaryTitle": "Carmencita",
"originalTitle": "Carmencita",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000002",
"titleType": "short",
"primaryTitle": "Le clown et ses chiens",
"originalTitle": "Le clown et ses chiens",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "5",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000003",
"titleType": "short",
"primaryTitle": "Pauvre Pierrot",
"originalTitle": "Pauvre Pierrot",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "4",
"genres": [
"Animation",
"Comedy",
"Romance"
]
},
{
"tconst": "tt0000004",
"titleType": "short",
"primaryTitle": "Un bon bock",
"originalTitle": "Un bon bock",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "12",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000005",
"titleType": "short",
"primaryTitle": "Blacksmith Scene",
"originalTitle": "Blacksmith Scene",
"isAdult": 0,
"startYear": "1893",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Comedy",
"Short"
]
},
{
"tconst": "tt0000006",
"titleType": "short",
"primaryTitle": "Chinese Opium Den",
"originalTitle": "Chinese Opium Den",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short"
]
},
{
"tconst": "tt0000007",
"titleType": "short",
"primaryTitle": "Corbett and Courtney Before the Kinetograph",
"originalTitle": "Corbett and Courtney Before the Kinetograph",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short",
"Sport"
]
},
{
"tconst": "tt0000008",
"titleType": "short",
"primaryTitle": "Edison Kinetoscopic Record of a Sneeze",
"originalTitle": "Edison Kinetoscopic Record of a Sneeze",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000009",
"titleType": "movie",
"primaryTitle": "Miss Jerry",
"originalTitle": "Miss Jerry",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "45",
"genres": [
"Romance"
]
},
{
"tconst": "tt0000010",
"titleType": "short",
"primaryTitle": "Leaving the Factory",
"originalTitle": "La sortie de l'usine Lumière à Lyon",
"isAdult": 0,
"startYear": "1895",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
}
]

File: example/nodejs-commonjs/document/package.json Copy Chapter

{
"name": "nodejs-commonjs-document",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-commonjs/document-export-import/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-commonjs/document-export-import/data.json Copy Chapter

[
{
"tconst": "tt0000001",
"titleType": "short",
"primaryTitle": "Carmencita",
"originalTitle": "Carmencita",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000002",
"titleType": "short",
"primaryTitle": "Le clown et ses chiens",
"originalTitle": "Le clown et ses chiens",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "5",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000003",
"titleType": "short",
"primaryTitle": "Pauvre Pierrot",
"originalTitle": "Pauvre Pierrot",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "4",
"genres": [
"Animation",
"Comedy",
"Romance"
]
},
{
"tconst": "tt0000004",
"titleType": "short",
"primaryTitle": "Un bon bock",
"originalTitle": "Un bon bock",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "12",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000005",
"titleType": "short",
"primaryTitle": "Blacksmith Scene",
"originalTitle": "Blacksmith Scene",
"isAdult": 0,
"startYear": "1893",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Comedy",
"Short"
]
},
{
"tconst": "tt0000006",
"titleType": "short",
"primaryTitle": "Chinese Opium Den",
"originalTitle": "Chinese Opium Den",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short"
]
},
{
"tconst": "tt0000007",
"titleType": "short",
"primaryTitle": "Corbett and Courtney Before the Kinetograph",
"originalTitle": "Corbett and Courtney Before the Kinetograph",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short",
"Sport"
]
},
{
"tconst": "tt0000008",
"titleType": "short",
"primaryTitle": "Edison Kinetoscopic Record of a Sneeze",
"originalTitle": "Edison Kinetoscopic Record of a Sneeze",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000009",
"titleType": "movie",
"primaryTitle": "Miss Jerry",
"originalTitle": "Miss Jerry",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "45",
"genres": [
"Romance"
]
},
{
"tconst": "tt0000010",
"titleType": "short",
"primaryTitle": "Leaving the Factory",
"originalTitle": "La sortie de l'usine Lumière à Lyon",
"isAdult": 0,
"startYear": "1895",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
}
]

File: example/nodejs-commonjs/document-export-import/package.json Copy Chapter

{
"name": "nodejs-commonjs-document-export-import",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-commonjs/document-persistent/README.md Copy Chapter

npm install

npm install [email protected]

node index.js

File: example/nodejs-commonjs/document-persistent/data.json Copy Chapter

[
{
"tconst": "tt0000001",
"titleType": "short",
"primaryTitle": "Carmencita",
"originalTitle": "Carmencita",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000002",
"titleType": "short",
"primaryTitle": "Le clown et ses chiens",
"originalTitle": "Le clown et ses chiens",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "5",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000003",
"titleType": "short",
"primaryTitle": "Pauvre Pierrot",
"originalTitle": "Pauvre Pierrot",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "4",
"genres": [
"Animation",
"Comedy",
"Romance"
]
},
{
"tconst": "tt0000004",
"titleType": "short",
"primaryTitle": "Un bon bock",
"originalTitle": "Un bon bock",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "12",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000005",
"titleType": "short",
"primaryTitle": "Blacksmith Scene",
"originalTitle": "Blacksmith Scene",
"isAdult": 0,
"startYear": "1893",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Comedy",
"Short"
]
},
{
"tconst": "tt0000006",
"titleType": "short",
"primaryTitle": "Chinese Opium Den",
"originalTitle": "Chinese Opium Den",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short"
]
},
{
"tconst": "tt0000007",
"titleType": "short",
"primaryTitle": "Corbett and Courtney Before the Kinetograph",
"originalTitle": "Corbett and Courtney Before the Kinetograph",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short",
"Sport"
]
},
{
"tconst": "tt0000008",
"titleType": "short",
"primaryTitle": "Edison Kinetoscopic Record of a Sneeze",
"originalTitle": "Edison Kinetoscopic Record of a Sneeze",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000009",
"titleType": "movie",
"primaryTitle": "Miss Jerry",
"originalTitle": "Miss Jerry",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "45",
"genres": [
"Romance"
]
},
{
"tconst": "tt0000010",
"titleType": "short",
"primaryTitle": "Leaving the Factory",
"originalTitle": "La sortie de l'usine Lumière à Lyon",
"isAdult": 0,
"startYear": "1895",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
}
]

File: example/nodejs-commonjs/document-persistent/package.json Copy Chapter

{
"name": "nodejs-commonjs-document-persistent",
"dependencies": {
"flexsearch": "^0.8.200",
"sqlite3": "^5.1.7"
}
}

File: example/nodejs-commonjs/document-resolver/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-commonjs/document-resolver/package.json Copy Chapter

{
"name": "nodejs-commonjs-document-resolver",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-commonjs/document-worker/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-commonjs/document-worker/data.json Copy Chapter

[
{
"tconst": "tt0000001",
"titleType": "short",
"primaryTitle": "Carmencita",
"originalTitle": "Carmencita",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000002",
"titleType": "short",
"primaryTitle": "Le clown et ses chiens",
"originalTitle": "Le clown et ses chiens",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "5",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000003",
"titleType": "short",
"primaryTitle": "Pauvre Pierrot",
"originalTitle": "Pauvre Pierrot",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "4",
"genres": [
"Animation",
"Comedy",
"Romance"
]
},
{
"tconst": "tt0000004",
"titleType": "short",
"primaryTitle": "Un bon bock",
"originalTitle": "Un bon bock",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "12",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000005",
"titleType": "short",
"primaryTitle": "Blacksmith Scene",
"originalTitle": "Blacksmith Scene",
"isAdult": 0,
"startYear": "1893",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Comedy",
"Short"
]
},
{
"tconst": "tt0000006",
"titleType": "short",
"primaryTitle": "Chinese Opium Den",
"originalTitle": "Chinese Opium Den",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short"
]
},
{
"tconst": "tt0000007",
"titleType": "short",
"primaryTitle": "Corbett and Courtney Before the Kinetograph",
"originalTitle": "Corbett and Courtney Before the Kinetograph",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short",
"Sport"
]
},
{
"tconst": "tt0000008",
"titleType": "short",
"primaryTitle": "Edison Kinetoscopic Record of a Sneeze",
"originalTitle": "Edison Kinetoscopic Record of a Sneeze",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000009",
"titleType": "movie",
"primaryTitle": "Miss Jerry",
"originalTitle": "Miss Jerry",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "45",
"genres": [
"Romance"
]
},
{
"tconst": "tt0000010",
"titleType": "short",
"primaryTitle": "Leaving the Factory",
"originalTitle": "La sortie de l'usine Lumière à Lyon",
"isAdult": 0,
"startYear": "1895",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
}
]

File: example/nodejs-commonjs/document-worker/package.json Copy Chapter

{
"name": "nodejs-commonjs-document-worker",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-commonjs/document-worker-export-import/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-commonjs/document-worker-export-import/data.json Copy Chapter

[
{
"tconst": "tt0000001",
"titleType": "short",
"primaryTitle": "Carmencita",
"originalTitle": "Carmencita",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000002",
"titleType": "short",
"primaryTitle": "Le clown et ses chiens",
"originalTitle": "Le clown et ses chiens",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "5",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000003",
"titleType": "short",
"primaryTitle": "Pauvre Pierrot",
"originalTitle": "Pauvre Pierrot",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "4",
"genres": [
"Animation",
"Comedy",
"Romance"
]
},
{
"tconst": "tt0000004",
"titleType": "short",
"primaryTitle": "Un bon bock",
"originalTitle": "Un bon bock",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "12",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000005",
"titleType": "short",
"primaryTitle": "Blacksmith Scene",
"originalTitle": "Blacksmith Scene",
"isAdult": 0,
"startYear": "1893",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Comedy",
"Short"
]
},
{
"tconst": "tt0000006",
"titleType": "short",
"primaryTitle": "Chinese Opium Den",
"originalTitle": "Chinese Opium Den",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short"
]
},
{
"tconst": "tt0000007",
"titleType": "short",
"primaryTitle": "Corbett and Courtney Before the Kinetograph",
"originalTitle": "Corbett and Courtney Before the Kinetograph",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short",
"Sport"
]
},
{
"tconst": "tt0000008",
"titleType": "short",
"primaryTitle": "Edison Kinetoscopic Record of a Sneeze",
"originalTitle": "Edison Kinetoscopic Record of a Sneeze",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000009",
"titleType": "movie",
"primaryTitle": "Miss Jerry",
"originalTitle": "Miss Jerry",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "45",
"genres": [
"Romance"
]
},
{
"tconst": "tt0000010",
"titleType": "short",
"primaryTitle": "Leaving the Factory",
"originalTitle": "La sortie de l'usine Lumière à Lyon",
"isAdult": 0,
"startYear": "1895",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
}
]

File: example/nodejs-commonjs/document-worker-export-import/package.json Copy Chapter

{
"name": "nodejs-commonjs-document-worker-export-import",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-commonjs/document-worker-extern-config/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-commonjs/document-worker-extern-config/data.json Copy Chapter

[
{
"tconst": "tt0000001",
"titleType": "short",
"primaryTitle": "Carmencita",
"originalTitle": "Carmencita",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000002",
"titleType": "short",
"primaryTitle": "Le clown et ses chiens",
"originalTitle": "Le clown et ses chiens",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "5",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000003",
"titleType": "short",
"primaryTitle": "Pauvre Pierrot",
"originalTitle": "Pauvre Pierrot",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "4",
"genres": [
"Animation",
"Comedy",
"Romance"
]
},
{
"tconst": "tt0000004",
"titleType": "short",
"primaryTitle": "Un bon bock",
"originalTitle": "Un bon bock",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "12",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000005",
"titleType": "short",
"primaryTitle": "Blacksmith Scene",
"originalTitle": "Blacksmith Scene",
"isAdult": 0,
"startYear": "1893",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Comedy",
"Short"
]
},
{
"tconst": "tt0000006",
"titleType": "short",
"primaryTitle": "Chinese Opium Den",
"originalTitle": "Chinese Opium Den",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short"
]
},
{
"tconst": "tt0000007",
"titleType": "short",
"primaryTitle": "Corbett and Courtney Before the Kinetograph",
"originalTitle": "Corbett and Courtney Before the Kinetograph",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short",
"Sport"
]
},
{
"tconst": "tt0000008",
"titleType": "short",
"primaryTitle": "Edison Kinetoscopic Record of a Sneeze",
"originalTitle": "Edison Kinetoscopic Record of a Sneeze",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000009",
"titleType": "movie",
"primaryTitle": "Miss Jerry",
"originalTitle": "Miss Jerry",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "45",
"genres": [
"Romance"
]
},
{
"tconst": "tt0000010",
"titleType": "short",
"primaryTitle": "Leaving the Factory",
"originalTitle": "La sortie de l'usine Lumière à Lyon",
"isAdult": 0,
"startYear": "1895",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
}
]

File: example/nodejs-commonjs/document-worker-extern-config/package.json Copy Chapter

{
"name": "nodejs-commonjs-document-worker-extern-config",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-commonjs/language-pack/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-commonjs/language-pack/package.json Copy Chapter

{
"name": "nodejs-commonjs-language-pack",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-commonjs/.document-worker-persistent/data.json Copy Chapter

[
{
"tconst": "tt0000001",
"titleType": "short",
"primaryTitle": "Carmencita",
"originalTitle": "Carmencita",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000002",
"titleType": "short",
"primaryTitle": "Le clown et ses chiens",
"originalTitle": "Le clown et ses chiens",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "5",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000003",
"titleType": "short",
"primaryTitle": "Pauvre Pierrot",
"originalTitle": "Pauvre Pierrot",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "4",
"genres": [
"Animation",
"Comedy",
"Romance"
]
},
{
"tconst": "tt0000004",
"titleType": "short",
"primaryTitle": "Un bon bock",
"originalTitle": "Un bon bock",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "12",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000005",
"titleType": "short",
"primaryTitle": "Blacksmith Scene",
"originalTitle": "Blacksmith Scene",
"isAdult": 0,
"startYear": "1893",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Comedy",
"Short"
]
},
{
"tconst": "tt0000006",
"titleType": "short",
"primaryTitle": "Chinese Opium Den",
"originalTitle": "Chinese Opium Den",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short"
]
},
{
"tconst": "tt0000007",
"titleType": "short",
"primaryTitle": "Corbett and Courtney Before the Kinetograph",
"originalTitle": "Corbett and Courtney Before the Kinetograph",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short",
"Sport"
]
},
{
"tconst": "tt0000008",
"titleType": "short",
"primaryTitle": "Edison Kinetoscopic Record of a Sneeze",
"originalTitle": "Edison Kinetoscopic Record of a Sneeze",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000009",
"titleType": "movie",
"primaryTitle": "Miss Jerry",
"originalTitle": "Miss Jerry",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "45",
"genres": [
"Romance"
]
},
{
"tconst": "tt0000010",
"titleType": "short",
"primaryTitle": "Leaving the Factory",
"originalTitle": "La sortie de l'usine Lumière à Lyon",
"isAdult": 0,
"startYear": "1895",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
}
]

File: example/nodejs-commonjs/.document-worker-persistent/package.json Copy Chapter

{
"name": "nodejs-worker-persistent",
"dependencies": {
"flexsearch": "^0.8.200",
"sqlite3": "^5.1.7"
}
}

File: example/nodejs-esm/basic/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-esm/basic/package.json Copy Chapter

{
"name": "nodejs-esm-basic",
"type": "module",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-esm/basic-export-import/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-esm/basic-export-import/package.json Copy Chapter

{
"name": "nodejs-esm-basic-export-import",
"type": "module",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-esm/basic-persistent/README.md Copy Chapter

npm install

npm install [email protected]

node index.js

File: example/nodejs-esm/basic-persistent/package.json Copy Chapter

{
"name": "nodejs-esm-basic-persistent",
"type": "module",
"dependencies": {
"flexsearch": "^0.8.200",
"sqlite3": "^5.1.7"
}
}

File: example/nodejs-esm/basic-resolver/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-esm/basic-resolver/package.json Copy Chapter

{
"name": "nodejs-esm-basic-resolver",
"type": "module",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-esm/basic-suggestion/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-esm/basic-suggestion/package.json Copy Chapter

{
"name": "nodejs-esm-basic-suggestion",
"type": "module",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-esm/basic-worker/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-esm/basic-worker/package.json Copy Chapter

{
"name": "nodejs-esm-basic-worker",
"type": "module",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-esm/basic-worker-export-import/package.json Copy Chapter

{
"name": "nodejs-esm-basic-worker-export-import",
"type": "module",
"dependencies": {
"flexsearch": "github:nextapps-de/flexsearch"
}
}

File: example/nodejs-esm/basic-worker-extern-config/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-esm/basic-worker-extern-config/package.json Copy Chapter

{
"name": "nodejs-esm-basic-worker-extern-config",
"type": "module",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-esm/document/README.md Copy Chapter

npm install

npm install [email protected]

node index.js

File: example/nodejs-esm/document/data.json Copy Chapter

[
{
"tconst": "tt0000001",
"titleType": "short",
"primaryTitle": "Carmencita",
"originalTitle": "Carmencita",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000002",
"titleType": "short",
"primaryTitle": "Le clown et ses chiens",
"originalTitle": "Le clown et ses chiens",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "5",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000003",
"titleType": "short",
"primaryTitle": "Pauvre Pierrot",
"originalTitle": "Pauvre Pierrot",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "4",
"genres": [
"Animation",
"Comedy",
"Romance"
]
},
{
"tconst": "tt0000004",
"titleType": "short",
"primaryTitle": "Un bon bock",
"originalTitle": "Un bon bock",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "12",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000005",
"titleType": "short",
"primaryTitle": "Blacksmith Scene",
"originalTitle": "Blacksmith Scene",
"isAdult": 0,
"startYear": "1893",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Comedy",
"Short"
]
},
{
"tconst": "tt0000006",
"titleType": "short",
"primaryTitle": "Chinese Opium Den",
"originalTitle": "Chinese Opium Den",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short"
]
},
{
"tconst": "tt0000007",
"titleType": "short",
"primaryTitle": "Corbett and Courtney Before the Kinetograph",
"originalTitle": "Corbett and Courtney Before the Kinetograph",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short",
"Sport"
]
},
{
"tconst": "tt0000008",
"titleType": "short",
"primaryTitle": "Edison Kinetoscopic Record of a Sneeze",
"originalTitle": "Edison Kinetoscopic Record of a Sneeze",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000009",
"titleType": "movie",
"primaryTitle": "Miss Jerry",
"originalTitle": "Miss Jerry",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "45",
"genres": [
"Romance"
]
},
{
"tconst": "tt0000010",
"titleType": "short",
"primaryTitle": "Leaving the Factory",
"originalTitle": "La sortie de l'usine Lumière à Lyon",
"isAdult": 0,
"startYear": "1895",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
}
]

File: example/nodejs-esm/document/package.json Copy Chapter

{
"name": "nodejs-esm-document",
"type": "module",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-esm/document-export-import/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-esm/document-export-import/data.json Copy Chapter

[
{
"tconst": "tt0000001",
"titleType": "short",
"primaryTitle": "Carmencita",
"originalTitle": "Carmencita",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000002",
"titleType": "short",
"primaryTitle": "Le clown et ses chiens",
"originalTitle": "Le clown et ses chiens",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "5",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000003",
"titleType": "short",
"primaryTitle": "Pauvre Pierrot",
"originalTitle": "Pauvre Pierrot",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "4",
"genres": [
"Animation",
"Comedy",
"Romance"
]
},
{
"tconst": "tt0000004",
"titleType": "short",
"primaryTitle": "Un bon bock",
"originalTitle": "Un bon bock",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "12",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000005",
"titleType": "short",
"primaryTitle": "Blacksmith Scene",
"originalTitle": "Blacksmith Scene",
"isAdult": 0,
"startYear": "1893",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Comedy",
"Short"
]
},
{
"tconst": "tt0000006",
"titleType": "short",
"primaryTitle": "Chinese Opium Den",
"originalTitle": "Chinese Opium Den",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short"
]
},
{
"tconst": "tt0000007",
"titleType": "short",
"primaryTitle": "Corbett and Courtney Before the Kinetograph",
"originalTitle": "Corbett and Courtney Before the Kinetograph",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short",
"Sport"
]
},
{
"tconst": "tt0000008",
"titleType": "short",
"primaryTitle": "Edison Kinetoscopic Record of a Sneeze",
"originalTitle": "Edison Kinetoscopic Record of a Sneeze",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000009",
"titleType": "movie",
"primaryTitle": "Miss Jerry",
"originalTitle": "Miss Jerry",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "45",
"genres": [
"Romance"
]
},
{
"tconst": "tt0000010",
"titleType": "short",
"primaryTitle": "Leaving the Factory",
"originalTitle": "La sortie de l'usine Lumière à Lyon",
"isAdult": 0,
"startYear": "1895",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
}
]

File: example/nodejs-esm/document-export-import/package.json Copy Chapter

{
"name": "nodejs-esm-document-export-import",
"type": "module",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-esm/document-persistent/README.md Copy Chapter

npm install

npm install [email protected]

node index.js

File: example/nodejs-esm/document-persistent/data.json Copy Chapter

[
{
"tconst": "tt0000001",
"titleType": "short",
"primaryTitle": "Carmencita",
"originalTitle": "Carmencita",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000002",
"titleType": "short",
"primaryTitle": "Le clown et ses chiens",
"originalTitle": "Le clown et ses chiens",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "5",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000003",
"titleType": "short",
"primaryTitle": "Pauvre Pierrot",
"originalTitle": "Pauvre Pierrot",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "4",
"genres": [
"Animation",
"Comedy",
"Romance"
]
},
{
"tconst": "tt0000004",
"titleType": "short",
"primaryTitle": "Un bon bock",
"originalTitle": "Un bon bock",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "12",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000005",
"titleType": "short",
"primaryTitle": "Blacksmith Scene",
"originalTitle": "Blacksmith Scene",
"isAdult": 0,
"startYear": "1893",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Comedy",
"Short"
]
},
{
"tconst": "tt0000006",
"titleType": "short",
"primaryTitle": "Chinese Opium Den",
"originalTitle": "Chinese Opium Den",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short"
]
},
{
"tconst": "tt0000007",
"titleType": "short",
"primaryTitle": "Corbett and Courtney Before the Kinetograph",
"originalTitle": "Corbett and Courtney Before the Kinetograph",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short",
"Sport"
]
},
{
"tconst": "tt0000008",
"titleType": "short",
"primaryTitle": "Edison Kinetoscopic Record of a Sneeze",
"originalTitle": "Edison Kinetoscopic Record of a Sneeze",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000009",
"titleType": "movie",
"primaryTitle": "Miss Jerry",
"originalTitle": "Miss Jerry",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "45",
"genres": [
"Romance"
]
},
{
"tconst": "tt0000010",
"titleType": "short",
"primaryTitle": "Leaving the Factory",
"originalTitle": "La sortie de l'usine Lumière à Lyon",
"isAdult": 0,
"startYear": "1895",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
}
]

File: example/nodejs-esm/document-persistent/package.json Copy Chapter

{
"name": "nodejs-esm-document-persistent",
"type": "module",
"dependencies": {
"flexsearch": "^0.8.200",
"sqlite3": "^5.1.7"
}
}

File: example/nodejs-esm/document-resolver/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-esm/document-resolver/package.json Copy Chapter

{
"name": "nodejs-esm-document-resolver",
"type": "module",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-esm/document-worker/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-esm/document-worker/data.json Copy Chapter

[
{
"tconst": "tt0000001",
"titleType": "short",
"primaryTitle": "Carmencita",
"originalTitle": "Carmencita",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000002",
"titleType": "short",
"primaryTitle": "Le clown et ses chiens",
"originalTitle": "Le clown et ses chiens",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "5",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000003",
"titleType": "short",
"primaryTitle": "Pauvre Pierrot",
"originalTitle": "Pauvre Pierrot",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "4",
"genres": [
"Animation",
"Comedy",
"Romance"
]
},
{
"tconst": "tt0000004",
"titleType": "short",
"primaryTitle": "Un bon bock",
"originalTitle": "Un bon bock",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "12",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000005",
"titleType": "short",
"primaryTitle": "Blacksmith Scene",
"originalTitle": "Blacksmith Scene",
"isAdult": 0,
"startYear": "1893",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Comedy",
"Short"
]
},
{
"tconst": "tt0000006",
"titleType": "short",
"primaryTitle": "Chinese Opium Den",
"originalTitle": "Chinese Opium Den",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short"
]
},
{
"tconst": "tt0000007",
"titleType": "short",
"primaryTitle": "Corbett and Courtney Before the Kinetograph",
"originalTitle": "Corbett and Courtney Before the Kinetograph",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short",
"Sport"
]
},
{
"tconst": "tt0000008",
"titleType": "short",
"primaryTitle": "Edison Kinetoscopic Record of a Sneeze",
"originalTitle": "Edison Kinetoscopic Record of a Sneeze",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000009",
"titleType": "movie",
"primaryTitle": "Miss Jerry",
"originalTitle": "Miss Jerry",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "45",
"genres": [
"Romance"
]
},
{
"tconst": "tt0000010",
"titleType": "short",
"primaryTitle": "Leaving the Factory",
"originalTitle": "La sortie de l'usine Lumière à Lyon",
"isAdult": 0,
"startYear": "1895",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
}
]

File: example/nodejs-esm/document-worker/package.json Copy Chapter

{
"name": "nodejs-esm-document-worker",
"type": "module",
"dependencies": {
"flexsearch": "github:nextapps-de/flexsearch"
}
}

File: example/nodejs-esm/document-worker-export-import/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-esm/document-worker-export-import/data.json Copy Chapter

[
{
"tconst": "tt0000001",
"titleType": "short",
"primaryTitle": "Carmencita",
"originalTitle": "Carmencita",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000002",
"titleType": "short",
"primaryTitle": "Le clown et ses chiens",
"originalTitle": "Le clown et ses chiens",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "5",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000003",
"titleType": "short",
"primaryTitle": "Pauvre Pierrot",
"originalTitle": "Pauvre Pierrot",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "4",
"genres": [
"Animation",
"Comedy",
"Romance"
]
},
{
"tconst": "tt0000004",
"titleType": "short",
"primaryTitle": "Un bon bock",
"originalTitle": "Un bon bock",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "12",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000005",
"titleType": "short",
"primaryTitle": "Blacksmith Scene",
"originalTitle": "Blacksmith Scene",
"isAdult": 0,
"startYear": "1893",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Comedy",
"Short"
]
},
{
"tconst": "tt0000006",
"titleType": "short",
"primaryTitle": "Chinese Opium Den",
"originalTitle": "Chinese Opium Den",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short"
]
},
{
"tconst": "tt0000007",
"titleType": "short",
"primaryTitle": "Corbett and Courtney Before the Kinetograph",
"originalTitle": "Corbett and Courtney Before the Kinetograph",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short",
"Sport"
]
},
{
"tconst": "tt0000008",
"titleType": "short",
"primaryTitle": "Edison Kinetoscopic Record of a Sneeze",
"originalTitle": "Edison Kinetoscopic Record of a Sneeze",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000009",
"titleType": "movie",
"primaryTitle": "Miss Jerry",
"originalTitle": "Miss Jerry",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "45",
"genres": [
"Romance"
]
},
{
"tconst": "tt0000010",
"titleType": "short",
"primaryTitle": "Leaving the Factory",
"originalTitle": "La sortie de l'usine Lumière à Lyon",
"isAdult": 0,
"startYear": "1895",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
}
]

File: example/nodejs-esm/document-worker-export-import/package.json Copy Chapter

{
"name": "nodejs-esm-document-worker-export-import",
"type": "module",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-esm/document-worker-extern-config/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-esm/document-worker-extern-config/data.json Copy Chapter

[
{
"tconst": "tt0000001",
"titleType": "short",
"primaryTitle": "Carmencita",
"originalTitle": "Carmencita",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000002",
"titleType": "short",
"primaryTitle": "Le clown et ses chiens",
"originalTitle": "Le clown et ses chiens",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "5",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000003",
"titleType": "short",
"primaryTitle": "Pauvre Pierrot",
"originalTitle": "Pauvre Pierrot",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "4",
"genres": [
"Animation",
"Comedy",
"Romance"
]
},
{
"tconst": "tt0000004",
"titleType": "short",
"primaryTitle": "Un bon bock",
"originalTitle": "Un bon bock",
"isAdult": 0,
"startYear": "1892",
"endYear": "",
"runtimeMinutes": "12",
"genres": [
"Animation",
"Short"
]
},
{
"tconst": "tt0000005",
"titleType": "short",
"primaryTitle": "Blacksmith Scene",
"originalTitle": "Blacksmith Scene",
"isAdult": 0,
"startYear": "1893",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Comedy",
"Short"
]
},
{
"tconst": "tt0000006",
"titleType": "short",
"primaryTitle": "Chinese Opium Den",
"originalTitle": "Chinese Opium Den",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short"
]
},
{
"tconst": "tt0000007",
"titleType": "short",
"primaryTitle": "Corbett and Courtney Before the Kinetograph",
"originalTitle": "Corbett and Courtney Before the Kinetograph",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Short",
"Sport"
]
},
{
"tconst": "tt0000008",
"titleType": "short",
"primaryTitle": "Edison Kinetoscopic Record of a Sneeze",
"originalTitle": "Edison Kinetoscopic Record of a Sneeze",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
},
{
"tconst": "tt0000009",
"titleType": "movie",
"primaryTitle": "Miss Jerry",
"originalTitle": "Miss Jerry",
"isAdult": 0,
"startYear": "1894",
"endYear": "",
"runtimeMinutes": "45",
"genres": [
"Romance"
]
},
{
"tconst": "tt0000010",
"titleType": "short",
"primaryTitle": "Leaving the Factory",
"originalTitle": "La sortie de l'usine Lumière à Lyon",
"isAdult": 0,
"startYear": "1895",
"endYear": "",
"runtimeMinutes": "1",
"genres": [
"Documentary",
"Short"
]
}
]

File: example/nodejs-esm/document-worker-extern-config/package.json Copy Chapter

{
"name": "nodejs-esm-document-worker-extern-config",
"type": "module",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: example/nodejs-esm/language-pack/README.md Copy Chapter

npm install

node index.js

File: example/nodejs-esm/language-pack/package.json Copy Chapter

{
"name": "nodejs-esm-language-pack",
"dependencies": {
"flexsearch": "^0.8.200"
}
}

File: src/db/clickhouse/package.json Copy Chapter

{
"public": true,
"preferGlobal": false,
"name": "flexsearch-clickhouse",
"version": "0.1.0",
"main": "index.js",
"dependencies": {
"clickhouse": "^2.6.0"
}
}

File: src/db/mongodb/package.json Copy Chapter

{
"public": true,
"preferGlobal": false,
"name": "flexsearch-mongodb",
"version": "0.1.0",
"main": "index.js",
"dependencies": {
"mongodb": "^6.13.0"
}
}

File: src/db/postgres/package.json Copy Chapter

{
"public": true,
"preferGlobal": false,
"name": "flexsearch-postgres",
"version": "0.1.0",
"main": "index.js",
"dependencies": {
"pg-promise": "^11.10.2"
}
}

File: src/db/redis/package.json Copy Chapter

{
"public": true,
"preferGlobal": false,
"name": "flexsearch-redis",
"version": "0.1.0",
"main": "index.js",
"dependencies": {
"redis": "^4.7.0"
}
}

File: src/db/sqlite/package.json Copy Chapter

{
"public": true,
"preferGlobal": false,
"name": "flexsearch-sqlite",
"version": "0.1.0",
"main": "index.js",
"dependencies": {
"sqlite3": "^5.1.7"
}
}

File: task/babel.bundle.json Copy Chapter

{
"plugins": [
["conditional-compile", {
"dropDebugger": true,
"define": {
"RELEASE": "module",
"DEBUG": false,
"PROFILER": false,
"POLYFILL": false,
"SUPPORT_WORKER": true,
"SUPPORT_ENCODER": true,
"SUPPORT_CHARSET": true,
"SUPPORT_CACHE": true,
"SUPPORT_ASYNC": true,
"SUPPORT_STORE": true,
"SUPPORT_SUGGESTION": true,
"SUPPORT_SERIALIZE": true,
"SUPPORT_DOCUMENT": true,
"SUPPORT_TAGS": true,
"SUPPORT_PERSISTENT": true,
"SUPPORT_KEYSTORE": true,
"SUPPORT_COMPRESSION": true,
"SUPPORT_RESOLVER": true,
"SUPPORT_HIGHLIGHTING": true
}
}],
"babel-plugin-minify-constant-folding",
"babel-plugin-minify-dead-code-elimination",
"babel-plugin-minify-flip-comparisons",
"babel-plugin-minify-guarded-expressions",
"babel-plugin-minify-infinity",
"babel-plugin-minify-replace",
"babel-plugin-minify-type-constructors",
"babel-plugin-transform-member-expression-literals",
"babel-plugin-transform-merge-sibling-variables",
"babel-plugin-transform-minify-booleans",
"babel-plugin-transform-property-literals",
"babel-plugin-transform-simplify-comparison-operators",
"babel-plugin-transform-undefined-to-void"
],
"ignore": [
"config.js"
],
"minified": false,
"compact": false,
"comments": true
}

File: task/babel.debug.json Copy Chapter

{
"plugins": [
["conditional-compile", {
"dropDebugger": true,
"define": {
"RELEASE": "module",
"DEBUG": true,
"PROFILER": false,
"POLYFILL": false,
"SUPPORT_WORKER": true,
"SUPPORT_ENCODER": true,
"SUPPORT_CHARSET": true,
"SUPPORT_CACHE": true,
"SUPPORT_ASYNC": true,
"SUPPORT_STORE": true,
"SUPPORT_SUGGESTION": true,
"SUPPORT_SERIALIZE": true,
"SUPPORT_DOCUMENT": true,
"SUPPORT_TAGS": true,
"SUPPORT_PERSISTENT": true,
"SUPPORT_KEYSTORE": true,
"SUPPORT_COMPRESSION": true,
"SUPPORT_RESOLVER": true,
"SUPPORT_HIGHLIGHTING": true
}
}],
"babel-plugin-minify-constant-folding",
"babel-plugin-minify-dead-code-elimination",
"babel-plugin-minify-flip-comparisons",
"babel-plugin-minify-guarded-expressions",
"babel-plugin-minify-infinity",
"babel-plugin-minify-replace",
"babel-plugin-minify-type-constructors",
"babel-plugin-transform-member-expression-literals",
"babel-plugin-transform-merge-sibling-variables",
"babel-plugin-transform-minify-booleans",
"babel-plugin-transform-property-literals",
"babel-plugin-transform-simplify-comparison-operators",
"babel-plugin-transform-undefined-to-void"
],
"ignore": [
"config.js"
],
"minified": false,
"compact": false,
"comments": true
}

File: task/babel.min.json Copy Chapter

{
"plugins": [
["conditional-compile", {
"dropDebugger": true,
"define": {
"RELEASE": "module",
"DEBUG": false,
"PROFILER": false,
"POLYFILL": false,
"SUPPORT_WORKER": true,
"SUPPORT_ENCODER": true,
"SUPPORT_CHARSET": true,
"SUPPORT_CACHE": true,
"SUPPORT_ASYNC": true,
"SUPPORT_STORE": true,
"SUPPORT_SUGGESTION": true,
"SUPPORT_SERIALIZE": true,
"SUPPORT_DOCUMENT": true,
"SUPPORT_TAGS": true,
"SUPPORT_PERSISTENT": true,
"SUPPORT_KEYSTORE": true,
"SUPPORT_COMPRESSION": true,
"SUPPORT_RESOLVER": true,
"SUPPORT_HIGHLIGHTING": true
}
}],
"babel-plugin-minify-constant-folding",
"babel-plugin-minify-dead-code-elimination",
"babel-plugin-minify-flip-comparisons",
"babel-plugin-minify-guarded-expressions",
"babel-plugin-minify-infinity",
"babel-plugin-minify-mangle-names",
"babel-plugin-minify-replace",
"babel-plugin-minify-simplify",
"babel-plugin-minify-type-constructors",
"babel-plugin-transform-member-expression-literals",
"babel-plugin-transform-merge-sibling-variables",
"babel-plugin-transform-minify-booleans",
"babel-plugin-transform-property-literals",
"babel-plugin-transform-simplify-comparison-operators",
"babel-plugin-transform-undefined-to-void"
],
"ignore": [
"config.js"
],
"minified": true,
"compact": true,
"comments": false
}

File: test/package.json Copy Chapter

{
"name": "flexsearch-test",
"type": "module",
"scripts": {
"test": "mocha ./.js --exit",
"test:ts": "npx tsc --noEmit --lib esnext ./types.ts",
"test:github": "npm run test:ts && mocha ./.js --exit flexsearch.light.min && mocha ./.js --exit flexsearch.light.module.min && mocha ./.js --exit flexsearch.compact.min && mocha ./.js --exit flexsearch.compact.module.min",
"test:coverage": "c8 -c ./.c8rc.json mocha ./.js --exit",
"test:keystore": "set NODE_OPTIONS=--max-old-space-size=16000 && c8 -c ./.c8rc.json mocha ./keystore.js --exit",
"test:db": "c8 -c ./.c8rc.json mocha ./persistent..js --exit",
"test:all": "npm run test:ts && npx mocha ./.js --exit module/bundle && npx mocha ./.js --exit module-debug/bundle && npx mocha ./.js --exit module-min/bundle && npx mocha ./.js --exit flexsearch.bundle.debug && npx mocha ./.js --exit flexsearch.bundle.min && npx mocha ./.js --exit flexsearch.bundle.module.debug && npx mocha ./.js --exit flexsearch.bundle.module.min && npx mocha ./.js --exit flexsearch.compact.debug && npx mocha ./.js --exit flexsearch.compact.min && npx mocha ./.js --exit flexsearch.compact.module.debug && npx mocha ./.js --exit flexsearch.compact.module.min && npx mocha ./.js --exit flexsearch.es5.debug && npx mocha ./.js --exit flexsearch.es5.min && npx mocha ./.js --exit flexsearch.light.debug && npx mocha ./.js --exit flexsearch.light.min && npx mocha ./.js --exit flexsearch.light.module.debug && npx mocha ./.js --exit flexsearch.light.module.min"
},
"devDependencies": {
"c8": "^10.1.3",
"chai": "^5.2.0",
"clickhouse": "^2.6.0",
"flexsearch": "^0.8.205",
"mocha": "^11.1.0",
"mongodb": "^6.13.0",
"pg-promise": "^11.10.2",
"redis": "^5.1.0",
"rollup": "^4.59.0",
"sqlite3": "^6.0.1",
"typescript": "^5.8.3"
}
}

File: test/.c8rc.json Copy Chapter

{
"reporter": ["text", "text-summary", "html"],
"reports-dir": "./test/report/",
"branches": 80,
"functions": 80,
"lines": 80,
"statements": 80,
"src": [
"src"
],
"include": [
"src/**"
],
"exclude": [
"test/.js",
"src/compress.js",
"src/type.js",
"src/profiler.js",
"src/db/indexeddb/",
"src/worker/node.js",
"src/worker/worker.js"
]
}

File: .github/FUNDING.yml Copy Chapter

These are supported funding model platforms

github: [ts-thomas]
open_collective: flexsearch
liberapay: ts-thomas
patreon: user?u=96245532
custom: ["https://www.paypal.com/donate/?hosted_button_id=GEVR88FC9BWRW", "https://salt.bountysource.com/teams/ts-thomas"]

File: .github/workflows/node.js.yml Copy Chapter

This workflow will do a clean installation of node dependencies, cache/restore them, build the source code and run tests across different versions of node

For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-nodejs

name: Node.js CI

permissions:
contents: read
pull-requests: write

on:
push:
branches: [ "master" ]
pull_request:
branches: [ "master" ]

jobs:

Label of the container job

build:
# Containers must run in Linux based operating systems
runs-on: ubuntu-latest
timeout-minutes: 10

# Docker Hub image that `container-job` executes in

container: node:20-bookworm-slim

# Service containers to run with `container-job`

services:

# Label used to access the service container

postgres:

# Docker Hub image

image: postgres

# Provide the password for postgres

env:

POSTGRES_USER: postgres

POSTGRES_PASSWORD: postgres

POSTGRES_DATABASE: postgres

POSTGRES_HOST: postgres

POSTGRES_PORT: 5432

# Set health checks to wait until postgres has started

options: >-

--health-cmd pg_isready

--health-interval 10s

--health-timeout 5s

--health-retries 5

strategy:
  matrix:
    node-version: [20.x, 22.x, 24.x] # 18.x
    # See supported Node.js release schedule at https://nodejs.org/en/about/releases/

steps:
- uses: actions/checkout@v4
- name: Use Node.js ${{ matrix.node-version }}
  uses: actions/setup-node@v4
  with:
    node-version: ${{ matrix.node-version }}
    cache: 'npm'
- uses: actions/setup-java@v1
  with:
    java-version: 21
- run: npm install
- run: npm run build:compact
- run: npm run build:module:compact
- run: npm run build:light
- run: npm run build:module:light
- run: npm install
  working-directory: test
- run: npm run test:github
  working-directory: test