llm-app Documentation

serviette

A universal, no-code, always up-to-date RAG server for any vector database
— powered by the Pathway Live Data Framework.

serviette stands up Retrieval-Augmented Generation over your own documents —
text, Office files, PDFs, scans, and (with API keys) audio and video —
without writing code. Point it at a folder, choose a vector database and an
embedder in a YAML file (or generate one with the wizard), and run one
command (serviette up) or the two components separately:

• serviette indexer — a Pathway streaming pipeline that watches your files,
  parses and chunks them, embeds the chunks and keeps your vector DB in sync
  (additions, modifications and deletions) in real time.
• serviette server — a FastAPI service that embeds incoming queries and
  retrieves the most relevant chunks (and, optionally, answers them with an
  LLM). It also serves the web chat UI on / (same port, same origin) unless
  server.serve_frontend is disabled; the REST API lives under /api/v1.

The indexer and the server are fully decoupled: they share only the vector
database, so you can scale them independently.

1. Installation Copy Chapter

serviette requires Python ≥ 3.10 (the minimum supported by Pathway).

pip install "serviette[local]" --extra-index-url https://download.pytorch.org/whl/cpu

The local extra brings sentence-transformers — the free, credential-free
embedder that serviette wizard selects by default. It pulls the PyTorch
stack, and the --extra-index-url flag makes pip take PyTorch's CPU build
(~0.8 GB). Without it, PyTorch's default Linux wheel is installed, which
carries the CUDA libraries (~5 GB) whether or not the machine has an NVIDIA
GPU — leave the flag out only if you want to embed on such a GPU. (With
uv, add --index-strategy unsafe-best-match as well.) If you intend to
embed with OpenAI instead, a plain pip install serviette is enough (pick
openai in the wizard).
From a checkout, pip install -e ".[dev,local]" — see the README's
Development section for the full walkthrough.
OpenAI support (the default /rag LLM and the OpenAI embedder) is built
in — just set OPENAI_API_KEY.

Optional extras: docling (layout-aware PDF + Office parsing), ocr (scanned images),
pyfilesystem (FTP/SFTP/WebDAV/ZIP sources), gdrive, sharepoint, plus
one extra per vector-DB client (qdrant, pgvector, …, or all).

2. Config wizard Copy Chapter

The fastest way to a chat is serviette demo (see the README); the fastest
way to a config for your documents is the interactive wizard:

serviette wizard

It asks only what it must (the DuckDB + local-embeddings happy path is seven
questions) and writes a YAML file; everything else — port, persistence,
chunking, parser routing — gets silent, documented defaults you can edit
later. You only have to provide document paths, vector-DB connection
details (for client-server backends), API keys for keyed providers, and your
Pathway license key.

Then run the two components (in separate terminals or on separate machines):

# One command: indexer + server supervised together (dev/demo convenience)
serviette up --config config.yaml         # open http://localhost:8989

# ... or run the components separately (how production deploys them):
# 1. Index your documents into the vector DB and keep it live
serviette indexer --config config.yaml

# 2. Serve the chat UI + retrieval / RAG API on one port
serviette server --config config.yaml     # open http://localhost:8989

# (optional, split deployments only) standalone UI tier on another host
serviette frontend --config config.yaml   # open http://localhost:3000

indexer, server and up check the obvious mistakes before starting
anything: a source folder that does not exist, a ${VAR} in the config that
is not exported, an openai-typed embedder or llm with no key in the config
or in OPENAI_API_KEY, a backend that cannot be shared by two processes. Each
stops with a one-paragraph message, not a traceback. If the indexer or the
server dies later, serviette up ends with a framed serviette up STOPPED
block that names the cause when it recognises it (missing or rejected API
key, OpenAI rate limit or quota, a port already in use, an unreachable
database) and what to do.

By default serviette up shows only its own progress, warnings, errors and
the URL to open; the indexer's and the server's routine INFO lines (the
engine's persistence and connector monitoring, one line per HTTP request)
are hidden. serviette up --verbose passes them all through. The hidden
lines are still kept for the STOPPED report, so a diagnosis never depends
on the flag.

Query the API (versioned under /api/v1; the pre-versioning /retrieve,
/rag and /health paths still work as deprecated aliases):

curl -X POST http://localhost:8989/api/v1/retrieve \
  -H 'Content-Type: application/json' \
  -d '{"query": "how does persistence work?", "k": 5}'

{
  "results": [
    {"text": "...", "metadata": {"path": "/data/docs/guide.pdf", ...}, "score": 0.95}
  ]
}

If you configured an llm section, the /rag endpoint also answers questions:

curl -X POST http://localhost:8989/api/v1/rag \
  -H 'Content-Type: application/json' \
  -d '{"query": "summarize the onboarding guide", "k": 5}'

{ "answer": "...", "sources": [ {"text": "...", "metadata": {...}, "score": 0.91} ] }

GET /api/v1/documents lists the indexed documents whose name or path
contains q (case_sensitive=true to match case; the default ignores it).
The chat page's Documents panel is built on it and asks only when a
search is submitted, never on load — an index may hold tens of thousands of
files. Over limit matches (default 100, at most 1000) only the counts are
returned and the page asks to narrow the search:

curl 'http://localhost:8989/api/v1/documents?q=report&case_sensitive=false'

{ "total": 1240, "matched": 2, "truncated": false,
  "documents": [ {"id": "/data/docs/report_2026.pdf", "name": "report_2026.pdf",
                  "metadata": {"path": "...", "modified_at": 1790000000, "size": 12345}, "chunks": 41}, ... ] }

name is what a person calls the document — the file name for folder, S3
and SharePoint sources, the Drive title for Google Drive — and id is the
full identity the source reports (path or Drive id). Pinecone cannot
enumerate documents; there the endpoint answers 501.

3. Pathway License Key Copy Chapter

Pathway requires a free license key. Get yours in one click at
https://pathway.com/framework/get-license (email or LinkedIn sign-in).

Provide it either via the config file:

pathway_license_key: ${PATHWAY_LICENSE_KEY}

or directly as a string. Using the ${PATHWAY_LICENSE_KEY} form keeps the key
out of the file — set it in the environment:

export PATHWAY_LICENSE_KEY="your-key-here"

serviette wizard notices an exported PATHWAY_LICENSE_KEY and offers to
write the ${PATHWAY_LICENSE_KEY} reference instead of asking for the key;
serviette demo uses the variable the same way.

The key is applied when the indexing graph is initialized.

4. Config reference Copy Chapter

A single YAML file can configure both the indexer and the server (a universal
config), or you can split it into an indexer-only and a server-only file. Any
string value supports ${ENV_VAR} interpolation, so credentials never need to be
hardcoded. A reference to a variable that is not set is a startup error naming
the variable and the config key using it (it does not silently become an empty
string, which would only surface later as a provider 401).

Supported embedders (from pathway.xpacks.llm.embedders): openai,
litellm (≈100 providers), sentence_transformer (fully local, no
credentials — install serviette[local]), gemini (serviette[gemini]),
bedrock (serviette[bedrock] for the server side). Every family has a
matching async client on the server side, so one embedder section serves both
components; make sure the indexer and server use the same model so vectors
are comparable.

Which local model. The sentence_transformer default,
sentence-transformers/all-MiniLM-L6-v2, is an English-only model: with
French, German or Polish documents retrieval degrades to "the context does not
contain the information" even though the files were indexed. For non-English
or mixed corpora set model: intfloat/multilingual-e5-small (~100 languages,
same size and speed) together with the prefixes it was trained with,
query_prefix: "query: " and document_prefix: "passage: "; for the highest
quality and long chunks, BAAI/bge-m3 (1024-dim, ~2.3 GB, several times slower
on CPU). serviette wizard asks this as its "Embedding model" step and writes
the prefixes for you. Changing the model means re-indexing (the fingerprint
check will say so).

Sources Copy Section

serviette only supports sources that offer Pathway's only_metadata read mode —
the graph holds just paths/identifiers and metadata (never the file bytes), and
the bytes are fetched on demand during parsing. That keeps the memory footprint
tiny (1 GB of PDFs stays a few KB in the pipeline). Today that means:

• fs — a local directory, watched recursively per glob.

• gdrive — a Google Drive folder or file. Create a Google service
  account, download its JSON key, share the target folder with the service
  account's email, and point service_user_credentials_file at the key. See the
  Pathway Google Drive connector guide.
  Install the extra: pip install "serviette[gdrive]".

• s3 — an S3 bucket or key prefix, including S3-compatible stores
  (MinIO, DigitalOcean, Wasabi) via endpoint + with_path_style: true.
  Credentials fall back to the standard AWS chain when omitted. Install the
  extra: pip install "serviette[s3]".

• sharepoint — a SharePoint directory or file, authenticated with an
  app-registration certificate (tenant, client_id, cert_path,
  thumbprint). Requires a Pathway Scale license and
  pip install "serviette[sharepoint]".

• pyfilesystem — anything the
  PyFilesystem2 library opens: FTP, SFTP,
  WebDAV, even ZIP/TAR archives — selected by fs_url (e.g.
  ftp://user:pass@host/dir, zip://./docs.zip). Install
  pip install "serviette[pyfilesystem]"; some protocols need their own driver
  package (fs.sshfs, fs.webdavfs).

You can list several sources of mixed types in one config.

Vector database backends Copy Section

Every backend is written through Pathway's native pw.io.* output
connector in snapshot/upsert mode keyed by the chunk id, so file additions,
edits and deletions become real inserts/updates/deletes in the store. The
server retrieves through each database's own vector-search API — there are no
linear scans in Python anywhere.

Targets are created for you. At startup the indexer runs
prepare_backend: it creates the missing table / collection / index for
every backend (pgvector table + HNSW index, Milvus collection, Chroma and
Weaviate collections, the Pinecone serverless index, the MongoDB Atlas
vectorSearch index), sizing the vector dimension from
vector_db.embedding_dimension when set, or by introspecting the configured
embedder otherwise. The schemas below are shown for reference — e.g. when a
DBA manages the database — not as required manual steps.

DuckDB (embedded — the zero-setup default)

vector_db:
  type: duckdb
  path: ./embeddings.duckdb
  table: serviette_embeddings

No external service and no setup: the file and table are created automatically
(pw.io.duckdb, snapshot mode). Embeddings are stored as native DOUBLE[]
columns and queried inside DuckDB with list_cosine_similarity — a
vectorized, columnar scan that answers in milliseconds for local corpora.

Concurrency: DuckDB allows one read-write process or several read-only
processes per file — never both. serviette resolves this with Pathway's
detach_between_batches: the streaming indexer releases the file lock
between minibatches, and the server's short-lived read-only connections query
it concurrently — live indexing and serving work on one file. A request that
arrives while a batch is being committed waits for the lock (up to 30 s; the
first commit of a large corpus can hold it for ~10 s), and only past that
answers 503 with "the index is being updated — try again in a few seconds"
(the chat shows the same text). On older Pathway builds without the flag, serviette
warns and falls back to the hold-the-lock behavior (use mode: static
there).

pgvector

Auto-created (extension, table, HNSW index). Reference schema, e.g. for
text-embedding-3-small (1536 dims):

CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE serviette_embeddings (
  chunk_id  text PRIMARY KEY,
  text      text,
  metadata  jsonb,
  embedding vector(1536)
);

(The indexer writes the chunk primary key to the chunk_id column.)

Milvus

Auto-created: a collection with a VARCHAR primary key, text (VARCHAR),
metadata (JSON) and an embedding (FLOAT_VECTOR) field of the embedder's
dimension, AUTOINDEX with the COSINE metric.

Milvus Lite (uri: ./milvus.db, a local file) is an embedded,
single-process database: the process that opens the file locks it. It works
for the one-process-at-a-time flow — serviette indexer with mode: static
sources, then serviette server — but not with serviette up, which runs
the indexer and the server as two processes against one store; up refuses
such a config up front. For up, a live chat over a changing folder, or
scaling the server, use a Milvus server (uri: http://host:19530, e.g. the
milvus run standalone docker image).

Qdrant

Nothing to create: the indexer (pw.io.qdrant, gRPC) auto-creates the
collection with the cosine metric on first write. The chunk's text and
metadata travel in the point payload; the server queries over REST.
Set api_key for Qdrant Cloud.

ChromaDB

Auto-created with the cosine distance (hnsw:space). The chunk text is
stored as the Chroma document and the source metadata as a JSON string
(Chroma metadata values must be scalars).

Weaviate

Auto-created (capitalized name, e.g. ServietteEmbeddings; vectorizer none).
The chunk vector is stored as the object vector; text / metadata (JSON
string) / chunk_id become properties. The server's v4 client needs both the
HTTP and gRPC ports.

Pinecone

Auto-created: a serverless index (vector_db.cloud / .region) with the
embedder's dimension and the cosine metric. The text and JSON-string
metadata are stored as record metadata. The API key comes from api_key
or $PINECONE_API_KEY.

MongoDB Atlas Vector Search

The indexer writes one document per chunk (pw.io.mongodb, snapshot mode).
The collection and the Atlas vectorSearch index (default name
vector_index, cosine, numDimensions from the embedder) are auto-created;
on Atlas Local the indexer rides out the search service's slow boot with
retries. The server retrieves with $vectorSearch.

5. Persistence Copy Chapter

When persistence.enabled: true, the indexer uses Pathway's built-in
persistence backend plus a DefaultCache UDF call-cache on the embedder. On
restart, Pathway replays its persisted state and re-emits only the diffs since
the last run, so:

• unchanged documents are not re-embedded (the embedder cache and operator
  state are restored), and
• documents removed while the indexer was down are correctly retracted from
  the vector DB on the next run.

The engine's state lives under <workdir_path>/persistence/ (the top level of
the working directory is reserved for serviette's own files, such as the
configuration fingerprint — the engine logs an ERROR for every foreign entry
in the directory it is given, so the two never share a folder); the same
persistence/ also hosts the parse cache (runtime_calls/, via
pw.udfs.DefaultCache — diskcache, LRU-bounded by
indexer.parse_cache_size_gb, default 8): extracted document text stays warm
across restarts, so unchanged documents are neither re-downloaded nor
re-parsed. Disabling persistence also disables the parse cache.

The configuration fingerprint (<workdir_path>/serviette-fingerprint.json)
records the chunking-relevant settings the working directory was indexed with
(splitter, parser routing, embedder identity, library versions, chunk-id
scheme). A change is reported as a diff on startup and the indexer refuses to
run unless confirmed (interactively, or with
SERVIETTE_ACCEPT_FINGERPRINT_CHANGES=1), because rows already in the vector
DB keep their old chunk ids and would be left behind as orphans. The check
runs with persistence on or off: a restart without persistence re-indexes
everything, but does not remove those rows either.

A document whose bytes cannot be fetched (after indexer.fetch_retries
retries) or parsed does not stop the indexer: it is indexed as empty and
logged at ERROR level with its path. That result sticks for this version of
the object — a restart alone does not retry it — so fix the cause and then
touch the file (or re-upload the object) to have the source re-emit it.

Disabling persistence still produces identical vectors for a given set of files;
it only forgoes the cross-restart diffing and caching.

6. Parsing & modalities Copy Chapter

serviette never implements its own parsers — each file is routed by extension to
a pathway.xpacks.llm.parsers parser. Every format is enabled by default,
preferring parsers that need no API key, and modalities whose only parser
requires an absent key are skipped with a single clear warning (never a
crash):

• text / Markdown → as-is;
• .pdf → DoclingParser (layout-aware, tables — serviette[docling]), falling
  back to PypdfParser;
• Office & co. (DOCX, PPTX, XLSX, HTML, EML, CSV, RTF, EPUB, …) →
  UnstructuredParser;
• scanned images (PNG, JPG, TIFF, …) → PaddleOCRParser (serviette[ocr]);
• audio (MP3, WAV, …) → Whisper, enabled when OPENAI_API_KEY is set;
• video (MP4, WebM, MOV, …) → TwelveLabs Pegasus (a searchable text
  description), enabled when TWELVELABS_API_KEY is set.

The routing is overridable per file pattern via the parser: config section
(first matching rule wins; unmatched files use the defaults above) — e.g.
route images to a vision model instead of OCR, or set a custom video prompt.
Routing changes are fingerprint-guarded like splitter changes. Expensive
parses (video) are cached on disk, so restarts cost nothing.

Still not supported: encrypted / password-protected PDFs; Google-native files
(Docs/Sheets/Slides) from a gdrive source — they have no binary download
and would need an export step (TODO); any format the installed unstructured
version cannot handle. Unsupported files are skipped and produce no chunks.

7. Architecture Copy Chapter

Inside the indexer, documents move through a Pathway dataflow — every stage
below is an incremental operator, so a change to one file re-runs only that
file's slice of the graph:

Why only_metadata? The filesystem connector only ever puts paths and
metadata into the graph, never file contents — so 1 GB of PDFs costs a few KB in
the pipeline. Text is extracted on demand inside the parse UDF.

Deletion handling. The parse UDF is non-deterministic, so Pathway memoizes
its output and, when a file is removed, re-emits that stored text negated
without re-reading the (now gone) file; the retraction flows through chunking and
embedding and the sink deletes exactly the matching vectors.

8. Scaling Copy Chapter

The indexer and the server are separate processes that share only the vector
database:

• Server. Stateless and read-only against the vector DB — run as many
  instances as you like behind a load balancer, across machines or availability
  zones. Because it is fully async (coroutines, not a thread pool), each instance
  handles many concurrent network-bound retrieval calls efficiently. Each
  instance also serves the chat UI (static, zero overhead), so the UI scales
  with the API for free.
• Frontend (optional split tier). When the UI must live on a different
  host than the API, serviette frontend is a separate, stateless web tier that
  serves the same chat page and proxies to the API at frontend.api_url
  server-side (the browser connects to the frontend, which calls the API).
• Indexer. Runs as its own long-lived process and shards across worker
  processes with one config line (indexer.workers, via pathway spawn) —
  the published benchmarks run with 8; its persistence lets it resume
  without re-embedding.
• Vector database. Scaling and replicating the vector DB itself is the
  user's responsibility and follows that database's own guidance. (The embedded
  DuckDB backend is the exception: it lives in one file next to the indexer and
  is meant for local / single-node setups.)

Engine observability: with indexer.monitoring_http_port set, every worker
process serves the Pathway engine's built-in endpoints on
127.0.0.1:(port + worker index) — GET /metrics (Prometheus: input/output
latency gauges, per-operator row counters) and GET /status.

Security & exposing the server Copy Chapter

The API server binds to 127.0.0.1 by default: on a laptop, serviette up
serves only you. It deliberately ships no built-in authentication —
that is a perimeter concern, and a reverse proxy does it better than any
hand-rolled API key (TLS, key rotation, SSO, rate limits, audit logs come
for free). This is the norm for tools of this class: Qdrant self-hosted
ships with auth off, Ollama has none at all — and internet scans keep
finding thousands of Ollama instances exposed by a careless 0.0.0.0.
Don't join them.

To serve beyond localhost:

1. Set the bind address explicitly — it is an intentional decision:

   yaml Copy

   server:
     host: 0.0.0.0     # containers / remote access; keep 127.0.0.1 otherwise
     port: 8989

2. Put an authenticating reverse proxy in front. A complete example with
   Caddy (TLS is automatic):

   text Copy

   rag.example.com {
       basic_auth {
           # generate the hash with: caddy hash-password
           alice $2a$14$...bcrypt-hash...
       }
       reverse_proxy 127.0.0.1:8989
   }

   With this in place serviette itself can stay on 127.0.0.1 — only the proxy
   is exposed. Any equivalent (nginx + auth_basic, oauth2-proxy, Cloudflare
   Access, a VPN like Tailscale) works the same way.

CORS is deliberately not emitted by default: serviette's own UIs never
need it (the embedded chat is same-origin; the standalone frontend proxies
server-side, so frontend.api_url may point at a backend on another host —
or another fleet — without any browser-visible cross-origin traffic), and
the absence of CORS headers is what stops arbitrary web pages from reading
responses off localhost instances. If you build your own browser frontend
that calls the API directly from another origin, opt in with an explicit
allowlist:

server:
  cors_origins: ["https://app.example.com"]   # never "*" unless you truly mean it

The engine's monitoring endpoints (indexer.monitoring_http_port) are
loopback-only by construction and cannot be exposed directly.

Appendix: regenerating the demo GIF Copy Chapter

The README demo GIF is rendered with Pillow (no terminal recorder or browser
needed), so it regenerates deterministically anywhere:

pip install pillow
python docs/generate_demos.py     # -> docs/assets/demo.gif

The frames show real output: the terminal lines are those of a recorded
serviette demo run (heartbeats thinned), the answers are what the model
produced on the bundled corpus before and after the sed edit, and the header
statistics are the real /api/v1/stats values. They live as constants in the
script (SCRIPT_UP, SCRIPT_EDIT, ANSWER_*, STATS_*) — when the output
of demo/up or the chat UI changes, re-record and update them rather than
inventing lines.