## File: README.md # lossless-claw Lossless Context Management plugin for [OpenClaw](https://github.com/openclaw/openclaw), based on the [LCM paper](https://papers.voltropy.com/LCM) from [Voltropy](https://x.com/Voltropy). Replaces OpenClaw's built-in sliding-window compaction with a DAG-based summarization system that preserves every message while keeping active context within model token limits. ## Table of contents - [What it does](#what-it-does) - [Quick start](#quick-start) - [Configuration](#configuration) - [Commands And Skill](#commands-and-skill) - [Session Migration CLI](#session-migration-cli) - [Documentation](#documentation) - [Development](#development) - [Line endings](#line-endings) - [Security](#security) - [License](#license) ## What it does Two ways to learn: read the below, or [check out this super cool animated visualization](https://losslesscontext.ai). When a conversation grows beyond the model's context window, OpenClaw (just like all of the other agents) normally truncates older messages. LCM instead: 1. **Persists every message** in a SQLite database, organized by conversation 2. **Summarizes chunks** of older messages into summaries using your configured LLM 3. **Condenses summaries** into higher-level nodes as they accumulate, forming a DAG (directed acyclic graph) 4. **Assembles context** each turn by combining summaries + recent raw messages 5. **Provides tools** (`lcm_grep`, `lcm_describe`, `lcm_expand`) so agents can search and recall details from compacted history Nothing is lost. Raw messages stay in the database. Summaries link back to their source messages. Agents can drill into any summary to recover the original detail. **It feels like talking to an agent that never forgets. Because it doesn't. In normal operation, you'll never need to think about compaction again.** ## Commands And Skill The package installs an agent-oriented `lcm` shell CLI and includes a bundled `lossless-claw` skill plus plugin commands for supported OpenClaw chat/native command providers. The shell CLI reads `lcm.db` without modifying conversation data. Its only write command sets one validated Lossless config value in `openclaw.json`: ```bash lcm status lcm conversations show --session-key 'agent:main:example' lcm messages tail --conversation-id 42 lcm summaries list --conversation-id 42 --depth 0 --recency 7d lcm config get freshTailCount lcm config set freshTailCount 96 ``` JSON is the default output. List commands use bounded keyset pagination. See [Lossless Claw CLI](docs/cli.md) for commands, filters, path precedence, output fields, config-write safety, and exit codes. The native OpenClaw command surface provides in-session operations: - `/lossless` shows version, enablement/selection state, DB path and size, summary counts, and summary-health status - `/lossless backup` creates a timestamped backup of the current LCM SQLite database - `/lossless rotate` rewrites the active session transcript into a compact tail-preserving form without changing the live OpenClaw session identity or current LCM conversation - `/lossless doctor` scans for broken or truncated summaries - `/lossless doctor apply` repairs broken summaries in the current conversation after the normal safety preflight - `/lossless doctor apply confirm-offline` repairs a specific conversation after its active channel path has been paused or moved away; targeted repair is restricted to authorized OpenClaw command senders and always requires the explicit offline confirmation - `/lossless doctor clean` shows read-only high-confidence junk diagnostics for archived subagents and cron sessions under every configured OpenClaw agent id, plus NULL-key orphaned subagent runs - `/lossless status` shows plugin, conversation, and maintenance state including deferred compaction debt - `/lcm` is the shorter alias for `/lossless` Supported native command examples: - `/lossless` - `/lossless backup` - `/lossless rotate` - `/lossless doctor` - `/lossless doctor apply 42 confirm-offline` - `/lossless doctor clean` - `/lcm` The package does not register these OpenClaw root subcommands: - `openclaw lossless` - `openclaw lcm` - `openclaw /lossless` - `openclaw /lcm` The bundled skill focuses on configuration, diagnostics, architecture, and recall-tool usage. Its reference set lives under `skills/lossless-claw/references/`. ### Programmatic control Lossless-claw also exposes an optional host-facing context-engine control contract for OpenClaw gateways that support context-engine capabilities and control dispatch. The contract is intentionally smaller than the native slash command surface: - `status` returns whether an LCM conversation is active and the current stored message count. - `doctor` returns a bounded, sanitized warning list for summary-health issues. - `rotate` runs the same safe transcript rotation path as `/lossless rotate` and returns the post-rotate message count plus the timestamp for that successful rotate operation. Programmatic control never returns transcript text, local database paths, backup paths, credentials, provider debug, or shell output. `status` does not report `lastRotatedAt`; hosts that need durable rotation timestamps should persist that product state themselves after a successful `rotate` result. This surface is capability-gated by the OpenClaw host. At the time of this change there is not yet a stable OpenClaw release with the required context-engine control endpoints; downstream users should treat it as unavailable unless their host advertises the matching capability, for example through the pending `openclaw/openclaw#98060` contract or an equivalent downstream gateway. ## Session Migration CLI `lossless-claw-migrate-sessions` is a one-time shell CLI for backfilling OpenClaw JSONL session files into `lcm.db` after lossless-claw was disabled, missing, or installed after sessions already existed. It is not a background replay loop and it does not run summarization. Run it in dry-run mode first: ```bash npx --package @martian-engineering/lossless-claw@latest lossless-claw-migrate-sessions --state-dir ~/.openclaw ``` Apply the import only after reviewing the dry-run output: ```bash npx --package @martian-engineering/lossless-claw@latest lossless-claw-migrate-sessions --state-dir ~/.openclaw --apply ``` The command defaults to `${OPENCLAW_STATE_DIR:-~/.openclaw}` and `${OPENCLAW_STATE_DIR:-~/.openclaw}/lcm.db`. `--apply` creates a timestamped SQLite backup before writing when the database already exists. Use `--file ` or repeatable `--sessions-dir ` for targeted imports, `--since ` or `--limit ` to narrow a batch, and `--json` for machine-readable output. ## Quick start ### Prerequisites - OpenClaw with plugin context engine support - Node.js 22+ - An LLM provider configured in OpenClaw (used for summarization) > **Compatibility:** Lossless Claw 0.x supports file-backed OpenClaw from `2026.5.28` through `2026.7.1`. OpenClaw `2026.7.2` prereleases and later use SQLite-backed session storage and require Lossless Claw 1.0 from the `next/1.0` release line. If you cannot move to 1.0, remain on OpenClaw `2026.7.1` or select OpenClaw's `legacy` context engine. Lossless Claw `0.9.4` remains the fallback for OpenClaw versions older than `2026.5.28`. ### Install the plugin Use OpenClaw's plugin installer (recommended): ```bash openclaw plugins install @martian-engineering/lossless-claw@latest ``` If you're running from a local OpenClaw checkout, use: ```bash pnpm openclaw plugins install @martian-engineering/lossless-claw@latest ``` Use exact versions only for rollback or reproducible canary testing. OpenClaw records an exact install spec such as `@martian-engineering/lossless-claw@0.12.0` as a pinned update track, so OpenClaw plugin update sync will keep that version until you move back to the stable track: ```bash openclaw plugins update @martian-engineering/lossless-claw@latest ``` For local plugin development, build your working copy first, then link it instead of copying files: ```bash cd /path/to/lossless-claw pnpm build openclaw plugins install --link /path/to/lossless-claw # or from a local OpenClaw checkout: # pnpm openclaw plugins install --link /path/to/lossless-claw ``` Re-run `pnpm build` after local source changes so the linked plugin's `dist/` output stays current. The install command records the plugin, enables it, and applies compatible slot selection (including `contextEngine` when applicable). > **Note:** If your OpenClaw config uses `plugins.allow`, allowlist the plugin id `lossless-claw` plus any other active plugins you rely on. Do not add command tokens or aliases like `lossless` or `/lcm` to `plugins.allow`; that setting only accepts plugin ids. In some setups, narrowing the allowlist can prevent plugin-backed integrations from loading, even if `lossless-claw` itself is installed correctly. Restart the gateway after plugin config changes. ### Configure OpenClaw In most cases, no manual JSON edits are needed after `openclaw plugins install`. If you need to set it manually, ensure the context engine slot points at lossless-claw: ```json { "plugins": { "slots": { "contextEngine": "lossless-claw" } } } ``` Restart OpenClaw after configuration changes. ## Configuration LCM is configured through a combination of plugin config and environment variables. Environment variables take precedence for backward compatibility. ### Plugin config Add a `lossless-claw` entry under `plugins.entries` in your OpenClaw config: ```json { "plugins": { "entries": { "lossless-claw": { "enabled": true, "llm": { "allowModelOverride": true, "allowedModels": ["openai/gpt-5.4-mini"] }, "config": { "freshTailCount": 64, "leafChunkTokens": 80000, "newSessionRetainDepth": 2, "contextThreshold": 0.75, "contextThresholdOverrides": [ { "name": "large-context-models", "match": { "modelContextWindowMin": 900000 }, "contextThreshold": 0.15 }, { "name": "telegram-sessions", "match": { "sessionPattern": "agent:*:telegram:**" }, "contextThreshold": 0.3 } ], "incrementalMaxDepth": 1, "cacheAwareCompaction": { "enabled": true, "cacheTTLSeconds": 300 }, "ignoreSessionPatterns": [ "agent:*:cron:**", "agent:*:**:active-memory:**", "agent:*:dreaming-narrative-**" ], "transcriptGcEnabled": false, "proactiveThresholdCompactionMode": "deferred", "summaryModel": "openai/gpt-5.4-mini", "expansionModel": "openai/gpt-5.4-mini", "delegationTimeoutMs": 300000, "summaryTimeoutMs": 60000, "summaryCallWindowMs": 600000, "summaryMaxCallsPerWindow": 24, "summarySpendBackoffMs": 1800000 } } } } } ``` The `ignoreSessionPatterns` entries in this example are storage exclusions. Matching cron, active-memory, and OpenClaw memory-core dreaming narrative sessions do not create LCM conversation rows or store messages in LCM. `leafChunkTokens` controls how many source tokens can accumulate in a leaf compaction chunk before summarization is triggered. The default is `20000`, but quota-limited summary providers may benefit from a larger value to reduce compaction frequency. `summaryModel` and `summaryProvider` let you request a cheaper or faster compaction model through OpenClaw's `api.runtime.llm.complete` capability; OpenClaw still owns provider dispatch and auth. Explicit summary model requests require `llm.allowModelOverride` and matching `llm.allowedModels` policy entries for `lossless-claw`. `expansionModel` does the same for `lcm_expand_query` sub-agent calls (drilling into summaries to recover detail). `delegationTimeoutMs` controls how long `lcm_expand_query` waits for that delegated sub-agent to finish before returning a timeout error; it defaults to `120000` (120s). `summaryTimeoutMs` controls the per-call timeout for model-backed LCM summarization; it defaults to `60000` (60s). `summaryMaxCallsPerWindow`, `summaryCallWindowMs`, and `summarySpendBackoffMs` bound repeated non-auth summarization spend per session. When unset, the model settings still fall back to OpenClaw's configured default model/provider. See [Expansion model override requirements](#expansion-model-override-requirements) for the required `subagent` trust policy when using `expansionModel`. ### Environment variables | Variable | Default | Description | |----------|---------|-------------| | `LCM_ENABLED` | `true` | Enable/disable the plugin | | `LCM_DATABASE_PATH` | `~/.openclaw/lcm.db` | Path to the SQLite database | | `LCM_IGNORE_SESSION_PATTERNS` | `""` | Comma-separated glob patterns for session keys to exclude from LCM storage | | `LCM_STATELESS_SESSION_PATTERNS` | `""` | Comma-separated glob patterns for session keys that may read from LCM but never write to it | | `LCM_SKIP_STATELESS_SESSIONS` | `true` | Enable stateless-session write skipping for matching session keys | | `LCM_CONTEXT_THRESHOLD` | `0.75` | Fraction of context window that triggers compaction (0.0–1.0) | | `LCM_FRESH_TAIL_COUNT` | `64` | Number of recent messages protected from compaction | | `LCM_NEW_SESSION_RETAIN_DEPTH` | `2` | Context retained after `/new` (`-1` keeps all context, `2` keeps d2+) | | `LCM_LEAF_MIN_FANOUT` | `8` | Minimum raw messages per leaf summary | | `LCM_CONDENSED_MIN_FANOUT` | `4` | Minimum summaries per condensed node | | `LCM_CONDENSED_MIN_FANOUT_HARD` | `2` | Relaxed fanout for forced compaction sweeps | | `LCM_INCREMENTAL_MAX_DEPTH` | `1` | How deep incremental compaction goes (0 = leaf only, 1 = one condensed pass, -1 = unlimited) | | `LCM_LEAF_CHUNK_TOKENS` | `20000` | Max source tokens per leaf compaction chunk | | `LCM_LEAF_TARGET_TOKENS` | `2400` | Target token count for leaf summaries | | `LCM_CONDENSED_TARGET_TOKENS` | `2000` | Target token count for condensed summaries | | `LCM_MAX_EXPAND_TOKENS` | `4000` | Token cap for sub-agent expansion queries | | `LCM_LARGE_FILE_TOKEN_THRESHOLD` | `25000` | File blocks above this size are intercepted and stored separately | | `LCM_LARGE_FILE_SUMMARY_PROVIDER` | `""` | Provider override for large-file summarization | | `LCM_LARGE_FILE_SUMMARY_MODEL` | `""` | Model override for large-file summarization | | `LCM_SUMMARY_MODEL` | `""` | Model override for compaction summarization; falls back to OpenClaw's default model when unset | | `LCM_SUMMARY_PROVIDER` | `""` | Provider override for compaction summarization; falls back to `OPENCLAW_PROVIDER` or the provider embedded in the model ref | | `LCM_SUMMARY_BASE_URL` | *(from OpenClaw / provider default)* | Base URL override for summarization API calls | | `LCM_EXPANSION_MODEL` | *(from OpenClaw)* | Model override for `lcm_expand_query` sub-agent (e.g. `openai/gpt-5.4-mini`) | | `LCM_EXPANSION_PROVIDER` | *(from OpenClaw)* | Provider override for `lcm_expand_query` sub-agent | | `LCM_DELEGATION_TIMEOUT_MS` | `120000` | Max time to wait for delegated `lcm_expand_query` sub-agent completion | | `LCM_SUMMARY_TIMEOUT_MS` | `60000` | Max time to wait for a single model-backed LCM summarizer call | | `LCM_SUMMARY_CALL_WINDOW_MS` | `600000` | Rolling window used by the per-session summarization spend guard | | `LCM_SUMMARY_MAX_CALLS_PER_WINDOW` | `24` | Max model-backed summarization calls per session/window before spend backoff opens | | `LCM_SUMMARY_SPEND_BACKOFF_MS` | `1800000` | Cooldown after the summarization spend guard opens | | `LCM_PRUNE_HEARTBEAT_OK` | `false` | Retroactively delete `HEARTBEAT_OK` turn cycles from LCM storage | | `LCM_TRANSCRIPT_GC_ENABLED` | `false` | Enable transcript rewrite GC during `maintain()` | | `LCM_PROACTIVE_THRESHOLD_COMPACTION_MODE` | `deferred` | Choose whether proactive threshold compaction is deferred into maintenance debt or kept inline for legacy behavior | | `LCM_CACHE_TTL_SECONDS` | `300` | Cache TTL used by cache-aware deferred compaction when provider/runtime telemetry does not supply a more specific retention window | Transcript GC rewrites are disabled by default. Set `transcriptGcEnabled` or `LCM_TRANSCRIPT_GC_ENABLED` to turn them on explicitly. Deferred proactive compaction is also the default. Set `proactiveThresholdCompactionMode` or `LCM_PROACTIVE_THRESHOLD_COMPACTION_MODE` to `inline` only if you need legacy foreground compaction behavior. In deferred mode, lossless-claw records one coalesced prompt-mutating debt item after the turn, leaves background `maintain()` to process only non-prompt-mutating work while Anthropic cache is still hot, and then consumes that debt pre-assembly once the cache is cold or the prompt is approaching overflow. ### Expansion model override requirements If you want `lcm_expand_query` to run on a dedicated model via `expansionModel` or `LCM_EXPANSION_MODEL`, OpenClaw must explicitly trust the plugin to request sub-agent model overrides. For most setups, `openai/gpt-5.4-mini` is a better starting point than Anthropic Haiku because it is cheap, fast, and does not depend on Anthropic quota remaining. Add a `subagent` policy under `plugins.entries.lossless-claw` and allowlist the canonical `provider/model` target you want the plugin to use: ```json { "models": { "openai/gpt-4.1-mini": {} }, "plugins": { "entries": { "lossless-claw": { "enabled": true, "subagent": { "allowModelOverride": true, "allowedModels": ["openai/gpt-4.1-mini"] }, "config": { "expansionModel": "openai/gpt-4.1-mini" } } } } } ``` - `subagent.allowModelOverride` is required for OpenClaw to honor plugin-requested per-run `provider`/`model` overrides. - `subagent.allowedModels` is optional but recommended. Use `"*"` only if you intentionally want to trust any target model. - The chosen expansion target must also be available in OpenClaw's normal model catalog. If it is not already configured elsewhere, add it under the top-level `models` map as shown above. - If you prefer splitting provider and model, set `config.expansionProvider` and use a bare `config.expansionModel`. - `openclaw doctor --fix` can add the required `subagent` policy for a configured `expansionModel`. If a host still rejects a stale or unavailable override, `lcm_expand_query` retries once without the override so recall does not fail hard. Plugin config equivalents: - `ignoreSessionPatterns` - `statelessSessionPatterns` - `skipStatelessSessions` - `transcriptGcEnabled` - `newSessionRetainDepth` - `summaryModel` - `summaryProvider` - `delegationTimeoutMs` - `summaryTimeoutMs` Environment variables still win over plugin config when both are set. ### Summary model priority For compaction summarization, lossless-claw resolves the model in this order: 1. `LCM_SUMMARY_MODEL` / `LCM_SUMMARY_PROVIDER` 2. Plugin config `summaryModel` / `summaryProvider` 3. OpenClaw's default compaction model/provider 4. Runtime/session model/provider hints from OpenClaw If `summaryModel` already includes a provider prefix such as `anthropic/claude-sonnet-4-20250514`, `summaryProvider` is ignored for that choice. Otherwise, the provider falls back to the matching override, then `OPENCLAW_PROVIDER`, then the provider inferred by the caller. Summary calls are dispatched through OpenClaw's runtime LLM layer, so auth profiles, OAuth refresh, API keys, base URLs, and provider-specific request preparation remain host-owned. Run `openclaw doctor --fix` after adding explicit summary model overrides if you want OpenClaw to add the matching `plugins.entries.lossless-claw.llm` policy. ### Recommended starting configuration ``` LCM_FRESH_TAIL_COUNT=64 LCM_LEAF_CHUNK_TOKENS=20000 LCM_INCREMENTAL_MAX_DEPTH=1 LCM_CONTEXT_THRESHOLD=0.75 LCM_SUMMARY_MODEL=openai/gpt-5.4-mini LCM_EXPANSION_MODEL=openai/gpt-5.4-mini ``` - **freshTailCount=64** protects the last 64 messages from compaction, expanding through the newest user when needed so its assistant/tool suffix cannot be separated from the active instruction. - **leafChunkTokens=20000** limits how large each leaf compaction chunk can grow before LCM summarizes it. Increase this when your summary provider is quota-limited and frequent leaf compactions are exhausting that quota. - **incrementalMaxDepth=1** runs one condensed pass after each leaf compaction by default. Set to `0` for leaf-only behavior, a larger positive integer for a deeper cap, or `-1` for unlimited cascading. - **contextThreshold=0.75** triggers compaction when context reaches 75% of the model's window, leaving headroom for the model's response. - **contextThresholdOverrides** optionally picks a different threshold for matching model ids, model context-window ranges, or session patterns. If no rule matches, LCM falls back to `contextThreshold`. ### Session exclusion patterns ### Session reset semantics Lossless-claw distinguishes OpenClaw's two session-reset commands: - `/new` keeps the active conversation row and all stored summaries, but prunes `context_items` so the next turn rebuilds context from retained summaries instead of the fresh tail. - `/reset` archives the active conversation row and creates a new active row for the same stable `sessionKey`, giving the next turn a clean LCM conversation while preserving prior history. For large sessions, neither command is a perfect “keep my live agent context, but stop writing into this giant active LCM row” tool: - `/new` keeps writing into the same active LCM conversation row. - `/reset` changes OpenClaw session flow, which is heavier than users often want when their real problem is just LCM row size. `/lossless rotate` fills that gap. Before trimming the transcript, it forces leaf-only compaction for raw context outside the preserved live tail so older transcript messages are represented by LCM summaries. It then replaces one rolling `rotate-latest` SQLite backup, rewrites the current session transcript down to the preserved live tail plus current session settings, and refreshes the bootstrap frontier on the same active LCM conversation so dropped transcript history is not replayed. Existing durable messages, summaries, context items, and conversation identity stay in place; only the transcript backing is compacted. If you want additional timestamped snapshots instead, run `/lossless backup`. `newSessionRetainDepth` (or `LCM_NEW_SESSION_RETAIN_DEPTH`) controls how much summary structure survives `/new`: - `-1`: keep all existing context items - `0`: keep all summaries, drop only fresh-tail messages - `1`: keep d1+ summaries - `2`: keep d2+ summaries; recommended default - `3+`: keep only deeper, more abstract summaries Lossless-claw applies `/new` pruning through `before_reset` and uses `session_end` to catch transcript rollovers such as `/reset`, idle or daily session rotation, compaction session replacement, and deletions. User-facing confirmation text after `/new` or `/reset` must still be emitted by OpenClaw's command handlers. Use `ignoreSessionPatterns` or `LCM_IGNORE_SESSION_PATTERNS` to keep low-value sessions completely out of LCM. Matching sessions do not create conversations, do not store messages, and do not participate in compaction or delegated expansion grants. Cron scheduler keys (`agent::cron:...`) are isolated automatically when OpenClaw reuses the same `sessionKey` for a new runtime `sessionId`: lossless-claw archives the prior active run and creates a fresh LCM conversation for the new run. Leave cron sessions out of `ignoreSessionPatterns` when they need in-run LCM compaction. Pattern rules: - `*` matches any characters except `:` - `**` matches anything, including `:` - Patterns match the full session key Examples: - `agent:*:cron:**` excludes cron sessions for any agent when you want to bypass LCM entirely - `agent:*:**:active-memory:**` excludes active-memory sessions under nested session-key prefixes; the `**` segment is intentionally broad because it spans colon-separated session-key segments - `agent:*:dreaming-narrative-**` excludes OpenClaw memory-core dreaming narrative sessions; OpenClaw builds those keys with the `dreaming-narrative-` prefix ([source](https://github.com/openclaw/openclaw/blob/b81666ca6af25c86cc099983a4358cdc5ea9ced8/extensions/memory-core/src/dreaming-narrative.ts)) - `agent:main:subagent:**` excludes all main-agent subagent sessions - `agent:ops:**` excludes every session under the `ops` agent id Treat these examples as storage exclusions. Matching sessions do not create LCM conversation rows or store messages in LCM, so use them only for lanes whose history can stay outside LCM. Environment variable example: ```bash LCM_IGNORE_SESSION_PATTERNS=agent:*:cron:**,agent:main:subagent:** ``` Plugin config example: ```json { "plugins": { "entries": { "lossless-claw": { "config": { "ignoreSessionPatterns": [ "agent:*:cron:**", "agent:main:subagent:**" ] } } } } } ``` ### Stateless session patterns Use `statelessSessionPatterns` or `LCM_STATELESS_SESSION_PATTERNS` for sessions that should still be able to read from existing LCM context, but should never create or mutate LCM state themselves. This is useful for delegated or temporary sub-agent sessions that should benefit from retained context without polluting the database. When `skipStatelessSessions` or `LCM_SKIP_STATELESS_SESSIONS` is enabled, matching sessions: - skip bootstrap imports - skip message persistence during ingest and after-turn hooks - skip compaction writes and delegated expansion grant writes - can still assemble context from already-persisted conversations when a matching conversation exists Pattern rules are the same as `ignoreSessionPatterns`, and matching is done against the full session key. Environment variable example: ```bash LCM_STATELESS_SESSION_PATTERNS=agent:*:subagent:**,agent:ops:subagent:** LCM_SKIP_STATELESS_SESSIONS=true ``` Plugin config example: ```json { "plugins": { "entries": { "lossless-claw": { "config": { "statelessSessionPatterns": [ "agent:*:subagent:**", "agent:ops:subagent:**" ], "skipStatelessSessions": true } } } } } ``` ### OpenClaw session reset settings LCM preserves history through compaction, but it does **not** change OpenClaw's core session reset policy. If sessions are resetting sooner than you want, increase OpenClaw's `session.reset.idleMinutes` or use a channel/type-specific override. ```json { "session": { "reset": { "mode": "idle", "idleMinutes": 10080 } } } ``` - `session.reset.mode: "idle"` keeps a session alive until the idle window expires. - `session.reset.idleMinutes` is the actual reset interval in minutes. - OpenClaw does **not** currently enforce a maximum `idleMinutes`; in source it is validated only as a positive integer. - If you also use daily reset mode, `idleMinutes` acts as a secondary guard and the session resets when **either** the daily boundary or the idle window is reached first. - Legacy `session.idleMinutes` still works, but OpenClaw prefers `session.reset.idleMinutes`. Useful values: - `1440` = 1 day - `10080` = 7 days - `43200` = 30 days - `525600` = 365 days For most long-lived LCM setups, a good starting point is: ```json { "session": { "reset": { "mode": "idle", "idleMinutes": 10080 } } } ``` ## Documentation - [Configuration guide](docs/configuration.md) - [Architecture](docs/architecture.md) - [Agent tools](docs/agent-tools.md) - [TUI Reference](docs/tui.md) - [lcm-tui](tui/README.md) - [Optional: enable FTS5 for fast full-text search](docs/fts5.md) ## Development ```bash # Build (bundles TypeScript to dist/index.js) pnpm build # Run tests npx vitest # Type check npx tsc --noEmit # Run a specific test file npx vitest test/engine.test.ts ``` ### Project structure ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Line endings This repository codifies LF line endings through `.gitattributes` so diffs stay stable across macOS, Linux, and Windows development environments. - Git should store text files with LF endings in the repository. - The root `* text=auto eol=lf` rule normalizes newly added text files. - Common source, documentation, lockfile, template, and config extensions are listed explicitly so contributors and tooling see the expected policy. - Binary assets and SQLite/database files are marked `binary` to avoid unsafe newline conversion. - If your editor or operating system prefers CRLF locally, keep the repository rules authoritative and avoid committing CRLF-only rewrites. - After changing line-ending rules, prefer a focused normalization commit so future functional changes remain easy to review. ## Security Please report suspected vulnerabilities privately. See [SECURITY.md](SECURITY.md) for the supported disclosure process and response expectations. ## License MIT --- ## File: .changeset/README.md # Changesets Hello and welcome! This folder has been automatically generated by `@changesets/cli`, a build tool that works with multi-package repos, or single-package repos to help you version and publish your code. You can find the full documentation for it [in our repository](https://github.com/changesets/changesets). We have a quick list of common questions to get you started engaging with this project in [our documentation](https://github.com/changesets/changesets/blob/main/docs/common-questions.md). --- ## File: docs/agent-tools.md # Agent tools LCM provides four tools for agents to search, inspect, and recall information from compacted conversation history. ## Usage patterns ### Escalation pattern: grep → describe → expand_query Most recall tasks follow this escalation: 1. **`lcm_grep`** — Find relevant summaries, messages, or externalized file prefixes by keyword/regex 2. **`lcm_describe`** — Inspect a specific summary's full content (cheap, no sub-agent) 3. **`lcm_expand_query`** — Deep recall: spawn a sub-agent to expand the DAG and answer a focused question Start with grep. If the snippet is enough, stop. If you need full summary content, use describe. If you need details that were compressed away, use expand_query. ### When to expand Summaries are lossy by design. The "Expand for details about:" footer at the end of each summary lists what was dropped. Use `lcm_expand_query` when you need: - Exact commands, error messages, or config values - File paths and specific code changes - Decision rationale beyond what the summary captured - Tool call sequences and their outputs - Verbatim quotes or specific data points `lcm_expand_query` is bounded (~120s, scoped sub-agent) and relatively cheap. Don't ration it, but use `lcm_grep` first when you need broad discovery across many sessions. ## Tool reference ### lcm_grep Search across messages, summaries, and/or the bounded prefix of externalized large files using regex or full-text search. Use `mode: "full_text"` for keyword or topical recall. Full-text queries are not regexes: alternation (`A|B`), regex wildcards (`.*`), character classes (`[abc]`), and anchors (`^foo`, `foo$`) require `mode: "regex"`. Wrap exact multi-word phrases in quotes to preserve phrase matching. Keep the default `sort: "recency"` for recent events, switch to `sort: "relevance"` when looking for the best older match on a topic, and use `sort: "hybrid"` when you want relevance without giving up recency entirely. **Parameters:** | Param | Type | Required | Default | Description | |-------|------|----------|---------|-------------| | `pattern` | string | ✅ | — | Search pattern | | `mode` | string | | `"regex"` | `"regex"` or `"full_text"` | | `scope` | string | | `"both"` | `"messages"`, `"summaries"`, `"both"`, or `"files"` | | `fileIds` | string[] | | — | Optional `file_xxx` IDs to restrict `scope: "files"` searches | | `conversationId` | number | | current session family | Specific physical conversation to search | | `allConversations` | boolean | | `false` | Search all conversations | | `since` | string | | — | ISO timestamp lower bound | | `before` | string | | — | ISO timestamp upper bound | | `limit` | number | | 50 | Max results (1–200) | | `sort` | string | | `"recency"` | `"recency"`, `"relevance"`, or `"hybrid"` for full-text ranking | **Returns:** Array of matches with: - `id` — Message, summary, or file ID - `type` — `"message"`, `"summary"`, or `"file"` - `snippet` — Truncated content around the match - `conversationId` — Which conversation - `createdAt` — Timestamp - For summaries: `depth`, `kind`, `summaryId` - For files: line number, byte offset, matched text, and a snippet from the first 512,000 bytes scanned per file **Examples:** ``` # Full-text search across all conversations lcm_grep(pattern: "database migration", mode: "full_text", allConversations: true) # Older-topic recall ranked by FTS relevance lcm_grep(pattern: "\"error handling\" retries", mode: "full_text", sort: "relevance") # Regex search in summaries only lcm_grep(pattern: "config\\.threshold.*0\\.[0-9]+", scope: "summaries") # Recent messages containing a specific term lcm_grep(pattern: "deployment", since: "2026-02-19T00:00:00Z", scope: "messages") # Search the bounded scanned prefix of an externalized file lcm_grep(pattern: "CRITICAL_MARKER", scope: "files", fileIds: ["file_789abc012345"]) ``` ### lcm_describe Look up metadata and content for a specific summary or stored file. **Parameters:** | Param | Type | Required | Default | Description | |-------|------|----------|---------|-------------| | `id` | string | ✅ | — | `sum_xxx` for summaries, `file_xxx` for files | | `conversationId` | number | | current session family | Scope to a specific physical conversation | | `allConversations` | boolean | | `false` | Allow cross-conversation lookups | **Returns for summaries:** - Full summary content - Metadata: depth, kind, token count, created timestamp - Time range (earliestAt, latestAt) - Descendant count - Parent summary IDs (for condensed summaries) - Child summary IDs - Source message IDs (for leaf summaries) - File IDs referenced in the summary **Returns for files:** - File content, capped by `expandFileMaxBytes` - Metadata: fileName, mimeType, byteSize - Exploration summary - Storage path **Examples:** ``` # Inspect a summary from context lcm_describe(id: "sum_abc123def456") # Retrieve a stored large file lcm_describe(id: "file_789abc012345") ``` ### lcm_expand_query Answer a focused question by expanding summaries through the DAG. Spawns a bounded sub-agent that walks parent links down to source material and returns a compact answer. When `allConversations: true` is set, `lcm_expand_query` can synthesize one answer across multiple conversations. That cross-conversation mode is bounded, not exhaustive: it ranks conversation buckets, expands only the top few under one shared deadline, and marks the result truncated when lower-ranked buckets are skipped or fail. The selected buckets share the existing `tokenCap`, so concurrent recall does not multiply the retrieval budget. **Parameters:** | Param | Type | Required | Default | Description | |-------|------|----------|---------|-------------| | `prompt` | string | ✅ | — | The question to answer | | `query` | string | ✅* | — | Text query to find summaries (if no `summaryIds`) | | `summaryIds` | string[] | ✅* | — | Specific summary IDs to expand (if no `query`) | | `maxTokens` | number | | 2000 | Answer length cap | | `timeoutMs` | number | ✅ | `delegationTimeoutMs + 30000` | Total OpenClaw dynamic tool RPC timeout; use the schema default so delegated recall can finish before the host watchdog fires | | `conversationId` | number | | current session family | Scope to a specific physical conversation | | `allConversations` | boolean | | `false` | Search across all conversations | *One of `query` or `summaryIds` is required. **Returns:** - `answer` — The focused answer text - `citedIds` — Summary IDs that contributed to the answer - `sourceConversationIds` — Conversations that were successfully expanded - `expandedSummaryCount` — How many summaries were expanded - `totalSourceTokens` — Total tokens read from the DAG - `truncated` — Whether source expansion was truncated or any selected conversation was skipped or failed - `conversationBreakdown` — Optional per-conversation success/failure diagnostics for bounded multi-conversation runs Successful single-conversation results keep the response shape above. When delegated recall fails, the result keeps the human-readable `error` and adds `errorCode`, empty source counters, and a `conversationBreakdown`. Failed entries identify the conversation, attempted summary IDs, failure phase, elapsed time, and error code. Timed-out child work is cancelled through the host-owned temporary-session cleanup path. Completed conversation buckets still contribute evidence when another bucket times out; timed-out buckets do not contribute guessed answer text or citations. **Examples:** ``` # Find and expand summaries about a topic lcm_expand_query( query: "OAuth authentication fix", prompt: "What was the root cause and what commits fixed it?", timeoutMs: 150000 ) # Expand specific summaries you already have lcm_expand_query( summaryIds: ["sum_abc123", "sum_def456"], prompt: "What were the exact file changes?", timeoutMs: 150000 ) # Cross-conversation synthesis lcm_expand_query( query: "deployment procedure", prompt: "What's the current deployment process?", allConversations: true, timeoutMs: 150000 ) ``` ### lcm_expand Low-level DAG expansion tool. **Only available to sub-agents** spawned by `lcm_expand_query`. Main agents should always use `lcm_expand_query` instead. This tool is what the expansion sub-agent uses internally to walk the summary DAG, read source messages, and build its answer. ## Tips for agent developers ### Configuring agent prompts Add instructions to your agent's system prompt so it knows when to use LCM tools: ```markdown ## Memory & Context Use LCM tools for recall: 1. `lcm_grep` — Search all conversations by keyword/regex. Prefer `mode: "full_text"` for short topic terms, use `mode: "regex"` for alternation or other regex syntax, quote exact phrases, use `sort: "relevance"` for older-topic lookups, and `sort: "hybrid"` when recency should still matter. 2. `lcm_describe` — Inspect a specific summary (cheap, no sub-agent) 3. `lcm_expand_query` — Deep recall with bounded sub-agent expansion When summaries in context have an "Expand for details about:" footer listing something you need, use `lcm_expand_query` to get the full detail. ``` ### Conversation scoping By default, tools operate on the current session family: the active conversation plus archived segments that share the same stable session identity. This keeps recall continuous across session rotation and `/reset` replacement rows without widening the search to unrelated sessions. Use `lcm_grep(..., allConversations: true)` when you need broad global discovery. Use `lcm_expand_query(..., allConversations: true)` when you want bounded synthesis across sessions. Use `conversationId` when you already know the exact physical conversation to inspect or expand. ### Performance considerations - `lcm_grep` and `lcm_describe` are fast (direct database queries) - `lcm_expand_query` spawns a sub-agent and takes ~30–120 seconds - The sub-agent has a 120-second timeout with cleanup guarantees by default, and the tool schema advertises a 150-second OpenClaw dynamic RPC timeout so the host watchdog stays open long enough for delegated recall plus result cleanup - Token caps (`LCM_MAX_EXPAND_TOKENS`) prevent runaway expansion - Cross-conversation `lcm_expand_query` expands only a bounded set of top-ranked conversations --- ## File: docs/architecture.md # Architecture This document describes how lossless-claw works internally — the data model, compaction lifecycle, context assembly, and expansion system. ## Data model ### Conversations and messages Every OpenClaw session maps to a **conversation**. The first time a session ingests a message, LCM creates a conversation record keyed by the runtime session ID. Messages are stored with: - **seq** — Monotonically increasing sequence number within the conversation - **role** — `user`, `assistant`, `system`, or `tool` - **content** — Plain text extraction of the message - **tokenCount** — Estimated token count (~4 chars/token) - **createdAt** — Insertion timestamp Each message also has **message_parts** — structured content blocks that preserve the original shape (text blocks, tool calls, tool results, reasoning, file content, etc.). This allows the assembler to reconstruct rich content when building model context, not just flat text. ### The summary DAG Summaries form a directed acyclic graph with two node types: **Leaf summaries** (depth 0, kind `"leaf"`): - Created from a chunk of raw messages - Linked to source messages via `summary_messages` - Contain a narrative summary with timestamps - Typically 800–1200 tokens **Condensed summaries** (depth 1+, kind `"condensed"`): - Created from a chunk of summaries at the same depth - Linked to parent summaries via `summary_parents` - Each depth tier uses a progressively more abstract prompt - Typically 1500–2000 tokens Every summary carries: - **summaryId** — `sum_` + 16 hex chars (SHA-256 of content + timestamp) - **conversationId** — Which conversation it belongs to - **depth** — Position in the hierarchy (0 = leaf) - **earliestAt / latestAt** — Time range of source material - **descendantCount** — Total number of ancestor summaries (transitive) - **fileIds** — References to large files mentioned in the source - **tokenCount** — Estimated tokens ### Context items The **context_items** table maintains the ordered list of what the model sees for each conversation. Each entry is either a message reference or a summary reference, identified by ordinal. When compaction creates a summary from a range of messages (or summaries), the source items are replaced by a single summary item. This keeps the context list compact while preserving ordering. ## Compaction lifecycle ### Ingestion When OpenClaw processes a turn, it calls the context engine's lifecycle hooks: 1. **bootstrap** — On session start, reconciles the JSONL session file with the LCM database. Imports any messages that exist in the file but not in LCM (crash recovery). 2. **ingest** / **ingestBatch** — Persists new messages to the database and appends them to context_items. 3. **afterTurn** — After the model responds, ingests new messages, then evaluates whether `contextThreshold` requires compaction. ### Leaf compaction The **leaf pass** converts raw messages into leaf summaries: 1. Identify the oldest contiguous chunk of raw messages outside the **fresh tail** (protected recent messages). 2. Cap the chunk at `leafChunkTokens` (default 20k tokens). 3. Concatenate message content with timestamps. 4. Resolve the most recent prior summary for continuity (passed as `previous_context` so the LLM avoids repeating known information). 5. Send to OpenClaw's host-owned `runtime.llm.complete` capability with the leaf prompt. 6. Normalize runtime LLM response text into plain text while preserving provider/model diagnostics from the host result. 7. If normalization is empty, log provider/model diagnostics and fall back to deterministic truncation. 8. If the summary is larger than the input (LLM failure), retry with the aggressive prompt. If still too large, fall back to deterministic truncation. 9. Persist the summary, link to source messages, and replace the message range in context_items. ### Condensation The **condensed pass** merges summaries at the same depth into a higher-level summary: 1. Find the shallowest depth with enough contiguous same-depth summaries (≥ `leafMinFanout` for d0, ≥ `condensedMinFanout` for d1+). 2. Concatenate their content with time range headers. 3. Send to the LLM with the depth-appropriate prompt (d1, d2, or d3+). 4. Apply the same escalation strategy (normal → aggressive → truncation fallback). 5. Persist with depth = targetDepth + 1, link to parent summaries, replace the range in context_items. ### Compaction modes **Automatic threshold sweep (after each turn):** - Checks if the assembled context crosses `contextThreshold` - Below threshold, does not compact and does not record leaf debt - In deferred mode, records one `"threshold"` maintenance row for background, `maintain()`, or pre-assembly execution - In inline mode, runs a full sweep before `afterTurn()` completes **Full sweep (threshold, manual `/compact`, or overflow):** - Phase 1: Repeatedly runs leaf passes until no more eligible chunks - Phase 2: If the summarized prefix is above `summaryPrefixTargetTokens`, repeatedly runs condensation passes starting from the shallowest eligible depth, respecting the preferred `sweepMaxDepth` (`0` for leaf-only, `-1` for unlimited) - Pressure phase: If summarized-prefix pressure remains, condensation may go beyond `sweepMaxDepth` using the hard fanout floor - Each pass checks for progress; stops if no tokens were saved **Budget-targeted (`compactUntilUnder`):** - Runs up to `maxRounds` (default 10) of full sweeps - Stops when context is under the target token count - Used by the overflow recovery path ### Three-level escalation Every summarization attempt follows this escalation: 1. **Normal** — Standard prompt, temperature 0.2 2. **Aggressive** — Tighter prompt requesting only durable facts, temperature 0.1, lower target tokens 3. **Fallback** — Deterministic truncation to ~512 tokens with `[Truncated for context management]` marker This ensures compaction always makes progress, even if the LLM produces poor output. ## Context assembly The assembler runs before each model turn and builds the message array: ``` [summary₁, summary₂, ..., summaryₙ, message₁, message₂, ..., messageₘ] ├── budget-constrained ──┤ ├──── fresh tail (always included) ────┤ ``` ### Steps 1. Fetch all context_items ordered by ordinal. 2. Resolve each item — summaries become user messages with XML wrappers; messages are reconstructed from parts. 3. Split into evictable prefix and protected fresh tail (last `freshTailCount` raw messages). 4. Compute fresh tail token cost (always included, even if over budget). 5. Fill remaining budget from the evictable set. By default this keeps newest older items and drops the oldest; when `promptAwareEviction` is enabled and a searchable prompt is present, the evictable prefix is ranked by prompt relevance first and then restored to chronological order. 6. Normalize assistant content to array blocks (Anthropic API compatibility). 7. Sanitize tool-use/result pairing (ensures every tool_result has a matching tool_use). ### XML summary format Summaries are presented to the model as user messages wrapped in XML: ```xml ...summary text with timestamps... Expand for details about: exact error messages, full config diff, intermediate debugging steps ``` Condensed summaries also include parent references: ```xml ... ``` The XML attributes give the model enough metadata to reason about summary age, scope, and how to drill deeper. The `` section enables targeted expansion of specific source summaries. ## Expansion system When summaries are too compressed for a task, agents use `lcm_expand_query` to recover detail. ### How it works 1. Agent calls `lcm_expand_query` with a `prompt` and either `summaryIds` or a `query`. 2. If `query` is provided, `lcm_grep` finds matching summaries first. 3. A **delegation grant** is created, scoping the sub-agent to the relevant conversation(s) with a token cap. 4. A sub-agent session is spawned with the expansion task. 5. The sub-agent walks the DAG: it can read summary content, follow parent links, access source messages, and inspect stored files. 6. The sub-agent returns a focused answer (default ≤ 2000 tokens) with cited summary IDs. 7. The grant is revoked and the sub-agent session is cleaned up. ### Security model Expansion uses a delegation grant system: - **Grants** are created at spawn time, scoped to specific conversation IDs - **Token caps** limit how much content the sub-agent can access - **TTL** ensures grants expire even if cleanup fails - **Revocation** happens on completion, cancellation, or sweep The sub-agent only gets `lcm_expand` (the low-level tool), not `lcm_expand_query` — preventing recursive sub-agent spawning. ## Large file handling Files embedded in user messages (typically via `` blocks from tool output) are checked at ingestion: 1. Parse file blocks from message content. 2. For each block exceeding `largeFileTokenThreshold` (default 25k tokens): - Generate a unique file ID (`file_` prefix) - Store the content to `largeFilesDir//.` (default `~/.openclaw/lcm-files/...`) - Generate a ~200 token exploration summary (structural analysis, key sections, etc.) - Insert a `large_files` record with metadata - Replace the file block in the message with a compact reference 3. The `lcm_describe` tool can retrieve bounded file content by ID, and `lcm_grep(scope="files")` can search the first 512,000 bytes of externalized text files. This prevents a single large file paste from consuming the entire context window while keeping the content accessible. ## Session reconciliation LCM handles crash recovery through **bootstrap reconciliation**: 1. On session start, read the JSONL session file (OpenClaw's ground truth). 2. Compare against the LCM database. 3. Find the most recent message that exists in both (the "anchor"). 4. Import any messages after the anchor that are in JSONL but not in LCM. 5. If an existing session key moves to a different transcript file and no anchor exists, treat the new file as a bounded transcript epoch and import its recoverable messages. The same flood cap used for tail reconciliation prevents large unrelated transcripts from being appended automatically. 6. Advance the bootstrap checkpoint only after an overlap is found or a bounded epoch import succeeds. No-anchor reads that import nothing leave the old checkpoint in place so a later turn can retry. This handles the case where OpenClaw wrote messages to the session file but crashed before LCM could persist them. For forked child sessions, LCM treats a host-copied parent JSONL branch as a first-time bootstrap source and imports only the newest messages that fit within `bootstrapMaxTokens`. That keeps child LCM state bounded even if the host fork payload still contains a long raw parent branch. The remaining fork-continuity contract belongs to the host: when lossless-claw advertises the `subagent-spawn` requirement for `thread-bootstrap-projection`, OpenClaw should bootstrap the child model thread from the context-engine projection rather than from the raw copied transcript. If the host cannot provide that capability, lossless-claw can preserve bounded durable state, but it cannot stop the host from replaying raw JSONL into the model before assembly. ## Operation serialization All mutating operations (ingest, compact) are serialized per-session using a promise queue. This prevents races between concurrent afterTurn/compact calls for the same conversation without blocking operations on different conversations. ## Runtime LLM boundary LCM needs model inference for summarization, but it does not resolve provider credentials, base URLs, or provider transport settings directly. Summarization calls go through OpenClaw's `runtime.llm.complete` capability, which owns model preparation, credential resolution, OAuth refresh, provider dispatch, and usage attribution. Configured Lossless summary model overrides (`summaryModel`, `largeFileSummaryModel`, and `fallbackProviders`) are sent as runtime LLM model override requests. OpenClaw enforces those requests with `plugins.entries.lossless-claw.llm.allowModelOverride` and `plugins.entries.lossless-claw.llm.allowedModels`; denied overrides fail closed instead of silently falling back to a different model. ## Stable event identity The `messages` table carries a `stable_event_key` column (TEXT, nullable) indexed by a partial unique index `(conversation_id, stable_event_key) WHERE stable_event_key IS NOT NULL`. The key is computed at `ingestSingle` time from a message's `responseId` or a single `toolCallId` and acts as a third identity axis alongside `transcript_entry_id` and `identity_hash`. Aggregate tool-result messages and messages without either stable identifier continue to use the existing identity-hash and redaction-aware deduplication. When a message with a key that is already present for the conversation is ingested, the duplicate is rejected before any side effects (large-file interception, parts, context items) so the originally-persisted row stays canonical. --- ## File: docs/configuration.md # Configuration Lossless-claw reads plugin configuration from `plugins.entries.lossless-claw.config`. Lossless Claw 0.x supports file-backed OpenClaw from `2026.5.28` through `2026.7.1`. OpenClaw `2026.7.2` prereleases and later use SQLite-backed session storage and require Lossless Claw 1.0 from the `next/1.0` release line. If you cannot move to 1.0, remain on OpenClaw `2026.7.1` or select OpenClaw's `legacy` context engine. The `2026.5.28` minimum lets the host enforce context-engine runtime capabilities before an agent run starts. Agent runs need a native host that provides the full context-engine lifecycle: session bootstrap, pre-prompt assembly, after-turn ingestion, maintenance, compaction, and runtime LLM completion. Native Codex and Pi embedded runs provide those capabilities; generic CLI harnesses such as `claude-cli` and `codex-cli` do not. If you must use a generic CLI harness, either set `plugins.slots.contextEngine` to `legacy` or explicitly set `hostFallbackMode` to `capture-only`. Capture-only mode lowers the installation-wide `agent-run` requirement to bootstrap, after-turn ingestion, and maintenance. Generic CLI runs can persist transcripts and use recall tools, but they do not receive Lossless prompt assembly or host-triggered Lossless compaction. Backend-native compaction remains host-owned. Explicit Lossless compaction requires `fallbackProviders` because generic CLI hosts do not provide runtime LLM completion. Fully capable native hosts still advertise and execute the full lifecycle, and Lossless retains compaction ownership for those runs. Subagent forks continue to require `thread-bootstrap-projection`. The optional programmatic `status` / `doctor` / `rotate` control surface requires a host that separately advertises context-engine capabilities/control dispatch. That host contract is not covered by the baseline plugin API version above. As of this documentation update, no stable OpenClaw release includes those gateway endpoints; downstream control planes should probe host capabilities and treat control as unavailable until `openclaw/openclaw#98060` or an equivalent stable host contract lands. The plugin's normal context-engine behavior and slash commands continue to work on the baseline supported OpenClaw versions. Configuration precedence is: 1. Environment variables 2. `plugins.entries.lossless-claw.config` 3. Built-in defaults from [`src/db/config.ts`](../src/db/config.ts) Most installations only need to override a handful of keys. If you want a complete starting point, use the full example below and then delete entries you do not need. ## Complete `plugins.entries.lossless-claw.config` example ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Notes on the example: - Values shown are the runtime defaults when a fixed default exists. - `databasePath` shows the expanded default path shape. Use an absolute path in config rather than `~`. - `largeFilesDir` shows the expanded default path shape. Both `databasePath` and `largeFilesDir` default to paths under `OPENCLAW_STATE_DIR` (which in turn falls back to `~/.openclaw`). - `timezone` has no fixed hardcoded default; at runtime it resolves from `TZ` first, then the system timezone. The example uses `America/Los_Angeles`. - `maxAssemblyTokenBudget` has no default. The example uses `30000` as a realistic cap for a 32k-class model. - `summaryPrefixTargetTokens` has no fixed default. The example uses `20000`, which matches the derived default for large-context models with the default `leafChunkTokens`. - `databasePath` is the preferred key. `dbPath` is an accepted alias. - `largeFileThresholdTokens` is the preferred key. `largeFileTokenThreshold` is an accepted alias. ## Install and enable Install with OpenClaw's plugin installer: ```bash openclaw plugins install @martian-engineering/lossless-claw@latest ``` If you are running from a local OpenClaw checkout: ```bash pnpm openclaw plugins install @martian-engineering/lossless-claw@latest ``` Use exact versions only for rollback or reproducible canary testing. OpenClaw treats an exact install spec such as `@martian-engineering/lossless-claw@0.12.0` as pinned, so plugin update sync will not follow newer LCM releases until you return to the moving track: ```bash openclaw plugins update @martian-engineering/lossless-claw@latest ``` For local plugin development, link a working copy: ```bash openclaw plugins install --link /path/to/lossless-claw ``` ## Reference ### Programmatic context-engine control When the OpenClaw host supports context-engine control dispatch, lossless-claw advertises three sanitized operations: | Operation | Result | Notes | | --- | --- | --- | | `status` | `active`, `messageCount` | Reports current LCM conversation state only. It does not include `lastRotatedAt` because that timestamp is product/runtime state, not durable LCM state. | | `doctor` | `ok`, `warnings[]` | Returns a bounded summary-health warning list without transcript text, paths, or raw provider/debug output. | | `rotate` | `messageCount`, `lastRotatedAt` | Reuses the `/lossless rotate` implementation and returns the timestamp for the successful rotate operation. Hosts that need durable rotation history should persist this result outside lossless-claw. | The control surface never accepts arbitrary slash commands and never exposes local database paths, transcript paths, backup paths, credentials, or raw shell output. ### Core storage and session behavior | Key | Type | Default | Env override | Purpose | | --- | --- | --- | --- | --- | | `enabled` | `boolean` | `true` | `LCM_ENABLED` | Enables or disables lossless-claw without uninstalling it. | | `databasePath` | `string` | `${OPENCLAW_STATE_DIR}/lcm.db` | `LCM_DATABASE_PATH` | Preferred path for the SQLite database. | | `dbPath` | `string` | alias of `databasePath` | `LCM_DATABASE_PATH` | Legacy alias for `databasePath`. Prefer `databasePath` in new config. | | `largeFilesDir` | `string` | `${OPENCLAW_STATE_DIR}/lcm-files` | `LCM_LARGE_FILES_DIR` | Directory where externalized large files and inline images are persisted. Automatically follows the active state directory. | | `ignoreSessionPatterns` | `string[]` | `[]` | `LCM_IGNORE_SESSION_PATTERNS` | Session-key glob patterns that skip LCM entirely. | | `statelessSessionPatterns` | `string[]` | `[]` | `LCM_STATELESS_SESSION_PATTERNS` | Session-key glob patterns that may read from LCM but never write to it. | | `skipStatelessSessions` | `boolean` | `true` | `LCM_SKIP_STATELESS_SESSIONS` | Enforces `statelessSessionPatterns` when enabled. | | `hostFallbackMode` | `"error" \| "capture-only"` | `"error"` | `LCM_HOST_FALLBACK_MODE` | `error` requires the full agent-run lifecycle. `capture-only` accepts bootstrap, after-turn ingestion, and maintenance so generic CLI runs keep transcript capture and recall without Lossless prompt assembly or host-triggered Lossless compaction. Backend-native compaction remains host-owned. Subagent projection requirements remain strict. | | `newSessionRetainDepth` | `integer` | `2` | `LCM_NEW_SESSION_RETAIN_DEPTH` | Controls what survives `/new`. `-1` keeps all context, `0` keeps summaries only, higher values keep only deeper summaries. | | `timezone` | `string` | `TZ` or system timezone | `TZ` | IANA timezone used for timestamp rendering in summaries. | | `pruneHeartbeatOk` | `boolean` | `false` | `LCM_PRUNE_HEARTBEAT_OK` | Retroactively removes `HEARTBEAT_OK` turn cycles from persisted storage. | | `transcriptGcEnabled` | `boolean` | `false` | `LCM_TRANSCRIPT_GC_ENABLED` | Enables transcript rewrite GC during `maintain()`; disabled by default so transcript rewrites stay opt-in. | | `enableSummaryThinking` | `boolean` | `true` | `LCM_ENABLE_SUMMARY_THINKING` | When true, requests low reasoning budget from the model during summarization calls. Set to false to disable reasoning and keep summarization output concise. | | `proactiveThresholdCompactionMode` | `"deferred" \| "inline"` | `"deferred"` | `LCM_PROACTIVE_THRESHOLD_COMPACTION_MODE` | Controls whether proactive threshold compaction is deferred into maintenance debt by default or run inline for legacy behavior. | | `autoRotateSessionFiles.enabled` | `boolean` | `true` | `LCM_AUTO_ROTATE_SESSION_FILES_ENABLED` | Enables automatic rotation for oversized LCM-managed session JSONL files. | | `autoRotateSessionFiles.createBackups` | `boolean` | `false` | `LCM_AUTO_ROTATE_SESSION_FILES_CREATE_BACKUPS` | Creates or replaces the rolling `rotate-latest` SQLite backup before automatic session-file rotation. Manual `/lossless rotate` backups are always created. | | `autoRotateSessionFiles.sizeBytes` | `integer` | `2097152` | `LCM_AUTO_ROTATE_SESSION_FILES_SIZE_BYTES` | Byte threshold that triggers automatic session-file rotation. | | `autoRotateSessionFiles.startup` | `"rotate" \| "warn" \| "off"` | `"rotate"` | `LCM_AUTO_ROTATE_SESSION_FILES_STARTUP` | Startup behavior for oversized indexed OpenClaw session transcripts that also have active LCM bootstrap state. | | `autoRotateSessionFiles.runtime` | `"rotate" \| "warn" \| "off"` | `"rotate"` | `LCM_AUTO_ROTATE_SESSION_FILES_RUNTIME` | Runtime behavior after post-turn checks. Runtime `rotate` logs deferral for active session JSONL rewrites and leaves direct rotation to startup or manual `/lossless rotate`. | | `independentLogFile.enabled` | `boolean` | `true` | `LCM_LOG_FILE_ENABLED` | Writes lossless-claw JSONL logs to an independent plugin-owned file in addition to OpenClaw's runtime logger. | | `independentLogFile.file` | `string` | `/tmp/openclaw/lossless-claw-YYYY-MM-DD.log` | `LCM_LOG_FILE` | Optional log path. A dated `lossless-claw-YYYY-MM-DD.log` path rolls over daily. | | `independentLogFile.maxFileBytes` | `integer` | `104857600` | `LCM_LOG_MAX_FILE_BYTES` | Size threshold for rotating the active lossless-claw log file to `.1.log` through `.5.log`. | > **Multi-profile note:** `OPENCLAW_STATE_DIR` (set by the host OpenClaw gateway) controls where state is stored. When two gateways run on the same host (e.g. separate bot personas), each gateway sets its own `OPENCLAW_STATE_DIR` and lossless-claw automatically uses that directory for the database, large-file payloads, auth-profile lookups, and legacy secrets — no per-profile plugin config is needed. Automatic session-file rotation rewrites only the live session transcript, keeps the active LCM conversation and durable history intact, and refreshes the bootstrap checkpoint. Before manual or startup rewrites, rotation forces leaf-only compaction for raw context outside the preserved tail so trimmed transcript messages are covered by LCM summaries without running unrelated summary-condensation passes. Startup rotation first scans OpenClaw's current indexed session stores for configured agents, then intersects those candidates with active LCM conversations and matching bootstrap file mappings. Runtime rotation checks from `afterTurn()` and `maintain()` intentionally do not directly rewrite active session JSONL because embedded prompt-lock fences can still be open while tool-call loops and host background maintenance overlap; runtime `rotate` logs a deferral until startup, manual `/lossless rotate`, or a future host-owned full-transcript rewrite primitive is available. Automatic rotation does not create a SQLite backup by default; set `autoRotateSessionFiles.createBackups` to `true` to make startup rotation create one pre-rotation LCM database backup for the batch before any transcript is rewritten. Manual `/lossless rotate` always keeps its backup-backed behavior regardless of this flag. Rotation never runs for ignored sessions, stateless sessions, or sessions without active LCM state. The preserved JSONL tail follows `freshTailCount` and expands through the newest user when necessary so rotation cannot split its following assistant/tool suffix. Transcript GC uses the host-provided `rewriteTranscriptEntries` primitive and defers until host-approved background maintenance when `transcriptGcEnabled` is enabled. Lossless-claw writes routine operational JSONL logs by default at `/tmp/openclaw/lossless-claw-YYYY-MM-DD.log`, beside OpenClaw's `/tmp/openclaw/openclaw-YYYY-MM-DD.log`. Routine info and debug lines go to the independent file instead of the shared OpenClaw log. Startup banners and warning/error lines still go through OpenClaw's runtime logger so gateway-level startup and failure diagnostics remain visible. The independent file follows the same practical rotation model as OpenClaw: a dated filename rolls over when the local date changes, stale dated files are pruned after 3 days, and an oversized active file is rotated through `.1.log` to `.5.log`. Every automatic decision emits grep-able log lines prefixed with `[lcm] auto-rotate:`. Startup emits one compact summary line with `phase=startup`, `action=summary`, `scanned`, `eligible`, `rotated`, `warned`, `skipped`, `durationMs`, `bytesRemoved`, and backup fields when a batch backup was created; quiet skips such as missing files, missing bootstrap mappings, and below-threshold files are counted there instead of producing one line per candidate. Rotation detail lines include `phase`, `action`, `sessionId`, `sessionKey`, `sessionFile`, `sizeBytes`, `thresholdBytes`, `durationMs`, `backupPath`, `bytesRemoved`, `preservedTailMessageCount`, and `checkpointSize`; real warning lines include the same available context plus `reason` or `error`. ### Compaction thresholds and summary sizing | Key | Type | Default | Env override | Purpose | | --- | --- | --- | --- | --- | | `contextThreshold` | `number` | `0.75` | `LCM_CONTEXT_THRESHOLD` | Fraction of the active model context window that triggers compaction. | | `contextThresholdOverrides` | `Array<{ name?: string; match: object; contextThreshold: number; freshTailCount?: integer; leafChunkTokens?: integer }>` | `[]` | none | Optional ordered rules that override `contextThreshold` and, optionally, `freshTailCount` and `leafChunkTokens` by model id, model context-window range, or session glob pattern. | | `freshTailCount` | `integer` | `64` | `LCM_FRESH_TAIL_COUNT` | Number of newest messages always kept raw. If this count would split the newest user turn, the protected tail expands to include that user and its following assistant/tool suffix. | | `freshTailMaxTokens` | `integer` | unset | `LCM_FRESH_TAIL_MAX_TOKENS` | Optional token cap for the protected fresh tail. The newest user message and its following assistant/tool suffix are always preserved even if they exceed the cap. | | `promptAwareEviction` | `boolean` | `false` | `LCM_PROMPT_AWARE_EVICTION_ENABLED` | When enabled, budget-constrained assembly keeps older evictable items by prompt relevance instead of pure chronology. This improves retrieval under tight budgets, but it can reduce prompt-cache hit rates because the preserved prefix changes as prompts change. | | `stubLargeToolPayloads` | `boolean` | `false` | `LCM_STUB_LARGE_TOOL_PAYLOADS` | When enabled, evictable tool-result rows backfilled with `messages.large_content` are assembled as `[LCM Tool Output: file_xxx ...]` stubs while the fresh tail stays inline. Requires `scripts/lcm-blob-migrate.mjs`, which defaults to the same large-files root as runtime LCM (`LCM_LARGE_FILES_DIR` or `${OPENCLAW_STATE_DIR}/lcm-files`). | | `leafMinFanout` | `integer` | `8` | `LCM_LEAF_MIN_FANOUT` | Minimum number of raw messages required before a leaf pass runs. | | `condensedMinFanout` | `integer` | `4` | `LCM_CONDENSED_MIN_FANOUT` | Number of same-depth summaries needed before condensation is attempted. | | `condensedMinFanoutHard` | `integer` | `2` | `LCM_CONDENSED_MIN_FANOUT_HARD` | Hard floor for condensation grouping during maintenance and repair flows. | | `sweepMaxDepth` | `integer` | `1` | `LCM_SWEEP_MAX_DEPTH` | Preferred maximum condensation source depth during routine threshold sweeps. Use `0` for leaf-only and `-1` for unlimited depth. Pressure sweeps may go deeper when summarized context remains above target. | | `incrementalMaxDepth` | `integer` | alias of `sweepMaxDepth` | `LCM_INCREMENTAL_MAX_DEPTH` | Deprecated alias for `sweepMaxDepth`. Kept so existing configs continue to load. | | `leafChunkTokens` | `integer` | `20000` | `LCM_LEAF_CHUNK_TOKENS` | Maximum source-token budget for a leaf compaction chunk. Larger chunks reduce sweep frequency at the cost of slower individual summary calls. | | `summaryPrefixTargetTokens` | `integer` | derived | `LCM_SUMMARY_PREFIX_TARGET_TOKENS` | Optional target for summarized-prefix tokens after a full sweep. If unset, Lossless derives `max(condensedTargetTokens, min(leafChunkTokens, floor(contextThreshold * tokenBudget * 0.5)))`. | | `maxSweepIterations` | `integer` | `12` | `LCM_MAX_SWEEP_ITERATIONS` | Hard cap on summarizer passes within a single full sweep. On hitting the cap the sweep stops cleanly and returns the partial result; bounds how long a sweep can run on the turn-critical path. | | `sweepDeadlineMs` | `integer` | `120000` | `LCM_SWEEP_DEADLINE_MS` | Wall-clock budget for a single full sweep, in milliseconds. When exceeded the sweep stops before starting another pass, so a slow or rate-limited summarizer cannot hang the agent turn. | | `compactUntilUnderDeadlineMs` | `integer` | `300000` | `LCM_COMPACT_UNTIL_UNDER_DEADLINE_MS` | Wall-clock budget for a whole `compactUntilUnder` operation, in milliseconds. `compactUntilUnder` runs up to `maxRounds` sweeps; without this the worst case is `maxRounds × sweepDeadlineMs` (~20 min at the defaults). The deadline is shared into each round's sweep and checked before the next round. | | `bootstrapMaxTokens` | `integer` | `max(6000, floor(leafChunkTokens * 0.3))` | `LCM_BOOTSTRAP_MAX_TOKENS` | Maximum parent-history tokens imported when a new LCM conversation bootstraps. | | `leafTargetTokens` | `integer` | `2400` | `LCM_LEAF_TARGET_TOKENS` | Prompt target for leaf summary size. | | `condensedTargetTokens` | `integer` | `2000` | `LCM_CONDENSED_TARGET_TOKENS` | Prompt target for condensed summary size. | | `summaryMaxOverageFactor` | `number` | `3` | `LCM_SUMMARY_MAX_OVERAGE_FACTOR` | Hard ceiling multiplier before oversized summaries are deterministically truncated. | | `fallbackMaxTokens` | `integer` | `512` | `LCM_FALLBACK_MAX_TOKENS` | Maximum token budget for deterministic fallback summaries when the LLM summarizer is unavailable. Values below 64 are ignored. | | `largeFileThresholdTokens` | `integer` | `25000` | `LCM_LARGE_FILE_TOKEN_THRESHOLD` | Preferred key for the token threshold that routes text attachments into large-file summarization. | | `largeFileTokenThreshold` | `integer` | alias of `largeFileThresholdTokens` | `LCM_LARGE_FILE_TOKEN_THRESHOLD` | Legacy alias accepted by the runtime. Prefer `largeFileThresholdTokens` in new config. | | `maxAssemblyTokenBudget` | `integer` | unset | `LCM_MAX_ASSEMBLY_TOKEN_BUDGET` | Optional hard cap for assembly and threshold evaluation, useful with smaller-context models. | | `maxExpandTokens` | `integer` | `4000` | `LCM_MAX_EXPAND_TOKENS` | Default token cap for `lcm_expand_query` responses. | Forked child transcripts are also bounded by `bootstrapMaxTokens` when a host copies a raw parent JSONL branch into the child file. This protects the LCM database from importing unbounded parent history, but the host must still honor the `thread-bootstrap-projection` context-engine capability for subagent or thread forks so the model starts from the LCM-assembled compact view instead of the raw copied transcript. ### Model selection, execution, and prompts | Key | Type | Default | Env override | Purpose | | --- | --- | --- | --- | --- | | `summaryModel` | `string` | `""` | `LCM_SUMMARY_MODEL` | Summarizer model override. Bare model names reuse the chosen provider; `provider/model` strings force a specific provider. | | `summaryProvider` | `string` | `""` | `LCM_SUMMARY_PROVIDER` | Provider hint used only when `summaryModel` is a bare model name. | | `largeFileSummaryModel` | `string` | `""` | `LCM_LARGE_FILE_SUMMARY_MODEL` | Large-file summarizer model override. | | `largeFileSummaryProvider` | `string` | `""` | `LCM_LARGE_FILE_SUMMARY_PROVIDER` | Large-file summarizer provider hint for bare model names. | | `expansionModel` | `string` | `""` | `LCM_EXPANSION_MODEL` | `lcm_expand_query` sub-agent model override. | | `expansionProvider` | `string` | `""` | `LCM_EXPANSION_PROVIDER` | `lcm_expand_query` sub-agent provider hint for bare model names. | | `delegationTimeoutMs` | `integer` | `120000` | `LCM_DELEGATION_TIMEOUT_MS` | Maximum wall-clock budget for delegated expansion work across one `lcm_expand_query` call. Cross-conversation buckets share this deadline. The dynamic tool advertises a `timeoutMs` default with 30 seconds of extra RPC headroom for cancellation, cleanup, and result delivery. | | `summaryTimeoutMs` | `integer` | `60000` | `LCM_SUMMARY_TIMEOUT_MS` | Maximum time to wait for one model-backed summarizer call. | | `summaryCallWindowMs` | `integer` | `600000` | `LCM_SUMMARY_CALL_WINDOW_MS` | Rolling window for the per-session summarization spend guard. | | `summaryMaxCallsPerWindow` | `integer` | `24` | `LCM_SUMMARY_MAX_CALLS_PER_WINDOW` | Maximum model-backed summarization calls per session/window before Lossless opens a non-auth spend backoff. | | `summarySpendBackoffMs` | `integer` | `1800000` | `LCM_SUMMARY_SPEND_BACKOFF_MS` | Cooldown after the summarization spend guard opens. | | `customInstructions` | `string` | `""` | `LCM_CUSTOM_INSTRUCTIONS` | Extra natural-language instructions injected into every summarization prompt. | Summary calls are executed through OpenClaw's `api.runtime.llm.complete` capability. If you configure an explicit Lossless summary model (`summaryModel`, `largeFileSummaryModel`, or `fallbackProviders`), OpenClaw must allow that runtime LLM override under `plugins.entries.lossless-claw.llm.allowModelOverride` and `plugins.entries.lossless-claw.llm.allowedModels`. `openclaw doctor --fix` can add the minimal policy entries for configured Lossless summary models. Delegated expansion calls use OpenClaw's runtime sub-agent layer; explicit `expansionModel` values require `plugins.entries.lossless-claw.subagent.allowModelOverride` and a matching `subagent.allowedModels` entry, or `"*"` if you intentionally trust any expansion target. `openclaw doctor --fix` can add the minimal subagent policy, and `lcm_expand_query` retries once without the override if the host rejects it. ### Fallbacks, circuit breaking, and safety rails | Key | Type | Default | Env override | Purpose | | --- | --- | --- | --- | --- | | `fallbackProviders` | `Array<{ provider: string; model: string }>` | `[]` | `LCM_FALLBACK_PROVIDERS` | Explicit provider/model fallback chain for compaction summarization. Format for env vars is `provider/model,provider/model`. | | `circuitBreakerThreshold` | `integer` | `5` | `LCM_CIRCUIT_BREAKER_THRESHOLD` | Consecutive auth failures before the summarization circuit breaker trips. | | `circuitBreakerCooldownMs` | `integer` | `1800000` | `LCM_CIRCUIT_BREAKER_COOLDOWN_MS` | Cooldown before the summarization circuit breaker resets automatically. | | `stripInjectedContextTags` | `string[]` | `["active_memory_plugin", "relevant-memories", "relevant_memories", "hindsight_memories"]` | `LCM_STRIP_INJECTED_CONTEXT_TAGS` | XML tag names whose blocks are stripped from message content before compaction summarization. Memory/context plugins inject these via `prependContext`; stripping prevents ephemeral retrieval context from polluting compacted summaries. Env var format is comma-separated tag names. Set to `[]` (or empty env string) to disable. | | `replayFloodThresholdExternal` | `integer` | `3` | `LCM_REPLAY_FLOOD_THRESHOLD_EXTERNAL` | Max replay-like messages allowed in a single SQLite-second for `role=user` before `assertNoReplayTimestampFlood` refuses the batch. Defaults to `3` to preserve replay defense for third-partyly-rebroadcastable input. | | `replayFloodThresholdInternal` | `integer` | `32` | `LCM_REPLAY_FLOOD_THRESHOLD_INTERNAL` | Max identical messages allowed in a single SQLite-second for `role=tool/assistant/system` before the anti-replay guard refuses the batch. Defaults to `32` to absorb legitimate idempotent sub-agent bursts (same-second tool returns like `{"status":"ok"}`). | ### Nested objects #### `cacheAwareCompaction` | Key | Type | Default | Env override | Purpose | | --- | --- | --- | --- | --- | | `cacheAwareCompaction.enabled` | `boolean` | `true` | `LCM_CACHE_AWARE_COMPACTION_ENABLED` | Deprecated. Accepted for config compatibility but no longer used for automatic compaction decisions. | | `cacheAwareCompaction.cacheTTLSeconds` | `integer` | `300` | `LCM_CACHE_TTL_SECONDS` | Deprecated. Accepted for config compatibility; threshold debt no longer waits for cache TTL. | | `cacheAwareCompaction.maxColdCacheCatchupPasses` | `integer` | `2` | `LCM_MAX_COLD_CACHE_CATCHUP_PASSES` | Deprecated. Automatic cold-cache catch-up passes were removed. | | `cacheAwareCompaction.hotCachePressureFactor` | `number` | `4` | `LCM_HOT_CACHE_PRESSURE_FACTOR` | Deprecated. Hot-cache raw-history pressure no longer drives automatic compaction. | | `cacheAwareCompaction.hotCacheBudgetHeadroomRatio` | `number` | `0.2` | `LCM_HOT_CACHE_BUDGET_HEADROOM_RATIO` | Deprecated. Hot-cache budget headroom no longer defers automatic threshold compaction. | | `cacheAwareCompaction.coldCacheObservationThreshold` | `integer` | `3` | `LCM_COLD_CACHE_OBSERVATION_THRESHOLD` | Deprecated. Cold-cache streaks remain observable telemetry only. | | `cacheAwareCompaction.criticalBudgetPressureRatio` | `number` | `0.90` | `LCM_CRITICAL_BUDGET_PRESSURE_RATIO` | Deprecated. `contextThreshold` is the only automatic compaction threshold. | #### `dynamicLeafChunkTokens` | Key | Type | Default | Env override | Purpose | | --- | --- | --- | --- | --- | | `dynamicLeafChunkTokens.enabled` | `boolean` | `true` | `LCM_DYNAMIC_LEAF_CHUNK_TOKENS_ENABLED` | Deprecated. Accepted for config compatibility but no longer used by automatic compaction. | | `dynamicLeafChunkTokens.max` | `integer` | `max(leafChunkTokens, floor(leafChunkTokens * 2))` | `LCM_DYNAMIC_LEAF_CHUNK_TOKENS_MAX` | Deprecated. With the default `leafChunkTokens=20000`, this resolves to `40000`, but automatic compaction uses `leafChunkTokens`. | ### Threshold full-sweep compaction Automatic compaction is threshold-only: - `afterTurn()` evaluates the resolved context threshold against the active token budget - below threshold, no automatic compaction runs and no leaf debt is recorded - at or above threshold, inline mode runs a threshold full sweep immediately - deferred mode records one coalesced `"threshold"` maintenance row and normally drains it in the background or host-approved `maintain()` - pre-assembly drain is reserved as an emergency safeguard when the live prompt is already over the active token budget Lossless still records prompt-cache telemetry for status and diagnostics, but cache hotness no longer delays threshold debt. Legacy `cacheAwareCompaction.*` and `dynamicLeafChunkTokens.*` settings remain accepted so existing OpenClaw config continues to load, but they do not change automatic compaction behavior. `contextThresholdOverrides` are optional and never replace the global fallback. Each rule's `match` object can include `model`, `modelContextWindowMin`, `modelContextWindowMax`, and `sessionPattern`; all fields in a rule must match. If several rules match, Lossless picks the highest-specificity rule, then the earliest rule in the array for ties. Exact `model` matches have higher specificity than `sessionPattern` matches, and session-pattern matches have higher specificity than context-window range matches. A matching rule may also set `freshTailCount`, which overrides the global fresh-tail count for assembly and threshold compaction, and `leafChunkTokens`, which overrides the global leaf chunk size for matching threshold sweeps. Threshold selection logs include the chosen threshold, source, rule index/name, token budget, threshold tokens, fresh-tail count, leaf chunk size, model, context-window value, and match reason. Context-window matchers only apply when the OpenClaw host reports explicit model context-window metadata to Lossless. Lossless does not infer `modelContextWindowMin` or `modelContextWindowMax` matches from the active token budget. If an override must affect assemble-time `freshTailCount` on all currently supported OpenClaw hosts, prefer an exact `model` or `sessionPattern` matcher. Full sweeps first run leaf passes until there are no more eligible raw-message chunks outside the fresh tail. Condensation is then driven by summarized-prefix pressure: the routine condensation phase obeys `sweepMaxDepth`, and if the summarized prefix still exceeds `summaryPrefixTargetTokens`, a pressure phase may use `condensedMinFanoutHard` and condense deeper. Total context pressure starts the sweep, but does not by itself force deeper condensation once the raw prefix has been summarized. A single sweep is bounded by both `maxSweepIterations` (a hard cap on summarizer passes) and `sweepDeadlineMs` (a wall-clock budget). When either limit is reached the sweep stops before starting another pass and returns the consistent partial result built so far, logging a `compactFullSweep stopped at …` warning. This keeps a slow or rate-limited summarizer from hanging the agent turn — remaining context pressure is picked up by the next sweep. Overflow recovery (`compactUntilUnder`) runs up to `maxRounds` sweeps to drive context under a target. Because every sweep re-arms its own `sweepDeadlineMs`, the whole operation is separately bounded by `compactUntilUnderDeadlineMs` (default 300000): the operation deadline is shared into each round's sweep — a sweep stops at whichever deadline is sooner — and is also checked before starting the next round. On hitting it, `compactUntilUnder` returns the consistent partial result and logs a `compactUntilUnder stopped at …` warning, so the worst case is the operation budget rather than `maxRounds × sweepDeadlineMs`. ### Prompt-aware eviction When `promptAwareEviction` is enabled: - the protected fresh tail is still preserved exactly as usual - only the older evictable prefix is affected - if the evictable prefix does not fit and the current prompt has searchable terms, lossless-claw keeps the most relevant older items instead of just the newest older items Tradeoff: - this can improve retrieval quality when the prompt is asking about an older topic and the assembled context is tight - it also makes the assembled prefix less stable for providers with prefix-based prompt caching, because different prompts can keep different older items If Anthropic prompt-cache stability matters more than topical recall under pressure, set `promptAwareEviction: false`. ## Behavior notes ### Summary model resolution Compaction summarization resolves candidates in this order: 1. `LCM_SUMMARY_MODEL` and `LCM_SUMMARY_PROVIDER` 2. `plugins.entries.lossless-claw.config.summaryModel` and `summaryProvider` 3. OpenClaw's default compaction model 4. Runtime/session provider and model hints from OpenClaw 5. `fallbackProviders` If `summaryModel` already contains a provider prefix such as `anthropic/claude-sonnet-4-20250514`, `summaryProvider` is ignored for that candidate. Lossless does not resolve provider credentials directly for compaction summaries. OpenClaw's runtime LLM layer owns provider/model preparation, auth profiles, OAuth refresh, base URLs, and dispatch. Lossless only selects the requested summary target and passes it to the host runtime, where model override policy is enforced. A practical starting point for cost-sensitive setups is: ```env LCM_SUMMARY_MODEL=openai/gpt-5.4-mini LCM_EXPANSION_MODEL=openai/gpt-5.4-mini ``` ### Session pattern matching `ignoreSessionPatterns` and `statelessSessionPatterns` use full session keys. - `*` matches any characters except `:` - `**` matches anything, including `:` Cron scheduler keys (`agent::cron:...`) are isolated automatically when a new runtime `sessionId` reuses the same `sessionKey`. Configure `ignoreSessionPatterns` for cron only when the run should bypass LCM entirely; leave cron sessions included when they need in-run compaction. When OpenClaw exposes its runtime compaction delegate, `/compact` and overflow recovery for ignored sessions fall back to OpenClaw's built-in compaction path instead of LCM's summary DAG. Older hosts that do not expose that delegate keep the previous safe skip behavior. These examples are storage exclusions, not compaction preferences. Matching sessions do not create LCM conversation rows or store messages in LCM. The `agent:*:**:active-memory:**` pattern is intentionally broad because `**` spans colon-separated session-key segments, including nested prefixes before `active-memory`. The `agent:*:dreaming-narrative-**` example matches OpenClaw memory-core keys built with the `dreaming-narrative-` prefix ([source](https://github.com/openclaw/openclaw/blob/b81666ca6af25c86cc099983a4358cdc5ea9ced8/extensions/memory-core/src/dreaming-narrative.ts)). Example: ```json { "ignoreSessionPatterns": [ "agent:*:cron:**", "agent:*:**:active-memory:**", "agent:*:dreaming-narrative-**" ], "statelessSessionPatterns": [ "agent:*:subagent:**", "agent:ops:subagent:**" ], "skipStatelessSessions": true } ``` ### `/new` and `/reset` Lossless-claw treats OpenClaw reset commands differently: - `/new` keeps the active LCM conversation and prunes active context according to `newSessionRetainDepth` - `/reset` archives the active conversation row and creates a fresh active row for the same stable `sessionKey` This keeps long-term history available while still giving users a real clean-slate reset. ### Deferred proactive compaction Lossless-claw now defaults `proactiveThresholdCompactionMode` to `deferred`. - deferred mode records a single coalesced maintenance debt row per conversation - new deferred compaction debt is only created for `contextThreshold` pressure and uses reason `"threshold"` - `maintain()` consumes threshold debt when the host explicitly opts in to deferred execution - `assemble()` leaves pending threshold debt for after-turn background drain or host-approved `maintain()` while the live prompt is still within budget - `assemble()` only consumes pending threshold debt synchronously as an emergency safeguard when the live prompt estimate is already over the active token budget - old non-threshold debt from earlier builds is revalidated; if the conversation is no longer over threshold, it is cleared as a no-op - `/lossless status` (`/lcm status` alias) shows the current maintenance state, including pending/running/last-failure details - status output also surfaces the latest API/cache telemetry as diagnostics, not as a deferral gate - set `proactiveThresholdCompactionMode` to `inline` only if you need the legacy inline proactive compaction behavior for compatibility ### `/lossless rotate` `/lossless rotate` exists for a different use case than `/new` or `/reset`: - `/new` keeps the same active LCM conversation row and only prunes context. - `/reset` changes OpenClaw session flow, which is sometimes more disruptive than users want. - `/lossless rotate` keeps the live OpenClaw session identity and the same active LCM conversation row, but rewrites the backing transcript into a compact preserved-tail form. Before rotating, Lossless-claw replaces one rolling `rotate-latest` SQLite backup. It then rewrites the current session transcript and checkpoints the same conversation at the new transcript frontier so bootstrap does not replay the dropped transcript history. Existing summaries, context items, and conversation identity stay in place. If you want additional timestamped snapshots, run `/lossless backup` explicitly before `/lossless rotate`. ## Environment-only knobs outside plugin config These settings are not part of `plugins.entries.lossless-claw.config`, but they still affect the system: | Env var | Default | Purpose | | --- | --- | --- | | `OPENCLAW_STATE_DIR` | `~/.openclaw` | Active state directory for the OpenClaw gateway. When set, all path defaults (database, large files, auth profiles, secrets) resolve relative to this directory instead of `~/.openclaw`. Set automatically by OpenClaw for non-default profiles. | | `LCM_OPENCLAW_DIR` | unset | Lossless shell CLI override for the OpenClaw state directory. Takes precedence over `OPENCLAW_STATE_DIR` for `lcm` commands only. | | `LCM_TUI_CONVERSATION_WINDOW_SIZE` | `200` | Number of messages `lcm-tui` loads per keyset-paged conversation window. | ## Database operations The SQLite database lives at `databasePath` or `LCM_DATABASE_PATH`. The default path is `${OPENCLAW_STATE_DIR}/lcm.db` (resolves to `~/.openclaw/lcm.db` when `OPENCLAW_STATE_DIR` is not set). Inspect it with: ```bash sqlite3 "${OPENCLAW_STATE_DIR:-$HOME/.openclaw}/lcm.db" SELECT COUNT(*) FROM conversations; SELECT * FROM context_items WHERE conversation_id = 1 ORDER BY ordinal; SELECT depth, COUNT(*) FROM summaries GROUP BY depth; SELECT summary_id, depth, token_count FROM summaries ORDER BY token_count DESC LIMIT 10; ``` Back it up with: ```bash cp "${OPENCLAW_STATE_DIR:-$HOME/.openclaw}/lcm.db" "${OPENCLAW_STATE_DIR:-$HOME/.openclaw}/lcm.db.backup" sqlite3 "${OPENCLAW_STATE_DIR:-$HOME/.openclaw}/lcm.db" ".backup ${OPENCLAW_STATE_DIR:-$HOME/.openclaw}/lcm.db.backup" ``` Or from a supported OpenClaw chat/native command surface: ```text /lossless backup ``` ## Disabling lossless-claw To disable the plugin but keep it installed: ```json { "plugins": { "entries": { "lossless-claw": { "enabled": false } } } } ``` To switch back to OpenClaw's legacy context engine instead: ```json { "plugins": { "slots": { "contextEngine": "legacy" } } } ``` ## Stable event identity deduplication Lossless-claw persists a `stable_event_key` on messages that carry a `responseId` or tool call id. This prevents duplicate ingestion when the same semantic event arrives in two different content representations (typical case: the JSONL transcript is redacted by `logging.redactPatterns` while the live `afterTurn` batch is not). The key is derived from the message as follows, in order: 1. assistant messages with `responseId` (or `response_id`): `assistant-response:`. 2. tool / toolResult messages that represent exactly one tool call id: `tool-result:`. 3. Otherwise, including aggregate tool-result messages: no key is persisted, and the row falls back to the existing content-based deduplication. A partial unique index on `(conversation_id, stable_event_key)` ensures that two rows in the same conversation can never share a key. The mechanism is independent of `logging.redactPatterns` and requires no user configuration. --- ## File: docs/fts5.md # Optional: enable FTS5 for fast full-text search `lossless-claw` works without FTS5 as of the current release. When FTS5 is unavailable in the Node runtime that runs the OpenClaw gateway, the plugin: - keeps persisting messages and summaries - falls back from `"full_text"` search to a slower `LIKE`-based search - loses FTS ranking/snippet quality If you want native FTS5 search performance and ranking, the **exact Node runtime that runs the gateway** must have SQLite FTS5 compiled in. ## Probe the gateway runtime Run this with the same `node` binary your gateway uses: ```bash node --input-type=module - <<'NODE' import { DatabaseSync } from 'node:sqlite'; const db = new DatabaseSync(':memory:'); const options = db.prepare('pragma compile_options').all().map((row) => row.compile_options); console.log(options.filter((value) => value.includes('FTS')).join('\n') || 'no fts compile options'); try { db.exec("CREATE VIRTUAL TABLE t USING fts5(content)"); console.log("fts5: ok"); } catch (err) { console.log("fts5: fail"); console.log(err instanceof Error ? err.message : String(err)); } NODE ``` Expected output: ```text ENABLE_FTS5 fts5: ok ``` If you get `fts5: fail`, build or install an FTS5-capable Node and point the gateway at that runtime. ## Build an FTS5-capable Node on macOS This workflow was verified with Node `v22.15.0`. ```bash cd ~/Projects git clone --depth 1 --branch v22.15.0 https://github.com/nodejs/node.git node-fts5 cd node-fts5 ``` Edit `deps/sqlite/sqlite.gyp` and add `SQLITE_ENABLE_FTS5` to the `defines` list for the `sqlite` target: ```diff 'defines': [ 'SQLITE_DEFAULT_MEMSTATUS=0', + 'SQLITE_ENABLE_FTS5', 'SQLITE_ENABLE_MATH_FUNCTIONS', 'SQLITE_ENABLE_SESSION', 'SQLITE_ENABLE_PREUPDATE_HOOK' ], ``` Important: - patch `deps/sqlite/sqlite.gyp`, not only `node.gyp` - `node:sqlite` uses the embedded SQLite built from `deps/sqlite/sqlite.gyp` Build the runtime: ```bash ./configure --prefix="$PWD/out-install" make -j8 node ``` Expose the binary under a Node-compatible basename that OpenClaw recognizes: ```bash mkdir -p ~/Projects/node-fts5/bin ln -sfn ~/Projects/node-fts5/out/Release/node ~/Projects/node-fts5/bin/node-22.15.0 ``` Use a basename like `node-22.15.0`, `node`, or `nodejs`. Names like `node-v22.15.0-fts5` may not be recognized correctly by OpenClaw's CLI/runtime parsing. Verify the new runtime: ```bash ~/Projects/node-fts5/bin/node-22.15.0 --version ~/Projects/node-fts5/bin/node-22.15.0 --input-type=module - <<'NODE' import { DatabaseSync } from 'node:sqlite'; const db = new DatabaseSync(':memory:'); db.exec("CREATE VIRTUAL TABLE t USING fts5(content)"); console.log("fts5: ok"); NODE ``` ## Point the OpenClaw gateway at that runtime on macOS Back up the existing LaunchAgent plist first: ```bash cp ~/Library/LaunchAgents/ai.openclaw.gateway.plist \ ~/Library/LaunchAgents/ai.openclaw.gateway.plist.bak-$(date +%Y%m%d-%H%M%S) ``` Replace the runtime path, then reload the agent: ```bash /usr/libexec/PlistBuddy -c 'Set :ProgramArguments:0 /Users/youruser/Projects/node-fts5/bin/node-22.15.0' \ ~/Library/LaunchAgents/ai.openclaw.gateway.plist launchctl bootout gui/$UID ~/Library/LaunchAgents/ai.openclaw.gateway.plist 2>/dev/null || true launchctl bootstrap gui/$UID ~/Library/LaunchAgents/ai.openclaw.gateway.plist launchctl kickstart -k gui/$UID/ai.openclaw.gateway ``` Verify the live runtime: ```bash launchctl print gui/$UID/ai.openclaw.gateway | sed -n '1,80p' ``` You should see: ```text program = /Users/youruser/Projects/node-fts5/bin/node-22.15.0 ``` ## Verify `lossless-claw` Check the logs: ```bash tail -n 60 ~/.openclaw/logs/gateway.log tail -n 60 ~/.openclaw/logs/gateway.err.log ``` You want: - `[gateway] [lcm] Plugin loaded ...` - no new `no such module: fts5` Then force one turn through the gateway and verify the DB fills: ```bash /Users/youruser/Projects/node-fts5/bin/node-22.15.0 \ /path/to/openclaw/dist/index.js \ agent --session-id fts5-smoke --message 'Reply with exactly: ok' --timeout 60 sqlite3 ~/.openclaw/lcm.db ' select count(*) as conversations from conversations; select count(*) as messages from messages; select count(*) as summaries from summaries; ' ``` Those counts should increase after a real turn. --- ## File: docs/tui.md # TUI Reference The Lossless Claw TUI (`lcm-tui`) is an interactive terminal application for inspecting, debugging, and maintaining the LCM database. It provides direct visibility into what the model sees (context assembly), how summaries are structured (DAG hierarchy), and tools for surgical repairs when things go wrong. ## Installation **From GitHub releases:** Download the latest binary for your platform from [Releases](https://github.com/Martian-Engineering/lossless-claw/releases). **Build from source:** ```bash cd tui go build -o lcm-tui . # or: make build # or: go install github.com/Martian-Engineering/lossless-claw/tui@latest ``` Requires Go 1.24+. ## Quick Start ```bash lcm-tui # default: ~/.openclaw/lcm.db lcm-tui --db /path/to/lcm.db # custom database path ``` The TUI auto-discovers agent session directories from `~/.openclaw/agents/`. ## Navigation Model The TUI is organized as a drill-down hierarchy. You navigate deeper with Enter and back with `b`/Backspace. ``` Agents → Sessions → Conversation → [Summary DAG | Context View | Large Files] ``` ### Screen 1: Agent List Lists all agents discovered under `~/.openclaw/agents/`. Select an agent to see its sessions. | Key | Action | |-----|--------| | `↑`/`↓` or `k`/`j` | Move cursor | | `Enter` | Open agent's sessions | | `r` | Reload agent list | | `q` | Quit | ### Screen 2: Session List Shows JSONL session files for the selected agent, sorted by last modified time. Each entry shows the filename, last update time, message count, conversation ID (if LCM-tracked), summary count, and large file count. If an OpenClaw session has a Codex app-server binding, the row also shows a `codex:` marker with the local backend rollout row count when available. Sessions load in batches of 50. Scrolling near the bottom automatically loads more. | Key | Action | |-----|--------| | `↑`/`↓` or `k`/`j` | Move cursor | | `Enter` | Open conversation | | `x` | Open bound Codex backend rollout transcript, when available | | `v` | Compare bound Codex backend rollout against the LCM active context | | `b`/`Backspace` | Back to agents | | `r` | Reload sessions | | `q` | Quit | ### Screen 3: Conversation View A scrollable, color-coded view of the raw session messages. Each message shows its timestamp, role (user/assistant/system/tool), and content. Roles are color-coded: - **Green** — user messages - **Blue** — assistant messages - **Yellow** — system messages - **Gray** — tool calls and results This is the raw session data, not the LCM-managed context. Use it to understand what actually happened in the conversation. For sessions with an LCM `conv_id`, the conversation view uses keyset-paged windows by `message_id` (newest window first) instead of hydrating full history. | Key | Action | |-----|--------| | `↑`/`↓` or `k`/`j` | Scroll one line | | `PgUp`/`PgDn` | Scroll half page | | `g` | Jump to top | | `G` | Jump to bottom | | `[` | Load older message window | | `]` | Load newer message window | | `l` | Open **Summary DAG** view | | `c` | Open **Context** view | | `o` | Open **Focus Briefs** view | | `f` | Open **Large Files** view | | `v` | Open **Codex ↔ LCM** comparison view | | `b`/`Backspace` | Back to sessions | | `r` | Reload messages | | `q` | Quit | ### Codex ↔ LCM Comparison For Codex app-server bound sessions, the comparison view renders native Codex backend rollout rows beside the Lossless-managed active context items for the same OpenClaw session. The panes are index-aligned for inspection rather than treated as a causal one-to-one mapping: Codex rows show what the backend session recorded, while LCM rows show summaries and fresh-tail messages that Lossless would assemble. ## Summary DAG View The core inspection tool. Shows the full hierarchy of LCM summaries for a conversation as an expandable tree. Each row shows: ``` [marker] summary_id [kind, tokens] content preview ``` - **Marker**: `>` (collapsed, has children), `v` (expanded), `-` (leaf, no children) - **Kind**: `leaf` for depth-0 summaries, `d1`/`d2`/`d3` for condensed summaries at each depth - **Tokens**: token count of the summary content The bottom panel shows the detail view for the selected summary: full content text and source messages (the raw messages that were summarized to create this node). ### When to Use - **Verify summarization quality** — read what the model will actually see - **Check DAG structure** — ensure the depth hierarchy is balanced - **Find corrupted nodes** — look for suspiciously short content, "[LCM fallback summary]" markers, or raw tool output that leaked into summaries - **Understand temporal coverage** — each summary's source messages show exactly which conversation segment it covers ### Navigation | Key | Action | |-----|--------| | `↑`/`↓` or `k`/`j` | Move cursor in list | | `Enter`/`l`/`Space` | Expand/collapse node | | `h` | Collapse current node | | `g` | Jump to first summary | | `G` | Jump to last summary | | `Shift+J` | Scroll detail panel down | | `Shift+K` | Scroll detail panel up | | `w` | **Rewrite** selected summary | | `W` | **Subtree rewrite** (selected + all descendants) | | `d` | **Dissolve** selected condensed summary | | `r` | Reload DAG | | `b`/`Backspace` | Back to conversation | | `q` | Quit | ## Context View Shows exactly what the model sees: the ordered list of context items (summaries + fresh tail messages) that LCM assembles for the next turn. This is the ground truth for "what does the agent know right now?" Each row shows: ``` ordinal kind [id, tokens] content_preview ``` - **Summaries** show as `leaf`, `d1`, `d2`, etc. with their summary ID - **Messages** show their role (user/assistant/system/tool) with message ID The status bar shows totals: how many summaries, how many messages, total items, and total tokens. ### When to Use - **Debug context overflow** — see total token count and identify what's consuming the budget - **Verify assembly order** — summaries should appear before fresh tail messages, ordered chronologically - **Check after dissolve/rewrite** — confirm your changes are reflected in what the model sees - **Compare with raw conversation** — the conversation view shows everything; the context view shows what survives compaction | Key | Action | |-----|--------| | `↑`/`↓` or `k`/`j` | Move cursor | | `g` | Jump to first item | | `G` | Jump to last item | | `Shift+J` | Scroll detail panel down | | `Shift+K` | Scroll detail panel up | | `r` | Reload context | | `b`/`Backspace` | Back to conversation | | `q` | Quit | ## Focus Briefs View Lists focus briefs generated for the selected LCM conversation. Each row shows status, creation time, brief ID, token count, and prompt preview. The detail panel shows generator metadata, source/citation counts, post-focus drift diagnostics, cited and expanded summary IDs, the original focus prompt, and the generated brief content. This view is read-only. When a focus brief is active, the conversation and active-context screens show a compact focus banner with the brief ID, prompt preview, token count, and stale/source-snapshot diagnostics. | Key | Action | |-----|--------| | `↑`/`↓` or `k`/`j` | Move cursor | | `g`/`G` | Jump to first/last | | `Shift+J` | Scroll detail panel down | | `Shift+K` | Scroll detail panel up | | `r` | Reload focus briefs | | `b`/`Backspace` | Back to conversation | | `q` | Quit | ## Large Files View Lists files that exceeded the large file threshold (default 25k tokens) and were intercepted by LCM. Shows file ID, display name, MIME type, byte size, and creation time. The detail panel shows the exploration summary that was generated as a lightweight stand-in. | Key | Action | |-----|--------| | `↑`/`↓` or `k`/`j` | Move cursor | | `g`/`G` | Jump to first/last | | `r` | Reload files | | `b`/`Backspace` | Back to conversation | | `q` | Quit | ## Operations ### Rewrite (`w`) Re-summarizes a single summary node using the current depth-aware prompt templates. The process: 1. **Preview** — shows the prompt that will be sent, including source material, target token count, previous context, and time range 2. **API call** — sends to the configured provider API (Anthropic by default) 3. **Review** — shows old and new content side-by-side with token delta. Toggle unified diff view with `d`. Scroll with `j`/`k`. | Key (Preview) | Action | |-----|--------| | `Enter` | Send to API | | `Esc` | Cancel | | Key (Review) | Action | |-----|--------| | `y`/`Enter` | Apply rewrite to database | | `n`/`Esc` | Discard | | `d` | Toggle unified diff view | | `j`/`k` | Scroll content | **When to use:** A summary has poor quality (too verbose, missing key details, or was generated before the depth-aware prompts were implemented). Rewriting regenerates it from its original source material using the current prompts. ### Subtree Rewrite (`W`) Rewrites the selected summary and all its descendants, bottom-up. Leaves are rewritten first so that condensed parents pick up the improved content. Nodes are processed one at a time through the same preview→API→review cycle. | Key (additional) | Action | |-----|--------| | `A` | **Auto-accept** — apply current and all remaining automatically | | `n` | Skip current node, advance to next | | `Esc` | Abort entire subtree rewrite | The status bar shows progress as `[N/total]`. Auto-accept pauses on errors so you can inspect failures. **When to use:** A whole branch of the DAG has outdated formatting (e.g., pre-depth-aware summaries). Subtree rewrite regenerates everything from the leaves up. ### Dissolve (`d`) Reverses a condensation: removes a condensed summary from the active context and restores its parent summaries in its place. This is a surgical undo of a compaction step. The confirmation screen shows: - The target summary (kind, depth, tokens, context ordinal) - Token impact (condensed tokens → total restored parent tokens) - Ordinal shift (how many items after the target will be renumbered) - Parent summaries that will be restored (with previews) | Key | Action | |-----|--------| | `y`/`Enter` | Execute dissolve | | `n`/`Esc` | Cancel | **When to use:** - A condensed summary is too lossy — you want the original finer-grained summaries back - A corrupted condensed node needs to be removed so its parents can be individually repaired - You want to re-do a condensation after improving the leaf summaries **Important:** Dissolving increases the number of context items and total token count. Check the context view afterward to verify you haven't exceeded the context window threshold. ## CLI Subcommands Each interactive operation also has a standalone CLI equivalent for scripting and batch operations. ### `lcm-tui doctor` Scans for genuinely truncated summaries and can rewrite them in place. This is narrower than `repair`: it looks for specific truncation marker shapes instead of the generic fallback-summary marker. ```bash # Preview repairs for one conversation lcm-tui doctor 44 --show-diff # Apply repairs through Codex CLI OAuth after `codex login` lcm-tui doctor 44 --apply --provider openai-codex --model gpt-5.3-codex # Scan only across every conversation lcm-tui doctor --all ``` | Flag | Description | |------|-------------| | `--apply` | Write repaired summaries to the database | | `--summary` | Scan only and show counts | | `--all` | Scan all conversations (discovery mode only) | | `--provider ` | API provider (default: anthropic) | | `--model ` | API model (default: `claude-haiku-4-5`) | | `--base-url ` | Custom API base URL (overrides config and env) | | `--show-diff` | Show unified diff for each fix | | `--timestamps` | Inject timestamps into rewrite source text | Use `--provider openai-codex` when you want ChatGPT Plus/Pro OAuth from the Codex CLI. Keep `--provider openai` for direct OpenAI-compatible HTTP calls with a raw `OPENAI_API_KEY`, including custom `--base-url` proxies. #### MiniMax regional endpoints Set `MINIMAX_API_KEY`, then select `--provider minimax` for the global endpoint or `--provider minimax-cn` for the China endpoint. Both provider IDs default to `MiniMax-M3` and use the Anthropic-compatible Messages API: | Provider ID | Default base URL | |-------------|------------------| | `minimax` | `https://api.minimax.io/anthropic` | | `minimax-cn` | `https://api.minimaxi.com/anthropic` | `--base-url`, `LCM_TUI_SUMMARY_BASE_URL`, and configured provider `baseUrl` values continue to override these defaults. ### `lcm-tui repair` Finds and fixes corrupted summaries (those containing the `[LCM fallback summary]` marker from failed summarization attempts). ```bash # Scan a specific conversation (dry run) lcm-tui repair 44 # Scan all conversations lcm-tui repair --all # Apply repairs lcm-tui repair 44 --apply # Repair a specific summary lcm-tui repair 44 --summary-id sum_abc123 --apply # Repair through Codex CLI OAuth after `codex login` lcm-tui repair 44 --apply --provider openai-codex --model gpt-5.3-codex # Repair through a custom OpenAI-compatible proxy with a raw API key lcm-tui repair 44 --apply --provider openai --model gpt-5.3-codex --base-url https://proxy.example.com/openai ``` The repair process: 1. Identifies corrupted summaries by scanning for the fallback marker 2. Orders them bottom-up: leaves first (in context ordinal order), then condensed nodes by ascending depth 3. Reconstructs source material from linked messages (leaves) or child summaries (condensed) 4. Resolves `previous_context` for each node (for deduplication in the prompt) 5. Sends to the resolved provider API with the appropriate depth prompt 6. Updates the database in a single transaction | Flag | Description | |------|-------------| | `--apply` | Write repairs to database (default: dry run) | | `--all` | Scan all conversations | | `--summary-id ` | Target a specific summary | | `--provider ` | API provider (inferred from `--model` when omitted) | | `--model ` | API model (default depends on provider) | | `--base-url ` | Custom API base URL (overrides config and env) | | `--verbose` | Show content hashes and previews | ### `lcm-tui rewrite` Re-summarizes summaries using current depth-aware prompts. Unlike repair, this works on any summary, not just corrupted ones. ```bash # Rewrite a single summary (dry run) lcm-tui rewrite 44 --summary sum_abc123 # Rewrite all depth-0 summaries lcm-tui rewrite 44 --depth 0 --apply # Rewrite everything bottom-up lcm-tui rewrite 44 --all --apply --diff # Rewrite with Codex CLI OAuth after `codex login` lcm-tui rewrite 44 --summary sum_abc123 --provider openai-codex --model gpt-5.3-codex --apply # Rewrite through a custom OpenAI-compatible proxy with a raw API key lcm-tui rewrite 44 --summary sum_abc123 --provider openai --model gpt-5.3-codex --base-url https://proxy.example.com/openai --apply # Use custom prompt templates lcm-tui rewrite 44 --all --apply --prompt-dir ~/.config/lcm-tui/prompts ``` | Flag | Description | |------|-------------| | `--summary ` | Rewrite a single summary | | `--depth ` | Rewrite all summaries at depth N | | `--all` | Rewrite all summaries (bottom-up by depth, then timestamp) | | `--apply` | Write changes to database | | `--dry-run` | Show before/after without writing (default) | | `--diff` | Show unified diff | | `--provider ` | API provider (inferred from `--model` when omitted) | | `--model ` | API model (default depends on provider) | | `--base-url ` | Custom API base URL (overrides config and env) | | `--prompt-dir ` | Custom prompt template directory | | `--timestamps` | Inject timestamps into source text (default: true) | | `--tz ` | Timezone for timestamps (default: system local) | Exactly one of `--summary`, `--depth`, or `--all` is required. ### `lcm-tui dissolve` Reverses a condensation, restoring parent summaries to the active context. ```bash # Preview (dry run) lcm-tui dissolve 44 --summary-id sum_abc123 # Execute lcm-tui dissolve 44 --summary-id sum_abc123 --apply # Keep the condensed summary record (don't purge from DB) lcm-tui dissolve 44 --summary-id sum_abc123 --apply --purge=false ``` | Flag | Description | |------|-------------| | `--summary-id ` | Condensed summary to dissolve (required) | | `--apply` | Execute changes | | `--purge` | Also delete the condensed summary record (default: true) | ### `lcm-tui transplant` Deep-copies a summary DAG from one conversation to another. Used when an agent gets a new conversation (session rollover) but you want to carry forward summaries from the old one. ```bash # Preview what would be copied lcm-tui transplant 18 653 # Execute lcm-tui transplant 18 653 --apply ``` The transplant: 1. Identifies all summary context items in the source conversation 2. Recursively collects the full DAG (all ancestor summaries) 3. Deep-copies every summary with new IDs, owned by the target conversation 4. Deep-copies all linked messages and message_parts with new IDs 5. Rewires summary_messages and summary_parents edges 6. Prepends transplanted summaries to the target's context (existing items shift) 7. Detects duplicates via content SHA256 and aborts if any match Everything runs in a single transaction. | Flag | Description | |------|-------------| | `--apply` | Execute transplant | | `--dry-run` | Show what would be transplanted (default) | ### `lcm-tui backfill` Imports a pre-LCM JSONL session into `conversations/messages/context_items`, runs iterative depth-aware compaction with the configured provider + prompt templates, optionally forces a single-root fold, and can transplant the result to another conversation. ```bash # Preview import + compaction plan (no writes) lcm-tui backfill my-agent session_abc123 # Import + compact lcm-tui backfill my-agent session_abc123 --apply # Re-run compaction for an already-imported session lcm-tui backfill my-agent session_abc123 --apply --recompact # Force a single summary root when possible lcm-tui backfill my-agent session_abc123 --apply --recompact --single-root # Import + compact + transplant into an active conversation lcm-tui backfill my-agent session_abc123 --apply --transplant-to 653 # Backfill using Codex CLI OAuth after `codex login` lcm-tui backfill my-agent session_abc123 --apply --provider openai-codex --model gpt-5.3-codex # Backfill through a custom OpenAI-compatible proxy with a raw API key lcm-tui backfill my-agent session_abc123 --apply --provider openai --model gpt-5.3-codex --base-url https://proxy.example.com/openai ``` All write paths are transactional: 1. Import transaction (conversation/messages/message_parts/context) 2. Per-pass compaction transactions (leaf/condensed replacements) 3. Optional transplant transaction (reuse of transplant command internals) An idempotency guard prevents duplicate imports for the same `session_id`. | Flag | Description | |------|-------------| | `--apply` | Execute import/compaction/transplant | | `--dry-run` | Show what would run, without writes (default) | | `--recompact` | Re-run compaction for already-imported sessions (message import remains idempotent) | | `--single-root` | Force condensed folding until one summary remains when possible | | `--transplant-to ` | Transplant backfilled summaries into target conversation | | `--title ` | Override imported conversation title | | `--leaf-chunk-tokens ` | Max source tokens per leaf chunk | | `--leaf-target-tokens ` | Target output tokens for leaf summaries | | `--condensed-target-tokens ` | Target output tokens for condensed summaries | | `--leaf-fanout ` | Min leaves required for d1 condensation | | `--condensed-fanout ` | Min summaries required for d2+ condensation | | `--hard-fanout ` | Min summaries for forced single-root passes | | `--fresh-tail ` | Preserve freshest N raw messages from leaf compaction | | `--provider ` | API provider (inferred from model when omitted) | | `--model ` | API model (default depends on provider) | | `--base-url ` | Custom API base URL (overrides config and env) | | `--prompt-dir ` | Custom depth-prompt directory | ### `lcm-tui prompts` Manage and inspect depth-aware prompt templates. Templates control how the LLM summarizes at each depth level. ```bash # List active template sources (embedded vs filesystem override) lcm-tui prompts --list # Export default templates to filesystem for customization lcm-tui prompts --export # default: ~/.config/lcm-tui/prompts/ lcm-tui prompts --export /path/to/my/prompts # Show a specific template's content lcm-tui prompts --show leaf # Diff a filesystem override against the embedded default lcm-tui prompts --diff condensed-d1 # Render a template with test variables lcm-tui prompts --render leaf --target-tokens 800 ``` | Flag | Description | |------|-------------| | `--list` | Show which templates are active and their source | | `--export [dir]` | Export embedded defaults to filesystem | | `--show ` | Print the active template content | | `--diff ` | Unified diff between override and embedded default | | `--render ` | Render template with provided variables | | `--prompt-dir ` | Custom prompt template directory | **Template names:** `leaf`, `condensed-d1`, `condensed-d2`, `condensed-d3` (`.tmpl` suffix optional). **Customization workflow:** 1. `lcm-tui prompts --export` to get the defaults 2. Edit the templates in `~/.config/lcm-tui/prompts/` 3. `lcm-tui prompts --diff condensed-d1` to verify changes 4. Templates are automatically picked up by rewrite/repair operations ## Depth-Aware Prompt Templates The TUI uses four distinct prompt templates, one per depth level. This matches the plugin's depth-dispatched summarization strategy: | Template | Depth | Strategy | Receives `previous_context` | |----------|-------|----------|-----------------------------| | `leaf.tmpl` | d0 | Narrative preservation with timestamps, file tracking | Yes | | `condensed-d1.tmpl` | d1 | Chronological session narrative, delta-oriented (avoids repeating previous context) | Yes | | `condensed-d2.tmpl` | d2 | Arc-focused: goal → outcome → what carries forward. Self-contained. | No | | `condensed-d3.tmpl` | d3+ | Maximum abstraction. Durable context only. Self-contained. | No | **d0/d1** summaries receive `previous_context` (the content of the preceding summary at the same depth) so they can avoid repeating information. **d2+** summaries are self-contained — they're designed to be independently useful for `lcm_expand_query` retrieval without requiring sibling context. All templates end with an `"Expand for details about:"` footer listing topics available for deeper retrieval via the agent tools. ## Authentication The TUI resolves API keys by provider for rewrite, repair, and backfill compaction operations. - Anthropic: `ANTHROPIC_API_KEY` - OpenAI: `OPENAI_API_KEY` Resolution order: 1. Provider API key environment variable 2. OpenClaw config (`~/.openclaw/openclaw.json`) — checks matching provider auth profile mode 3. OpenClaw env file 4. `~/.zshrc` export 5. Credential file candidates under `~/.openclaw/` If the provider auth profile mode is `oauth` (not `api_key`), set the provider API key environment variable explicitly. Summary-producing operations (`doctor`, `repair`, `rewrite`, `backfill`, and interactive rewrite `w`/`W`) can be configured with: - `LCM_TUI_SUMMARY_PROVIDER` - `LCM_TUI_SUMMARY_MODEL` - `LCM_TUI_SUMMARY_BASE_URL` It also honors `LCM_SUMMARY_PROVIDER` / `LCM_SUMMARY_MODEL` / `LCM_SUMMARY_BASE_URL` as fallback. Separately, the conversation browser window size uses `LCM_TUI_CONVERSATION_WINDOW_SIZE` (default `200`). ## Database The TUI operates directly on the SQLite database at `~/.openclaw/lcm.db`. All write operations (rewrite, dissolve, repair, transplant, backfill) use transactions. Changes take effect on the next conversation turn — the running OpenClaw instance picks up database changes automatically. **Backup recommendation:** Before batch operations (repair `--all`, rewrite `--all`, transplant, backfill), copy the database: ```bash cp ~/.openclaw/lcm.db ~/.openclaw/lcm.db.bak-$(date +%Y%m%d) ``` ## Troubleshooting **"No LCM summaries found"** — The session may not have an associated conversation in the LCM database. Check that the `conv_id` column shows a non-zero value in the session list. Sessions without LCM tracking won't have summaries. **Rewrite returns empty/bad content** — Check provider/model access and API key. If normalization still yields empty text, the TUI now returns diagnostics including `provider`, `model`, and response `block_types` to help pinpoint adapter mismatches. **Dissolve fails with "not condensed"** — Only condensed summaries (depth > 0) can be dissolved. Leaf summaries have no parent summaries to restore. **Transplant aborts with duplicates** — The target conversation already has summaries with identical content hashes. This prevents accidental double-transplants. If intentional, delete the duplicates from the target first. **Token count discrepancies** — The TUI estimates tokens as `len(content) / 4`. This is a rough heuristic, not a precise tokenizer count. The plugin uses the same estimate for consistency. --- ## File: tui/README.md # lcm-tui Interactive terminal UI for inspecting, debugging, and maintaining the [Lossless Claw](https://github.com/Martian-Engineering/lossless-claw) database. Browse conversations, navigate the summary DAG, see exactly what the model sees in context, and perform surgical repairs — all from the terminal. ## Install **From releases:** Download the latest binary from [GitHub Releases](https://github.com/Martian-Engineering/lossless-claw/releases). **From source:** ```bash go build -o lcm-tui . # or: go install github.com/Martian-Engineering/lossless-claw/tui@latest ``` Requires Go 1.24+. ## Usage ```bash lcm-tui # default: ~/.openclaw/lcm.db lcm-tui --db /path/to/lcm.db # custom database path ``` ## Features **Browse & Inspect** - **Agent/session browser** — drill down from agents → sessions → conversations - **Codex backend inspection** — open native Codex app-server rollout JSONL from a bound OpenClaw session - **Codex ↔ LCM comparison** — render Codex backend rows beside the Lossless-managed context window - **Windowed conversation paging** — keyset pagination by `message_id` for large LCM conversations - **Summary DAG** — expandable tree of the full summary hierarchy with depth, kind, token counts, and source messages - **Context view** — see the exact ordered list of summaries + messages the model receives each turn - **Large files** — inspect intercepted oversized files and their exploration summaries **Repair & Maintain** - **Rewrite** (`w`) — re-summarize a node using current depth-aware prompts - **Subtree rewrite** (`W`) — bottom-up rewrite of an entire branch with auto-accept mode - **Doctor** — detect and repair genuinely truncated summaries with position-aware marker checks - **Dissolve** (`d`) — reverse a condensation, restoring parent summaries to active context - **Repair** — find and fix corrupted summaries (fallback truncations from failed API calls) - **Transplant** — deep-copy summary DAGs between conversations with full message/edge rewiring - **Backfill** — import pre-LCM JSONL sessions, compact depth-aware history, optional single-root fold + transplant **Prompt Management** - Four depth-aware templates: leaf, d1 (session), d2 (arc), d3+ (durable) - Export, customize, diff, and render templates via `lcm-tui prompts` ## CLI Subcommands Each interactive operation has a standalone CLI equivalent for scripting: ```bash lcm-tui doctor 44 --apply --provider openai-codex --model gpt-5.3-codex lcm-tui repair 44 --apply --provider openai-codex --model gpt-5.3-codex lcm-tui rewrite 44 --all --apply --diff --provider openai-codex --model gpt-5.3-codex lcm-tui dissolve 44 --summary-id sum_abc --apply # undo a condensation lcm-tui transplant 18 653 --apply # copy DAG between conversations lcm-tui backfill my-agent session_abc --apply --provider openai-codex --model gpt-5.3-codex lcm-tui backfill my-agent session_abc --apply --recompact --single-root # re-fold existing import to one root lcm-tui prompts --list # show active prompt sources ``` Use `--provider openai-codex` after `codex login` when you want the TUI to delegate through the Codex CLI OAuth session. Keep `--provider openai` for direct OpenAI-compatible HTTP calls with a raw `OPENAI_API_KEY`. ## Documentation Full reference with keybindings, screen descriptions, flag tables, and troubleshooting: **[docs/tui.md](../docs/tui.md)** ## Architecture The TUI reads directly from the LCM SQLite database (`~/.openclaw/lcm.db`) and session JSONL files (`~/.openclaw/agents/`). Write operations (rewrite, repair, dissolve, transplant, backfill) use transactions. Changes take effect on the next conversation turn — no restart needed. Doctor, repair, rewrite, and backfill compaction operations all accept `--provider`, `--model`, and `--base-url`, and they also honor `LCM_TUI_SUMMARY_PROVIDER`, `LCM_TUI_SUMMARY_MODEL`, and `LCM_TUI_SUMMARY_BASE_URL` before falling back to the legacy `LCM_SUMMARY_*` settings. By default, repair/rewrite/backfill use Anthropic (`claude-sonnet-4-20250514`), while doctor keeps its lighter default (`claude-haiku-4-5`). MiniMax API-key calls use `--provider minimax` for the global endpoint or `--provider minimax-cn` for the China endpoint. Both read `MINIMAX_API_KEY` and default to `MiniMax-M3`. ## License Part of the [Lossless Claw](https://github.com/Martian-Engineering/lossless-claw) monorepo.