Attention Architecture

The context window is a cache. Treat it like one.

Design doc, July 2026. The third axis of the sem storage engine.

The observation nobody builds on

Every code tool ever made — grep, LSP, ctags, even sem until today — is
stateless per call. Ask the same question twice, get the same bytes twice.
That was fine when a human read the answers: humans remember.

Agents remember too. Everything a tool returns lands in the model's context
window and STAYS there for the rest of the session. But the tool doesn't know
that, so it keeps re-sending what the model already holds:

- The agent reads resolve_entity (200 lines). Forty turns later it asks
again — different phrasing, same entity — and pays 200 lines again.
- Two answers overlap (a caller listed by impact is re-included as a
dependent by context). The overlap ships twice.
- Nothing changed between the two reads. The second copy carries zero
information. It costs real dollars and real context-window pressure anyway.

Measured across our agent benchmarks, repeated and overlapping fills are the
single largest token cost after first-read — and first-read is irreducible.
The repeats are pure waste.

The architecture

Model the agent's context window as a cache tier, and make the code
intelligence layer its coherence protocol.

text

L0  agent context window     what the model has already been shown
L1  resident server (RAM)    warm entity graph · <10ms socket answers
L2  cache.db (disk)          entities · content store · commit index
L3  cloud                    the same layers, hosted

L1-L3 exist in sem today. The new piece is that L0 becomes tracked state:
the resident server keeps a per-session ledger of every fill it has emitted
— entity id plus the structural hash of the version it sent.

Three behaviors fall out, and each one attacks the token axis directly:

1. Duplicate fills are suppressed.
A re-ask of something in the ledger answers with one line —
resolve_entity: unchanged since you read it (L910 calls diffy_merge) —
instead of the body. ~10 tokens instead of ~800. The agent loses nothing:
the body is already in its own context.

2. Changes ship as deltas.
If the entity's structural hash moved since the ledger's version, the answer
is the entity-level diff against what this agent saw, not the full body.
"You read version A; here is what changed" is the minimum-token truthful
answer, and only a layer that remembers what it sent can give it.

3. Prefetch becomes free to be aggressive.
Speculative context injection (sem's prompt-time prefetch) has always had one
risk: shipping something the agent would have read anyway costs nothing, but
shipping it twice costs double. With the ledger, a speculative fill is
recorded like any other, so the later explicit ask collapses to one line.
Prefetch and pull stop competing and start composing.

Why this is agent-native and not human-native

A human's screen is not a cache with perfect retention — people scroll away,
forget, re-open files. Tools for humans are RIGHT to be stateless. An LLM's
context window has perfect retention within a session and hard capacity
limits, which is exactly the profile of a cache tier. Coherence protocols,
not search engines, are the correct discipline for feeding it. That inversion
is why this design looks obvious in hindsight and appears nowhere: it only
makes sense once the reader is a machine with a context window.

How it completes the storage engine

sem's storage engine now manages three axes, each one turning a repeated
computation into data paid for once:

| axis | structure | what stops being recomputed |
|---|---|---|
| space | entity graph + content store | parsing, cross-file resolution |
| time | semantic commit index | history walks (150x measured) |
| attention | session fill ledger | re-reads and overlaps in the conversation |

Files were never the unit any of this needed. Entities address space,
commits-as-deltas address time, and fills address attention. A hosted service
serves all three from the same store: the graph is an artifact, history is
rows, and the ledger is a session key away.

Implementation sketch (v1 is small)

- The ledger lives in the resident server: session_key -> {entity_id -> structural_hash}, LRU-bounded, TTL ~2h. No new storage layer.
- Session key: the socket client passes one (agents already carry a session
id in their environment); no key means no ledger, exactly today's behavior.
- Fill suppression handles context first (bodies dominate token spend),
then text hits and impact lists.
- Deltas reuse the differ that already powers sem diff — the machinery
exists; it just needs the "since the version YOU saw" anchor the ledger
provides.
- Escape hatch: --fresh re-sends unconditionally.

Nothing about the protocol requires sem: it is the natural contract between
any long-lived code intelligence layer and any consumer with a context
window. But it requires entity identity (ids + structural hashes) to say
"unchanged" honestly, per-session state to know who saw what, and a resident
process to hold it — which is exactly the stack sem now has and grep never
will.

---

Llms

sem

Entity-level semantic diff on top of Git. Functions, classes, methods instead of lines.

Overview

sem extends Git with entity-level operations. Instead of tracking lines, sem tracks functions, classes, methods, and types. Uses tree-sitter for parsing and AST-normalized structural hashing to detect cosmetic vs structural changes.

Git tracks lines. Developers think in functions. sem bridges the gap.

Install

text

brew install sem-cli

Or build from source:

text

git clone https://github.com/Ataraxy-Labs/sem
cd sem/crates && cargo install --path sem-cli

Binary at crates/target/release/sem.

Commands

sem diff

Entity-level diff. Shows which functions/classes were added, modified, deleted, or renamed. Distinguishes cosmetic changes (whitespace/formatting) from structural changes (logic).

text

sem diff                          # working changes
sem diff --staged                 # staged only
sem diff --commit abc1234         # specific commit
sem diff --from HEAD~5 --to HEAD  # commit range
sem diff file1.ts file2.ts       # compare two files (no git needed)
sem diff --format json            # JSON output for agents/CI
sem diff --format plain           # git-status style
sem diff --format markdown        # markdown tables
sem diff --stdin --format json    # read file changes from stdin
sem diff --file-exts .py .rs      # filter by extension
sem diff -v                       # verbose inline content diffs

sem blame

Entity-level blame. Who last modified each function/class, not each line.

text

sem blame src/auth.ts
sem blame src/auth.ts --json

sem graph

Cross-file entity dependency graph. Shows what each function calls and what calls it.

text

sem graph
sem graph --entity validateToken
sem graph --file-exts .py
sem graph --format json
sem graph --no-default-excludes

sem impact

Transitive impact analysis. If this entity changes, what else is affected? BFS through dependency graph.

text

sem impact validateToken
sem impact validateToken --json
sem impact validateToken --file-exts .py
sem impact validateToken --no-default-excludes

Key Features

- 27 languages with full entity extraction via tree-sitter
- Structural hashing: AST-normalized hashes that ignore whitespace, comments, formatting
- Cosmetic vs structural change detection in diff
- Entity-level blame (per function, not per line)
- Cross-file dependency graph via call/reference analysis
- Transitive impact analysis (BFS through dependency graph)
- Three-phase entity matching: exact ID, structural hash (rename detection), fuzzy similarity
- JSON output for AI agents and CI pipelines
- Stdin mode for non-git usage

Language Support

27 programming languages:

| Language | Extensions | Entity Types |
|----------|-----------|--------------|
| TypeScript | .ts .tsx .mts .cts | functions, classes, interfaces, types, enums, exports |
| JavaScript | .js .jsx .mjs .cjs | functions, classes, variables, exports |
| Python | .py | functions, classes, decorated definitions |
| Go | .go | functions, methods, types, vars, consts |
| Rust | .rs | functions, structs, enums, impls, traits, mods, consts |
| Java | .java | classes, methods, interfaces, enums, fields, constructors |
| C | .c .h | functions, structs, enums, unions, typedefs |
| C++ | .cpp .cc .hpp | functions, classes, structs, enums, namespaces, templates |
| C# | .cs | classes, methods, interfaces, enums, structs, properties |
| Ruby | .rb | methods, classes, modules |
| PHP | .php | functions, classes, methods, interfaces, traits, enums |
| Swift | .swift | functions, classes, protocols, structs, enums, properties |
| Elixir | .ex .exs | modules, functions, macros, guards, protocols |
| Bash | .sh | functions |
| HCL/Terraform | .hcl .tf .tfvars | blocks, attributes (qualified names) |
| Kotlin | .kt .kts | classes, interfaces, objects, functions, properties |
| Fortran | .f90 .f95 .f | functions, subroutines, modules, programs |
| Vue | .vue | template/script/style blocks + inner TS/JS entities |
| XML | .xml .plist .svg .csproj | elements (nested, tag-name identity) |
| ERB | .erb | blocks, expressions, code tags |
| Svelte | .svelte .svelte.js .svelte.ts | component blocks, rune modules + inner JS/TS entities |
| Dart | .dart | classes, mixins, extensions, enums, type aliases, functions |
| Perl | .pl .pm .t | subroutines, packages |
| OCaml | .ml .mli | values, modules, types, classes, externals |
| Scala | .scala .sc .sbt | classes, objects, traits, enums, functions, vals, extensions |
| Nix | .nix | bindings, inherit declarations |
| Zig | .zig | functions, tests, variables |

Plus structured data formats:

| Format | Extensions | Entity Types |
|--------|-----------|--------------|
| JSON | .json | properties, objects (RFC 6901 paths) |
| YAML | .yml .yaml | sections, properties (dot paths) |
| TOML | .toml | sections, properties |
| CSV | .csv .tsv | rows (first column as ID) |
| Markdown | .md .mdx | heading-based sections |

Everything else falls back to chunk-based diffing.

JSON Output

text

sem diff --format json

Returns:

json

{
  "summary": { "fileCount": 2, "added": 1, "modified": 1, "deleted": 1, "moved": 0, "renamed": 0, "reordered": 0, "orphan": 0, "total": 3 },
  "changes": [
    {
      "entityId": "src/auth.ts::function::validateToken",
      "changeType": "added",
      "entityType": "function",
      "entityName": "validateToken",
      "startLine": 12,
      "endLine": 18,
      "oldStartLine": null,
      "oldEndLine": null,
      "filePath": "src/auth.ts"
    }
  ]
}

The named change-type buckets (added, modified, deleted, moved, renamed, reordered) always sum to total. Orphan is metadata for module-level changes already included in those buckets.

Architecture

Cargo workspace: sem-core (library) + sem-cli (binary).

- tree-sitter for code parsing (native Rust, not WASM)
- git2 for in-process Git operations
- rayon for parallel file processing
- xxhash for structural hashing
- Plugin system for adding new languages and formats

Performance

- Small commit (1 file): 5ms
- Medium commit (5 files): 8ms
- Large commit (13 files): 19ms
- Range (8 commits, 30 files): 24ms
- Faster than git diff for equivalent operations

As a Library

sem-core can be used as a Rust library dependency:

toml

[dependencies]
sem-core = { git = "https://github.com/Ataraxy-Labs/sem", version = "0.3" }

Used by weave (semantic merge driver) and inspect (entity-level code review).

Changelog

All notable changes to sem are documented in this file.

[Unreleased]

Fixed

- Building a graph over Svelte components no longer crashes (SIGSEGV) on Linux/glibc. sem graph/context/orient over .svelte files deterministically exited 139 from an invalid free in the tree-sitter-htmlx-svelte 0.1.8 grammar's scanner, hit during parallel graph construction (macOS's allocator tolerated the bad free, so it only showed on Linux). Bumped the grammar to 0.1.16, which carries the scanner fixes; the existing version constraint already permitted it, so this is a lock-only dependency update. Added a parallel-Svelte-graph regression test. Thanks @XF-FW for the exhaustive isolation and the verified fix (#471).

[0.21.0] - 2026-07-10

Added

- Cloud-enabled sem diff now creates an immutable, owner-private hosted review URL. Snapshot uploads include changed-entity caller/callee relations and truthful Git provenance: branch, scope, base/head refs, and available SHAs. Working-tree reviews explicitly report HEAD → WORKTREE rather than implying that uncommitted changes exist on GitHub.
- GitHub login can now be exchanged for a revocable CLI session. Raw GitHub credentials are not persisted as sem credentials, and the CLI session resolves to the same stable principal used by web reviews.

Fixed

- Hosted-review upload failures are visible without breaking the local diff. sem diff still exits successfully with its complete local result, while stderr explains that the private review could not be uploaded.
- Transitional cloud repository states refresh immediately. sem whoami no longer leaves a repository stuck at a cached pending state after cloud indexing has completed.
- sem diff now collapses contiguous line chunks on unsupported files into one summary line. When a file has no grammar, sem falls back to fixed 20-line chunks, so deleting or adding one previously printed a wall of ⊖ chunk lines 1-20 [deleted] / 21-40 / 41-60 … lines that ate context for no information. Contiguous chunks of the same change type now consolidate to a single line, e.g. ⊖ 13 chunks lines 1-246 [deleted]. Verbose mode (-v) is unchanged, since it still prints per-chunk content. Thanks @graipher for the report (#466).

Performance

- sem context now answers from an indexed point query instead of loading the whole graph, so it scales to millions of entities. It previously hydrated every entity (plus decompressed bodies) just to answer about one, so on a 2.3M-entity repo each call took ~15s. The SQLite cache is already normalized and indexed, so when the git oracle proves the cache fresh (no filesystem walk) sem context now builds only the k-hop neighbourhood around the target straight from the store: batched IN (...) edge queries, bodies fetched per hop, stopping once there is enough content to cover the token budget so a hub entity's fan-out does not explode the fetch. It reuses the existing packer on that subgraph, so output is byte-for-byte identical to the full-graph path, and falls back to the full load whenever the oracle declines. On the Linux kernel (2.31M entities) sem context drops from ~15s to 0.44s per call; on Kubernetes (520k) from ~5s to ~1.2s.
- Default (All-mode) sem impact now answers from the indexed cache too, instead of hydrating the whole graph. A full cache stored entities and edges but not test flags, so All/Tests-mode impact (which includes the "covered by N tests" answer) fell through to a full-graph load — ~3.5s on Kubernetes, ~10.7s for an unbounded --depth 0, even for a tiny blast radius. The full save now records test flags (shared with the topology save) behind a metadata marker, so the existing point-query path can serve All-mode impact straight from indexed edge queries. Output is identical to the full-load path — verified byte-for-byte against it, including the tests field. On Kubernetes (520k entities) default sem impact drops from ~3.5s to 0.04s, and --depth 0 from ~10.7s to 0.04s. Caches built before the marker still take the full path, so nothing regresses.
- The resident MCP server no longer holds the whole graph in RAM by default, cutting idle memory dramatically. It used to proactively build and keep the entire deserialized graph in memory on startup so the first query would be warm — ~671MB on Kubernetes, ~5GB on the Linux kernel. But context and impact now answer from the indexed cache directly, and the CLI's fast paths bypass the resident entirely, so that proactive hold is mostly wasted memory. Prewarm is now opt-in (SEM_PREWARM); by default the resident stays light and builds the full graph lazily, only when a query that genuinely needs it (graph/diff/text) runs. On Kubernetes an idle resident drops from 671MB to 10MB.

[0.20.0] - 2026-07-05

Changed

- Indexing now shows a staged loader with a real, whole-build progress bar, not a single "Building entity graph" spinner. A cold graph build renders each phase sem-core reports as a persistent ◆ line — Scanning files — N found, Parsing code — done — and a single filling bar with a live percentage spans the entire build: the build is two passes over the file set (parse, then resolve), so the bar's length is 2×files and its position is (files parsed + files resolved). It tops out at ~50% when parsing finishes and only reaches 100% when resolution actually completes — so 100% means genuinely done, not "parsing done." Fed by two lock-free per-file counters (graph_parse_done, graph_resolve_done) via phase hooks. Ends with the existing ✓ N entities · M files in …ms summary. Warm cache fires nothing and stays instant; TTY-only, so agents, pipes, the MCP server, and CI see nothing (the bar's poll thread never even spawns off a terminal).
- sem setup now shows a staged progress loader instead of a flat list of check lines. Setup runs as a small tree of steps — git diff → sem diff, Claude Code hooks, pre-commit hook — each with a live braille spinner that resolves to a green ◆ (did something), a dim · (nothing to do / not applicable, e.g. not in a git repo), or a yellow ⚠ (left a file untouched on purpose, e.g. an unparseable settings.json). It ends with a one-line summary and the sem unsetup revert hint. Same idempotent behaviour, just legible at a glance.

Added

- sem-core: set_build_phase_hook / clear_build_phase_hook / BuildPhase + graph_parse_done + graph_resolve_done — an optional per-thread callback at graph-build phase boundaries (parsing, resolving) plus a lock-free counter of files parsed, so a front-end can render staged progress and a live parse bar. No-op for every caller that doesn't read them.

[0.19.0] - 2026-07-05

Removed

- Removed sem orient and all fuzzy/ranked retrieval. The orient command and its --pack briefing, the sem-core ranking (lexical scoring, IDF, recall net, structural priming), the sem_entities query= intent-search mode, and the resident server's orient socket op are all gone. Ranking a natural-language task to the right entity proved unreliable — a 45-task validation showed the ranker's hit-rate (~47%) could not be lifted by heuristics without causing regressions — and it was the only non-deterministic thing in sem. sem is now purely deterministic: context (read an entity plus its callers/callees), impact (blast radius), diff, entities (list by path, or text= for exact-substring search), blame, log. To find code whose name you don't know, use a plain text search to get a candidate name, then hand it to sem context for the structure grep can't give. The prompt-submit hook keeps its deterministic exact-name prefetch and no longer shells out to the ranker.

Fixed

- Dot-chain extraction is now linear, not quadratic, in file size. extract_dot_chains_with_positions computed each match's line number by counting newlines from the start of the file every time, so on a large file dense with a.b chains the cost was O(matches times filelen). Since the regex yields matches in increasing byte order, it now tracks the line number incrementally and counts only the newlines since the previous match, which is linear overall and produces identical one-based line numbers. Verified byte-for-byte identical graph output on React (34,251 entities, 73,702 edges). No change for typical files; it removes a cliff on very large generated or minified sources.

- Structural hashing no longer allocates a Vec per AST node. The two structural-hash walkers (hash_structural_tokens and its name-excluding variant) collected every internal node's children into a fresh heap Vec (plus a fresh tree-sitter cursor) just to push them in reverse, despite a comment claiming zero allocations. They now reuse a single cursor and push children in place, reversing the appended slice, which is byte-for-byte identical output. On a cold graph build this removes roughly 300k allocations (structural hashing alone dropped from about 319k allocations to 13.5k, measured with dhat on deno). Peak RSS is unchanged and wall time is within noise under mimalloc, but the churn reduction helps memory-constrained and non-mimalloc builds. Hashes are verified identical across 3,337 entities, so rename detection and existing caches are unaffected.

Added

- sem setup now makes sem a Claude Code session default (macOS/Linux). Beyond the git diff alias, it installs two session hooks into ~/.claude/settings.json: a warm resident graph (SessionStart runs sem mcp --resident detached, so structural queries answer in single-digit ms instead of rebuilding) and prompt-time context injection (sem hook prompt-submit). The JSON edit is idempotent, backs up settings.json first, refuses to touch a file it can't parse, and preserves every existing user hook and key; sem unsetup removes exactly the sem hooks and cleans up empty arrays. Local warmth is free and login-free — cloud (sem login) is repositioned in the README as the scale/team/CI upgrade, not the way to get warmth.

Documentation

- Benchmarks page rebuilt on the July 2026 paired-run data. The docs site's benchmarks page now reports the real agent A/B numbers (grep+read agent vs sem agent on SWE-bench Verified bugs, hidden-test graded): 50-65% faster code understanding, verify loop 2.90s to 0.59s per iteration when call-graph edges resolve (bimodal, 1.2x floor disclosed), token parity stated plainly, and an explicit "what sem does not do" section including the unchanged solve rate. Retired the stale "75% fewer tokens" and "2.3x agent accuracy" hero claims. Changelog page gains entries for v0.17-v0.18 work with the lessons that produced them.

[0.18.0] - 2026-07-03

Fixed

- sem now works on repos using git's reftable ref storage (git init --ref-format=reftable, git 2.45+). Previously every command died with libgit2's cryptic unsupported extension name extensions.refstorage. libgit2 can't read reftable refs, but the object database and index are unchanged, so GitBridge now tolerates the extension and routes just the ref resolutions (HEAD, refspecs, revwalk starts) through the git CLI while libgit2 keeps doing everything else by OID. Verified end to end on a real reftable repo: working/staged/commit/range diffs, blame, and per-file history all produce identical results to a files-backend repo. One residual gap: the cache freshness oracle's direct git2::Repository::open is .ok()-guarded, so on reftable repos it just skips the acceleration (correctness unaffected). Requires git on PATH for the ref lookups. Thanks @bengry for the report and clean repro (#451).

Added

- Unique-method-name call edges (dynamic languages). Attribute calls on receivers of unknown type (index.keep_levels(...)) previously produced no graph edge, hiding real dependents and blinding --tests. In Python/Ruby — where receiver types are statically unknowable — a method name with exactly one definition repo-wide now resolves to it: one candidate, one edge; any ambiguity, no edge. Static languages keep precision-first resolution (an unresolved receiver there is deliberate: shadowed import, instance property).
- Parent::child entity qualifiers. sem impact "Dataset::set_index" and friends now work everywhere Parent.child does (graph and cached lookups).

- sem impact --tests lexical fallback + fast-path fallthrough. Graph edges miss tests that call a target through a module namespace (xr.where(...) resolves to no entity), so --tests could answer "No tests found" for a function with dozens of tests. Now: an empty tests answer from the sidecar or disk cache is treated as non-authoritative and falls through to the full path, which backstops zero graph edges with lexical reachability — test entities naming the target as a whole word — clearly labeled as weaker evidence. Found live: an agent's graph-selected verify loop (run only the tests that reach your change) went from 0 selected tests to a 56-test net on xr.where, versus the 1,900+ tests of whole-file runs.
- Sharper --pack briefings. Term ranking is now IDF-weighted (a term appearing in half the repo is worth almost nothing), <details> environment dumps in issue text are stripped before extraction, attribute accesses glued to receivers ("d2.loc") also emit their .attr suffixes as terms, and one of the three briefing slots goes to the top name-echo orient hit — for bugs where the issue names a surface API the culprit's body never mentions.

- sem orient --pack <tokens>: turn-zero briefings from task text. Feed orient a whole issue or task description and it returns a packed briefing — the top matching functions' bodies plus their immediate callers/callees — sized to the token budget. Ranking is body-term convergence: code-ish terms are extracted from the task text (flags, dotted names, identifiers) and entities are ranked by how many distinct terms their bodies contain, since issue vocabulary lives in bodies, not names. Built for prompt-time injection (the agent-side analog of the prompt-submit prefetch hook): the code an agent would spend its first turns foraging for arrives at turn zero. Honest calibration: on three ground-truth issues it put the exact target function first on two; issues that quote the tool's own output can still poison term extraction.

- sem is published to the official MCP registry as io.github.Ataraxy-Labs/sem, so MCP clients that browse the registry (VS Code, Cursor, Claude Code, and others) can discover and install the server directly. The release workflow now publishes each release to the registry via mcp-publisher (authenticated with GitHub OIDC, no extra secrets), backed by a server.json manifest and an mcpName field in the npm wrapper.

Performance

- Delta-fills: changed entities answer with a diff against the version your session saw. The attention ledger now stores fill contents, so when a session re-asks about an entity that changed since its last look, the answer is an entity-level delta (∆ alpha · changed since you read it … - x = 1 / + x = 42) instead of the whole packed body. Measured end to end: a post-edit re-ask that previously re-sent the body now costs 2 diff lines. Completes the ledger's answer set — new entity: full fill; unchanged: one line; changed: delta; SEM_FRESH=1 / fresh: true always forces the full re-pack. Deltas larger than 120 lines fall back to a full fill.

- Attention ledger covers the MCP path. sem_context (the tool agent sessions actually call) now runs through the same per-session fill ledger: an MCP server process serves exactly one session, so re-asks for unchanged entities collapse to one ≡ unchanged since you read it line automatically — no environment variable needed. New optional fresh: true param forces a full re-send (for when context compaction dropped the earlier fill).

- Attention ledger v1: repeated context fills collapse to one line. The resident server now keeps a per-session ledger of every context fill it has emitted (entity id + content fingerprint). When the same session re-asks for an unchanged entity, the answer is a single ≡ unchanged since you read it line instead of the full packed body — the body is already sitting in the asking model's context window, so re-sending it is pure token waste. Measured through the CLI socket path: 8,586 bytes first fill, 139 bytes on repeat (98.4% suppressed). Opt-in via SEM_SESSION=<id> in the environment; SEM_FRESH=1 bypasses; anonymous calls are never suppressed. Any change to the target entity misses the fingerprint and re-sends in full. This is the first piece of the attention architecture (docs/attention-architecture.md): space (graph), time (commit index), attention (ledger).

- sem entities --text: entity-addressed text search from the CLI (the MCP tool already had it) — one line per hit (file, innermost entity, line, matched text) instead of whole bodies, served from the resident server's warm graph in milliseconds with a local-graph fallback. This is the token-cheap way for an agent to verify a call site or find a string: a body-level sem context costs hundreds of tokens where a text hit costs ~15.

- Auto-resident server: every sem CLI query after the first answers in milliseconds, on any repo. A socket miss now spawns sem mcp --resident (hidden plumbing) detached in the background: a server that holds the repo's graph warm and serves ONLY the per-repo unix socket, exiting on its own when idle for 30 minutes or when it loses the bind race to a live session. sem context and sem orient gain sidecar fast paths (impact already had one), so the full structural read loop runs against the warm graph: measured on a 30K-LOC repo, first sem context 0.68s cold (spawning the resident), then context/orient/impact all under 10ms. In a controlled agent benchmark (6 verified code-understanding questions, classic grep/read agent vs sem agent, identical prompts and batching guidance), the sem agent answered in 13s vs 37s with equal 6/6 correctness — 65% faster. SEM_NO_AUTOWARM=1 disables the auto-spawn, SEM_NO_SIDECAR=1 the fast path.

- Token-efficient tool output: the same answers at a fraction of the tokens the consuming model has to read (and pay for). (1) The context packer stops enumerating noise: related test entities are folded into per-role counts instead of packed as one-line "#[test]" stubs (unless the target itself is a test, when its test neighborhood is the question), bare attribute/comment signatures are skipped, and transitive tiers are capped at 25 entries per role with the remainder counted. What was dropped is stated explicitly in one line ("not packed: +64 direct dependents (64 tests) · sem_impact lists them"), so the signal survives at a fraction of the cost. Measured: sem context on a hot sem-core entity 7,272 to 4,113 tokens (-43%, 119 to 48 entries); on a hot weave entity 5,216 to 3,498 (-33%, 146 to 27 entries, the 119 test stubs now one line). (2) The sem_entities MCP tool renders compact per-line trees (name · type · lines, children indented, files as group headers) instead of pretty-printed JSON: measured 3.7x fewer tokens on a 115-entity file (5,028 to 1,361). Applies to the MCP path and query modes; CLI --json output is unchanged for scripts.

- Semantic commit index (storage engine layer 2): history is now stored as entity deltas. Each commit is semantic-diffed against its first parent exactly once and persisted as entity-change rows in the cache (commits + entity_changes tables, sha-keyed and branch-agnostic); every later history query is a SQLite lookup plus a diff of only the commits git gained since. Measured on the sem repo: sem log over 500 commits drops from 4.46s to 0.03s on the second query (~150x), with the first query as the one-time indexing pass and each new commit costing one incremental diff. Applies to sem log repo analytics (hotspots + co-change pairs) and the MCP sem_log tool; per-entity traces are unchanged. Aggregation is shared code between the live git walk and the store (aggregate_history_analytics), tested to produce identical hotspot/co-change output, so the two paths cannot drift. Merge-heavy history gets better semantics: each commit is attributed its own first-parent diff instead of a diff against its arbitrary revwalk neighbor, and merge commits contribute no changes of their own (their first-parent diff restates the merged-in commits, which index individually). File-filtered queries index full-repo diffs once and filter at aggregation, so the first filtered query costs more than the old pathspec-scoped walk but every later query on any filter is instant. Cache schema v9; existing caches rebuild automatically on first use.

- CLI sidecar fast path: sem impact now answers from the resident sem mcp server's warm graph via its unix socket before doing any local work — measured 4.5ms end-to-end on a 158K-LOC repo, versus 22.4ms for the local cold path and 7.7ms for a ripgrep scan of the same repo: the full blast radius (callers, dependencies, depth-bounded transitive impact, affected tests) is now cheaper than a raw text match. Output is byte-identical to the local path (the sidecar ships serialized EntityInfos that the CLI feeds to its existing printers; verified across all modes and --json). The fast path is an accelerator, never a requirement: bounded socket timeouts and silent fallback mean no resident server (or SEM_NO_SIDECAR=1, --no-cache, custom scopes, .semignore, --entity-id) just runs the normal local path. Server-side, the new sidecar impact op classifies affected tests only among the entities the impact BFS actually reached, instead of walking the whole corpus per call (6.8ms → 0.1ms on a 4.7K-entity graph).

Fixed

- Internal: rustfmt line-wrap missed in the #445 sidecar change; no behavior change.

- The context packer's token estimator was undercounting real tokens 2-3x on dense code (words x 1.3 vs the ~4 chars/token reality: a context reported as 661 tokens measured ~2,400 real tokens), silently overshooting every budget. It now takes the max of the word- and character-based estimates, so nominal budgets match what the consuming model actually pays.

- The context packer's "not packed" summary line pluralizes roles correctly ("transitive dependencies", not "dependencys").

- Workspace version bumped to 0.16.0: ContextResult gained the public omitted field (a breaking change for struct-literal constructors, flagged by cargo-semver-checks), and 0.x semantics put breaking changes in the minor version.

- The docs site deploys through workflow-based GitHub Pages (.github/workflows/docs-pages.yml: upload /docs verbatim, deploy) instead of the legacy branch-based Jekyll builder, which began failing repo-wide with zero-duration "Page build failed" errors on commits that didn't touch docs — including on direct build requests via the Pages API. The site is pure static HTML, so the legacy builder added nothing but a failure mode; deploys now also skip entirely on commits that don't change docs/.

Changed

- The GitHub Action's PR-comment footer now tells the reader what to do next — "add it to your repo in 2 minutes", linking to the action's install snippet — instead of only naming the tool. Every entity-diff comment is seen by all of a repo's collaborators; the footer is the loop that turns viewers into installs.

Added

Added

- Entity-level history analytics: sem log with no entity now analyzes recent repo history in one pass and reports hotspots (the most-changed code entities, with commit counts, distinct authors, and the last commit that touched each) and co-change pairs (entities that repeatedly change in the same commits, with a confidence score — "these two never change apart"). Counts are per commit, code entities only (doc headings, config properties, and lockfile chunks are excluded so the signal is about code), and bulk commits touching >50 entities are excluded from pair-counting to keep quadratic noise out. Same via MCP: sem_log without entity_name. --file scopes to one file; --json returns everything. This is the time axis a snapshot dependency graph cannot see: which code churns, and which code moves together.

Changed

- sem_impact MCP results now render as a blast-radius tree (◉ header, one ├─▶ branch per file, real callers first, all-test files sunk to the bottom, nothing elided) — expanding the tool widget is the live graph, no separate viewer process needed. The bundled skill also instructs agents to draw the blast radius as a small ASCII tree directly in their reply when an impact result drives the answer.

- sem_impact and sem_context MCP results now render as a compact entity tree instead of pretty-printed JSON: dependents/dependencies/transitive impact grouped one line per file, every entity name preserved, with the elapsed time and source in a footer. The same information lands in about 15% of the tokens, and the expanded tool widget in agent UIs reads at a glance (⊕ entity · file, ← 29 dependents · 10 files, ⚡ 70 transitively affected). Context entries keep their verbatim content under a per-entry header.

[0.15.1] - 2026-07-01

Added

- npx @ataraxy-labs/sem-skill --badge (opt-in) installs a live sem badge in the Claude Code statusline: it shows how many structural queries ran this session, the last command and the entity it analyzed, its latency, a sparkline of recent latencies, and a rotating stat (distinct entities analyzed, top command) (⊕ sem ×12 impact validateToken 9ms ▁▂▃▅▂ · 7 entities analyzed). It is fed by a PostToolUse hook that catches sem via both the MCP tools and the sem CLI (Bash), and falls back to recent activity so the badge never stalls on "idle". Non-destructive: it backs up settings and never overwrites an existing statusline (it prints how to add the badge yourself instead).
- GitHub Action (Ataraxy-Labs/sem/action): entity-level semantic diff comments on pull requests. One sticky comment per PR showing which functions/classes/methods were added, modified, or deleted, updated in place on every push; cosmetic-only PRs (formatting/comments) are called out explicitly. Installs the prebuilt binary (~2s), needs no config or API keys, and never fails the build. sem's own PRs now dogfood it via .github/workflows/pr-entity-diff.yml.
- The savings meter now lives in the statusline itself — no extra process. The --badge badge always shows the live estimated time + tokens this session's sem calls saved vs grep+read (⊕ sem ×5 diff · ≈ 4m · ≈ 25k tokens saved), and when idle it shows the lifetime total (⊕ sem idle · ≈ 3h · ≈ 190k tokens saved). The PostToolUse hook is the single writer of the persisted lifetime tally (~/.claude/sem-savings.json), so the counter grows from real usage whether or not the live viewer is open. Estimates stay anchored to the measured benchmark and labelled ≈.
- Live viewer for the --badge install: ~/.claude/sem-live.py (run it in a spare terminal pane). It redraws an ASCII blast-radius graph each time sem runs — the analyzed entity, its direct callers (real ones surfaced, test fan-out collapsed), and the transitive count — plus a savings meter: a running, honestly-estimated tally of the grep+read round-trips, time, and tokens sem saved this session, and a lifetime counter persisted across sessions (~/.claude/sem-savings.json). Estimates are anchored to a measured benchmark and labelled ≈. The badge hook now also records --file and cwd so the graph can be reconstructed.

Fixed

- Repository discovery now tolerates Git worktrees that use the extensions.relativeworktrees config key, avoiding libgit2's unsupported-extension error when plain git can open the checkout.
- Cloud-backed sem impact / sem context no longer answer queries they can't answer correctly. Two gates added: --no-cache now always computes fresh locally (previously the cloud snapshot was served anyway), and file-hinted queries (--file) stay local — the cloud resolves entities by name with a silent name-only fallback, so for same-named entities (e.g. ten fn run command handlers) it could return the wrong entity's graph, and a stale cloud index could drop dependents that exist locally. Local resolution disambiguates exactly; the cloud path returns once the server resolves name+file strictly and exposes its indexed commit for a freshness check.
- Impact/dependency resolution now follows type-qualified associated calls (Type::method()) when the receiver is a known repo type, so a caller reached only through a static/associated path is no longer dropped from sem impact. Previously, e.g., a test helper calling SemPlugin::detect_changes() was invisible to the reverse-dependency graph, and its transitive callers were missing from the blast radius. Resolution stays precise: a bare module path (foo::bar::baz()) still does not bind to a same-name local function, and common associated names (Type::new, ::default) are not guessed.

Performance

- Faster graph hydrate on large repos. The public EntityGraph maps now use rustc-hash (FxHashMap) instead of std SipHash, matching the build's internal maps, and the SQLite cache sets read pragmas (mmap_size, cache_size, temp_store=MEMORY) on every connection. On a 200K-entity / 800K-edge graph this is about 9% faster to hydrate (0.42s to 0.39s, no overlap across repeats); negligible on small repos. Output is byte-identical.

[0.15.0] - 2026-06-30

- Release pipeline: the Intel macOS cross-build failed on openssl-sys (no target-arch OpenSSL when cross-compiling on Apple Silicon). It now builds OpenSSL from source via --features vendored-openssl, the same approach the Linux arm64 cross-build uses. 0.14.0's binaries never published because of this; 0.14.1 is the first release to ship binaries for every platform, including Intel macOS (#374).

[0.14.0] - 2026-06-23

Added

- Progress spinner for slow graph builds. sem graph and sem context now show a uv-style spinner and a summary line (e.g. 135,298 entities, 7,743 files in 6.6s) while building the entity graph. Strictly stderr and TTY-only, so pipes, JSON, and agent/MCP sessions are unaffected. Disable with SEM_NO_PROGRESS=1.

- SQL support (.sql, .psql, .pgsql, .ddl) via the official DerekStride/tree-sitter-sql grammar. Extracts tables, views, materialized views, functions, indexes, types, schemas, triggers, sequences, and databases. Thanks @robahtou for the request (#339).
- Start tracking project changes in CHANGELOG.md.
- Add a pull request check that asks contributors to include a changelog entry.
- sem entities accepts multiple file or directory path arguments.

Changed

- Sparse checkouts now work. libgit2 cannot read a sparse index (unsupported mandatory extension: 'sdir') and its workdir diff reported sparse-excluded files as deleted; sem now routes working and staged diffs through the git CLI when a sparse checkout is detected. Thanks for the report (#330).

- README now documents adding the MCP server to coding agents (claude mcp add sem -- sem mcp) and explains why sem mcp exists. The old section pointed at a separate sem-mcp binary; sem mcp ships in the main binary.

- sem stats now counts every diff, including runs that find no changes (previously those returned early and were never recorded, so diffs performed undercounted).

- Telemetry no longer records development builds (debug builds, or binaries run from a Cargo target/ directory), so contributor and CI-of-our-own usage stays out of the numbers.

- Cloud sync only auto-registers repos that GitHub confirms are public. Private repos run locally unless you opt in with SEM_SYNC_PRIVATE=1.
- install.sh verifies the release archive against checksums.txt before installing.
- Switched the Perl grammar to the official ts-parser-perl crate (was the unattributed tree-sitter-perl-next copy). Properly attributed, correctly MIT-licensed, and includes upstream fixes: an infinite-loop hang on malformed input, better error recovery, and faster parsing. Thanks @rabbiveesh for the report (#355).

Removed

- sem verify (function call-arity checker). It saw negligible use and overlapped with compilers/LSPs; removing it keeps the surface area focused.

---

CONTRIBUTING

Contributing to sem

Changelog Entries

Every pull request should add a CHANGELOG.md entry under ## [Unreleased].
Use the section that best describes the change:

- Added
- Changed
- Fixed
- Removed
- Security
- Maintenance

Keep entries concise and user-facing. For maintenance-only work, add a short
Maintenance bullet. CI checks that each pull request adds at least one line to
CHANGELOG.md.

Adding a New Language

sem uses tree-sitter grammars to extract semantic entities (functions, classes, etc.) from source code. Adding a new language is straightforward: you define a config struct and add a cargo dependency. No parser code needed.

This guide walks through the process step by step.

Overview

All language support lives in two files:

- crates/sem-core/Cargo.toml (tree-sitter grammar dependency)
- crates/sem-core/src/parser/plugins/code/languages.rs (language config)

Each language gets a LanguageConfig that tells sem which AST node types represent code entities.

Step 1: Add the tree-sitter dependency

Add the grammar crate to crates/sem-core/Cargo.toml:

toml

[dependencies]
tree-sitter-scala = "0.23"

Most grammars are published on crates.io as tree-sitter-{lang}. Check crates.io for the latest version. The 0.23 series works with tree-sitter 0.26.

Step 2: Add a getter function

In languages.rs, add a function that returns the tree-sitter Language:

rust

fn get_scala() -> Option<Language> {
    Some(tree_sitter_scala::LANGUAGE.into())
}

Some crates export the language differently. Check the crate's docs. Common patterns:

- tree_sitter_python::LANGUAGE (most languages)
- tree_sitter_typescript::LANGUAGE_TYPESCRIPT (when a crate has multiple grammars)
- tree_sitter_php::LANGUAGE_PHP (same)

Step 3: Define the language config

Add a static config in languages.rs:

rust

static SCALA_CONFIG: LanguageConfig = LanguageConfig {
    id: "scala",
    extensions: &[".scala", ".sc"],
    entity_node_types: &[
        "function_definition",
        "class_definition",
        "object_definition",
        "trait_definition",
        "val_definition",
        "var_definition",
        "type_definition",
    ],
    container_node_types: &["template_body", "block"],
    call_entity_identifiers: &[],
    suppressed_nested_entities: &[],
    get_language: get_scala,
};

Step 4: Register the config

Add a reference to the ALL_CONFIGS array:

rust

static ALL_CONFIGS: &[&LanguageConfig] = &[
    // ... existing configs ...
    &SCALA_CONFIG,
];

Step 5: Register the extensions

Add all file extensions to get_all_code_extensions():

rust

static EXTENSIONS: &[&str] = &[
    // ... existing extensions ...
    ".scala", ".sc",
];

Step 6: Add a test

Add a test in crates/sem-core/src/parser/plugins/code/mod.rs:

rust

#[test]
fn test_scala_entity_extraction() {
    let code = r#"
class UserService {
  def getUsers(): List[User] = {
    db.findAll()
  }
}
object AppConfig {
  val version = "1.0"
}
trait Repository[T] {
  def findById(id: String): Option[T]
}
"#;
    let plugin = CodeParserPlugin;
    let entities = plugin.extract_entities(code, "UserService.scala");
    let names: Vec<&str> = entities.iter().map(|e| e.name.as_str()).collect();
    eprintln!("Scala entities: {:?}", entities.iter().map(|e| (&e.name, &e.entity_type)).collect::<Vec<_>>());    assert!(names.contains(&"UserService"), "got: {:?}", names);
    assert!(names.contains(&"AppConfig"), "got: {:?}", names);
    assert!(names.contains(&"Repository"), "got: {:?}", names);
}

Step 7: Run tests

bash

cd crates && cargo test

All existing tests plus your new test should pass.

LanguageConfig Fields

`entity_node_types`

The tree-sitter AST node types that represent top-level code entities. These are the things sem tracks: functions, classes, interfaces, etc.

For most languages, this is all you need. Examples:

| Language | Node types |
|----------|-----------|
| Python | function_definition, class_definition, decorated_definition |
| Rust | function_item, struct_item, enum_item, impl_item, trait_item |
| Go | function_declaration, method_declaration, type_declaration |

`container_node_types`

AST nodes that can contain nested entities. When sem finds a container, it looks inside for child entities and sets up parent-child relationships.

For example, in Java a class_body contains method declarations. Setting container_node_types: &["class_body"] lets sem extract methods as children of the class.

Common containers: block, class_body, declaration_list, compound_statement.

`call_entity_identifiers`

For languages where entities are defined via function calls rather than syntax. Elixir is the primary example:

elixir

defmodule MyApp do    # "defmodule" is a call, not a keyword
  def greet(name) do  # "def" is a call
    "Hello #{name}"
  end
end

Set entity_node_types to &[] and list the call identifiers instead:

rust

call_entity_identifiers: &["defmodule", "def", "defp", "defmacro", ...],

Most languages don't need this. Leave it as &[].

`suppressed_nested_entities`

Prevents double-extraction when a child entity type shouldn't be extracted inside a parent entity type. Used by HCL to suppress nested attribute nodes inside block nodes (since the block already captures that content).

rust

suppressed_nested_entities: &[SuppressedNestedEntity {
    parent_entity_node_type: "block",
    child_entity_node_type: "attribute",
}],

Most languages don't need this. Leave it as &[].

Finding the Right Node Types

The hardest part is figuring out which AST node types your language uses. Here's how:

Option 1: Tree-sitter Playground

Go to tree-sitter.github.io/tree-sitter/playground. Paste some sample code and look at the parse tree. The node type names in the tree are exactly what you put in entity_node_types.

Option 2: Check the grammar repo

Every tree-sitter grammar has a grammar.js or src/node-types.json in its repo. Search for the node types you need. The GitHub repos are usually at tree-sitter/tree-sitter-{lang} or tree-sitter-grammars/tree-sitter-{lang}.

Option 3: Use `tree-sitter parse`

If you have the tree-sitter CLI installed:

bash

tree-sitter parse sample.scala

This prints the full AST with node types.

Tips

- Start with the obvious ones: function_definition, class_definition, etc.
- Use eprintln! in your test to see what entities are extracted. The existing tests all do this.
- If something isn't extracted, the node type name is probably different. Check the AST.
- If too many things are extracted, you may be including container nodes or low-level syntax.

Questions?

Open an issue or check the existing language configs in languages.rs for reference. The simplest configs (Python, Bash) are good starting points. The Elixir config shows the call-based approach.

Performance Fixtures

For graph-build performance work, use the generated JS/TS fixture benchmark in benchmarks/large-js-fixture/. It can regenerate large public fixtures and measure cold cache, warm cache, and one-file incremental rebuild timings without relying on private repositories.

---

README

Part of the Ataraxy Labs stack — agent-native infrastructure for software development. See also: weave (entity-level git merge driver) · inspect (semantic code review) · opensessions (tmux sidebar for coding agents).

> Read the manifesto: https://ataraxy-labs.com/#thesis · Essays: https://ataraxy-labs.com/blogs · LLMs: https://ataraxy-labs.com/llms.txt

Semantic version control built on Git. 
Instead of lines changed, sem tells you what entities changed: functions, methods, classes.

<a href="https://ataraxy-labs.com/blogs/code-is-not-text">Why sem?</a> ·
<a href="#install">Install</a> ·
<a href="#commands">Commands</a> ·
<a href="#use-with-ai-agents-mcp">Agents (MCP)</a> ·
<a href="docs/cloud-consent.html">Cloud consent</a> ·
<a href="https://github.com/Ataraxy-Labs/sem/releases/latest">Releases</a>

sem is a semantic version control tool that works on top of Git. It parses your code with tree-sitter, extracts every function, class, and method as an entity, and diffs at the entity level instead of lines. This means you see "function blahh was modified" instead of "lines x-y changed."

It works in any Git repo with no setup.

Cloud-backed queries are opt-in per repo: logging in does not upload a repo or send a query. See the cloud consent flow for the public/private repo states, preview screen, local audit log, and forget controls.

Install

bash

curl -fsSL https://raw.githubusercontent.com/Ataraxy-Labs/sem/main/install.sh | sh

Or via Homebrew:

bash

brew install sem-cli

Or via winget on Windows:

powershell

winget install AtaraxyLabs.sem

Or install the npm wrapper into node_modules:

bash

npm install --save-dev @ataraxy-labs/sem

With Bun, trust the package so its postinstall script can download the binary:

bash

bun add -d @ataraxy-labs/sem
bun pm trust @ataraxy-labs/sem

Once installed, update to the latest release any time:

bash

sem update

Or build from source (requires Rust):

bash

cargo install --git https://github.com/Ataraxy-Labs/sem sem-cli

Or grab a binary from GitHub Releases.

Or run via Docker:

bash

docker build -t sem .
docker run --rm -it -u "$(id -u):$(id -g)" -v "$(pwd):/repo" sem diff

Name conflict with GNU Parallel

GNU Parallel ships a sem binary (/usr/bin/sem) as a symlink to parallel. If you have both installed, they'll collide. Run sem --version to check which one you're using. (#77)

Quick fixes:

bash

Option 1: alias in your shell profile (~/.bashrc, ~/.zshrc)

alias sem="$HOME/.cargo/bin/sem"
Option 2: make sure cargo bin comes first in PATH

export PATH="$HOME/.cargo/bin:$PATH"
Option 3: if installed via Homebrew

export PATH="$(brew --prefix)/bin:$PATH"

If you installed via npm/bun, the binary lives in node_modules/.bin/sem and is invoked through npx sem or bunx sem, which avoids the conflict entirely.

Commands

Works in any Git repo. No setup required. Also works outside Git for arbitrary file comparison.

sem stores its SQLite entity cache outside the repository, under the OS cache directory by default. Set SEM_CACHE_DIR=/path/to/cache to override the cache root; repo-local overrides are ignored so cache files do not dirty the working tree.

sem diff

Entity-level diff with rename detection, structural hashing, and word-level inline highlights.

bash

Semantic diff of working changes

sem diff
Staged changes only

sem diff --staged
Specific commit

sem diff --commit abc1234
Commit range

sem diff --from HEAD~5 --to HEAD
Verbose mode (word-level inline diffs for each entity)

sem diff -v
Plain text output (git status style)

sem diff --format plain
JSON output (for AI agents, CI pipelines)

sem diff --format json
Markdown output (for PRs, reports)

sem diff --format markdown
Compare any two files (no git repo needed)

sem diff file1.ts file2.ts
Read file changes from stdin (no git repo needed)

echo '[{"filePath":"src/main.rs","status":"modified","beforeContent":"...","afterContent":"..."}]' \
  | sem diff --stdin --format json
Only specific file types

sem diff --file-exts .py .rs

sem impact

Cross-file dependency graph shows what breaks if an entity changes.

bash

Full impact analysis

sem impact authenticateUser
Direct dependencies only

sem impact authenticateUser --deps
Direct dependents only

sem impact authenticateUser --dependents
Affected tests only

sem impact authenticateUser --tests
JSON output

sem impact authenticateUser --json
Disambiguate by file

sem impact authenticateUser --file src/auth.ts
Include default-excluded paths such as generated, fixture, vendor, benchmark, and build trees

sem impact authenticateUser --no-default-excludes

sem blame

Entity-level blame showing who last modified each function, class, or method.

bash

sem blame src/auth.ts
JSON output

sem blame src/auth.ts --json

sem log

Track how a single entity evolved through git history.

bash

sem log authenticateUser
Verbose mode (show content diff between versions)

sem log authenticateUser -v
Limit commits scanned

sem log authenticateUser --limit 20
JSON output

sem log authenticateUser --json

With no entity, sem log analyzes recent repo history at the entity level:
hotspots (most-changed functions/classes, with author counts) and
co-change pairs (entities that repeatedly change in the same commits —
"if you touch one, don't forget the other"):

bash

sem log                 # repo hotspots + co-change pairs (last 50 commits)
sem log --limit 200     # deeper history
sem log --file src/auth.ts   # scoped to one file
sem log --json          # full data

sem entities

List all entities under a file or directory path. No path is the same as ..

bash

sem entities
sem entities .
sem entities src/auth.ts
JSON output

sem entities --json
sem entities src/auth.ts --json
Include default-excluded paths such as generated, fixture, vendor, benchmark, and build trees

sem entities --no-default-excludes

sem context

Token-budgeted context for LLMs: the entity, its dependencies, and its dependents, fitted to a strict content token budget.
When the target signature itself does not fit, JSON output reports target_omitted: true.

bash

sem context authenticateUser
Custom token budget

sem context authenticateUser --budget 4000
JSON output

sem context authenticateUser --json
Include default-excluded paths such as generated, fixture, vendor, benchmark, and build trees

sem context authenticateUser --no-default-excludes

Use as default Git diff

Replace git diff output with entity-level diffs. Agents and humans get sem output automatically without changing any commands.

bash

sem setup

Now git diff shows entity-level changes instead of line-level. No prompts, no agent configuration needed. Everything that calls git diff gets sem output automatically. Also installs a pre-commit hook that shows entity-level blast radius of staged changes.

On macOS and Linux, sem setup also wires sem into your Claude Code sessions (free, local, no login): a warm resident graph so structural queries answer in single-digit milliseconds instead of rebuilding each time, and prompt-time context so the code an agent would otherwise forage for arrives at the start of the turn. It edits ~/.claude/settings.json idempotently, backs it up first, and leaves any hooks you already have untouched.

To disable and go back to normal git diff (also removes the session hooks):

bash

sem unsetup

Entity-level diffs on every pull request

Add the GitHub Action and every PR gets one sticky comment showing which
functions, classes, and methods changed — updated in place on each push, and
calling out cosmetic-only PRs (formatting/comments) explicitly:

yaml

.github/workflows/entity-diff.yml

name: Entity diff
on: pull_request
permissions:
  contents: read
  pull-requests: write
jobs:
  entity-diff:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: Ataraxy-Labs/sem/[email protected]

No config, no API keys, never fails your build. See action/ for details.

Cloud acceleration (for scale and teams)

Local is always free and, after sem setup, always warm — the resident graph keeps your repo hot on your own machine, so day-to-day queries are instant with no login. You do not pay to make your laptop fast.

Cloud is for what a laptop can't do. On a very large monorepo the first local graph build can take a few seconds; a shared team graph shouldn't be rebuilt per developer; and CI wants the graph without checking anything out. sem login connects those cases to sem cloud, which keeps a warm, pre-built graph for your registered repos and serves the heavy queries from it (on a large repo like deno, an impact query is ~86ms from the cloud vs ~573ms rebuilt locally).

bash

sem login                              # GitHub device flow, one time
sem impact myFunc --file src/foo.rs    # served from the cloud's warm graph

It is fully optional and transparent:

- Not logged in, or the cloud is unreachable? sem computes locally and prints the exact same output. No failures, no difference in results.
- SEM_LOCAL=1 forces local computation even when logged in.
- Small repos see no change, local is already fast. The win is for large codebases where rebuilding the graph each time is the bottleneck.

What it parses

32 programming languages with full entity extraction via tree-sitter:

| Language | Extensions | Entities |
|----------|-----------|----------|
| TypeScript | .ts .tsx .mts .cts | functions, classes, interfaces, types, enums, exports |
| JavaScript | .js .jsx .mjs .cjs | functions, classes, variables, exports |
| Python | .py | functions, classes, decorated definitions |
| Go | .go | functions, methods, types, vars, consts |
| Rust | .rs | functions, structs, enums, impls, traits, mods, consts |
| Java | .java | classes, methods, interfaces, enums, fields, constructors |
| C | .c .h | functions, structs, enums, unions, typedefs |
| C++ | .cpp .cc .hpp | functions, classes, structs, enums, namespaces, templates |
| C# | .cs | classes, methods, interfaces, enums, structs, properties |
| Ruby | .rb | methods, classes, modules |
| PHP | .php | functions, classes, methods, interfaces, traits, enums |
| Swift | .swift | functions, classes, protocols, structs, enums, properties |
| Elixir | .ex .exs | modules, functions, macros, guards, protocols |
| Bash | .sh | functions |
| Fish | .fish | functions |
| Lua | .lua | functions (global, local, table, and method forms) |
| HCL/Terraform | .hcl .tf .tfvars | blocks, attributes (qualified names for nested blocks) |
| Kotlin | .kt .kts | classes, interfaces, objects, functions, properties, companion objects |
| Fortran | .f90 .f95 .f | functions, subroutines, modules, programs |
| Vue | .vue | template/script/style blocks + inner TS/JS entities |
| XML | .xml .plist .svg .csproj | elements (nested, tag-name identity) |
| ERB | .erb .html.erb | blocks, expressions, code tags |
| Svelte | .svelte .svelte.js .svelte.ts | component blocks + rune JS/TS modules |
| Perl | .pl .pm .t | subroutines, packages |
| Dart | .dart | classes, mixins, extensions, enums, type aliases, functions |
| OCaml | .ml .mli | values, modules, types, classes, externals |
| Scala | .scala .sc .sbt | classes, objects, traits, enums, functions, vals, extensions |
| Nix | .nix | bindings, inherit declarations |
| Haskell | .hs | functions, signatures, data types, newtypes, classes, instances, type synonyms |
| Elm | .elm | value declarations, type aliases, type declarations, port annotations, infix declarations |
| Clojure | .clj .cljs .cljc | vars, functions, macros, multimethods, protocols, records, types |
| D | .d .di | modules, functions, classes, structs, interfaces, unions, enums, templates, aliases, unittests |
| Zig | .zig | functions, tests, variables |
| SQL | .sql .psql .pgsql .ddl | tables, views, functions, indexes, types, schemas, triggers, sequences |

Plus structured data formats:

| Format | Extensions | Entities |
|--------|-----------|----------|
| JSON | .json | properties, objects (RFC 6901 paths) |
| YAML | .yml .yaml | sections, properties (dot paths) |
| TOML | .toml | sections, properties |
| EDN | .edn | top-level map entries (keyword keys) |
| CSV | .csv .tsv | rows (first column as identity) |
| Markdown | .md .mdx | heading-based sections |

Everything else falls back to chunk-based diffing.

Custom extensions and extensionless files

For files with non-standard extensions, create a .semrc in your project root:

text

.xyz = cpp
.j = json
.mypy = python

sem also reads .gitattributes patterns (diff= and linguist-language=) if you already have those set up. .semrc takes priority when both define the same extension.

For files with no extension at all, sem detects the language automatically from content (imports, declarations, shebang lines, vim modelines). This covers 19 languages with no config needed.

How matching works

Three-phase entity matching:

1. Exact ID match — same entity in before/after = modified or unchanged
2. Structural hash match — same AST structure, different name = renamed or moved (ignores whitespace/comments)
3. Fuzzy similarity — >80% token overlap = probable rename

This means sem detects renames and moves, not just additions and deletions. Structural hashing also distinguishes cosmetic changes (whitespace, formatting) from real logic changes.

Use with AI agents (MCP)

sem mcp starts a Model Context Protocol server over stdin/stdout. It's not a command you run and read yourself: it's a server your coding agent launches in the background so it can ask sem questions while it works. That's the reason mcp lives alongside the normal commands. The agent gets 6 tools, all entity-level: sem_impact, sem_context, sem_diff, sem_entities, sem_blame, sem_log.

Why an agent wants these: instead of reading whole files and burning tokens, it can ask "what breaks if I change submitOrder" (sem_impact) or "give me just the context to refactor this function" (sem_context) and get a precise answer from the dependency graph.

Add it once, then talk to your agent normally. It calls the tools on its own.

Claude Code:

bash

claude mcp add sem -- sem mcp

Or one command that also installs the skill, so the agent knows when to reach for sem:

bash

npx @ataraxy-labs/sem-skill

Cursor, Claude Desktop, or any client with an mcpServers config:

json

{
  "mcpServers": {
    "sem": {
      "command": "sem",
      "args": ["mcp"]
    }
  }
}

If sem isn't on the agent's PATH, use the absolute path to the binary. No separate install is needed: sem mcp ships in the same binary as every other command.

JSON output

bash

sem diff --format json

json

{
  "summary": {
    "fileCount": 2,
    "added": 1,
    "modified": 1,
    "deleted": 1,
    "moved": 0,
    "renamed": 0,
    "reordered": 0,
    "binary": 0,
    "orphan": 0,
    "total": 3
  },
  "changes": [
    {
      "entityId": "src/auth.ts::function::validateToken",
      "changeType": "added",
      "entityType": "function",
      "entityName": "validateToken",
      "startLine": 12,
      "endLine": 18,
      "oldStartLine": null,
      "oldEndLine": null,
      "filePath": "src/auth.ts"
    }
  ],
  "binaryChanges": []
}

The named change-type buckets (added, modified, deleted, moved, renamed, reordered) always sum to total. orphan is a cross-cutting metadata count for module-level changes, and those changes are already included in the named change-type buckets.

As a library

sem-core can be used as a Rust library dependency:

toml

[dependencies]
sem-core = { git = "https://github.com/Ataraxy-Labs/sem", version = "0.5" }

Used by weave (semantic merge driver) and inspect (entity-level code review).

Architecture

- tree-sitter for code parsing (native Rust, not WASM)
- git2 for Git operations
- rayon for parallel file processing
- xxhash for structural hashing
- Plugin system for adding new languages and formats

Telemetry

sem collects anonymous usage data: the command name (e.g. diff, impact), CLI version, and operating system. Nothing else — no code, file paths, repo names, or user identity. Events are batched locally and sent in the background, so commands never wait on the network.

Disable it any time:

bash

export SEM_NO_TELEMETRY=1   # or DO_NOT_TRACK=1

Contributing

Want to add a new language? See CONTRIBUTING.md for a step-by-step guide.

Star History

[](https://star-history.com/#Ataraxy-Labs/sem&Date)

License

MIT OR Apache-2.0

---

Attention Architecture Copy Section

The context window is a cache. Treat it like one.

The observation nobody builds on Copy Chapter

The architecture Copy Chapter

Why this is agent-native and not human-native Copy Chapter

How it completes the storage engine Copy Chapter

Implementation sketch (v1 is small) Copy Chapter

Llms Copy Section

sem

Overview Copy Chapter

Install Copy Chapter

Commands Copy Chapter

sem diff Copy Section

sem blame Copy Section

sem graph Copy Section

sem impact Copy Section

Key Features Copy Chapter

Language Support Copy Chapter

JSON Output Copy Chapter

Architecture Copy Chapter

Performance Copy Chapter

As a Library Copy Chapter

Links Copy Chapter

CHANGELOG Copy Section

Changelog

[Unreleased] Copy Chapter

Fixed Copy Section

[0.21.0] - 2026-07-10 Copy Chapter

Added Copy Section

Fixed Copy Section

Performance Copy Section

[0.20.0] - 2026-07-05 Copy Chapter

Changed Copy Section

Added Copy Section

[0.19.0] - 2026-07-05 Copy Chapter

Removed Copy Section

Fixed Copy Section

Added Copy Section

Documentation Copy Section

[0.18.0] - 2026-07-03 Copy Chapter

Fixed Copy Section

Added Copy Section

Performance Copy Section

Fixed Copy Section

Changed Copy Section

Added Copy Section

Documentation Copy Section

Performance Copy Section

Added Copy Section

Performance Copy Section

Added Copy Section

Added Copy Section

Added Copy Section

Performance Copy Section

Removed Copy Section

Added Copy Section

Fixed Copy Section

Added Copy Section

Changed Copy Section

[0.15.1] - 2026-07-01 Copy Chapter

Added Copy Section

Fixed Copy Section

Performance Copy Section

[0.15.0] - 2026-06-30 Copy Chapter

Changed Copy Section

Added Copy Section

Performance Copy Section

Documentation Copy Section

Added Copy Section

Added Copy Section

Added Copy Section

Added Copy Section

Changed Copy Section

[0.14.1] - 2026-06-23 Copy Chapter

Fixed Copy Section

[0.14.0] - 2026-06-23 Copy Chapter

Added Copy Section

Fixed Copy Section

[0.13.1] - 2026-06-23 Copy Chapter

Added Copy Section

Attention Architecture

The observation nobody builds on

The architecture

Why this is agent-native and not human-native

How it completes the storage engine

Implementation sketch (v1 is small)

Llms

Overview

Install

Commands

sem diff

sem blame

sem graph

sem impact

Key Features

Language Support

JSON Output

Architecture

Performance

As a Library

Links

CHANGELOG

[Unreleased]

Fixed

[0.21.0] - 2026-07-10

Added

Fixed

Performance

[0.20.0] - 2026-07-05

Changed

Added

[0.19.0] - 2026-07-05

Removed

Fixed

Added

Documentation

[0.18.0] - 2026-07-03

Fixed

Added

Performance

Fixed

Changed

Added

Documentation

Performance

Added

Performance

Added

Added

Added

Performance

Removed

Added

Fixed

Added

Changed

[0.15.1] - 2026-07-01

Added

Fixed

Performance

[0.15.0] - 2026-06-30

Changed

Added

Performance

Documentation

Added

Added

Added

Added

Changed

[0.14.1] - 2026-06-23

Fixed

[0.14.0] - 2026-06-23

Added

Fixed

[0.13.1] - 2026-06-23

Added

Changed

Fixed

[0.13.0] - 2026-06-16