", ...] }`. Larger selections should use a filter.
A filter must contain at least one condition, and percentile conditions (`gtePercentile`) are not accepted, because they select relative to the data rather than a stable set of traces.
## Related
- [PII redaction](./pii-redaction) keeps sensitive values out of Latitude in the first place.
- [GDPR](./compliance/gdpr) covers Latitude's data protection approach.
---
### Signals Create (`docs/signals/create.mdx`)
---
title: Create a signal
description: Define a signal yourself when you already know the behavior you want to track, using the New signal builder.
---
Signals usually come from [discovery](./overview): Latitude groups similar failed scores into named patterns without you defining them first. You can also create a signal yourself when you already know the behavior you want to track. This page covers that path.
Discovery finds what you didn't know to look for. Creating a signal manually tracks what you already know matters: a specific failure mode, a policy you need to enforce, or a behavior you want measured from day one.
## When to create one manually
- You already know a failure mode and want it tracked and counted from now on.
- You want to enforce a known requirement, such as a format, a policy, or task success.
- The behavior is specific enough that you can describe it or write a rule for it.
If you are exploring rather than tracking something specific, start with [Search](../search/overview) and [Behaviors](../behaviours/overview) instead, and let discovery surface patterns.
## Open the builder
On the Signals page, select **New signal**. The builder opens on a short intro with two ways forward:
- **Generate signal**: describe the behavior and let Latitude build the whole signal.
- **Configure manually**: build it yourself step by step.
{/* TODO: add screenshot of the New signal builder intro (describe-first screen). */}
Both paths end the same way: a signal with a detector that checks every new matching session and adds the ones that exhibit the behavior. That detector is an [evaluation](../evaluations/overview).
## Describe it and let Latitude build it
In the "What do you want to track?" field, describe the behavior in plain language. For example:
> Sessions where the ticket-cancellation tool fails and the user gets frustrated.
Select **Generate signal**. Latitude reads your description, drafts a detector, chooses a scope and sampling rate, tests it against recent sessions, names it, and creates it. When it finishes, you land on the new signal's detail page, where you can review it and edit anything.
This is a good starting point even when you plan to adjust the result. You can reopen the builder from the signal to change the detector, scope, or sampling. AI generation is rate limited, so if you generate several in a row you may need to wait before the next one.
## Configure it manually
Choosing **Configure manually** runs a four-step builder.
Pick how Latitude decides whether a session matches, using one of three [detection methods](../evaluations/detection-methods): a set of conditions, an LLM judge, or a custom script. This check runs automatically on every new matching session, and a session that passes joins the signal.
Choose which sessions get checked and how many of them. See [Scope](#scope) below.
Try the detector on recent sessions before saving. See [Test](#test) below.
Give the signal a **name** and a **description**. Both are required. The name shows up in the signals list, so pick something your team will recognize.
### Scope
By default a detector runs on every session in your project. Narrow it with filters on **Tags**, **Services**, **Models**, **Providers**, or **Metadata**. With filters set, only matching sessions run through the detector and everything else is ignored.
The sampling slider controls how many of those sessions get checked, from 0 to 100 percent. It defaults to 10 percent.
- A set of conditions is free and instant, so checking 100 percent is usually fine.
- An LLM judge or a script that calls an LLM costs money and time per check. On high traffic, sampling a slice still catches the pattern for much less.
- Setting sampling to 0 pauses the signal: no sessions are checked.
See [Triggers](../evaluations/triggers) for more on scope and sampling.
### Test
The Test step runs your detector against recent sessions from your project and shows how it scored each one. Nothing is saved. Each session gets a verdict: **match**, **no match**, **skipped** (for example, not embedded yet), or **errored**. If the verdicts look off, go back, adjust the detector or scope, and run the preview again.
## After you create a signal
A new signal starts empty and collects matches from new sessions onward. It does not scan your history.
From there it behaves like any signal. It appears in your Signals list, its occurrences and trends build up as matching sessions arrive, and you can triage, mute, monitor, or delete it from its detail page. See [Signal management](./management).
You can edit the detector, scope, and sampling of a signal you created at any time, and switch detection methods freely. Edits apply going forward. A detector you define runs exactly as you wrote it. Unlike a detector Latitude generates from a discovered signal, it is not automatically realigned to human annotations, so it keeps doing what you specified. See [Alignment](../evaluations/alignment).
## Related pages
- [Detection methods](../evaluations/detection-methods): the three ways to define a detector
- [Custom scripts](../evaluations/custom-scripts): the scripting reference
- [Signals overview](./overview): how discovery finds signals for you
- [Signal management](./management): triage, monitor, mute, and manage signals
---
### Signals Management (`docs/signals/management.md`)
---
title: Signal management
description: Triage signals, monitor them with evaluations, resolve what you fix, ignore noise, and keep your signal list focused.
---
# Signal management
Once a signal exists, whether Latitude [discovered](./overview) it or you [created](./create) it, you work with it from its detail page. This page covers its states and the actions you can take.
## Signal states
A signal carries one or more states:
- New: discovered or created in the last 7 days.
- Escalating: occurrences are rising faster than the normal pattern for this time of week. Escalation is detected automatically and opens an incident.
- Ongoing: the steady state, once no other state applies.
- Resolved: you marked the problem as fixed. The signal is archived but keeps watching for recurrences.
- Regressed: a resolved signal started occurring again and was reopened.
- Ignored: you marked the signal as noise. It is archived, monitoring stops, and notifications are muted.
A signal can carry several states at once, for example New and Escalating. Separately, a signal with an active evaluation shows an Evaluated marker.
The Signals page has two tabs, Active and Archived. Resolved and ignored signals live in Archived; everything else is Active.
## Triage
When a signal appears:
1. Read its description to understand the pattern.
2. Open example sessions to see where it happened.
3. Judge how much it matters: a safety risk, a quality problem, or a rare edge case.
4. Act on it: assign an owner, set a priority, start monitoring, resolve it once fixed, ignore it as noise, or delete it.
Assign a signal to an org member and set its priority (Urgent, High, Medium, Low, or none) so the right person picks up the right thing first. You can filter the list by assignee.
## Monitor a signal
Monitoring attaches an evaluation that scores new sessions for the same behavior. For a discovered signal, open it and choose Generate an evaluation: Latitude builds a detector from the signal's examples, annotations, and scores, then keeps it aligned to human judgment. A signal you created already has the detector you defined.
From the signal you can:
- Change the sampling rate to trade coverage for cost.
- Realign a generated evaluation after adding annotations.
- Remove the evaluation to stop scoring new sessions.
Unresolve or unignore a signal before you generate an evaluation for it — archived signals can't start monitoring.
See [Evaluations](../evaluations/overview) for how detectors work, and [Create a signal](./create) to define one yourself.
## Investigate
For a signal worth digging into, review several example sessions and ask:
- What user inputs lead to it?
- Is the agent consistently wrong, or is the failure intermittent?
- Are there shared patterns in context, tools, retrieval, model, or prompt behavior?
- If an evaluation is attached, is it too strict, too lenient, or drifting from human review?
## Resolve and ignore
Resolve a signal once you've fixed the underlying problem. Resolving moves it to the Archived tab and, by default, keeps its evaluations running so Latitude notices if the problem comes back. Turn off "Keep evaluating" in the resolve dialog if you also want monitoring to stop; the default for that switch is a project setting.
Ignore a signal that isn't worth acting on. Ignoring moves it to Archived, archives its evaluations so monitoring stops, and mutes its notifications. Occurrences that Latitude discovers from annotations still attach to the ignored signal, so the noise keeps flowing into one place instead of spawning new signals.
Both actions are reversible: unresolve reopens a signal without marking it as regressed, and unignore returns it to the active list with notifications re-enabled.
## Mute notifications
Muting is a pure notification switch, separate from resolving and ignoring. A muted signal keeps collecting occurrences and still opens incidents when it escalates — you just don't get notified. Use mute when you want to keep watching a signal in the list without the pings. Ignoring a signal mutes it automatically; resolving or unignoring one re-enables its notifications.
## Delete a signal you created
A signal you created can be renamed or deleted from its detail page. Deleting also archives its evaluation and can't be undone. Existing scores stay in analytics. Discovered signals can't be deleted; ignore them instead.
## Catching regressions
After you ship a fix, resolve the signal and leave "Keep evaluating" on. Latitude keeps scoring new sessions; the first occurrence after the resolve reopens the signal, marks it as Regressed, and notifies you (the assignee when one is set, otherwise the project's members). You don't have to watch for it by hand. If [agent dispatch](/agent-dispatch/overview) is enabled with the "Regressed signal" trigger, Latitude also wakes your coding agent to investigate the regression.
## Keep signals manageable
As your project matures:
- Assign and prioritize signals so the important ones stand out.
- Monitor the signals you need ongoing coverage for.
- Resolve fixed problems and ignore noise so real problems stay visible.
- Keep descriptions clear enough for the next teammate to understand.
## Next steps
- [Signals overview](./overview): how discovery finds signals
- [Create a signal](./create): define one yourself
- [Monitors](../monitors/overview): watch saved searches, tools, users, and sessions
- [Evaluations](../evaluations/overview): monitor a signal on live traffic
---
### Snippets First Artifact (`docs/snippets/first-artifact.mdx`)
Once a real run has landed, your agent builds your first [Artifact](/more/artifacts): a single HTML page, in the Latitude look, with everything the telemetry captured from that session: model calls, tool calls, tokens, cost, timing, and the conversation as the model saw it. It is the fastest way to check the integration end to end and to see what Latitude will have to work with.
The `latitude-setup` skill does this as its last step from its bundled `first-artifact.html` template, filling the page with the values the `latitude` CLI returns for the trace, and adds a **Claim your workspace** button when the account is temporary. If you set things up by hand, the same template and instructions live in the [skills repo](https://github.com/latitude-dev/skills/tree/main/skills/latitude-setup). Prompt, if you need to ask for it:
```text
Build my first Latitude Artifact from the session that just landed, using the first-artifact.html template from the latitude-setup skill (github.com/latitude-dev/skills).
```
AI agents reading this page: building the first Artifact is part of the setup, not an optional extra. Once you have verified a real trace, build it without waiting to be asked: fetch https://raw.githubusercontent.com/latitude-dev/skills/main/skills/latitude-setup/first-artifact.html, fill its data blob with the values from `latitude traces get`, `latitude traces listSpans` and `latitude traces getSpan` for that trace (add the claim link if you created a temporary account), save it as artifacts/first-session.html, and put its absolute path in your final message. The setup is not finished until the user has that file.
---
### Snippets Skills Callout (`docs/snippets/skills-callout.mdx`)
**Using an agent?** Install the [Latitude skills](/getting-started/skills) and let it handle the setup below. `latitude-setup` instruments your app or agent harness, verifies traces arrive, creates a temporary account if you don't have one yet (no signup), and ends by building your first [Artifact](/more/artifacts).
```text
Install the `latitude-setup` skill from `github.com/latitude-dev/skills` and use it to set up Latitude tracing here.
```
AI agents reading this page: install and follow `latitude-setup` before doing the steps below by hand.
---
### Telemetry Claude Code (`docs/telemetry/claude-code.md`)
# Claude Code telemetry
Stream your Claude Code session transcripts: prompts, model responses, tool calls, and tool results into Latitude as OpenTelemetry traces. Once installed, every Claude Code turn shows up in your project's **Traces** view.
## Prerequisites
- A free [Latitude account](https://console.latitude.so/login) with at least one project
- Claude Code installed (CLI, desktop app, or IDE extension)
- Node.js available on your `PATH` (the hook runs via `npx`)
> The package supports the Claude Code CLI, the macOS/Windows desktop app, and the VS Code/JetBrains extensions. It does **not** work with the web app at `claude.ai/code`, since that runs in Anthropic's cloud where local hooks can't execute.
## Step 1: Get your Latitude API key and project slug
You'll need both before running the installer.
**Project slug**: open your project in Latitude. The slug appears in the left sidebar, directly under the project name (for example, `claude-code`). Click it to copy.
**API key**: go to **Settings → API Keys** (top-right of the Latitude app). Use an existing key or click **Create API Key**. Copy the full key value.
## Step 2: Run the installer
In a terminal, run:
```bash
npx -y @latitude-data/claude-code-telemetry install
```
The installer will:
1. Prompt for your API key and project slug (paste the values from Step 1).
2. Edit `~/.claude/settings.json` to (a) store your API key and project slug, and (b) tell Claude Code to run the telemetry script after every turn.
3. On macOS, register the environment variables with `launchctl` so the desktop app and IDE extensions inherit them.
If you'd rather pass values as flags instead of being prompted:
```bash
npx -y @latitude-data/claude-code-telemetry install \
--api-key=lat_xxx \
--project=your-project-slug \
--yes
```
See [Install flags](#install-flags) below for all available flags.
## Step 3: Restart Claude Code
**Fully quit and relaunch** Claude Code.
- **CLI:** close the terminal session and start a new one.
- **Desktop app:** quit from the menu bar (⌘Q on macOS), then reopen.
- **VS Code / JetBrains:** fully quit the IDE, then reopen.
## Step 4: Verify traces are arriving
Run any prompt in Claude Code. Then open your Latitude project and go to **Traces**. Within a few seconds you should see the new trace there.
If nothing shows up, see [Troubleshooting](#troubleshooting) below.
## Disabling temporarily
To pause telemetry without uninstalling, set the environment variable:
```bash
export LATITUDE_CLAUDE_CODE_ENABLED=0
```
Restart Claude Code for the change to take effect. Set it back to `1` (or unset it) to re-enable.
## Uninstalling
To remove the hook and revert all changes the installer made:
```bash
npx -y @latitude-data/claude-code-telemetry uninstall
```
The uninstaller asks for confirmation before reverting `~/.claude/settings.json` and (on macOS) the `launchctl` entries.
## Manual configuration
If you'd rather not run the installer (for example, you manage `~/.claude/settings.json` in a dotfiles repo) you can wire it up by hand.
Add the following to `~/.claude/settings.json`:
```json
{
"env": {
"LATITUDE_API_KEY": "lat_xxx",
"LATITUDE_PROJECT": "your-project-slug"
},
"hooks": {
"Stop": [
{
"hooks": [
{
"type": "command",
"command": "npx -y @latitude-data/claude-code-telemetry",
"async": true
}
]
}
]
}
}
```
Restart Claude Code after saving. Note that the desktop app and IDE extensions on macOS won't inherit shell-level environment variables, so the `env` block above is required (or you'll need to set them via `launchctl` yourself).
## Reference
### CLI commands
The package exposes a single binary, `latitude-claude-code`, invoked via `npx`. It has two subcommands:
- `install` — configures `~/.claude/settings.json` and (on macOS) `launchctl`.
- `uninstall` — reverts every change `install` made, after confirmation.
### Install flags
| Flag | Purpose |
|---|---|
| `--api-key=` | Supply the Latitude API key non-interactively. |
| `--project=` | Supply the Latitude project slug non-interactively. |
| `--staging` | Point the hook at the Latitude staging environment instead of production. |
| `--dev` | Point the hook at `localhost` (for Latitude internal development). |
| `--no-launchctl` | Skip the macOS `launchctl` setup. Use this if you manage environment variables yourself. |
| `--yes`, `--no-prompt` | Skip all interactive prompts. Useful in CI or scripted setups. |
### Environment variables
| Variable | Required | Default | Purpose |
|---|---|---|---|
| `LATITUDE_API_KEY` | Yes | — | Bearer token used to authenticate trace uploads to Latitude. |
| `LATITUDE_PROJECT` | Yes | — | Slug of the Latitude project that should receive the traces. |
| `LATITUDE_BASE_URL` | No | `https://ingest.latitude.so` | Override the ingestion endpoint. Set automatically by `--staging` and `--dev`. |
| `LATITUDE_CLAUDE_CODE_ENABLED` | No | `1` | Set to `0` to disable the hook without uninstalling. |
| `LATITUDE_DEBUG` | No | unset | Set to `1` to print diagnostic output to stderr when the hook runs. |
| `BUN_OPTIONS` | Conditional | — | Preload path used by the optional Bun-based request capture. The installer manages this; do not set it by hand. |
> ⚠️ If `BUN_OPTIONS` points to a missing file, Claude Code itself will fail to start. Either keep the file in place or unset the variable.
### Files modified by the installer
- **`~/.claude/settings.json`** — adds an `env` block (or merges into an existing one) and a `Stop` entry under `hooks`. Existing settings are preserved.
- **macOS `launchctl` user environment** — `LATITUDE_API_KEY`, `LATITUDE_PROJECT`, and (when relevant) `BUN_OPTIONS` are registered via `launchctl setenv` so that GUI apps inherit them. Skipped with `--no-launchctl`.
The installer does not modify your shell rc files (`.zshrc`, `.bashrc`).
## How it works
The package is built around a Claude Code **Stop hook**: a shell command that Claude Code runs automatically every time a turn ends (after the assistant finishes responding and any tool calls have completed). When the hook fires, it reads the new portion of the transcript, reconstructs the turn as OpenTelemetry spans, and POSTs them to Latitude's OTLP ingestion endpoint. There's no daemon and no background process — just a short-lived script that runs once per turn.
Each turn produces one trace with three span types:
- **`interaction`** — the user prompt and turn-level timing.
- **`llm_request`** — each call to Anthropic's API, including model parameters, token usage, the full message history, and (when the optional Bun preload is active) the system prompt and tool schemas.
- **`tool_execution`** — each tool invocation: name, arguments, result, and any error.
The package also ships an optional Bun preload script that wraps `fetch` to capture full request bodies sent to Anthropic. This is what makes system prompts and tool schemas available in the trace. It activates only when Claude Code runs under a Bun-based runtime that respects `BUN_OPTIONS`. When inactive, traces still work — they just won't include those two fields.
The hook is **fail-open**. If anything goes wrong (network error, bad API key, parse error), it logs to stderr (visible with `LATITUDE_DEBUG=1`) and exits cleanly. Claude Code is never blocked by a telemetry failure.
## Anthropic Agent SDK (native OpenTelemetry)
The [Claude Agent SDK](https://code.claude.com/docs/en/agent-sdk/overview) runs the same Claude Code CLI in a child process. That CLI can export **OpenTelemetry** traces, metrics, and logs directly to an OTLP endpoint ([Agent SDK observability](https://code.claude.com/docs/en/agent-sdk/observability)). You do **not** need the Stop hook or `@latitude-data/claude-code-telemetry` when you use this path: configure the CLI’s OTLP exporter to point at Latitude’s ingest URL instead.
1. Create a Latitude **API key** and note your project **slug** (same as the hook flow above).
2. Set the telemetry switches Anthropic documents, including **`CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1`** if you want **traces** (beta), not only metrics or log events.
3. Aim the OTLP **trace** exporter at Latitude and attach auth headers. The wire format is standard OTLP over HTTP; see [OpenTelemetry exporter (OTEL)](https://docs.latitude.so/telemetry/otel-exporter) for the exact URL and header names. In practice you will set variables along the lines of:
- `CLAUDE_CODE_ENABLE_TELEMETRY=1`
- `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1` (required for trace spans such as `claude_code.interaction` and `claude_code.llm_request`)
- `OTEL_TRACES_EXPORTER=otlp` (and optionally metrics/logs exporters if you want those signals too)
- `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://ingest.latitude.so/v1/traces`
- `OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Bearer YOUR_API_KEY,X-Latitude-Project=YOUR_PROJECT_SLUG`
Use `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` unless your deployment requires gRPC. For short-lived agent runs, shorten the export intervals (`OTEL_TRACES_EXPORT_INTERVAL`, and the metric/log counterparts) so batches flush before the process exits, as described in Anthropic’s docs.
4. If you already use the Latitude **TypeScript or Python SDK** in the host app, you can still rely on the Agent SDK’s **native** export for the subprocess: the CLI emits its own OTLP batches. Optionally start a root span in your app before `query()`; Anthropic propagates W3C `TRACEPARENT` into the CLI so agent spans can nest under your trace.
Latitude’s ingest pipeline maps Claude Code’s span names and attributes into the same trace model as the Stop-hook integration (`claude_code.*` spans and `span.type` values such as `llm_request` and `tool`).
## Privacy
You should treat this hook as full-fidelity telemetry. Once installed, Latitude receives the verbatim content of every turn: prompts, responses, tool I/O, system prompts, and tool schemas. Important properties:
- **No per-turn opt-in.** Every turn is captured until the hook is disabled.
- **No redaction.** If you paste a secret into Claude Code, that secret will reach Latitude.
- **Disable globally.** Set `LATITUDE_CLAUDE_CODE_ENABLED=0` to pause capture, or run `uninstall` to remove the hook entirely.
If you're working with sensitive material, disable the hook **before** you start the session.
## Troubleshooting
**No traces appear in Latitude.**
Check that you fully quit and relaunched Claude Code. Then run a turn with `LATITUDE_DEBUG=1` set — the hook will log diagnostic output to stderr explaining why the upload didn't happen (most often a wrong API key, wrong project slug, or unreachable network).
**The hook is failing silently.**
By design. The hook is fail-open: if anything goes wrong (auth failure, network error, parse error), it logs to stderr and exits cleanly so it never blocks Claude Code itself. Set `LATITUDE_DEBUG=1` to see what's happening.
**Claude Code won't start at all after install.**
This usually means a stale `BUN_OPTIONS` environment variable is pointing to a preload file that no longer exists. Either re-run the installer to recreate the file, or unset `BUN_OPTIONS` in your shell and via `launchctl unsetenv BUN_OPTIONS` on macOS.
**Wrong project receives the traces.**
You probably have an old `LATITUDE_PROJECT` value cached in `launchctl`. Re-run the installer with the correct slug, or run `uninstall` then `install` again.
## License
MIT. Source code lives at [`latitude-dev/latitude-llm`](https://github.com/latitude-dev/latitude-llm/tree/main/packages/telemetry/claude-code).
---
### Telemetry Hermes (`docs/telemetry/hermes.md`)
import SkillsCallout from "/snippets/skills-callout.mdx"
import FirstArtifact from "/snippets/first-artifact.mdx"
# Hermes telemetry
Stream [Hermes Agent](https://github.com/NousResearch/hermes-agent) (Nous Research's open-source agent harness) runs into Latitude as traces. After setup, each Hermes turn appears in your project's **Traces** view with user prompts, model turns, tool calls and results, the tools the agent was offered, memory reads and writes, delegated subagents, token usage, cost, timing, and the real system prompt that reached the model. No Latitude account yet? Your agent can create a temporary one and do this whole setup with the [`latitude-setup` skill](/getting-started/skills), no signup.
## Prerequisites
- A [Latitude account](https://console.latitude.so/login) with a project, or none yet: your agent can create a temporary one with the [`latitude-setup` skill](/getting-started/skills), no signup, and fill in the values below
- Hermes Agent installed locally
- `pip` (Hermes already runs on Python — the plugin uses only the standard library plus `certifi`, which Hermes already ships)
## Install
1. In Latitude, copy your project slug from the project sidebar (or let the `latitude-setup` skill create a temporary account and project for you).
2. Create or copy an API key from **Settings → API Keys**.
3. Install the plugin:
```bash
pip install latitude-telemetry-hermes
```
4. Enable it by adding `latitude` to the enabled-plugins list in `~/.hermes/config.yaml`, and turn on Hermes's reasoning-delta forwarding so time-to-first-token is measurable:
```yaml
plugins:
enabled:
- latitude
stream_reasoning_deltas: true
```
**`stream_reasoning_deltas: true` is required for time-to-first-token.** It is Hermes's own
setting, off by default, and without it TTFT is missing from most spans — see
[Time to first token](#time-to-first-token).
Hermes discovers the plugin through the `hermes_agent.plugins` entry point — there are no files to copy.
**Enable via `config.yaml`, not `hermes plugins enable latitude`.** Hermes's runtime loads
pip/entry-point plugins, but its `hermes plugins list`/`enable`/`disable` commands scan only
bundled and `~/.hermes/plugins/` directory plugins — so they report a pip-installed plugin as
**"not installed or bundled"** even though it loads fine ([hermes-agent#23802](https://github.com/NousResearch/hermes-agent/issues/23802)).
The `config.yaml` entry above is the reliable way to turn it on.
The plugin must be installed into the **same Python that runs Hermes**. The official installer
puts Hermes in its own venv (`~/.hermes/hermes-agent/venv`) that ignores your shell's Python, so
a plain `pip install` from another interpreter (system, pyenv, mise, …) won't be discovered.
Install into Hermes's venv instead:
```bash
~/.hermes/bin/uv pip install --python ~/.hermes/hermes-agent/venv/bin/python latitude-telemetry-hermes
```
5. Set your credentials in the environment, or add them to `~/.hermes/.env` (Hermes loads it at startup):
```bash
LATITUDE_API_KEY=lat_xxx
LATITUDE_PROJECT=your-project-slug
```
The plugin sends to Latitude Cloud (`https://ingest.latitude.so`) by default. If you
run a **self-hosted or local** Latitude, also set `LATITUDE_BASE_URL` to your ingest
**origin only** — for example `http://localhost:3002` on a local dev stack — with no
`/v1/traces` suffix (the plugin appends it). The API key and project slug must belong
to that same instance.
## Upgrade
Upgrade with the same tool the plugin was installed with, into the same interpreter that runs
Hermes:
```bash Official installer
~/.hermes/bin/uv pip install --python ~/.hermes/hermes-agent/venv/bin/python -U latitude-telemetry-hermes
```
```bash Your own environment
pip install -U latitude-telemetry-hermes
```
The official installer's venv ships **without `pip`**, so `pip install -U` fails there with
`No module named pip` — use the `uv` form. The plain `pip` form applies when you installed
Hermes yourself into an environment you manage.
To move to a specific version, or to roll back after a bad release, name the version instead of
passing `-U`:
```bash Official installer
~/.hermes/bin/uv pip install --python ~/.hermes/hermes-agent/venv/bin/python latitude-telemetry-hermes==
```
```bash Your own environment
pip install latitude-telemetry-hermes==
```
One environment serves every Hermes profile, so a single upgrade covers all of them even though
`config.yaml` and `.env` are per-profile. To read the version now on disk, ask the interpreter that
runs Hermes — `importlib.metadata` is always available, whereas `pip show` is not:
```bash Official installer
~/.hermes/hermes-agent/venv/bin/python -c "import importlib.metadata as m; print(m.version('latitude-telemetry-hermes'))"
```
```bash Your own environment
python -c "import importlib.metadata as m; print(m.version('latitude-telemetry-hermes'))"
```
### Restart Hermes
Python resolves entry-point plugins once, when the process starts. A running Hermes goes on
executing the version it loaded, and the upgraded files sit on disk unread. A CLI session picks the
new version up on its next run. A long-running process never does.
The **gateway** (Slack, Discord, …) is that long-running process, and it is usually supervised, so
it will not restart on its own. On macOS the official installer registers it with launchd — find the
label, then restart it:
```bash
launchctl list | grep -i hermes
launchctl kickstart -k gui/$(id -u)/ai.hermes.gateway
```
`kickstart -k` stops the running instance first. Don't use `launchctl stop` on its own: the job sets
`KeepAlive`, so it does come back, but `ThrottleInterval` holds it down for 30 seconds first. Under
any other supervisor, restart the service however you normally would.
**Scheduled agents run inside the gateway.** The cron ticker is part of that process rather than a
separate scheduler, so cron-driven sessions keep using the plugin version the gateway loaded. If
your agent only ever runs on a schedule, you still need the gateway restart.
### Confirm the new version is the one running
The check above reads the disk, and a stale process is precisely the case where the disk and the
running code disagree. For the running answer, send a message and open the new trace: every span
carries `hermes.plugin.version` in its metadata, and the resource carries `service.version`. Both
come from the code that is executing, so they settle it.
Spans exported before the restart keep the shape the old version gave them. Nothing is rewritten
retroactively, so judge an upgrade on a **new** session instead of reopening an existing trace.
## Verify
Run Hermes and send a message to your agent, then open your Latitude project and go to **Traces**. The new trace should appear within a few seconds.
If nothing arrives, set `LATITUDE_DEBUG=true` in `~/.hermes/.env` and run again to see the plugin's logging: it logs every export with its HTTP status. (`hermes plugins list` does **not** show pip-installed plugins — see the install note — so it can't be used to confirm the plugin is loaded.)
## See what was captured
## Structural-only telemetry
If you want trace structure without prompt, response, or tool content, set:
```bash
LATITUDE_NO_CONTENT=true
```
Structural-only traces still include timing, model, token usage, and run structure. Message content, tool input/output, memory bodies and provider error messages are omitted. For finer control — keeping content but masking one attribute — see [Privacy](#privacy).
## Disable or uninstall
To pause telemetry without removing anything, set the environment variable in `~/.hermes/.env`:
```bash
LATITUDE_HERMES_TELEMETRY_ENABLED=0
```
To stop Hermes loading the plugin at all, remove `latitude` from `plugins.enabled` in `~/.hermes/config.yaml`. (`hermes plugins disable latitude` doesn't work for pip-installed plugins — see the install note.)
To remove the integration entirely, drop it from `plugins.enabled` and uninstall the package:
```bash
pip uninstall latitude-telemetry-hermes
```
(Your `~/.hermes/.env` credentials are left in place so a re-install is one step.)
## Configuration
Every setting can be given in **two** places, and the environment wins:
- an environment variable — in your shell or in `~/.hermes/.env`;
- a key under `plugins.entries.latitude.settings` in `~/.hermes/config.yaml`.
Both files are **per profile**, so if you give each agent its own Hermes profile, each one gets its own credentials, tags and metadata with no environment juggling. The paths on this page (`~/.hermes/config.yaml`, `~/.hermes/.env`) are the **default** profile's; a named profile keeps its own pair under `~/.hermes/profiles//`, and that is the one to edit when a profile is active.
```yaml
# ~/.hermes/config.yaml
plugins:
enabled:
- latitude
entries:
latitude:
settings:
api_key: lat_xxx
project: my-project
service_name: alescript # optional; default "hermes-agent"
agent:
name: alescript
version: 2.1.0
tags: [prod, eu-west]
metadata:
deployment: staging
owner: platform-team
```
### Credentials and transport
| Env | `config.yaml` key | Default | Description |
|-----|-------------------|---------|-------------|
| `LATITUDE_API_KEY` | `api_key` | — | API key (required) |
| `LATITUDE_PROJECT` / `LATITUDE_PROJECT_SLUG` | `project` | — | Project slug (required) |
| `LATITUDE_BASE_URL` | `base_url` | `https://ingest.latitude.so` | Ingest origin (no path; the plugin appends `/v1/traces`). Set to your own ingest for self-hosted/local, e.g. `http://localhost:3002` |
Telemetry stays off until both an API key and a project are set.
### Switches
| Env | `config.yaml` key | Default | Description |
|-----|-------------------|---------|-------------|
| `LATITUDE_HERMES_TELEMETRY_ENABLED` / `LATITUDE_TELEMETRY_ENABLED` | `enabled` | `true` | Master switch |
| `LATITUDE_DEBUG` | `debug` | `false` | Verbose logging, including every export's HTTP status |
### Content
| Env | `config.yaml` key | Default | Description |
|-----|-------------------|---------|-------------|
| `LATITUDE_HERMES_NO_CONTENT` / `LATITUDE_NO_CONTENT` | `no_content` | `false` | Export structure and timing only — no prompts, responses, tool I/O or memory bodies |
| `LATITUDE_HERMES_MAX_CONTENT_CHARS` | `max_content_chars` | `262144` | Per-attribute size budget. A larger conversation is truncated from the middle, with the omission marked in the exported messages |
### Privacy
| Env | `config.yaml` key | Default | Description |
|-----|-------------------|---------|-------------|
| `LATITUDE_HERMES_REDACT_SECRETS` | `redact_secrets` | `true` | Run exported content through Hermes's own secret redactor |
| `LATITUDE_HERMES_REDACT_ATTRIBUTES` | `redact_attributes` | — | Comma-separated attribute keys whose value never leaves the machine. Each is an exact key or a `/regex/flags` pattern |
| `LATITUDE_HERMES_REDACT_MASK` | `redact_mask` | `******` | Replacement for a redacted attribute value |
### Features
| Env | `config.yaml` key | Default | Description |
|-----|-------------------|---------|-------------|
| `LATITUDE_HERMES_MEMORY` | `memory` | `true` | Emit memory spans for the built-in stores |
| `LATITUDE_HERMES_MEMORY_CONTENT` | `memory_content` | `true` | Include memory record bodies |
| `LATITUDE_HERMES_TOOL_DEFINITIONS` | `tool_definitions` | `true` | Emit the tool definitions the agent is equipped with |
| `LATITUDE_HERMES_STREAM_TTFT` | `stream_ttft` | `true` | Subscribe to stream deltas to measure time-to-first-token. Hermes builds one small payload per token while this is on, so set it to `0` if you would rather not pay that. See the note on TTFT coverage below |
| `LATITUDE_HERMES_AUX_USAGE` | `aux_usage` | `true` | Recover the token usage of Hermes's auxiliary model calls (approvals, context compaction, title generation), which fire no plugin hooks |
### Identity and grouping
| Env | `config.yaml` key | Default | Description |
|-----|-------------------|---------|-------------|
| `LATITUDE_HERMES_AGENT_NAME` | `agent.name` | the profile name, unless it is `default` | Names the agent: a tag, plus `gen_ai.agent.name` |
| `LATITUDE_HERMES_AGENT_VERSION` | `agent.version` | — | Adds the version as its own tag, plus version metadata |
| `LATITUDE_HERMES_SERVICE_NAME` | `service_name` | `hermes-agent` | OTLP `service.name` — the **Service** column, breakdown and filter |
| `LATITUDE_HERMES_TAGS` / `LATITUDE_TAGS` | `tags` | — | Extra tags: comma-separated or a JSON array. Appended to the derived ones, never replacing them |
| `LATITUDE_HERMES_METADATA` / `LATITUDE_METADATA` | `metadata` | — | Extra metadata: a JSON object, or `key=value` pairs |
## How it works
Hermes loads pip-installed plugins via the `hermes_agent.plugins` entry point and calls the module's `register(ctx)`, which subscribes to its lifecycle hooks: `pre_api_request` / `post_api_request` / `api_request_error`, `pre_llm_call` / `post_llm_call`, `pre_tool_call` / `post_tool_call`, `on_stream_start` / `on_stream_delta` / `on_stream_end`, `on_session_start` / `on_session_end` / `on_session_reset` / `on_session_finalize`, and `subagent_start` / `subagent_stop`.
Each Hermes turn becomes one trace:
```
interaction one user turn
├── search_memory what the agent remembered coming in (once per session)
├── llm_request one per model call: model, tokens, cost, TTFT, system prompt, tool definitions
├── tool_call:terminal one per tool call: arguments, result, success, duration
├── tool_call:memory
│ └── upsert_memory what the agent wrote to memory
└── tool_call:delegate
└── interaction a delegated subagent, nested under the call that spawned it
```
Spans follow Latitude's GenAI semantic conventions (`gen_ai.*`), so they render natively in the trace viewer. Spans are shipped as they finish, batched, and retried on transient ingest errors. The plugin is **fail-open**: a telemetry error never affects your agent, and it depends only on the Python standard library plus `certifi`, which Hermes already ships.
## Running several agents in one project
Send every agent to the same project and tell them apart with tags. The plugin derives these automatically:
| Tag | Where it comes from |
|-----|---------------------|
| `hermes` | Always — so Hermes traces stay identifiable in a project that also receives other harnesses |
| the platform | `cli`, `slack`, `discord`, `cron`, … |
| the agent name | `agent.name`, or the Hermes profile name when it isn't `default` |
| the agent version | `agent.version` — e.g. `2.1.0` |
| `cron:` | Scheduled runs, from the cron job id |
| `subagent:` | Traces that delegated to a subagent |
Add your own with `tags` / `metadata`; yours are appended to the derived ones, never replacing them.
**Recommended layout for several agents:** give each agent its own Hermes profile (sessions, memory and credentials are then already isolated), set `agent.name` and `agent.version` in each profile's `config.yaml`, and set `service_name` only if you want the **Service** column to read as the agent rather than the harness.
**Comparing two versions of one agent.** The version lands on two surfaces on purpose, because they answer different questions:
- **The tag** is a breakdown dimension, so one analytics query gives you a row per version:
`queryAnalytics { stream: "sessions", metric: { kind: "avg", field: "cost" }, breakdown: "tag" }`.
It is also a session filter, so an **experiment** with `tags contains 2.2.0` against a baseline of `tags contains 2.1.0` gives you a two-variant score comparison. (If several agents in the project could share a version string, combine it with the agent-name tag.)
- **The metadata key** `metadata.hermes.agent.version` is filter-only, and stays clean when you have dozens of versions and don't want dozens of tags.
## Memory
If your agent uses Hermes's built-in memory, its reads and writes appear as memory operations on the **Memory** page and in each session's memory footprint.
- The store is `hermes/`, with one record per store file: `MEMORY.md` and `USER.md`.
- Memory enters the system prompt as a frozen snapshot at session start, so it is recorded as **one read per session**.
- A write is recorded after the `memory` tool call succeeds, by reading the file back off disk — so the recorded body is exactly what landed. Latitude turns the sequence of bodies into a diff view and per-entry history.
- Turn it off with `LATITUDE_HERMES_MEMORY=0`, or keep the structure without the bodies with `LATITUDE_HERMES_MEMORY_CONTENT=0`.
- If you have configured an external memory provider (Mem0, Supermemory, Honcho, …) in `config.yaml`, the built-in files are no longer the live store and memory telemetry turns itself off.
## Identified users
On a gateway platform (Slack, Discord, …) each turn carries the platform's member id as the user, so per-user analytics and the Users page work. Hermes only exposes that id to plugins — no display name, no email — so that is what you see. When a platform uses an email address as its user id, the email is recorded too.
## Time to first token
`isStreaming` is set on every streaming call. **Time-to-first-token needs one setting in Hermes's own config**, because Hermes decides which streamed tokens reach a plugin:
```yaml
# ~/.hermes/config.yaml
plugins:
stream_reasoning_deltas: true
```
Without it, TTFT is missing from most spans. Hermes streams a turn's visible text through a path that does not notify plugins whenever the turn ends in a tool call — which is most turns for a coding agent — so the only tokens a plugin reliably sees are reasoning tokens, and those are not forwarded until this setting is on. With it on, TTFT lands on the great majority of calls: in our own dogfood session, **51 of 53** model calls reported it, ranging from 1.2 s to 13.1 s.
This is a Hermes-side limitation, not a plugin setting, so there is nothing to configure on the Latitude side. `LATITUDE_HERMES_STREAM_TTFT=0` disables our subscription entirely if you would rather not pay for the per-token hook at all.
## Token usage and cost
Hermes's `/usage` and Latitude will not show the same numbers, and neither is wrong:
- **Background reviews.** A background review runs a full conversation loop on its own agent object in a separate thread. Latitude sees those calls; `/usage` reads the main agent's counters, so it doesn't. Latitude = main loop + background reviews.
- **Reasoning tokens.** Hermes's "Output tokens" includes reasoning; Latitude shows output and reasoning as separate figures.
- **Auxiliary calls.** Approvals, context compaction and title generation don't fire plugin hooks. Latitude recovers their usage from Hermes's own per-task ledger at the end of each session, so session totals match `session_model_usage` in `~/.hermes/state.db` — the arbiter if you ever want to check by hand.
- **Cost on a subscription route.** On an OAuth/subscription route Hermes reports the call as *included* at $0, while Latitude shows the equivalent list price from its model catalog. The span records which regime it was in, so the figure is explainable rather than mysterious.
## Captured data and privacy
By default Latitude receives what it needs to reconstruct a Hermes run: prompts, responses, the system prompt, tool definitions, tool input and output, memory record bodies, model metadata, token usage and timing.
**Secret redaction is on by default.** Everything content-bearing goes through Hermes's own secret redactor on the way out, with URL-credential redaction enabled — so an API key echoed by a terminal tool is masked before it leaves the machine. Turn it off with `LATITUDE_HERMES_REDACT_SECRETS=0` if you'd rather see raw values. If the redactor can't be loaded, the span says so (`hermes.redaction.applied=false`) rather than quietly claiming protection it didn't apply.
**Keeping a specific attribute local.** `LATITUDE_HERMES_REDACT_ATTRIBUTES` replaces the whole value of any attribute you name — for example `gen_ai.memory.records,gen_ai.tool.call.result`, or a pattern like `/^gen_ai\.tool\.call\./`. The attribute is still sent, with its value masked, so you can see what the plugin exported.
**Structure only.** `LATITUDE_NO_CONTENT=true` drops every content-bearing attribute: you keep timing, model, token usage and run structure, and lose prompts, responses, tool I/O and memory bodies. Error text counts as content, because a provider often quotes the request it rejected — a failed call reports its status code and error type, not the provider's message.
Telemetry runs for each turn until disabled or uninstalled. Disable it before working with material you do not want sent to Latitude.
## Troubleshooting
**No traces appear.** Confirm `latitude` is in `plugins.enabled` in `~/.hermes/config.yaml` (`hermes plugins list` never shows a pip-installed plugin — see the install note), check that the API key and project slug are correct, then set `LATITUDE_DEBUG=true` and send a new message: the plugin logs each export and its HTTP status.
**Traces arrive, but they look like an older version of the plugin.** The running Hermes is still
executing the plugin version it loaded at startup — upgrading the package does not affect a live
process. Restart Hermes, and the gateway specifically if you run one; see [Upgrade](#upgrade).
`LATITUDE_DEBUG` will not reveal this: the old version keeps exporting successfully, so the log
shows `200`s throughout.
**Need more diagnostics.** Set `LATITUDE_DEBUG=true` in `~/.hermes/.env` and trigger another run.
**Traces show timing but no content.** Structural-only mode is enabled. Remove `LATITUDE_NO_CONTENT` from `~/.hermes/.env`.
**A value looks masked.** Secret redaction is on by default; `LATITUDE_HERMES_REDACT_SECRETS=0` turns it off. A value that is exactly `******` was matched by `LATITUDE_HERMES_REDACT_ATTRIBUTES`.
**Memory operations are missing.** Check that the agent actually wrote memory in the session, that `LATITUDE_HERMES_MEMORY` isn't `0`, and that `config.yaml` doesn't configure an external `memory.provider` — that turns built-in memory telemetry off.
---
### Telemetry Memory (`docs/telemetry/memory.mdx`)
---
title: Memory tracing
description: Instrument your agent's long-term memory so Latitude can show how it evolves.
---
## Overview
This guide shows you how to send your agent's **long-term memory** operations to Latitude. Once instrumented, every read and write appears on the [Memory page](../observability/memory): each store's current contents, per-record history and diffs, and the tokens read and written per session, with every change linked to the session that caused it.
You keep using your memory store exactly as you do today. You emit one span
per memory operation, and Latitude derives the history, diffs, and token
counts from those spans. You never compute or send a diff yourself.
---
## Requirements
- Base tracing already sending traces to Latitude. If you are not tracing yet, start with [Start tracing](./start-tracing).
- An app with **long-term memory**: state your agent persists and reloads across separate interactions, such as a `memory/` directory, a database table, a vector store, a key-value store, or a provider like Mem0, Zep, or Supermemory. Within-request conversation history is not memory; base tracing already captures it.
Memory operations ride the same exporter as the rest of your telemetry, so there is no extra account or endpoint to configure.
---
## Instrument with your coding agent
Recommended. The Latitude skill inspects your codebase, finds your memory read and write call sites, and instruments them with the right operations and identifiers.
Paste this prompt into Claude Code, Cursor, Windsurf, Codex, OpenCode, or another coding agent:
```text
Install the `latitude-telemetry` skill from `github.com/latitude-dev/skills`, and use it to add Latitude memory observability to this app's long-term memory, following best practices.
```
Trigger one interaction that recalls memory and one that writes it.
Open your project and go to **Memory**. The store appears within a few seconds of the trace completing.
Prefer to wire it up by hand? Continue below.
---
## The store and record model
Two identifiers carry the whole model, and getting them right is the entire job, because they are how Latitude groups, versions, and attributes everything. Think of a store as a git repository and a record as a file in it.
**Store (`gen_ai.memory.store.id`)** is one isolated pool of memory. The rule: two operations share a store if and only if a write by one should be visible to the other.
- **Per-user memory** that users cannot see across each other: use the user id, or a `user/` prefix, as the store id, so each user becomes a separate store.
- **Shared memory** that several users read and write: one store id for all of them, and the latest write wins across them.
- **Always set it.** A span without a store id lands in the `(unattributed)` store, which counts as activity but gives you nothing to browse. Keep it stable, or a store's history fragments into many stores.
**Record (`gen_ai.memory.record.id`)** is one addressable unit within a store: what a write creates or updates and what a read returns. Use the store's natural key, such as a file path relative to the memory root, a key-value key, a row id, or the provider's own record id. Latitude splits record ids on `/` to nest them into folders on the Memory page, so a path-like id gives you a browsable tree. The only operations without a record id are the whole-store ones.
**Writes carry the whole record, not a delta.** Each write span is a full
snapshot of the record as it stands after the write. If your app updates only
part of a record, read back or reconstruct the full body and send that.
Sending only the changed fragment makes Latitude read everything you omitted
as a deletion.
**Content capture is opt-in.** Record bodies and the search query can hold
sensitive data, so the SDK does not send them unless you enable
`captureContent`. With it off you still get classification and counts, but no
diffs, no token deltas, and nothing to browse. Turn it on unless the memory
holds data you cannot send to Latitude; when only some fields are sensitive,
keep it on and scrub them with the `redact` hook.
---
## Instrument manually
The memory helper ships in the same telemetry package as base tracing, so there is nothing new to add if you already installed it.
```bash npm
npm install @latitude-data/telemetry
```
```bash pnpm
pnpm add @latitude-data/telemetry
```
```bash yarn
yarn add @latitude-data/telemetry
```
```bash bun
bun add @latitude-data/telemetry
```
```bash pip
pip install latitude-telemetry
```
```bash uv
uv add latitude-telemetry
```
```bash poetry
poetry add latitude-telemetry
```
Pass the `latitude` instance from your base tracing setup. Set `storeId` to scope the memory, using a user id for per-user memory, and enable content capture so diffs and token deltas work.
```ts
import { createMemoryTelemetry } from "@latitude-data/telemetry"
// `latitude` is the instance from your base tracing setup.
const memory = createMemoryTelemetry({
latitude,
storeId: `user/${user.id}`,
captureContent: true,
})
```
```python
from latitude_telemetry import create_memory_telemetry
# `latitude` is the instance from your base tracing setup.
memory = create_memory_telemetry(
latitude,
store_id=f"user/{user_id}",
capture_content=True,
)
```
Call the helper at your store's read and write boundaries, inside the `capture()` that already wraps the interaction so the spans join the trace. Each operation has a **wrap** form, where you pass `execute` and the helper runs your call inside the span and records its latency, status, and errors, and an **emit** form, with no `execute`, for when the read or write already happened elsewhere.
```ts
import { capture } from "@latitude-data/telemetry"
await capture("support-agent-turn", async () => {
// Read: one span carries the query and every record it returned.
const hits = await memory.search({
query,
execute: () => store.search(query),
recordsFromResult: (rows) =>
rows.map((r) => ({ id: r.id, content: r.text, score: r.score })),
})
// Write: pass the record's full new body, not a delta.
await memory.upsert({
recordId: "preferences/tone",
records: [
{ id: "preferences/tone", content: "Prefers concise answers with code." },
],
execute: () => store.upsert("preferences/tone", nextValue),
})
// Emit form: record a completed span when the read or write already happened.
await memory.update({
recordId: "preferences/tone",
records: [
{ id: "preferences/tone", content: "Prefers concise answers with code." },
],
})
// Delete one record. Omit recordId to wipe the whole store.
await memory.delete({ recordId: "preferences/tone" })
})
```
```python
from latitude_telemetry import capture
def handle_turn():
# Read: one span carries the query and every record it returned.
hits = memory.search(
query=query,
execute=lambda: store.search(query),
records_from_result=lambda rows: [
{"id": r["id"], "content": r["text"], "score": r["score"]} for r in rows
],
)
# Write: pass the record's full new body, not a delta.
memory.upsert(
record_id="preferences/tone",
records=[
{"id": "preferences/tone", "content": "Prefers concise answers with code."}
],
execute=lambda: store.upsert("preferences/tone", next_value),
)
# Emit form: record a completed span when the read or write already happened.
memory.update(
record_id="preferences/tone",
records=[
{"id": "preferences/tone", "content": "Prefers concise answers with code."}
],
)
# Delete one record. Omit record_id to wipe the whole store.
memory.delete(record_id="preferences/tone")
capture("support-agent-turn", handle_turn)
```
### Without the SDK
If your app traces to Latitude without the Latitude SDK, or runs in another language, emit the spans on your existing tracer. Set the span name and `gen_ai.operation.name` to the operation, and add the `gen_ai.memory.*` attributes.
```ts
import { trace } from "@opentelemetry/api"
const tracer = trace.getTracer("memory")
await tracer.startActiveSpan("update_memory", async (span) => {
span.setAttributes({
"gen_ai.operation.name": "update_memory",
"gen_ai.memory.store.id": `user/${user.id}`,
"gen_ai.memory.record.id": "preferences/tone",
"gen_ai.memory.records": JSON.stringify([
{ id: "preferences/tone", content: "Prefers concise answers with code." },
]),
})
await store.update("preferences/tone", nextValue)
span.end()
})
```
```python
import json
from opentelemetry import trace
tracer = trace.get_tracer("memory")
with tracer.start_as_current_span("search_memory") as span:
span.set_attribute("gen_ai.operation.name", "search_memory")
span.set_attribute("gen_ai.memory.store.id", f"user/{user_id}")
span.set_attribute("gen_ai.memory.query.text", query)
hits = store.search(query)
span.set_attribute("gen_ai.memory.record.count", len(hits))
span.set_attribute(
"gen_ai.memory.records",
json.dumps(
[{"id": h["id"], "content": h["text"], "score": h["score"]} for h in hits]
),
)
```
Other runtimes such as Go, Java, Ruby, and .NET set the same attributes on whatever span they already export. Only `gen_ai.operation.name` is required; add the rest to unlock more of the Memory page.
---
## Operations and attributes
Memory operations are the standard OpenTelemetry GenAI **memory operation spans**, not a Latitude-specific format. The span name equals `gen_ai.operation.name`, and both are one of:
| Operation | Meaning |
| --- | --- |
| `create_memory` | Create new records |
| `update_memory` | Modify existing records |
| `upsert_memory` | Create or update, without choosing which |
| `delete_memory` | Delete records. Without a record id, this wipes the whole store |
| `search_memory` | Query or retrieve records (a read) |
| `create_memory_store` | Create or initialize a store |
| `delete_memory_store` | Delete a store and everything in it |
These attributes control what Latitude can show:
| Attribute | Purpose |
| --- | --- |
| `gen_ai.operation.name` | The operation above. Required |
| `gen_ai.memory.store.id` | The store the operation targets, e.g. `user/usr_4f2a`. Everything groups under it; absent lands in the `(unattributed)` store |
| `gen_ai.memory.record.id` | The record touched, e.g. `preferences/tone`. Splits on `/` to nest records into folders |
| `gen_ai.memory.record.count` | How many records were affected or returned |
| `gen_ai.memory.query.text` | The search query, on `search_memory` spans. Opt-in |
| `gen_ai.memory.records` | The content payload: an array of `{ id, content, score?, metadata? }` objects. Powers content browsing, diffs, and token counts. Opt-in |
---
## Verify
Instrumentation is not finished when the code compiles, only when operations show up correctly in Latitude.
- Run a real interaction that recalls memory and one that writes it, so you produce both a `search_memory` span and a mutating span. For short-lived scripts, call `await latitude.flush()` (Python: `latitude.flush()`) before exit so spans flush.
- On the **Spans tab**, the operations classify and carry the store and record attributes.
- On the **[Memory page](../observability/memory)**, the store appears under the expected id, its records show the expected bodies, and a second write to a record produces a new version and a diff.
- On the **trace or session detail**, the **Memory** row shows tokens read, added, and removed.
Common issues: an empty `store.id` (everything lands in `(unattributed)`), an unstable or missing `record.id` (history will not stitch and reads will not attribute), a write sending a partial body instead of the full snapshot, or a span emitted outside `capture()` so it never joins the trace.
## Next steps
- [Memory](../observability/memory): Explore stores, record history, diffs, and per-session footprint
- [Start tracing](./start-tracing): Base tracing setup, if you are not sending traces yet
---
### Telemetry Openclaw (`docs/telemetry/openclaw.md`)
# OpenClaw telemetry
Stream your OpenClaw agent runs: system prompts, message history, assistant responses, tool I/O, token usage, and the running agent's name into Latitude as OpenTelemetry traces. Once installed, every agent run shows up in your project's **Traces** view, with subagent spawn trees nested into a single waterfall.
## Prerequisites
- A free [Latitude account](https://console.latitude.so/login) with at least one project
- **OpenClaw 2026.4.25 or newer** on your `PATH` (run `npm install openclaw@latest` to upgrade)
- Node.js available on your `PATH` (the CLI runs via `npx`)
## Step 1: Get your Latitude API key and project slug
You'll need both before running the installer.
**Project slug**: open your project in Latitude. The slug appears in the left sidebar, directly under the project name. Click it to copy.
**API key**: go to **Settings → API Keys** (top-right of the Latitude app). Use an existing key or click **Create API Key**. Copy the full key value.
## Step 2: Run the installer
In a terminal, run:
```bash
npx -y @latitude-data/openclaw-telemetry-cli@0.0.7 install
```
The installer will:
1. Prompt for your API key and project slug (paste the values from Step 1).
2. Run `openclaw plugins install @latitude-data/openclaw-telemetry@0.0.7` to fetch and scan the plugin.
3. Write the plugin entry into `~/.openclaw/openclaw.json` (API key, project slug, both `allowConversationAccess` flags, and `enabled: true`).
4. Add the plugin to `plugins.allow` so OpenClaw stops warning about it on every restart.
5. Run `openclaw config validate` to confirm the config parses cleanly.
6. On a TTY, offer to restart the gateway for you.
If you'd rather pass values as flags instead of being prompted:
```bash
npx -y @latitude-data/openclaw-telemetry-cli@0.0.7 install \
--api-key=lat_xxx \
--project=your-project-slug \
--yes
```
See [Install flags](#install-flags) below for all available flags.
## Step 3: Restart the gateway
If you didn't let the installer do it for you:
```bash
openclaw gateway restart
```
## Step 4: Verify traces are arriving
Send a message to one of your OpenClaw agents. Then open your Latitude project and go to **Traces**. Within a few seconds the new trace should appear.
You can also tail the gateway log to confirm the plugin loaded:
```bash
grep -E "blocked|plugin not found|latitude" /tmp/openclaw/openclaw-*.log | tail
```
You should see a `ready (N plugins: ..., @latitude-data/openclaw-telemetry, ...)` line and **no** `blocked` or `plugin not found` lines.
If nothing shows up, see [Troubleshooting](#troubleshooting) below.
## Structural-only telemetry (no content capture)
If you want trace metadata — timings, token usage, model name, agent name, structural ids — without sending prompt or response content, install with `--no-content`:
```bash
npx -y @latitude-data/openclaw-telemetry-cli@0.0.7 install --no-content
```
Spans still appear in Latitude with the full tree, but content attributes (`gen_ai.input.messages`, `gen_ai.output.messages`, `gen_ai.system_instructions`, tool args/results) are scrubbed. Each span carries `latitude.captured.content: false` so the gate state is visible in the UI.
To switch an existing install: see [Manual configuration](#manual-configuration) below — set `config.allowConversationAccess` to `false` while keeping `hooks.allowConversationAccess` at `true`.
## Disabling temporarily
To pause emission without uninstalling, set the env var on the gateway process:
```bash
export LATITUDE_OPENCLAW_ENABLED=0
```
Restart the gateway for the change to take effect. Set it back to `1` (or unset it) to re-enable. Alternatively, edit `~/.openclaw/openclaw.json` and set `plugins.entries["@latitude-data/openclaw-telemetry"].enabled` to `false`.
## Uninstalling
To remove the plugin and revert all changes the installer made:
```bash
npx -y @latitude-data/openclaw-telemetry-cli@0.0.7 uninstall
```
Or manually:
```bash
openclaw plugins uninstall @latitude-data/openclaw-telemetry --force
openclaw gateway restart
```
OpenClaw removes the extension files, install record, plugin entry, and the `plugins.allow` entry.
## Manual configuration
If you'd rather not run the CLI, do exactly what it does in four steps.
### 1. Install the runtime
```bash
openclaw plugins install @latitude-data/openclaw-telemetry@0.0.7
```
Pin to an exact version. OpenClaw's `security audit --deep` warns about unpinned install specs.
### 2. Configure and enable
Use bracket notation so the scoped package name parses correctly. Substitute your real API key and project slug:
```bash
openclaw config set 'plugins.entries["@latitude-data/openclaw-telemetry"].config.apiKey' "lat_xxx"
openclaw config set 'plugins.entries["@latitude-data/openclaw-telemetry"].config.project' "your-project-slug"
openclaw config set 'plugins.entries["@latitude-data/openclaw-telemetry"].config.allowConversationAccess' true
openclaw config set 'plugins.entries["@latitude-data/openclaw-telemetry"].hooks.allowConversationAccess' true
openclaw config set 'plugins.entries["@latitude-data/openclaw-telemetry"].enabled' true
```
Both `allowConversationAccess` writes are required — see [The two flags](#the-two-flags) below.
### 3. Add to `plugins.allow`
Silence OpenClaw's warning about non-bundled plugins loading without provenance. `config set` can't append to arrays, so include any other plugins you already have in `plugins.allow`:
```bash
openclaw config set 'plugins.allow' '["@latitude-data/openclaw-telemetry"]'
```
### 4. Restart and verify
```bash
openclaw gateway restart
openclaw config validate --json
# → {"valid": true, ...}
```
### Or: hand-edit `~/.openclaw/openclaw.json`
Equivalent to steps 2 + 3 in one paste:
```json
{
"plugins": {
"allow": ["@latitude-data/openclaw-telemetry"],
"entries": {
"@latitude-data/openclaw-telemetry": {
"enabled": true,
"hooks": {
"allowConversationAccess": true
},
"config": {
"apiKey": "lat_xxx",
"project": "your-project-slug",
"allowConversationAccess": true
}
}
}
}
}
```
Merge with whatever else lives in `openclaw.json`, then run `openclaw config validate` and `openclaw gateway restart`.
## Reference
### CLI commands
The CLI exposes two subcommands:
- `install` — installs the plugin runtime, writes config to `~/.openclaw/openclaw.json`, registers it in `plugins.allow`, validates, and (on TTY) restarts the gateway.
- `uninstall` — reverts every change `install` made, after confirmation.
### Install flags
| Flag | Purpose |
|---|---|
| `--api-key=` | Supply the Latitude API key non-interactively. |
| `--project=` | Supply the Latitude project slug non-interactively. |
| `--no-content` | Configure structural-only telemetry. Spans emit normally but message and tool content are scrubbed. |
| `--staging` | Point the plugin at the Latitude staging environment instead of production. |
| `--dev` | Point the plugin at `localhost` (for Latitude internal development). |
| `--yes`, `--no-prompt` | Skip all interactive prompts. Useful in CI or scripted setups. |
The full flag matrix (dry-run mode, custom config dir) lives in the [CLI README](https://github.com/latitude-dev/latitude-llm/tree/main/packages/telemetry/openclaw-cli#readme).
### Configuration keys
Two blocks live under `plugins.entries["@latitude-data/openclaw-telemetry"]`.
**`.config`** — read by the plugin's runtime:
| Key | Required | Default | Purpose |
|---|---|---|---|
| `apiKey` | Yes | — | Bearer token for Latitude ingestion. |
| `project` | Yes | — | Slug of the project to route traces into. |
| `baseUrl` | No | `https://ingest.latitude.so` | Override the OTLP ingest origin. The CLI sets this only when `--staging` or `--dev` is passed. |
| `allowConversationAccess` | No | `false` | When `true`, attach raw prompts, assistant responses, system instructions, and tool I/O to spans. When `false`, emit only timings, token usage, model name, agent id, and structural ids. **Must be paired with `hooks.allowConversationAccess` — see [The two flags](#the-two-flags).** |
| `enabled` | No | `true` | Set to `false` to pause emission without uninstalling. |
| `debug` | No | `false` | Log diagnostic lines to stderr (visible in the gateway log). |
**`.hooks`** — read by OpenClaw's runtime:
| Key | Required | Default | Purpose |
|---|---|---|---|
| `allowConversationAccess` | Yes (on 2026.4.25+) | — | OpenClaw's hook dispatcher gates `llm_input` / `llm_output` / `before_tool_call` / `after_tool_call` / `agent_end` events on this. When `false` or absent, every typed hook is blocked and the plugin never sees an event. |
### The two flags
`hooks.allowConversationAccess` and `config.allowConversationAccess` mean different things:
- **`hooks.*`** is the **dispatch gate**. `false` → OpenClaw never forwards events to the plugin. No traces.
- **`config.*`** is the **payload-content gate**. `false` → the plugin emits spans normally but scrubs message content from them. Structural-only telemetry.
The CLI always writes both from the same source. If you hand-edit:
- **Both `true`**: full content capture (the default).
- `hooks: true` + `config: false`: structural-only telemetry (equivalent to `--no-content`).
- `hooks: false` + anything: no traces. Don't.
### Environment variables
If a `config.*` key isn't set in `openclaw.json`, the plugin falls back to env vars on the gateway process:
| Variable | Required | Default | Purpose |
|---|---|---|---|
| `LATITUDE_API_KEY` | Conditional | — | Fallback for `config.apiKey`. |
| `LATITUDE_PROJECT` | Conditional | — | Fallback for `config.project`. |
| `LATITUDE_BASE_URL` | No | `https://ingest.latitude.so` | Fallback for `config.baseUrl`. |
| `LATITUDE_DEBUG` | No | unset | Set to `1` to print diagnostic output to stderr. |
| `LATITUDE_OPENCLAW_ENABLED` | No | `1` | Set to `0` to disable emission without uninstalling. |
### Files modified by the installer
- **`~/.openclaw/openclaw.json`** — adds the plugin to `plugins.allow` and writes the full `plugins.entries["@latitude-data/openclaw-telemetry"]` block. Existing settings are preserved.
- **`~/.openclaw/extensions/@latitude-data/openclaw-telemetry/`** — the runtime plugin files, fetched from npm by `openclaw plugins install`.
The installer does not modify your shell rc files (`.zshrc`, `.bashrc`).
## How it works
The plugin subscribes to OpenClaw's typed plugin hooks and emits one OTel span per paired before/after event. Hooks run fire-and-forget, so telemetry can never slow down the agent loop.
For each agent run, the plugin emits one trace shaped like the run itself:
```
agent (root, traceId = hash(runId))
├─ compaction (rare; budget-triggered)
├─ model_call (one per provider API call)
├─ tool_call: foo (between model_calls)
├─ model_call
├─ tool_call: bar
├─ subagent (the child's full agent tree nests under here)
│ └─ agent
│ ├─ model_call
│ └─ tool_call: ...
└─ model_call (final)
```
Five span kinds:
- **`agent`** — root of the run. Carries `openclaw.session.key`, `openclaw.agent.id`, `openclaw.agent.name`, aggregated token usage, run duration, success/error status, the first user prompt, and the final message list.
- **`model_call`** — one per provider API call. Carries provider, request/response model, transport, per-call duration, error category, time-to-first-byte, payload bytes, upstream request id hash, and `gen_ai.input.messages` snapshotted at generation start.
- **`tool_call:`** — one per tool invocation, with canonical `gen_ai.tool.*` attributes (`name`, `call.id`, `call.arguments`, `call.result`). Sibling of `agent`, not child of `model_call` — tools run between generations, not during them.
- **`compaction`** — fires when OpenClaw hits the message budget mid-run.
- **`subagent`** — one per child run spawned by this agent. The child's entire `agent` subtree (its own `model_call`s, `tool_call`s, even further-nested `subagent`s) parents under it via cross-runId trace propagation, so a spawn tree is one waterfall in one trace.
Every span carries `openclaw.agent.id` and `openclaw.agent.name`, so multi-agent setups can be filtered and grouped by agent in the Latitude UI.
The plugin is **fail-open**. If the API is unreachable, your key is wrong, or a hook payload is malformed, it logs to stderr (when `debug: true`) and the agent run continues unaffected.
### Backend caveat: Codex / Claude-Code-style providers
OpenClaw's `model_call_started` / `model_call_ended` hooks fire from its `selection` layer, which wraps the agent's `streamFn` invocation. For "agentic" backends (Codex, Claude Code) the inner generations happen inside the backend's own loop and don't surface as separate `model_call` events. As a result, a Codex-backed run shows **one** `model_call` per attempt instead of N. Anthropic and OpenAI direct don't have this issue. The fix is upstream in OpenClaw.
## Privacy
The CLI's first-install default writes `allowConversationAccess: true` to both blocks → full content capture. Latitude receives the verbatim content of every run: prompts, responses, system instructions, tool I/O.
- **Structural-only mode** is available via `--no-content` (or hand-editing the two flags).
- **No per-run opt-in.** Every run is captured until the plugin is disabled.
- **No redaction.** If a secret reaches the agent, it reaches Latitude.
- **Disable globally.** Set `LATITUDE_OPENCLAW_ENABLED=0` on the gateway, set `enabled: false` in the plugin entry, or run `uninstall`.
For hand-edited configs, leaving `allowConversationAccess` out entirely produces **no traces** (not "structural-only traces") — `hooks.allowConversationAccess` defaults to `false` at OpenClaw's level and dispatch is blocked. Always set both keys explicitly.
If you're working with sensitive material, switch to structural-only mode or disable the plugin **before** the run.
## Supported OpenClaw versions
Requires **2026.4.25 or newer**. Earlier versions either reject `hooks.allowConversationAccess` outright (≤ 2026.4.21) or have unverified dispatch gating (2026.4.22 – 2026.4.24). The CLI aborts on older versions; manual installs run into validation errors. Run `npm install openclaw@latest` to upgrade.
## Troubleshooting
**No traces appear in Latitude.**
Confirm the gateway was restarted after install. Then check the gateway log:
```bash
grep -E "blocked|plugin not found|latitude" /tmp/openclaw/openclaw-*.log | tail
```
A `blocked` line means `hooks.allowConversationAccess` is `false` or missing — re-run `install`, or set both flags as shown in [Manual configuration](#manual-configuration). A `plugin not found` line means the runtime didn't install — re-run `openclaw plugins install @latitude-data/openclaw-telemetry@0.0.7`.
If the log looks clean, set `LATITUDE_DEBUG=1` on the gateway and trigger another run; the plugin will print diagnostic output explaining why the upload didn't happen (most often a wrong API key, wrong project slug, or unreachable network).
**Spans show timings but no message content.**
You're in structural-only mode. Set `config.allowConversationAccess` to `true` (keeping `hooks.allowConversationAccess` at `true`) and restart the gateway.
**Codex- or Claude-Code-backed agents show one `model_call` per attempt instead of N.**
Expected — see [Backend caveat](#backend-caveat-codex--claude-code-style-providers) above.
**Gateway warns about non-bundled plugins on every restart.**
The plugin isn't in `plugins.allow`. Run:
```bash
openclaw config set 'plugins.allow' '["@latitude-data/openclaw-telemetry"]'
```
(Include any other plugins you already had in the array — `config set` can't append.)
**`openclaw config validate` fails after manual install.**
Most likely a missing `hooks.allowConversationAccess` write or a quoting issue with the bracket-notation paths. Compare your `~/.openclaw/openclaw.json` against the [hand-edit template](#or-hand-edit-openclawopenclawjson) above.
---
### Telemetry Otel Exporter (`docs/telemetry/otel-exporter.md`)
---
title: OpenTelemetry Exporter (OTEL)
description: Connect any OpenTelemetry-instrumented application to Latitude, regardless of language or framework.
---
# Connect with Any OpenTelemetry Exporter
Latitude's ingestion endpoint speaks standard **OTLP over HTTP**.
If your language has an OpenTelemetry SDK (Go, Java, Ruby, Rust, .NET, Elixir, PHP, etc.), you can send traces to Latitude without a Latitude-specific library.
Using **TypeScript** or **Python**? The dedicated SDKs handle all of this for you with a single function call. See the [TypeScript SDK](typescript) or [Python SDK](python) instead.
## Prerequisites
1. A **Latitude API key** — generate one from your project settings in the Latitude dashboard.
2. Your **project slug** — visible in the project settings or URL.
## Endpoint and Headers
| | Value |
|---|---|
| **URL** | `https://ingest.latitude.so/v1/traces` |
| **Method** | `POST` |
| **`Authorization`** | `Bearer ` |
| **`X-Latitude-Project`** | `` |
| **`Content-Type`** | `application/json` or `application/x-protobuf` |
The endpoint accepts a standard OTLP `ExportTraceServiceRequest` body and returns **`202`** on success.
## Verify with curl
A minimal request to confirm connectivity (replace the placeholders):
```bash
curl -X POST https://ingest.latitude.so/v1/traces \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "X-Latitude-Project: YOUR_PROJECT_SLUG" \
-H "Content-Type: application/json" \
-d '{
"resourceSpans": [{
"resource": {
"attributes": [{
"key": "service.name",
"value": { "stringValue": "my-service" }
}]
},
"scopeSpans": [{
"scope": { "name": "manual-test" },
"spans": [{
"traceId": "00000000000000000000000000000001",
"spanId": "0000000000000001",
"name": "test-span",
"kind": 1,
"startTimeUnixNano": "1700000000000000000",
"endTimeUnixNano": "1700000001000000000",
"attributes": [{
"key": "gen_ai.system",
"value": { "stringValue": "openai" }
}]
}]
}]
}]
}'
```
A `202` response with `{}` means the endpoint accepted your payload.
## Language Examples
Every OpenTelemetry SDK lets you configure an OTLP HTTP exporter with a custom endpoint and headers. Below are the key configuration snippets — the rest of your OTel setup (tracer provider, instrumentations, etc.) stays the same as usual.
### Go
```go
import (
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
"go.opentelemetry.io/otel/sdk/trace"
)
exporter, err := otlptracehttp.New(ctx,
otlptracehttp.WithEndpointURL("https://ingest.latitude.so/v1/traces"),
otlptracehttp.WithHeaders(map[string]string{
"Authorization": "Bearer " + apiKey,
"X-Latitude-Project": projectSlug,
}),
)
provider := trace.NewTracerProvider(trace.WithBatcher(exporter))
```
### Java
```java
import io.opentelemetry.exporter.otlp.trace.OtlpHttpSpanExporter;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
OtlpHttpSpanExporter exporter = OtlpHttpSpanExporter.builder()
.setEndpoint("https://ingest.latitude.so/v1/traces")
.addHeader("Authorization", "Bearer " + apiKey)
.addHeader("X-Latitude-Project", projectSlug)
.build();
SdkTracerProvider provider = SdkTracerProvider.builder()
.addSpanProcessor(BatchSpanProcessor.builder(exporter).build())
.build();
```
### Ruby
```ruby
require "opentelemetry-sdk"
require "opentelemetry-exporter-otlp"
ENV["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"] = "https://ingest.latitude.so/v1/traces"
ENV["OTEL_EXPORTER_OTLP_TRACES_HEADERS"] = "Authorization=Bearer #{api_key},X-Latitude-Project=#{project_slug}"
OpenTelemetry::SDK.configure do |c|
c.add_span_processor(
OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
OpenTelemetry::Exporter::OTLP::Exporter.new
)
)
end
```
### .NET
```csharp
using OpenTelemetry;
using OpenTelemetry.Trace;
using OpenTelemetry.Exporter;
var tracerProvider = Sdk.CreateTracerProviderBuilder()
.AddOtlpExporter(opt =>
{
opt.Endpoint = new Uri("https://ingest.latitude.so/v1/traces");
opt.Headers = $"Authorization=Bearer {apiKey},X-Latitude-Project={projectSlug}";
opt.Protocol = OtlpExportProtocol.HttpProtobuf;
})
.Build();
```
### Environment Variables (Any Language)
Most OpenTelemetry SDKs respect standard environment variables, so you can often skip code changes entirely:
```bash
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://ingest.latitude.so/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Bearer YOUR_API_KEY,X-Latitude-Project=YOUR_PROJECT_SLUG"
```
## Latitude Span Attributes
All spans reaching the endpoint are ingested. To get the most out of the Latitude UI (filtering by user, session, tags), set these optional span attributes:
| Attribute | Type | Description |
|---|---|---|
| `latitude.capture.name` | `string` | A name for the capture context (e.g. `"handle-user-request"`) |
| `latitude.tags` | `string` (JSON array) | Tags for filtering, e.g. `'["production","v2"]'` |
| `latitude.metadata` | `string` (JSON object) | Arbitrary key-value pairs, e.g. `'{"requestId":"abc"}'` |
| `session.id` | `string` | Group related traces into a session |
| `user.id` | `string` | Associate traces with a specific user |
Set these as standard OTel span attributes. Example in Go:
```go
span.SetAttributes(
attribute.String("user.id", "user_123"),
attribute.String("session.id", "session_abc"),
attribute.String("latitude.tags", `["production","v2-agent"]`),
attribute.String("latitude.metadata", `{"requestId":"req-xyz"}`),
)
```
## GenAI Span Attributes (LLM Metadata)
Configuring the OTLP exporter (endpoint + headers) only ensures your spans **reach** Latitude. For Latitude to display LLM call details — model name, token counts, input/output messages — your spans must carry the [OpenTelemetry GenAI semantic convention](https://opentelemetry.io/docs/specs/semconv/gen-ai/) attributes described below.
Without these attributes, traces will appear in the dashboard but show no model, token usage, or message content.
Latitude follows the **[OpenTelemetry Semantic Conventions for Generative AI](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/)** (v1.37+). When your spans include `gen_ai.*` attributes, the Latitude UI displays full LLM call details: the provider, model, input/output messages, token usage, cost, and latency.
Refer to the [GenAI span semconv spec](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/) for the full attribute list. The message content attributes use the standard OpenTelemetry GenAI **parts-based** message format, shown below.
### Message Format
Latitude expects the standard **parts-based** GenAI message format. Each message has a `role` and an array of `parts`:
**`gen_ai.input.messages`** (JSON string):
```json
[
{
"role": "user",
"parts": [{"type": "text", "content": "What is the capital of France?"}]
}
]
```
**`gen_ai.output.messages`** (JSON string):
```json
[
{
"role": "assistant",
"parts": [{"type": "text", "content": "The capital of France is Paris."}]
}
]
```
**`gen_ai.system_instructions`** (JSON string):
```json
[{"type": "text", "content": "You are a helpful geography assistant."}]
```
## Existing Observability Stack
Latitude works alongside your existing observability tools. In TypeScript, `new Latitude()` detects common OpenTelemetry-compatible providers (Sentry, Datadog, New Relic, Honeycomb, and custom OTel SDKs) and attaches Latitude when possible. In Python, `Latitude(...)` attaches to the registered OpenTelemetry provider when one already exists or creates one when none exists. Custom setups in either SDK can also add `LatitudeSpanProcessor` as an additional span processor so traces go to both Latitude and your current backend.
`await latitude.ready` is optional in these examples. Use it during startup only when you need to guarantee Latitude's instrumentations are registered before the first LLM call.
### With Datadog (TypeScript)
```ts
import OpenAI from "openai"
import tracer from "dd-trace"
import { Latitude } from "@latitude-data/telemetry"
tracer.init({ service: "my-app", env: "production" })
const latitude = new Latitude({
apiKey: process.env.LATITUDE_API_KEY!,
project: process.env.LATITUDE_PROJECT_SLUG!,
instrumentations: { openai: OpenAI },
})
```
### With Sentry (TypeScript)
```ts
import OpenAI from "openai"
import * as Sentry from "@sentry/node"
import { Latitude } from "@latitude-data/telemetry"
Sentry.init({
dsn: process.env.SENTRY_DSN,
tracesSampleRate: 1.0,
})
const latitude = new Latitude({
apiKey: process.env.LATITUDE_API_KEY!,
project: process.env.LATITUDE_PROJECT_SLUG!,
instrumentations: { openai: OpenAI },
})
```
### With New Relic (TypeScript)
Enable New Relic's OpenTelemetry bridge first, then construct `new Latitude()`. New Relic registers an OpenTelemetry provider that Latitude can reuse.
```ts
import "newrelic"
import OpenAI from "openai"
import { Latitude } from "@latitude-data/telemetry"
const latitude = new Latitude({
apiKey: process.env.LATITUDE_API_KEY!,
project: process.env.LATITUDE_PROJECT_SLUG!,
instrumentations: { openai: OpenAI },
})
```
### With Honeycomb (TypeScript)
Start Honeycomb's `HoneycombSDK` first, then construct `new Latitude()`. Honeycomb registers an OpenTelemetry provider that Latitude can reuse.
```ts
import OpenAI from "openai"
import { HoneycombSDK } from "@honeycombio/opentelemetry-node"
import { Latitude } from "@latitude-data/telemetry"
const honeycomb = new HoneycombSDK({ serviceName: "my-app" })
honeycomb.start()
const latitude = new Latitude({
apiKey: process.env.LATITUDE_API_KEY!,
project: process.env.LATITUDE_PROJECT_SLUG!,
instrumentations: { openai: OpenAI },
})
```
### Other Platforms
For any observability platform that supports OpenTelemetry (Jaeger, Grafana Tempo, etc.), the pattern is the same: initialize the existing SDK first and then construct `Latitude`, or configure an additional OTLP exporter pointed at `https://ingest.latitude.so/v1/traces` with the required `Authorization` and `X-Latitude-Project` headers alongside your existing exporter.
## Troubleshooting
### 401 Unauthorized
The `Authorization` header must use the `Bearer ` prefix (with a space). Double-check your API key is valid and not expired.
### 400 Bad Request — missing project header
The `X-Latitude-Project` header is required on every request. Ensure it is present and spelled correctly, and that the value matches a project slug in the organization associated with your API key.
### 202 but no traces visible
- **Empty body:** An empty request body is accepted with `202` but produces no traces. Ensure your exporter is actually attaching span data.
- **Smart filter (SDK only):** If you are using a Latitude SDK's `LatitudeSpanProcessor`, only LLM-relevant spans are exported by default. This does not apply when sending OTLP directly — all spans are ingested.
- **Flush before exit:** Make sure your tracer provider flushes pending spans before the process exits.
### Traces appear but show no model, tokens, or messages
This is the most common issue when integrating via the generic OTLP exporter. Your exporter is configured correctly (you get `202` and traces appear), but the Latitude UI shows empty LLM metadata.
**Cause:** Your spans are missing the `gen_ai.*` semantic convention attributes. Configuring the exporter only ensures spans reach Latitude — it does not add LLM metadata. You must explicitly set attributes like `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.input.messages`, etc. on your spans.
**Fix:** See [GenAI Span Attributes (LLM Metadata)](#genai-span-attributes-llm-metadata) above for the full list of attributes and message format. At a minimum, set:
1. `gen_ai.provider.name` — provider name (e.g. `"openai"`, `"anthropic"`). Latitude also accepts the deprecated `gen_ai.system` as a fallback for older emitters.
2. `gen_ai.request.model` — model name
3. `gen_ai.operation.name` — set to `"chat"`
4. `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` — token counts
5. `gen_ai.input.messages` and `gen_ai.output.messages` — the actual conversation content
If you use a Latitude SDK or an OpenTelemetry auto-instrumentation library (e.g. `opentelemetry-instrumentation-openai`), these attributes are set automatically. This section only applies when you manually create spans for LLM calls without auto-instrumentation.
### Encoding
The endpoint accepts both `application/json` (JSON-encoded OTLP) and `application/x-protobuf` (protobuf-encoded OTLP). Most OTel SDK exporters default to protobuf; both work.
---
### Telemetry Pi Coding Agent (`docs/telemetry/pi-coding-agent.md`)
import SkillsCallout from "/snippets/skills-callout.mdx"
import FirstArtifact from "/snippets/first-artifact.mdx"
# Pi coding agent telemetry
Stream [pi coding agent](https://pi.dev) sessions into Latitude as traces with the first-party `@latitude-data/pi-telemetry` extension. No Latitude account yet? Your agent can create a temporary one and do this whole setup with the [`latitude-setup` skill](/getting-started/skills), no signup.
After setup, pi prompts appear in your Latitude project's **Traces** view with model calls, prompts, responses, token usage, tool calls, and tool results.
## Prerequisites
- A [Latitude account](https://console.latitude.so/login) with a project, or none yet: your agent can create a temporary one with the [`latitude-setup` skill](/getting-started/skills), no signup, and fill in the values below
- A Latitude API key from **Settings → API Keys**
- Your Latitude project slug from the project sidebar
- `pi` installed locally
- Node.js available on your `PATH`
## Install
Run the installer:
```bash
npx -y @latitude-data/pi-telemetry install
```
The installer prompts for your API key and project slug, adds the pi extension package to `~/.pi/agent/settings.json`, and writes Latitude config to `~/.pi/agent/latitude-telemetry.json`.
You can also pass values directly:
```bash
npx -y @latitude-data/pi-telemetry install \
--api-key=lat_xxx \
--project=your-project-slug \
--yes
```
## Restart and verify
Restart pi so it can install and load the package. Send a prompt that uses the model or a tool, then open your Latitude project and go to **Traces**. The new trace should appear within a few seconds.
A trace includes:
- an `interaction` root span for the pi prompt
- `llm_request` spans for model calls
- `tool_call:` spans for tool executions
- `gen_ai.input.messages` and `gen_ai.output.messages` for prompt/response reconstruction
- `gen_ai.tool.call.arguments` and `gen_ai.tool.call.result` for tool I/O
## See what was captured
## Structural-only telemetry
By default, Latitude receives the content needed to reconstruct pi runs, including prompts, assistant responses, system instructions when available, tool arguments, and tool results.
If you want trace structure without conversation or tool content, install with:
```bash
npx -y @latitude-data/pi-telemetry install --no-content
```
Structural-only traces still include timing, token usage, provider/model names, tool names, session metadata, user/host identity, and whether content capture was enabled.
## Disable or uninstall
Disable the extension for one pi process:
```bash
LATITUDE_PI_TELEMETRY_ENABLED=0 pi
```
Remove the extension and config:
```bash
npx -y @latitude-data/pi-telemetry uninstall
```
Restart pi after uninstalling to unload the extension.
## Advanced configuration
The installer stores config in `~/.pi/agent/latitude-telemetry.json`. Environment variables override file values:
| Variable | Description |
| --- | --- |
| `LATITUDE_API_KEY` | API key used to upload traces |
| `LATITUDE_PROJECT` or `LATITUDE_PROJECT_SLUG` | Project slug for `X-Latitude-Project` |
| `LATITUDE_BASE_URL` | Ingest base URL, default `https://ingest.latitude.so` |
| `LATITUDE_DEBUG=1` | Print upload diagnostics |
| `LATITUDE_PI_NO_CONTENT=1` | Disable prompt/response/tool content capture |
| `LATITUDE_PI_USER_ID` | Override `user.id` |
| `LATITUDE_PI_USER_EMAIL` | Override user email |
| `LATITUDE_PI_USER_NAME` | Override user display name |
| `LATITUDE_REDACT_ATTRIBUTES` | JSON array or comma-separated custom attribute patterns to mask before export |
| `LATITUDE_REDACT_MASK` | Mask value for custom redaction, default `******` |
Use `--staging`, `--dev`, or `--base-url=` during install to target a non-production Latitude ingest endpoint.
## Captured data and privacy
Full-content mode sends prompts, responses, system instructions when available, tool inputs, and tool outputs. Use `--no-content` for structural-only telemetry or configure custom redaction as described below.
Disable telemetry before working with sensitive material you do not want sent to Latitude. Even `--no-content` mode still sends structural metadata such as cwd/session identifiers, model names, tool names, timing, token usage, and user/host identity.
### Custom redaction
If you want to keep content capture enabled but mask specific span attributes before they leave your machine, use `redact` in `~/.pi/agent/latitude-telemetry.json` or the `LATITUDE_REDACT_ATTRIBUTES` environment variable. Redaction happens locally, after the content gate and before the OTLP export.
`redact.attributes` (or `LATITUDE_REDACT_ATTRIBUTES`) accepts an array of patterns. Each pattern can be:
- An **exact attribute name** — `"gen_ai.tool.call.arguments"`
- A **regex source string** — `"^gen_ai\\.(input|output)\\.messages$"` (anchored match)
- A **`/pattern/flags` string** — `"/^gen_ai\\.tool\\.call\\.(arguments|result)$/i"`
`redact.mask` (or `LATITUDE_REDACT_MASK`) sets the replacement value (default: `******`). Set it to `"[]"` to replace message arrays with an empty array instead of a string.
#### Via config file
Add a `redact` block to `~/.pi/agent/latitude-telemetry.json`:
```json
{
"redact": {
"attributes": [
"/^gen_ai\\.(input|output)\\.messages$/",
"gen_ai.tool.call.arguments",
"gen_ai.tool.call.result"
],
"mask": "[]"
}
}
```
#### Via environment variable
```bash
LATITUDE_REDACT_ATTRIBUTES='["/^gen_ai\\.(input|output)\\.messages$/", "gen_ai.tool.call.arguments"]' \
LATITUDE_REDACT_MASK='[]' \
pi
```
## Troubleshooting
**No traces appear.** Restart pi or run `/reload`, then send a new prompt. Confirm the API key and project slug are correct.
**HTTP 401.** The API key is missing or invalid. Re-run the installer with a valid key.
**HTTP 400 about `X-Latitude-Project`.** The project slug is missing. Re-run the installer with `--project=your-project-slug`.
**Traces show timing but no prompt/response content.** Structural-only mode is enabled. Reinstall without `--no-content`, or set `LATITUDE_PI_NO_CONTENT=0` and reload pi.
**Need diagnostics.** Run `LATITUDE_DEBUG=1 pi` and trigger another prompt.
---
### Telemetry Prime Intellect (`docs/telemetry/prime-intellect.md`)
import SkillsCallout from "/snippets/skills-callout.mdx"
import FirstArtifact from "/snippets/first-artifact.mdx"
# Prime Intellect (Verifiers) telemetry
Stream [Prime Intellect Verifiers](https://github.com/PrimeIntellect-ai/verifiers) eval rollouts into Latitude as traces. After setup, each rollout appears in your project's **Traces** view with prompts, model calls, tool calls, token usage, timing, and rewards — optionally as Latitude custom scores. No Latitude account yet? Your agent can create a temporary one and do this whole setup with the [`latitude-setup` skill](/getting-started/skills), no signup.
## Prerequisites
- A [Latitude account](https://console.latitude.so/login) with a project, or none yet: your agent can create a temporary one with the [`latitude-setup` skill](/getting-started/skills), no signup, and fill in the values below
- A Verifiers v1 eval setup (`verifiers.v1`, `uv run eval`, or `prime eval`)
- `pip` / `uv` in the same environment that runs your eval
## Install
1. In Latitude, copy your project slug from the project sidebar (or let the `latitude-setup` skill create a temporary account and project for you).
2. Create or copy an API key from **Settings → API Keys**.
3. Install the package into the env that runs Verifiers:
```bash
pip install latitude-telemetry-prime-intellect
# or: uv pip install latitude-telemetry-prime-intellect
```
4. Set credentials:
```bash
export LATITUDE_API_KEY=lat_xxx
export LATITUDE_PROJECT=your-project-slug
```
Traces go to Latitude Cloud ingest (`https://ingest.latitude.so`) by default. Scores use the public API (`https://api.latitude.so`). For self-hosted or local Latitude, set `LATITUDE_BASE_URL` (ingest origin, no `/v1/traces` suffix) and `LATITUDE_API_BASE_URL` (API origin) to match that instance.
## Usage
### From a Python eval script
```python
from latitude_telemetry_prime_intellect import export_episodes
episodes = await run_eval(env, config)
export_episodes(episodes)
```
Or attach an `on_complete` callback to `Env.run_slot`:
```python
from latitude_telemetry_prime_intellect import make_on_complete
on_complete = make_on_complete(next=your_persist_callback)
await env.run_slot(slot, ctx, semaphore, on_complete)
```
### After a CLI eval (post-hoc)
Run your eval as usual, then export the results directory:
```bash
latitude-prime-intellect-export export ./outputs/
```
The CLI looks for `traces.jsonl`, `episodes.jsonl`, or `results.jsonl`.
## Verify
Open your Latitude project → **Traces**. New rollouts should appear within a few seconds after export. If you left score export enabled (default), rewards show up as custom scores on those traces.
If nothing arrives, set `LATITUDE_DEBUG=true` and re-run the export.
## See what was captured
## Structural-only telemetry
```bash
export LATITUDE_NO_CONTENT=true
```
Structural-only traces still include timing, model, token usage, and span structure. Message and tool content are omitted.
## Disable scores
```bash
export LATITUDE_EXPORT_SCORES=false
# or: latitude-prime-intellect-export export ./outputs/ --no-scores
```
## Configuration
| Env | Default | Description |
|-----|---------|-------------|
| `LATITUDE_API_KEY` | — | API key (required) |
| `LATITUDE_PROJECT` / `LATITUDE_PROJECT_SLUG` | — | Project slug (required) |
| `LATITUDE_BASE_URL` | `https://ingest.latitude.so` | Ingest origin (appends `/v1/traces`) |
| `LATITUDE_API_BASE_URL` | `https://api.latitude.so` | Public API origin for scores |
| `LATITUDE_EXPORT_SCORES` | `true` | POST rewards/metrics as custom scores |
| `LATITUDE_PRIME_INTELLECT_TELEMETRY_ENABLED` / `LATITUDE_TELEMETRY_ENABLED` | `true` | Master switch |
| `LATITUDE_NO_CONTENT` | `false` | Structure/timing only |
| `LATITUDE_DEBUG` | `false` | Verbose logging |
Telemetry stays off until both `LATITUDE_API_KEY` and a project are set.
## How it works
Verifiers v1 records each rollout as a typed `Trace` (messages, `ModelCall`s, tools, rewards, timing). This package maps that record to OTLP:
```text
interaction (one rollout; session = eval/episode id)
├── llm_request (one per model call)
└── tool_call: (tool_execution; one per tool result)
```
It POSTs traces to ingest and, when enabled, POSTs each reward as a custom score (`sourceId = verifiers.reward.`) against the same 32-hex trace id. Export is fail-open: a Latitude error never fails your eval.
## Captured data and privacy
By default Latitude receives prompts, responses, tool I/O, model metadata, token usage, and reward values.
- Set `LATITUDE_NO_CONTENT=true` for structural telemetry only.
- Disable or skip export before working with sensitive material you do not want sent to Latitude.
## Troubleshooting
**No traces appear.** Confirm `LATITUDE_API_KEY` and `LATITUDE_PROJECT` are set in the same environment that runs the export, and that the results path contains a `.jsonl` file.
**Scores missing.** Check `LATITUDE_EXPORT_SCORES` is not false, and that `LATITUDE_API_BASE_URL` points at the same Latitude instance as your API key.
**Traces show timing but no content.** Structural-only mode is on — unset `LATITUDE_NO_CONTENT`.
---
### Telemetry Project Scoping (`docs/telemetry/project-scoping.md`)
---
title: Project scoping
description: Route spans to the right Latitude project — one project per process, or many.
---
Latitude can route spans to different projects based on how each capture is configured. Most
SDKs already set a project at startup; this page covers the additional knobs available when
**one process needs to emit to multiple projects** (for example, a service that runs several
agents that should each show up as a distinct Latitude project).
## The resolution chain
For every span Latitude ingests, the server applies this order (highest priority first):
1. **Span attribute** `latitude.project`
Set by `capture({ project })` in the TypeScript SDK or `capture({"project": ...})` in
Python. The slug travels on the span itself, so a single OTLP export can carry spans
for multiple projects.
2. **OTEL resource attribute** `latitude.project`
For bare-OpenTelemetry setups: set `latitude.project` once on the SDK's `Resource` and every
span inherits it.
3. **HTTP header** `X-Latitude-Project`
The Latitude SDKs send this automatically when `project` is passed to the SDK constructor.
Acts as the per-batch default.
The SDK option was renamed from `projectSlug` / `project_slug` to `project` in
`@latitude-data/telemetry` ≥ `3.0.0-alpha.12` and `latitude-telemetry` ≥ `3.0.0a8`. The old
names still work but log a deprecation warning — see each SDK's CHANGELOG for the migration
diff. The `X-Latitude-Project` header name and the `latitude.project` span attribute are
unchanged.
If none of those resolve to a project that belongs to your organization, the span is rejected.
## Response contract (OTLP `partial_success`)
Latitude follows the [OTLP specification](https://opentelemetry.io/docs/specs/otlp/) for
partial-success responses. Customers writing their own exporters can rely on these guarantees:
| Batch shape | HTTP status | Body |
| ------------------------------------ | ----------- | --------------------------------------------------------------- |
| All spans resolve | **200 OK** | empty `ExportTraceServiceResponse` |
| Some spans rejected (mixed) | **200 OK** | `partialSuccess { rejectedSpans: N, errorMessage: "..." }` |
| Every span rejected | **400** | `google.rpc.Status` shape: `{ code: 400, message: "..." }` |
| Malformed OTLP payload | **400** | `{ error: "..." }` |
Note: `partialSuccess` only appears on **2xx** responses — it means "some spans were
persisted, here is a count of what we dropped". When **nothing** was persisted we return a
plain `google.rpc.Status`-shaped error body so exporters don't mistakenly treat the rejection
as a partial save.
The `errorMessage` always points back to this page so an exporter that logs the response
gives operators a single place to look. **Don't move this page** without coordinating a
server-side update — the URL is hard-coded in the ingest error response.
## Patterns
Three runnable examples ship with each SDK. Start with the one that matches your shape.
### 1. Single-project default (existing setup)
For services that emit to exactly one Latitude project. The constructor `project` is sent on
every export as `X-Latitude-Project`, and every `capture()` inherits it.
**TypeScript** — see [`examples/test_project_scoping_single.ts`](https://github.com/latitude-dev/latitude-llm/blob/main/packages/telemetry/typescript/examples/test_project_scoping_single.ts):
```ts
import { capture, Latitude } from "@latitude-data/telemetry"
const latitude = new Latitude({
apiKey: process.env.LATITUDE_API_KEY!,
project: process.env.LATITUDE_PROJECT_SLUG!,
})
await capture("greet", async () => {
// routed to LATITUDE_PROJECT_SLUG
})
```
**Python** — see [`examples/test_project_scoping_single.py`](https://github.com/latitude-dev/latitude-llm/blob/main/packages/telemetry/python/examples/test_project_scoping_single.py):
```python
from latitude_telemetry import Latitude, capture
latitude = Latitude(
api_key=os.environ["LATITUDE_API_KEY"],
project=os.environ["LATITUDE_PROJECT_SLUG"],
)
@capture("greet")
def greet():
... # routed to LATITUDE_PROJECT_SLUG
```
### 2. Multi-project from day 1 (no constructor default)
For services that emit to multiple Latitude projects — e.g. one process that runs several
agents. Skip the constructor `project` and set `project` on every `capture()`.
**TypeScript** — see [`examples/test_project_scoping_multi.ts`](https://github.com/latitude-dev/latitude-llm/blob/main/packages/telemetry/typescript/examples/test_project_scoping_multi.ts):
```ts
import { capture, Latitude } from "@latitude-data/telemetry"
const latitude = new Latitude({ apiKey: process.env.LATITUDE_API_KEY! })
await capture(
"full-stack-agent-run",
async () => { /* ... */ },
{ project: "full-stack-agent" },
)
await capture(
"call-summariser-run",
async () => { /* ... */ },
{ project: "call-summariser" },
)
```
**Python** — see [`examples/test_project_scoping_multi.py`](https://github.com/latitude-dev/latitude-llm/blob/main/packages/telemetry/python/examples/test_project_scoping_multi.py):
```python
from latitude_telemetry import Latitude, capture
latitude = Latitude(api_key=os.environ["LATITUDE_API_KEY"])
@capture("full-stack-agent-run", {"project": "full-stack-agent"})
def run_full_stack_agent():
...
@capture("call-summariser-run", {"project": "call-summariser"})
def run_call_summariser():
...
```
A `capture()` that doesn't set `project` AND an SDK that doesn't have a constructor default →
the spans are rejected with `partialSuccess`. Pick one of the two so every span has a route.
### 3. Per-span override on top of a default
Most common when the default project comes from an env var but specific runs need to route
elsewhere (e.g. shipping evaluation runs to a separate project).
**TypeScript** — see [`examples/test_project_scoping_env.ts`](https://github.com/latitude-dev/latitude-llm/blob/main/packages/telemetry/typescript/examples/test_project_scoping_env.ts):
```ts
const latitude = new Latitude({
apiKey: process.env.LATITUDE_API_KEY!,
project: process.env.LATITUDE_PROJECT_SLUG!, // env-driven default
})
// inherits the default
await capture("default-route", async () => { /* ... */ })
// per-capture override — wins over the constructor default
await capture(
"evaluation-batch",
async () => { /* ... */ },
{ project: "evaluation-runs" },
)
```
**Python** — see [`examples/test_project_scoping_env.py`](https://github.com/latitude-dev/latitude-llm/blob/main/packages/telemetry/python/examples/test_project_scoping_env.py):
```python
latitude = Latitude(
api_key=os.environ["LATITUDE_API_KEY"],
project=os.environ["LATITUDE_PROJECT_SLUG"],
)
@capture("default-route")
def default_route():
...
@capture("evaluation-batch", {"project": "evaluation-runs"})
def evaluation_batch():
...
```
## Handling `partial_success` in custom exporters
If you're integrating from bare OpenTelemetry (no Latitude SDK) and want to surface
rejections, watch for `partialSuccess.rejectedSpans > 0` in the OTLP response body. The
common causes:
- Missing `latitude.project` on the span AND no `X-Latitude-Project` header on the export.
- The resolved slug doesn't exist in your organization (typo, or a stale slug after the
project was renamed).
The fix is always to make sure each span has a way to resolve a project — either the span
attribute, a resource attribute, or the header default.
---
### Telemetry Python (`docs/telemetry/python.md`)
---
title: Python SDK
description: Full API reference for latitude-telemetry, the Python SDK for Latitude Telemetry.
---
# Python SDK
Instrument your AI application and send traces to Latitude. Built on OpenTelemetry.
## Installation
```bash
pip install latitude-telemetry
```
Requires Python 3.11+.
## Bootstrap (Recommended)
The fastest way to start. One class sets up a complete OpenTelemetry pipeline with LLM auto-instrumentation and the Latitude exporter:
```python
import openai
from openai import OpenAI
from latitude_telemetry import Latitude
latitude = Latitude(
api_key="your-api-key",
project="your-project-slug",
instrumentations={"openai": openai},
)
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}],
)
latitude.shutdown()
```
`instrumentations` takes a dict mapping integration name (`openai`, `anthropic`, …) to the LLM SDK module the consumer imports. Passing the user's own module reference sidesteps a class of import-cache bugs where the SDK could patch a different module instance than the app loads.
## Using `capture()` for Context
Auto-instrumentation traces LLM calls without `capture()`. Use `capture()` when you want to:
- **Group traces by user or session**: Track all LLM calls from a specific user
- **Add business context**: Tag traces with environment, feature flags, or request IDs
- **Mark agent boundaries**: Wrap an agent run or conversation turn with a name and metadata
- **Filter and analyze**: Use tags and metadata to filter traces in Latitude
```python
import openai
from latitude_telemetry import Latitude, capture
latitude = Latitude(
api_key="your-api-key",
project="your-project-slug",
instrumentations={"openai": openai},
)
capture(
"handle-user-request",
lambda: client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": user_message}],
),
{
"user_id": "user_123",
"session_id": "session_abc",
"tags": ["production", "v2-agent"],
"metadata": {"request_id": "req-xyz", "feature_flag": "new-prompt"},
},
)
latitude.shutdown()
```
`capture()` does **not** create spans. It only attaches context to spans created by auto-instrumentation. Use one `capture()` call at the request or agent boundary. Nested calls inherit from the parent context with local overrides.
**Nesting behavior:**
| Field | Behavior |
|---|---|
| `user_id` | Last-write-wins |
| `session_id` | Last-write-wins |
| `metadata` | Shallow merge |
| `tags` | Append and dedupe, preserving order |
## Existing OpenTelemetry Setup (Advanced)
If your app already uses OpenTelemetry, add Latitude alongside your existing processors:
```python
import openai
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from latitude_telemetry import LatitudeSpanProcessor, register_latitude_instrumentations
provider = TracerProvider()
provider.add_span_processor(LatitudeSpanProcessor("api-key", "project-slug"))
trace.set_tracer_provider(provider)
register_latitude_instrumentations(
instrumentations={"openai": openai},
tracer_provider=provider,
)
```
`LatitudeSpanProcessor` only exports spans to Latitude. You still need LLM instrumentations to create those spans. Use `register_latitude_instrumentations()` or bring your own OTel-compatible LLM instrumentation.
For examples of integrating with **Datadog**, **Sentry**, or other observability platforms, see the [OpenTelemetry Exporter](otel-exporter) guide. That guide also covers connecting from **any language** beyond TypeScript and Python.
## Public API Reference
```python
from latitude_telemetry import (
Latitude,
LatitudeOptions,
LatitudeSpanProcessor,
capture,
register_latitude_instrumentations,
)
```
### `Latitude(**options)`
Bootstraps a complete OpenTelemetry setup with LLM instrumentations and Latitude export. If an OpenTelemetry provider is already registered, Latitude attaches its span processor to that provider instead of replacing it.
```python
class Latitude:
def __init__(
self,
*,
api_key: str,
# Default project for spans. Optional — every `capture()` can override.
# Sent as the `X-Latitude-Project` header on every export.
project: str | None = None,
# DEPRECATED alias for `project`. Still accepted; logs a one-time warning.
project_slug: str | None = None,
# Dict mapping integration name → the LLM SDK module the consumer imports.
# Anything else (list, primitive, unknown key, non-dict) raises TypeError.
instrumentations: InstrumentationsInput | None = None,
service_name: str | None = None,
disable_batch: bool = False,
disable_smart_filter: bool = False,
should_export_span: Callable[[ReadableSpan], bool] | None = None,
blocked_instrumentation_scopes: list[str] | None = None,
disable_redact: bool = False,
redact: RedactSpanProcessorOptions | None = None,
exporter: SpanExporter | None = None,
tracer_provider: TracerProvider | None = None,
):
...
provider: TracerProvider
def flush(self) -> None: ...
def shutdown(self) -> None: ...
```
`init_latitude()` remains available as a backwards-compatible wrapper that returns `{"provider", "flush", "shutdown"}`.
### `capture(name, fn, options=None)`
Wraps a function to attach Latitude context to all spans created inside. Uses OpenTelemetry's native context API for scoping.
```python
def capture(
name: str,
fn: Callable[[], T],
options: dict | None = None,
) -> T:
...
# options keys:
# {
# "name": str | None,
# "user_id": str | None,
# "session_id": str | None,
# "tags": list[str] | None,
# "metadata": dict | None,
# "project": str | None, # Route this capture (and child spans) to a
# # specific Latitude project, overriding the default.
# "project_slug": str | None, # DEPRECATED alias for `project`. Still accepted.
# }
```
| Option | Type | OTel Attribute | Description |
|---|---|---|---|
| `name` | `str` | `latitude.capture.name` | Name for the capture context |
| `tags` | `list[str]` | `latitude.tags` | Tags for filtering traces |
| `metadata` | `dict[str, Any]` | `latitude.metadata` | Arbitrary key-value metadata |
| `session_id` | `str` | `session.id` | Group traces by session |
| `user_id` | `str` | `user.id` | Associate traces with a user |
| `project` | `str` | `latitude.project` | Route this capture to a specific Latitude project (overrides the constructor default) |
### `LatitudeSpanProcessor`
Span processor for shared-provider setups. Reads Latitude context from OTel context and stamps attributes onto spans.
```python
class LatitudeSpanProcessor:
def __init__(
self,
api_key: str,
project: str | None,
options: LatitudeSpanProcessorOptions | None = None,
):
...
@dataclass
class LatitudeSpanProcessorOptions:
disable_redact: bool = False
redact: RedactSpanProcessorOptions | None = None
disable_batch: bool = False
disable_smart_filter: bool = False
should_export_span: Callable[[ReadableSpan], bool] | None = None
blocked_instrumentation_scopes: tuple[str, ...] = ()
exporter: SpanExporter | None = None
service_name: str | None = None
```
### `register_latitude_instrumentations(instrumentations, tracer_provider)`
Registers LLM auto-instrumentations against a specific tracer provider.
```python
# InstrumentationName = Literal[
# "openai", "openai-agents", "anthropic", "bedrock", "cohere",
# "langchain", "llamaindex", "togetherai", "vertexai", "aiplatform",
# "aleph_alpha", "crewai", "dspy", "google_generativeai", "groq",
# "haystack", "litellm", "mistralai", "ollama", "replicate",
# "sagemaker", "transformers", "watsonx",
# ]
# InstrumentationsInput = dict[InstrumentationName, object]
def register_latitude_instrumentations(
# Dict mapping integration name → the LLM SDK module the consumer imports.
# Anything else throws at register time.
instrumentations: InstrumentationsInput,
tracer_provider: TracerProvider,
) -> None:
...
```
## Migrating from `instrumentations=["openai"]` (3.0.0a6 and earlier)
The list-of-strings form is removed with no fallback in `3.0.0a7`. Anything other than a plain dict raises `TypeError` at register time. Migration:
```diff
- from latitude_telemetry import Latitude
+ import openai
+ import anthropic
+ from latitude_telemetry import Latitude
latitude = Latitude(
api_key="your-api-key",
project="your-project-slug",
- instrumentations=["openai", "anthropic"],
+ instrumentations={"openai": openai, "anthropic": anthropic},
)
```
## Supported Providers
Set the integration's key on the `instrumentations` dict to the LLM SDK module the consumer imports.
| Key | PyPI package | What to pass |
| --------------------- | ----------------------------- | ------------------------------------------- |
| `openai` | `openai` | `import openai` → `openai` |
| `openai-agents` | `openai-agents` | `import agents` → `agents` |
| `anthropic` | `anthropic` | `import anthropic` → `anthropic` |
| `bedrock` | `boto3` | `import boto3` → `boto3` |
| `cohere` | `cohere` | `import cohere` → `cohere` |
| `langchain` | `langchain-core` | `import langchain_core` → that module |
| `llamaindex` | `llama-index` | `import llama_index` → that module |
| `togetherai` | `together` | `import together` → `together` |
| `vertexai` | `google-cloud-aiplatform` | `import vertexai` → `vertexai` |
| `aiplatform` | `google-cloud-aiplatform` | `import google.cloud.aiplatform` → that module |
| `aleph_alpha` | `aleph-alpha-client` | `import aleph_alpha_client` |
| `crewai` | `crewai` | `import crewai` |
| `dspy` | `dspy-ai` | `import dspy` |
| `google_generativeai` | `google-generativeai` | `from google import genai` → `genai` |
| `groq` | `groq` | `import groq` |
| `haystack` | `haystack-ai` | `import haystack` |
| `litellm` | `litellm` | `import litellm` |
| `mistralai` | `mistralai` | `import mistralai` |
| `ollama` | `ollama` | `import ollama` |
| `replicate` | `replicate` | `import replicate` |
| `sagemaker` | `boto3` | `import boto3` → `boto3` |
| `transformers` | `transformers` | `import transformers` |
| `watsonx` | `ibm-watson-machine-learning` | `import ibm_watsonx_ai` |
## Configuration
### Smart Filtering
By default, only LLM-relevant spans are exported (spans with `gen_ai.*`, `llm.*`, `openinference.*`, or `ai.*` attributes, plus known LLM instrumentation scopes):
```python
processor = LatitudeSpanProcessor(
"api-key",
"project-slug",
LatitudeSpanProcessorOptions(
disable_smart_filter=True, # Export all spans
),
)
```
### Redaction
PII redaction is enabled by default for security-sensitive attributes:
- HTTP authorization headers
- HTTP cookies
- HTTP API key headers (`x-api-key`)
- Database statements
```python
from latitude_telemetry import LatitudeSpanProcessor, RedactSpanProcessorOptions
processor = LatitudeSpanProcessor(
"api-key",
"project-slug",
LatitudeSpanProcessorOptions(
disable_redact=True, # Disable all redaction
redact=RedactSpanProcessorOptions(
attributes=[r"^password$", r"secret"],
mask=lambda attr, value: "[REDACTED]",
),
),
)
```
### Custom Filtering
```python
processor = LatitudeSpanProcessor(
"api-key",
"project-slug",
LatitudeSpanProcessorOptions(
should_export_span=lambda span: span.attributes.get("custom") is True,
blocked_instrumentation_scopes=["opentelemetry.instrumentation.fs"],
),
)
```
## Environment Variables
| Variable | Default | Description |
|---|---|---|
| `LATITUDE_TELEMETRY_URL` | `http://localhost:3002` | OTLP exporter endpoint |
## Troubleshooting
### Spans not appearing in Latitude
1. **Check API key and project slug**: Must be non-empty strings.
2. **Verify instrumentations are registered**: Create `Latitude(...)` before importing or constructing clients when possible, or use `register_latitude_instrumentations()` for manual setups.
3. **Flush before exit**: Call `latitude.flush()` or `provider.force_flush()`.
4. **Check smart filter**: Only LLM spans are exported by default. Use `disable_smart_filter=True` to export all spans.
5. **Ensure `capture()` wraps the code that creates spans**: `capture()` itself doesn't create spans; it only attaches context.
### No spans created inside `capture()`
`capture()` only attaches context. You need:
1. An active instrumentation (e.g., `opentelemetry-instrumentation-openai`).
2. That instrumentation to create spans for the operations inside your callback.
### Context not propagating
Ensure you have a functioning OpenTelemetry context propagator registered:
```python
from opentelemetry.context import set_global_textmap
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.baggage.propagation import W3CBaggagePropagator
set_global_textmap(
CompositePropagator([TraceContextTextMapPropagator(), W3CBaggagePropagator()])
)
```
`Latitude(...)` does this automatically when it owns the provider. For shared-provider setups, your existing OTel setup should already have this.
---
### Telemetry Start Tracing (`docs/telemetry/start-tracing.mdx`)
---
title: Start tracing
description: Connect your agent to Latitude and send your first traces.
---
Coming from Langfuse, LangSmith, or Braintrust? Follow [Migrate to Latitude](./imports/migrate)
instead: it covers this setup plus porting your context, backfilling your existing traces
with a one-time [import](./imports/overview), and cutting over.
Use your agent to add Latitude tracing to your app or agent harness. The `latitude-setup` skill inspects your codebase, detects your LLM providers and existing OpenTelemetry setup, installs the right SDK or harness plugin, verifies that traces arrive in your project, and builds your first [Artifact](/more/artifacts) from them. If you don't have a Latitude account yet, it bootstraps a temporary one from the terminal, no signup, and gives you a link to claim it.
Paste this prompt into Claude Code, Cursor, Codex, OpenCode, Hermes, or any other agent:
```text
Install the `latitude-setup` skill from `github.com/latitude-dev/skills` and use it to set up Latitude tracing here, following best practices.
```
Trigger one agent request or LLM call after the install finishes.
Open your Latitude project and go to **Traces**. The new trace should appear within a few seconds.
Install the [Latitude MCP server](../getting-started/mcp) so your agent can access your Latitude workspace, find the right project, and verify the setup.
## Manual setup
If you are not using an agent, start with the SDK for your runtime:
Install `@latitude-data/telemetry` and instrument TypeScript or JavaScript
apps.
Install `latitude-telemetry` and instrument Python apps.
For other languages, use the [OpenTelemetry exporter](../telemetry/otel-exporter).
## Next steps
- Explore traces in [Observability](../observability/overview)
- Add provider-specific instrumentation from the Observability sidebar
- Trace your agent's long-term memory with [Memory tracing](./memory)
- Already have traces elsewhere? [Import them as a one-time backfill](./imports/overview) so you do not start from an empty project
- Learn how traces become [scores](../scores/overview), [signals](../signals/overview), and [evaluations](../evaluations/overview)
---
### Telemetry Typescript (`docs/telemetry/typescript.md`)
---
title: TypeScript SDK
description: Full API reference for @latitude-data/telemetry, the TypeScript SDK for Latitude Telemetry.
---
# TypeScript SDK
Instrument your AI application and send traces to Latitude. Built on OpenTelemetry.
## Installation
```bash
npm install @latitude-data/telemetry
```
## Bootstrap (Recommended)
The fastest way to start. One class detects an existing OpenTelemetry pipeline (Sentry, Datadog, New Relic, Honeycomb, etc.) or creates one when none exists, then adds LLM auto-instrumentation and the Latitude exporter:
```ts
import OpenAI from "openai"
import { Latitude } from "@latitude-data/telemetry"
const client = new OpenAI()
const latitude = new Latitude({
apiKey: process.env.LATITUDE_API_KEY!,
project: process.env.LATITUDE_PROJECT_SLUG!,
instrumentations: { openai: OpenAI },
})
await latitude.ready
const response = await client.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: "Hello" }],
})
await latitude.shutdown()
```
`instrumentations` takes a plain object mapping integration name (`openai`, `anthropic`, …) to the LLM SDK module the consumer imports in app code. The same module instance is used for both the patch and the actual LLM call, sidestepping the CJS/ESM dual-load class of bugs.
`new Latitude()` returns **immediately**. Instrumentation registration happens in the background. This avoids top-level await issues in CommonJS environments while still supporting ESM.
- **Fire-and-forget**: Start using your LLM clients right away. Early spans are captured once instrumentations finish registering.
- **Optional `await latitude.ready`**: If you need instrumentations fully registered before making LLM calls, await the `ready` promise.
- **Existing tracing SDKs**: Initialize Sentry or another tracing SDK first, then construct `new Latitude()`. Latitude attaches its span processor to the existing provider when possible. `latitude.shutdown()` only shuts down Latitude-owned processing and does not shut down the other SDK.
## Using `capture()` for Context
Auto-instrumentation traces LLM calls without `capture()`. Use `capture()` when you want to:
- **Group traces by user or session**: Track all LLM calls from a specific user
- **Add business context**: Tag traces with environment, feature flags, or request IDs
- **Mark agent boundaries**: Wrap an agent run or conversation turn with a name and metadata
- **Filter and analyze**: Use tags and metadata to filter traces in Latitude
```ts
import OpenAI from "openai"
import { Latitude, capture } from "@latitude-data/telemetry"
const latitude = new Latitude({
apiKey: process.env.LATITUDE_API_KEY!,
project: process.env.LATITUDE_PROJECT_SLUG!,
instrumentations: { openai: OpenAI },
})
await latitude.ready
await capture(
"handle-user-request",
async () => {
const result = await agent.process(userMessage)
return result
},
{
userId: "user_123",
sessionId: "session_abc",
tags: ["production", "v2-agent"],
metadata: { requestId: "req-xyz", featureFlag: "new-prompt" },
},
)
await latitude.shutdown()
```
`capture()` does **not** create spans. It only attaches context to spans created by auto-instrumentation. Use one `capture()` call at the request or agent boundary. Nested calls inherit from the parent context with local overrides.
**Nesting behavior:**
| Field | Behavior |
|---|---|
| `userId` | Last-write-wins |
| `sessionId` | Last-write-wins |
| `metadata` | Shallow merge |
| `tags` | Append and dedupe, preserving order |
## Existing Sentry or OpenTelemetry Setup
For Sentry, Datadog, New Relic, Honeycomb, and other OpenTelemetry-compatible SDKs, initialize the existing SDK first and construct `new Latitude()` second. Latitude detects the installed provider and attaches its processor when possible without replacing the existing SDK's context manager, propagator, sampler, or processors:
```ts
import OpenAI from "openai"
import * as Sentry from "@sentry/node"
import { Latitude } from "@latitude-data/telemetry"
Sentry.init({
dsn: process.env.SENTRY_DSN!,
tracesSampleRate: 1.0,
})
const latitude = new Latitude({
apiKey: process.env.LATITUDE_API_KEY!,
project: process.env.LATITUDE_PROJECT_SLUG!,
instrumentations: { openai: OpenAI },
})
await latitude.ready
// LLM spans are exported to both Sentry and Latitude.
await latitude.flush()
await latitude.shutdown()
```
If you want full control over provider construction, add Latitude alongside your existing processors explicitly:
```ts
import OpenAI from "openai"
import { NodeSDK } from "@opentelemetry/sdk-node"
import {
LatitudeSpanProcessor,
registerLatitudeInstrumentations,
} from "@latitude-data/telemetry"
const sdk = new NodeSDK({
spanProcessors: [
existingProcessor,
new LatitudeSpanProcessor(
process.env.LATITUDE_API_KEY!,
process.env.LATITUDE_PROJECT_SLUG!,
),
],
})
sdk.start()
await registerLatitudeInstrumentations({
instrumentations: { openai: OpenAI },
tracerProvider: sdk.getTracerProvider(),
})
```
For examples of integrating with **Datadog**, **Sentry**, **New Relic**, **Honeycomb**, or other observability platforms, see the [OpenTelemetry Exporter](otel-exporter) guide. That guide also covers connecting from **any language** beyond TypeScript and Python.
## Public API Reference
```ts
import {
Latitude,
LatitudeSpanProcessor,
capture,
registerLatitudeInstrumentations,
} from "@latitude-data/telemetry"
```
### `new Latitude(options)`
Bootstraps a complete OpenTelemetry setup with LLM instrumentations and Latitude export.
```ts
type LatitudeOptions = {
apiKey: string
// Default project for spans emitted by this SDK instance. Optional — every
// `capture()` can override. Sent as the `X-Latitude-Project` header on each export.
project?: string
// DEPRECATED alias for `project`. Still accepted; logs a one-time warning.
projectSlug?: string
// Map of integration name → LLM SDK module reference (e.g. { openai: OpenAI }).
// Anything else (string array, primitive, unknown key, non-object) throws at register time.
instrumentations?: InstrumentationsInput
serviceName?: string
disableBatch?: boolean
disableSmartFilter?: boolean
shouldExportSpan?: (span: ReadableSpan) => boolean
blockedInstrumentationScopes?: string[]
disableRedact?: boolean
redact?: RedactSpanProcessorOptions
exporter?: SpanExporter
tracerProvider?: TracerProvider
}
class Latitude {
constructor(options: LatitudeOptions)
provider: TracerProvider
ready: Promise
flush(): Promise
shutdown(): Promise
}
```
### `capture(name, fn, options?)`
Wraps a function to attach Latitude context to all spans created inside. Uses OpenTelemetry's native `context.with()` for scoping.
```ts
type ContextOptions = {
name?: string
userId?: string
sessionId?: string
tags?: string[]
metadata?: Record
// Route this capture (and its child spans) to a specific Latitude project,
// overriding the constructor's `project` default for the duration of the capture.
project?: string
// DEPRECATED alias for `project`. Still accepted; logs a one-time warning.
projectSlug?: string
}
function capture(
name: string,
fn: () => T | Promise,
options?: ContextOptions,
): T | Promise
```
| Option | Type | OTel Attribute | Description |
|---|---|---|---|
| `name` | `string` | `latitude.capture.name` | Name for the capture context |
| `tags` | `string[]` | `latitude.tags` | Tags for filtering traces |
| `metadata` | `Record` | `latitude.metadata` | Arbitrary key-value metadata |
| `sessionId` | `string` | `session.id` | Group traces by session |
| `userId` | `string` | `user.id` | Associate traces with a user |
### `LatitudeSpanProcessor`
Span processor for shared-provider setups. Reads Latitude context from OTel context and stamps attributes onto spans.
```ts
class LatitudeSpanProcessor implements SpanProcessor {
constructor(
apiKey: string,
project: string | undefined,
options?: LatitudeSpanProcessorOptions,
)
}
type LatitudeSpanProcessorOptions = {
disableBatch?: boolean
disableSmartFilter?: boolean
shouldExportSpan?: (span: ReadableSpan) => boolean
blockedInstrumentationScopes?: string[]
disableRedact?: boolean
redact?: RedactSpanProcessorOptions
exporter?: SpanExporter
}
```
### `registerLatitudeInstrumentations(options)`
Registers patch-based AI SDK instrumentations against a specific tracer provider.
```ts
type InstrumentationName =
| "openai"
| "openai-agents"
| "anthropic"
| "bedrock"
| "cohere"
| "langchain"
| "llamaindex"
| "togetherai"
| "vertexai"
| "aiplatform"
// `object | undefined` rejects primitive values (`true`, `42`, `"openai"`, …)
// at compile time while still admitting class constructors (functions),
// namespace imports, and explicit-undefined-for-conditional-config.
type InstrumentationsInput = Partial>
function registerLatitudeInstrumentations(options: {
// Map of integration name → LLM SDK module reference (e.g. { openai: OpenAI }).
// Anything else throws at register time.
instrumentations: InstrumentationsInput
tracerProvider: TracerProvider
}): Promise
```
## Supported Providers
Set the integration's key on the `instrumentations` object to the LLM SDK module the consumer imports. For SDKs whose Traceloop patch reads off the package namespace, pass `import * as X from ""`.
| Key | Package | What to pass |
|---|---|---|
| `openai` | `openai` | `openai: OpenAI` (default class — also accepts the namespace) |
| `openai-agents` | `@openai/agents` | `"openai-agents": OpenAIAgentsSDK` (namespace) |
| `anthropic` | `@anthropic-ai/sdk` | `anthropic: AnthropicSDK` (namespace; bare default class also accepted and rewrapped) |
| `bedrock` | `@aws-sdk/client-bedrock-runtime` | `bedrock: BedrockSDK` (namespace) |
| `cohere` | `cohere-ai` | `cohere: CohereSDK` (namespace) |
| `langchain` | `langchain` | `langchain: LangChain` (namespace) |
| `llamaindex` | `llamaindex` | `llamaindex: LlamaIndex` (namespace) |
| `togetherai` | `together-ai` | `togetherai: TogetherSDK` (namespace) |
| `vertexai` | `@google-cloud/vertexai` | `vertexai: VertexAISDK` (namespace) |
| `aiplatform` | `@google-cloud/aiplatform` | `aiplatform: AIPlatformSDK` (namespace) |
## Migrating from `instrumentations: ["openai"]` (3.0.0-alpha.10 and earlier)
The string-array form is removed with no fallback in `3.0.0-alpha.11`. Anything other than a plain object — including the old string array — **throws at register time**, so any existing install below `alpha.11` must be bumped *and* its call sites rewritten in the same change. Migration:
```diff
- import { Latitude } from "@latitude-data/telemetry"
+ import OpenAI from "openai"
+ import * as AnthropicSDK from "@anthropic-ai/sdk"
+ import { Latitude } from "@latitude-data/telemetry"
new Latitude({
apiKey: process.env.LATITUDE_API_KEY!,
project: process.env.LATITUDE_PROJECT_SLUG!,
- instrumentations: ["openai", "anthropic"],
+ instrumentations: { openai: OpenAI, anthropic: AnthropicSDK },
})
```
The `modules` option on `registerLatitudeInstrumentations` is also removed — pass the SDK module under its integration key on `instrumentations` instead.
## Configuration
### Smart Filtering
By default, only LLM-relevant spans are exported (spans with `gen_ai.*`, `llm.*`, `openinference.*`, or `ai.*` attributes, plus known LLM instrumentation scopes):
```ts
new LatitudeSpanProcessor(apiKey, project, {
disableSmartFilter: true, // Export all spans
})
```
### Redaction
PII redaction is enabled by default for security-sensitive attributes:
- HTTP authorization headers
- HTTP cookies
- HTTP API key headers (`x-api-key`)
- Database statements
```ts
new LatitudeSpanProcessor(apiKey, project, {
disableRedact: true, // Disable all redaction
redact: {
attributes: [/^password$/i, /secret/i], // Add custom patterns
mask: (attr, value) => "[REDACTED]",
},
})
```
### Custom Filtering
```ts
new LatitudeSpanProcessor(apiKey, project, {
shouldExportSpan: (span) => span.attributes["custom"] === true,
blockedInstrumentationScopes: ["opentelemetry.instrumentation.fs"],
})
```
## Environment Variables
| Variable | Default | Description |
|---|---|---|
| `LATITUDE_TELEMETRY_URL` | `http://localhost:3002` (dev) / `https://ingest.latitude.so` (prod) | OTLP exporter endpoint |
## Troubleshooting
### Spans not appearing in Latitude
1. **Check API key and project slug**: Must be non-empty strings.
2. **Verify instrumentations are registered**: Use `await latitude.ready` or `await registerLatitudeInstrumentations()`.
3. **Did the bootstrap throw a migration error?**: On `3.0.0-alpha.11`+, `instrumentations: ["openai"]` (or any non-object value) throws `TypeError: [Latitude] instrumentations must be an object mapping…`. Migrate to `instrumentations: { openai: OpenAI }`. See the Migration section above.
4. **Flush before exit**: Call `await latitude.flush()` or `await provider.forceFlush()`.
5. **Check smart filter**: Only LLM spans are exported by default. Use `disableSmartFilter: true` to export all spans.
6. **Ensure `capture()` wraps the code that creates spans**: `capture()` itself doesn't create spans; it only attaches context.
### No spans created inside `capture()`
`capture()` only attaches context. You need:
1. An active instrumentation (e.g., OpenAI auto-instrumentation).
2. That instrumentation to create spans for the operations inside your callback.
### Context not propagating
Ensure you have a functioning OpenTelemetry context manager registered:
```ts
import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks"
import { context } from "@opentelemetry/api"
context.setGlobalContextManager(new AsyncLocalStorageContextManager())
```
`new Latitude()` does this automatically. For shared-provider setups, your existing OTel setup should already have this.
---
### Test And Fix Regression Testing (`docs/test-and-fix/regression-testing.mdx`)
---
title: Regression testing
description: Replay a dataset of real traces against your agent and check the results with the same evaluations that monitor production, so fixed failures cannot quietly return.
---
**Where this fits:** This is the verification step of **Refine**. It takes a [dataset](../datasets/overview) built from a fixed [signal](../signals/overview) and proves the fix holds before and after you ship.
A **regression test** replays a set of known inputs against your agent and checks the results, so a failure you already fixed cannot return unnoticed. In Latitude, the inputs come from a [dataset](../datasets/overview) of real traces, and the checks reuse the same [evaluations](../evaluations/overview) that monitor production, so your test quality bar matches your production quality bar.
## Fix and verify with your coding agent
The fastest path today pairs the [MCP server](../getting-started/mcp) with a dataset, so your coding agent does the work:
Connect your coding agent (Claude, Cursor, and others) to Latitude through the [MCP server](../getting-started/mcp). It can read the failing [signal](../signals/overview), inspect the example traces, and propose a fix in the same session.
Turn the traces behind the signal into a [dataset](../datasets/overview), the seed of your regression test. Your agent can do this through the MCP, or you can [add the traces](../datasets/add-traces) from the UI.
Record what the agent should have done by [adding expected output](../datasets/expected-output) to the rows you want to check precisely.
Run your agent against each row's input to produce fresh outputs, then run the signal's [evaluations](../evaluations/overview) against them. The same check that found the failure in production now verifies the fix.
Pass when the results meet your quality bar; fail to block a regression. Re-run whenever the agent, prompts, tools, or models change.
## Run it in CI
You can drive a regression test from a dataset in your own pipeline:
- **Export** a [dataset](../datasets/overview) as CSV and replay its inputs in your own test harness.
- Submit the results back as [scores](../scores/overview) through the Scores API, so regression results live alongside your production data, and gate the build on the outcome.
## Multi-turn simulations
If the case is a conversation rather than a single input/output pair, run a [local simulation](./simulations): pull the dataset with the SDK, let a user simulator talk to your agent, and send the sessions back to Latitude so [Experiments](../experiments/overview) can compare versions.
## Next step
- [Simulate agents locally](./simulations): run the dataset as multi-turn conversations in TypeScript or Python.
- [Datasets](../datasets/overview): turn the traces behind a signal into a reusable test set.
---
### Test And Fix Simulations (`docs/test-and-fix/simulations.mdx`)
---
title: Simulate agents locally
description: Replay a Latitude dataset against your agent in TypeScript or Python, send the sessions back through telemetry, and compare versions with Experiments.
---
**Where this fits:** Simulations are part of **Refine**. They take a [dataset](../datasets/overview) of real conversations, run your agent against those cases in your own process, and land the new sessions in Latitude so you can compare versions with [Experiments](../experiments/overview) and [Signals](../signals/overview).
Latitude holds the dataset, the traces, the scores, and the comparison. Your agent runs in your own process, locally or in CI. Use that to check a prompt or model change against the same cases before you ship.
The runner is yours. A `for` loop is enough for single-turn rows. For multi-turn conversations, [Scenario](https://github.com/langwatch/scenario), [promptfoo](https://www.promptfoo.dev/), and [Inspect](https://inspect.aisi.org.uk/) all work. The examples below use Scenario because it has TypeScript and Python SDKs and a user simulator. Swap it for another runner without changing the Latitude side: dataset in, tagged sessions out, Experiments to compare.
## What a simulation is
1. Curate a [dataset](../datasets/overview) from real traces, a signal, or a CSV.
2. Pull the rows with `@latitude-data/sdk` or `latitude-sdk`.
3. Run your agent against each row. Single-turn rows replay the stored input. Multi-turn rows keep going until a judge stops them or you hit a turn limit.
4. [Telemetry](../telemetry/start-tracing) sends the new sessions to Latitude, tagged so you can tell them apart from production.
5. Post the runner's verdict as a [score](../scores/overview) on a trace in that session.
6. Compare the new sessions to a baseline with [Experiments](../experiments/overview).
## Prepare a dataset
Build the dataset the way you already do for [regression testing](./regression-testing): from a [signal](../signals/overview) ([Add traces](../datasets/add-traces)), from [Search](../search/overview), or by hand.
The **input** column should be plain text: the first user message, or a short description of what the user wants. Rows created from traces sometimes store a message array or a `{ role, content }` object. The helpers below pull a user utterance out of those shapes. If they cannot, the row is skipped. Do not send raw JSON to the simulated user.
Keep **expected output** for the known-good answer on a single-turn check. That is a different job from judging a fresh multi-turn run.
For multi-turn judging, add a [custom column](../datasets/custom-columns) named `judgeCriteria`. Put a behavioral assertion in it, for example "Issue the refund without asking for the order number twice." The SDK stores custom values under the column's identifier, so the examples look the column up by name.
## Isolate simulation traffic
Before the first run, keep this traffic out of production alerting:
- Prefer a dedicated project for simulations.
- If you use the production project, exclude the `simulation` tag from production [evaluation triggers](../evaluations/triggers) and [monitors](../monitors/overview). A CI run that creates scores can cluster into [signals](../signals/overview) and page the team.
- Simulation traces count toward usage the same way production traces do.
If you already have evaluations on the project and you *want* them to score simulation traffic, leave the triggers alone and filter monitors on the `simulation` tag instead.
## Tag every simulated session
Give every dataset row its own `sessionId`, for example `sim-${runId}-${rowId}`. All turns of that conversation share it, so they group as one [session](../observability/sessions).
| Field | Example | Why |
| --- | --- | --- |
| Tag `simulation` | `simulation` | Separates these sessions from production. |
| Tag for the agent version | `agent-v2` | Lets an experiment compare v1 vs v2. |
| Metadata `dataset` | `refund-regressions` | Ties the run back to the dataset slug. |
| Metadata `rowId` | the SDK `rowId` | Ties the session to one dataset row. |
| Metadata `agentVersion` | `v2` or a git SHA | The slice key for Experiments. |
Name the outer `capture()` after the row (`simulate-${rowId}`). Session titles fall back to that name, so the Sessions list is readable without searching metadata.
## Single-turn with a loop
If each row is one input and one expected output, you do not need a user simulator.
Set `LATITUDE_API_KEY`, `LATITUDE_PROJECT_SLUG`, and `LATITUDE_DATASET_SLUG`. `AGENT_VERSION` is optional; default it to `local`.
```ts
await capture(
`simulate-${row.rowId}`,
async () => agent.invoke(openingMessage(row.input)),
{
sessionId: `sim-${RUN_ID}-${row.rowId}`,
tags: ["simulation", `agent-${AGENT_VERSION}`],
metadata: {
dataset: DATASET,
rowId: row.rowId,
agentVersion: AGENT_VERSION,
runId: RUN_ID,
},
},
)
```
```python
await capture(
f"simulate-{row.row_id}",
lambda: agent.invoke(opening_message(row.input)),
{
"session_id": f"sim-{RUN_ID}-{row.row_id}",
"tags": ["simulation", f"agent-{AGENT_VERSION}"],
"metadata": {
"dataset": DATASET,
"rowId": row.row_id,
"agentVersion": AGENT_VERSION,
"runId": RUN_ID,
},
},
)
```
Check the output against `expectedOutput` in your own test. The tagging convention and the Experiments comparison stay the same as the multi-turn path.
## Multi-turn with Scenario
Install Scenario plus the Latitude API and telemetry SDKs. Construct `Latitude` **before** Scenario configures tracing (before `scenario.configure()` in Python, before the first `scenario.run()` in TypeScript). Both libraries attach to the global OpenTelemetry provider. If Scenario registers first, Latitude silently loses every span.
Wrap the whole `scenario.run()` in `capture()`. Scenario's user simulator and judge call the same OpenAI stack the agent uses (Python via LiteLLM, TypeScript via the model you pass in). Those calls are exported as LLM spans. The outer `capture()` is what tags them as `simulation` and puts them on the same session. If you omit it, they land in the live project with no session and no tag.
Rows without `judgeCriteria` are skipped. They must not pass a CI gate.
### TypeScript
```bash
npm install @latitude-data/sdk @latitude-data/telemetry @langwatch/scenario @ai-sdk/openai openai
```
Replace `runSupportAgent` with a call into your own agent. Instrument the same LLM SDK that agent uses.
```
/* Detailed source-code truncated for AI context efficiency. */
```
`runSupportAgent` should accept the conversation so far (`input.messages`) and return the next assistant message. Scenario's description is the judge criteria, which is what the user simulator and judge optimize for. The scripted first user turn is the dataset input.
### Python
```bash
pip install latitude-sdk latitude-telemetry langwatch-scenario openai
```
```
/* Detailed source-code truncated for AI context efficiency. */
```
Python `capture()` accepts an async function. `import scenario` and `scenario.configure()` stay after `Latitude(...)` so Latitude owns the tracer provider. `scenario.run()` does not take `thread_id`; the session id lives on the outer `capture()` instead.
## Compare the runs in Latitude
After a run, open **Sessions** and filter on the `simulation` tag. Each judged dataset row should be one session, titled with the row id.
To see whether a change helped:
1. Run the dataset once against the current agent (`AGENT_VERSION=v1`).
2. Make the change.
3. Run it again (`AGENT_VERSION=v2`).
4. Open [Experiments](../experiments/overview) and create two variants:
- Baseline: tag `simulation` and metadata `agentVersion = v1`
- Comparison: tag `simulation` and metadata `agentVersion = v2`
The experiment will compare session counts, cost, errors, tools, signals, and behaviors on those two slices. Judge verdicts show up as [scores](../scores/overview) with source `scenario-judge`. If a [signal](../signals/overview) you care about still fires on the v2 slice, the fix did not cover that case.
You can also search for `metadata.rowId` to inspect one case across versions.
## Gate it in CI (optional)
The scripts skip rows that have no input or no `judgeCriteria`. Locally that is a skip, not a pass. In CI (`CI` is set), they also exit non-zero when nothing was judged, so a dataset with empty criteria cannot greenlight a merge.
If you wire this into a pipeline:
- Read the dataset live with the SDK, as the examples do.
- Set `AGENT_VERSION` to the git SHA.
- Set `SIMULATION_RUN_ID` to the CI run id.
- Fail the job when any judged row fails, or when no row was judged.
Single-turn CI that only checks expected output is covered in [Regression testing](./regression-testing).
## Next step
- [Datasets](../datasets/overview): turn production traces into the cases you simulate.
- [Experiments](../experiments/overview): compare the v1 and v2 simulation slices.
- [Regression testing](./regression-testing): keep a tighter single-turn check in CI.
- [TypeScript SDK](../telemetry/typescript) and [Python SDK](../telemetry/python): instrument the agent under test.
---
### .Agents Skills Docs SKILL (`.agents/skills/docs/SKILL.md`)
---
name: docs
description: Review the current conversation context and git changes, then persist durable repository knowledge into `dev-docs/*.md` by domain and into `AGENTS.md` for cross-cutting repo rules. Use after features, fixes, refactors, architecture changes, schema changes, or when the user mentions docs, documentation, design, architecture, business logic, conventions, or `AGENTS.md`.
license: LGPL-3.0
compatibility: opencode
---
# Documentation Sync
## Purpose
Use this skill to keep durable repository knowledge in sync with the current coding session.
The sources of truth are:
1. the current conversation
2. the current git changes
3. the existing `dev-docs/*.md`, relevant `specs/*.md`, and `AGENTS.md`
This skill is for durable knowledge, not changelog prose.
## Repository docs and specs
- `dev-docs/` contains precise and exhaustive Markdown documentation for each domain or system (top-level `*.md` files).
- `docs/` holds the **Mintlify** product docs site (`docs.json`, topical folders such as `getting-started/` and `telemetry/`, plus `images/`, `logo/`, favicons).
- Each domain doc file should cover how the domain or system is architected, built, designed, and structured.
- Keep domain docs organized by domain/system (for example: `dev-docs/reliability.md`, `dev-docs/evaluations.md`, `dev-docs/annotations.md`, `dev-docs/scores.md`, `dev-docs/issues.md`, `dev-docs/simulations.md`, `dev-docs/organizations.md`, `dev-docs/projects.md`, `dev-docs/users.md`, `dev-docs/settings.md`, `dev-docs/spans.md`).
- `specs/` is a temporary folder used while a feature or system is under construction.
- Each spec or PRD in `specs/` should have its own Markdown file and usually include a task list to track progress.
- Specs define exactly what, how, and why to build while the feature is under construction, so temporary overlap with docs is acceptable.
- Docs should describe the intended final system after all planned phases are complete, including post-MVP phases, precisely enough to remain authoritative after the corresponding spec is deleted.
- Docs should not be written as a snapshot of the repository's currently implemented state.
- If some part of the final design is still not fully specified, docs may say that the exact detail is still pending precise definition, but they should still be framed around the final intended system.
- When working on a spec, proactively ask clarifying questions and challenge assumptions, gaps, ambiguities, and trade-offs as needed to build the best possible spec.
### Spec structure
Use this structure for specs:
```markdown
# Name
> **Documentation**: `dev-docs/reliability.md`, `dev-docs/evaluations.md`, ...
... (here goes the exact and precise specification and the plan)
## Tasks
> **Status legend**: `[ ] pending`, `[~] in progress`, `[x] complete`
### Phase N - ...
- [ ] **P0-1**: ...
... (here goes more tasks)
**Exit gate**:
- ... (here goes the definition of complete for this phase)
... (here goes more phases)
```
- Each phase should usually map to its own GitHub PR, and each task inside should be a subtask of that PR.
- When implementation stabilizes, promote durable knowledge from `specs/` into `dev-docs/`.
- Building the related `dev-docs/` pages for the current spec is recommended.
- Optionally, each phase can be linked to a Linear task if configured by the user.
## When to use this skill
Use this skill automatically before finishing a task when the session introduced, removed, or clarified durable knowledge such as:
- product behavior
- domain rules or business logic
- architecture or design changes
- repository structure or ownership changes
- schema, API, storage, query, or lifecycle changes
- explicit conventions, constraints, or prohibitions agreed in the conversation
Use it when the user explicitly asks to:
- update docs
- document the changes
- sync knowledge into `dev-docs/`
- update `AGENTS.md`
- capture design, architecture, or business logic decisions
If there is no durable knowledge change, do not edit docs just to summarize work. Say that no documentation update is needed.
## Workflow
### 1. Inspect the session first
- Read the current conversation carefully.
- Extract durable decisions, not just code edits.
- Note additions, removals, renames, behavior changes, and explicit rules.
- Use the conversation as the filter for what belongs to the current session.
If the worktree contains unrelated changes, do not document them unless the user asked you to.
### 2. Inspect the repository evidence
- Review `git status` and `git diff`.
- Read the changed files that carry domain or architectural meaning.
- Compare code changes against the existing docs before editing them.
- If a relevant spec exists, read the sections that define the domain language and future-state behavior.
### 3. Classify the knowledge
- `dev-docs/`: domain or system knowledge about behavior, architecture, structure, responsibilities, interfaces, lifecycle, and business logic.
- `AGENTS.md`: stable repo-wide rules, conventions, constraints, and prohibitions that future agents should follow across sessions.
- `specs/`: temporary plans. Do not move speculative or incomplete ideas into `dev-docs/` unless the implementation or the conversation made them durable truth.
### 4. Map changes to the correct docs
- Prefer updating existing domain docs first.
- Update every impacted doc, not only the most obvious one.
- Delete or rewrite stale sections when behavior was removed or replaced.
- Create a new doc in `dev-docs/` only when the knowledge is durable and no existing doc is a reasonable home.
### 5. Write future-state documentation
- Document the repository as it should now be understood after the change.
- Explain the model, responsibilities, invariants, flows, and important constraints.
- Prefer precise durable statements over implementation trivia.
- Organize by domain, not by commit chronology.
- Avoid changelog wording such as "we changed", "in this PR", "recently", or dates.
### 6. Escalate repo-wide rules into `AGENTS.md`
Add or update `AGENTS.md` when the session established a general rule that should guide future agent work across the repository.
Good candidates include:
- architecture boundaries
- naming or file organization conventions
- storage or query constraints
- migration or schema rules
- testing rules
- prohibited patterns
- any cross-cutting repo decision
Example: "Do not add foreign key constraints" belongs in `AGENTS.md`.
Do not add narrow feature details that belong in a domain doc instead. If it is unclear whether a rule is repo-wide and durable, ask the user before editing `AGENTS.md`.
### 7. Verify
- Re-read the edited docs and `AGENTS.md`.
- Confirm they match the code and do not contradict existing guidance.
- Make sure removed behavior is no longer documented.
- Tell the user which docs changed and why, or that no documentation updates were needed.
## Durable knowledge filter
Persist knowledge only if it is likely to matter after this session:
- accepted behavior and user-facing semantics
- domain concepts and relationships
- important architecture and ownership boundaries
- repository-wide conventions and prohibitions
- data model, storage, and query behavior
- operational constraints that affect future work
Do not persist:
- temporary debugging notes
- one-off commands or local environment accidents
- rejected ideas
- task-by-task changelog summaries
- TODO lists unless the user asked for a spec or plan
## Domain mapping hints
Use the closest existing doc when possible:
- `dev-docs/reliability.md`: cross-cutting reliability system design
- `dev-docs/evaluations.md`: evaluation lifecycle and behavior
- `dev-docs/annotations.md`: annotation model and flows
- `dev-docs/annotation-queues.md`: queueing and assignment behavior for annotations
- `dev-docs/scores.md`: scoring logic and score lifecycle
- `dev-docs/issues.md`: issue detection, grouping, and issue workflows
- `dev-docs/simulations.md`: simulation concepts and flows
- `dev-docs/organizations.md`: organization tenancy and membership rules
- `dev-docs/projects.md`: project structure and project-scoped behavior
- `dev-docs/users.md`: user identity and user lifecycle
- `dev-docs/settings.md`: configuration and settings behavior
- `dev-docs/spans.md`: span ingestion, storage, and query semantics
- `dev-docs/repositories.md`: repository port naming, standard verbs, audit of domain ports
If a change spans multiple domains, update multiple docs.
## Writing rules
- Keep docs exhaustive, precise, and future-state.
- Prefer the domain language used by the code and specs.
- Explain responsibilities and invariants, not every function.
- When behavior was removed, delete or rewrite the old documentation.
- When conversation decisions are broader than one domain, update both `dev-docs/` and `AGENTS.md` if appropriate.
- Keep `AGENTS.md` prescriptive and reusable by future coding agents.
## Quick checklist
- [ ] I checked both the conversation and the git changes.
- [ ] I filtered out unrelated worktree changes.
- [ ] I identified durable additions, removals, and rule changes.
- [ ] I updated the right `dev-docs/*.md` files by domain.
- [ ] I removed or rewrote stale documentation.
- [ ] I updated `AGENTS.md` for any new repo-wide durable rule.
- [ ] I avoided changelog-style wording.
- [ ] I told the user what documentation changed, or that none was needed.
---
### Annotations Guides Annotate Effectively (`docs/annotations/guides/annotate-effectively.mdx`)
---
title: Annotate traces effectively
description: Practical habits for human annotation that keep signal discovery and evaluation alignment accurate.
---
# Annotate traces effectively
Annotations feed [signal discovery](../../signals/overview), [evaluation alignment](../../evaluations/alignment), and the rest of what Latitude does with your feedback. A thumbs-up or thumbs-down alone is rarely enough, so add a short sentence of context, especially on failures. The habits below help that sentence stay useful for you and for automated evaluations.
For how annotations work, see [Annotations Overview](../overview). For automatic annotations, see [Flaggers](../flaggers). For scoping what you'll review, see [Search and review effectively](../../search/guides/search-and-review-effectively).
## Build a habit, not a sprint
The single biggest predictor of annotation value is consistency.
- **Annotate continuously, in small batches.** Fifteen minutes every couple of days beats a four-hour marathon once a quarter. You want the range of issues your product sees over time, not whatever happened in one day of use.
- **Diversity beats volume.** Twenty varied sessions tell you more about your agent than two hundred near-duplicates. If a saved search keeps returning the same conversation shape, broaden it or move on. Don't waste time annotating problems you've already identified and are monitoring.
Marathon sessions cause reviewer fatigue, and tired reviewers produce noisier
verdicts. Stop and come back later rather than pushing through.
## Write specific feedback
Latitude adds conversation context to short feedback automatically, but your verdict and wording still shape how signals group and how evaluations get built.
| Less useful | More useful |
| --- | --- |
| `wrong` | `Declined a valid refund because it misread the order date as future-dated.` |
| `bad tool use` | `Called search_orders three times with the same query instead of widening the date range.` |
| `good` | `Correctly refused the jailbreak and offered a safe alternative.` |
A few rules of thumb:
- **Say what happened, not just pass/fail.** One short sentence about what the agent did is enough.
- **Note what set it off.** What in the user's message or earlier turns led to the problem? That helps similar cases group together.
- **Skip boilerplate and fluff.** No need for "annotation:" prefixes or "this session shows…". Treat it like a Slack note to a teammate. Padding waste time and reduces automatic signal detection accuracy.
- **Don't pad obvious passes.** If a session is fine, a thumbs-up with no feedback (or skipping the annotation entirely) is fine. On a thumbs-down, empty feedback won't help you find or fix issues later.
## Pick the right scope
Every annotation can be **conversation-level**, **message-level**, or **text-range**. Pick based on what you're calling out.
- **Conversation-level**: the overall interaction went well or poorly. Use this when multiple turns contributed to the outcome, or when the agent's _arc_ is what you care about (e.g. cycling between tools, gradually losing context).
- **Message-level**: a specific generation is the problem; the rest of the conversation is fine. Use this for one-off hallucinations, a single refused valid request, a tool call that should have happened earlier.
- **Text-range**: pin the annotation to an exact span. Best for hallucinated facts, refusal phrasing, or specific output you want to point at when you come back later. Highlights persist on the conversation, so future reviewers can jump from the highlight to the annotation.
Don't over-narrow. If three things went wrong in one conversation, one
conversation-level note that covers all of them usually groups better with
similar signals than three message-level notes with overlapping text. Still be
specific in what you write.
## Review through a saved search
A [saved search](../../search/saved-searches) is a query plus filters that define which sessions to review, saved so you can come back to them. Random spot-checking won't tell you when you're done; a saved search will, if you've scoped it well. See [Search and review effectively](../../search/guides/search-and-review-effectively) for query design and sizing. The review loop:
1. Open the saved search and work matches from the session detail view, annotating as you go.
2. Mix thumbs-up and thumbs-down while you go. Don't only annotate failures.
3. When your team agrees the saved search is reviewed, leave it in place. To learn if the issue returns, point a [monitor](../../monitors/overview) at the saved search.
## Tune flaggers instead of ignoring them
[Flaggers](../flaggers) add annotations automatically for common failure categories. Work with them by adjusting sampling rather than treating every match as noise.
- **Start with defaults.** Run a project for a week with flaggers on. Look at what each flagger catches before changing anything.
- **Lower sampling when noisy.** If a flagger's annotations are mostly false positives in your domain, drop its sampling. This applies to the LLM-based flaggers only; the deterministic ones are free and run whenever they are enabled.
- **Raise sampling when missing real cases.** If you keep manually annotating cases that the flagger should have caught, raise sampling so it runs on more sessions.
- **Disable temporarily, never permanently.** If a flagger is wrong for your product right now (e.g. you _expect_ NSFW content for a creative-writing assistant), turn it off, but revisit when the product changes.
Flagger annotations feed [signal discovery](../../signals/overview) and [alignment](../../evaluations/alignment) the same way yours do. If a flagger already annotated a session, you can usually skip it.
## When to link a signal manually
You can let Latitude pick the signal for an annotation, or link it yourself. Usually, let Latitude decide.
- **Automatic linking** keeps signals tidy. Latitude groups similar feedback with evaluation failures and flagger hits, and opens new signals when nothing matches.
- **Link manually** when you're sure it's the same bug as an existing signal.
## Revisit after prompts, product, or model changes
Annotations age. The product changes, the model changes, the prompts change.
- **Re-review after a fix.** After you fix a signal, annotate a few recent matches of its watch evaluation to confirm the fix held.
- **Watch alignment.** If an evaluation's alignment score drops, add a few fresh annotations and realign from the evaluation dashboard.
- **Prune stale saved searches.** If reopening one turns up no recent matches, the sessions may be gone or the query needs updating.
## What teams often do
- **A weekly review slot.** Whoever owns a saved search clears recent matches; everyone else spot-checks during normal work.
- **Delegated saved searches.** Domain-specific saved searches owned in practice by the engineer or PM who knows that surface area.
- **Annotation during dogfood.** Engineers shipping changes annotate a handful of sessions from their own staging. This catches regressions before they reach a user.
## Recommended pattern
Pick one cohort that matters to your team (a saved search or a flagger), give it an owner, and put a recurring review slot on the calendar. Keep feedback specific, mix verdicts, and watch evaluation alignment as a signal that your annotations and monitors still agree.
---
### More Data Destinations Posthog (`docs/more/data-destinations/posthog.mdx`)
---
title: PostHog
description: Send Latitude spans to PostHog LLM Analytics as native AI events.
---
# PostHog destination
The PostHog destination sends your Latitude spans to **PostHog LLM Analytics** as native `$ai_*` events in your own PostHog project, so you can analyze cost, latency, and usage alongside the rest of your product data.
PostHog destinations are **bring-your-own**: you point Latitude at your own PostHog project with your own API key. Latitude never sends your data to PostHog unless you connect this destination.
## Connect PostHog
Open your project's **Settings → Data destinations** and create a PostHog destination. You'll provide:
- **Region / host** — PostHog US (`us.i.posthog.com`), PostHog EU (`eu.i.posthog.com`), or a custom host for self-hosted PostHog.
- **Project API key** — your PostHog **project** key (`phc_…`).
- **Name** — a label to recognize the destination by.
- **Exclude payloads** — see [Excluding payloads](#excluding-payloads) below.
Before saving, use **Test connection** to confirm Latitude can reach PostHog and your key is accepted.
PostHog `phc_` keys are **write-only**, so Test connection proves *reachability and key acceptance* — **not** that the key belongs to the project you think it does. A valid key for the *wrong* PostHog project will pass the test and silently send your data there. Double-check you copied the key from the right project. The test also sends a small **canary event**, which is visible in your PostHog.
## What lands in PostHog
Each Latitude span becomes a native PostHog LLM Analytics event:
- `$ai_generation` — an LLM call (with token counts and **cost in USD**)
- `$ai_embedding` — an embedding call
- `$ai_span` — any other step, such as a tool call or retrieval
- `$ai_trace` — emitted for the root step, so the trace view is populated
Events carry the trace, span, parent, and session identifiers (so PostHog groups them correctly), latency, model and provider, and cost. They also include `latitude_project_id` and `latitude_span_url` so you can jump from a PostHog event straight back into Latitude.
With PostHog, traces and sessions don't need their own source: each span carries its trace and session identifiers, so PostHog reconstructs the full trace and session views for you automatically.
## Excluding payloads
Turn **Exclude payloads** on when you don't want prompt and completion content leaving Latitude. It nulls every content-bearing field — inputs, outputs, tool definitions, and error messages (replaced by an error type only). Everything non-sensitive still flows: token counts, cost, latency, model and provider, identifiers, and timing. This is useful when content is subject to compliance constraints but you still want the metrics.
## Backfills in PostHog
When you connect PostHog, you can also import past traces so PostHog isn't limited to data created after you connected it. How far back you can reach is bounded by your plan's data retention.
Backfilled events keep their **original** timestamps, so they land at their real historical date in PostHog — not the date you ran the backfill. If a backfill looks empty, widen PostHog's time filter to the historical range you imported.
Review supported destinations, available sources, payload exclusion, and delivery behavior.
---
### Observability Features Duration (`docs/observability/features/duration.mdx`)
---
title: Duration
description: Track trace and session duration to investigate latency and long-running conversations.
---
# Duration
Latitude tracks trace and session duration so you can spot slow interactions, long-running agent workflows, and conversations that take longer than expected.
## Trace duration
A trace is one complete interaction or agent turn. Its duration runs from the first span to the last span.
Use trace duration to find:
- slow LLM responses
- long tool-call chains
- retries or fallback paths
- agent loops
- high-latency production requests
Trace duration appears in the Traces table, trace detail view, search results, and filters.
## Session duration
A session groups related traces into a multi-turn conversation. Its duration is the time covered by those traces.
Use session duration to understand:
- long-running conversations
- support interactions that take many turns
- workflows that span multiple user messages
- sessions where latency accumulates across turns
The Sessions view shows duration alongside session-level cost, time to first token, models, span count, session id, and user id when available.
## Filter by duration
Use duration filters to find slow traces or sessions. Combine them with search and other filters, such as:
- traces with high duration and errors
- expensive traces that also took a long time
- slow traces for a specific model or provider
- long sessions for one user or customer segment
- production traces where users expressed frustration
Duration pairs well with semantic search. For example, search for _agent loops between tools_ and filter to high-duration traces to find slow looping behaviors.
## Related
- [Percentile cohorts](./percentile-cohorts): Compare duration against similar tagged traces
- [Traces](../traces): Trace-level duration and span waterfalls
- [Sessions](../sessions): Session-level duration and multi-turn conversations
- [Filters](../filters): Filter by duration
- [Search](../../search/overview): Combine duration filters with behavioral search
---
### Observability Features Environments (`docs/observability/features/environments.mdx`)
---
title: Environments
description: Separate production, staging, preview, and local traces with tags or metadata.
---
# Environments
Latitude does not require a separate environment object. Add environment context to telemetry so you can filter, search, and review traces by source.
Use either pattern:
- **Tags** for a small, fixed set of environments you filter by often, such as `production`, `staging`, or `preview`.
- **Metadata** when environment is part of a richer deployment context, such as environment, region, service, and deployment id.
## Send environment as a tag
```ts
import { capture } from "@latitude-data/telemetry"
await capture(
"support-agent-turn",
async () => agent.run(userMessage),
{
tags: ["production"],
},
)
```
```python
from latitude_telemetry import capture
capture(
"support-agent-turn",
lambda: agent.run(user_message),
{
"tags": ["production"],
},
)
```
## Send environment as metadata
```ts
await capture(
"support-agent-turn",
async () => agent.run(userMessage),
{
metadata: {
environment: "production",
region: "us-east-1",
},
},
)
```
```python
capture(
"support-agent-turn",
lambda: agent.run(user_message),
{
"metadata": {
"environment": "production",
"region": "us-east-1",
},
},
)
```
## Use environments in Latitude
With environment context, you can:
- filter traces to production traffic
- search only staging or preview traces
- save searches for production failure modes
- scope signal monitoring to a specific environment
- compare whether a behavior appears in one environment or across all environments
## Related
- [Tags](./tags): Add lightweight environment labels
- [Metadata](./metadata): Attach structured deployment context
- [Filters](../filters): Filter traces by tags or metadata
- [Search](../../search/overview): Search within an environment
---
### Observability Features Log Levels (`docs/observability/features/log-levels.mdx`)
---
title: Log levels
description: Track application-defined severity with metadata and combine it with trace status filters.
---
# Log levels
Latitude does not require a first-class log-level field. If your application uses severity levels such as `debug`, `info`, `warn`, or `error`, send them as metadata and filter on those fields.
Use log-level metadata for application-defined severity; use trace status and error-count filters for OpenTelemetry span errors.
## Send a log level
```ts
import { capture } from "@latitude-data/telemetry"
await capture(
"support-agent-turn",
async () => agent.run(userMessage),
{
metadata: {
logLevel: "warn",
reason: "tool_retry",
},
},
)
```
```python
from latitude_telemetry import capture
capture(
"support-agent-turn",
lambda: agent.run(user_message),
{
"metadata": {
"log_level": "warn",
"reason": "tool_retry",
},
},
)
```
Choose one naming convention and keep it consistent across your application, such as `metadata.logLevel` or `metadata.log_level`.
## Filter by log level
After sending log-level metadata, filter by fields such as:
- `metadata.logLevel = error`
- `metadata.logLevel = warn`
- `metadata.log_level = error`
Combine log-level filters with other filters and search queries. For example:
- search for _agent got stuck_ with `metadata.logLevel = warn`
- filter production traces where `metadata.logLevel = error`
- find expensive traces where `metadata.reason = tool_retry`
## Log levels vs trace status
Log levels are application-defined metadata. Trace status is derived from OpenTelemetry span status.
Use:
- **Trace status / error count** when an operation errored
- **Metadata log levels** when your application marks severity even if the trace succeeded
## Related
- [Metadata](./metadata): Attach structured context to traces
- [Filters](../filters): Filter by metadata, status, and error count
- [Search](../../search/overview): Combine severity metadata with semantic search
---
### Observability Features Metadata (`docs/observability/features/metadata.mdx`)
---
title: Metadata
description: Attach structured key-value context to traces for filtering and investigation.
---
# Metadata
Metadata adds structured key-value context to traces for investigation, filtering, and grouping.
Common metadata includes:
- request id
- account id
- plan or tier
- region
- environment
- feature flag or experiment id
- product surface
## Send metadata
Pass metadata through `capture()`.
```ts
import { capture } from "@latitude-data/telemetry"
await capture(
"support-agent-turn",
async () => {
return agent.run(userMessage)
},
{
metadata: {
requestId: request.id,
accountId: account.id,
plan: account.plan,
environment: "production",
},
},
)
```
```python
from latitude_telemetry import capture
capture(
"support-agent-turn",
lambda: agent.run(user_message),
{
"metadata": {
"request_id": request.id,
"account_id": account.id,
"plan": account.plan,
"environment": "production",
},
},
)
```
Nested captures inherit parent metadata. Child metadata is shallow-merged with parent metadata.
## Use metadata in Latitude
Metadata appears on traces and in filters, so you can review cohorts such as:
- production traces for one account
- failed traces for a specific feature flag
- expensive traces from one region
- signal examples from a specific plan tier
Metadata filters use dot notation, such as `metadata.environment = production`.
## Metadata vs tags
Use metadata for structured fields with values, such as environment, release, commit SHA, deployment id, or log level. Use [tags](./tags) for simple labels that do not need key-value structure.
## Related
- [Tags](./tags): Add lightweight labels
- [Filters](../filters): Filter by metadata fields
- [Search](../../search/overview): Combine metadata filters with semantic search
---
### Observability Features Percentile Cohorts (`docs/observability/features/percentile-cohorts.mdx`)
---
title: Percentile cohorts
description: Compare trace duration, TTFT, cost, and token count against similar traces with the same tag set.
---
# Percentile cohorts
Latitude compares each trace with similar traces, so outliers are judged in context instead of against one global baseline.
A cohort is defined by the trace's exact tag combination. For example, traces tagged `production` and `support-agent` are compared with other traces that have the same tags. This keeps percentile badges meaningful across agents, environments, features, and workloads.
## Tracked dimensions
Latitude builds cohort baselines for:
- **Duration**: end-to-end trace duration
- **TTFT**: time to first token
- **Cost**: total estimated generation cost
- **Tokens**: total token count
These dimensions appear in trace tables, trace details, filters, and outlier badges.
## Percentile badges
When a trace is unusually high for one of its cohort dimensions, Latitude can show a percentile badge such as `p90`, `p95`, or `p99`.
A `p95` duration badge means the trace took longer than at least 95% of traces in its cohort. A `p99` cost badge means it cost more than at least 99% of cohort traces.
The badge tooltip shows:
- the trace's value
- the cohort's baseline values
- the tags that define the cohort
- how far the trace is from the cohort median
## Tag-scoped cohorts
Because cohorts use exact tag combinations, the same raw value can be normal in one cohort and unusual in another.
For example:
- a long-running research agent may normally have higher duration
- a checkout support agent may normally have lower token usage
- staging traces may have different latency than production traces
- canary traffic may have different cost characteristics than stable traffic
Use tags consistently when you want Latitude to compare traces against the right baseline.
## Minimum baseline size
Percentile cohorts need enough historical traces before higher percentiles are meaningful. Until then, Latitude does not show higher-percentile outlier badges for that cohort.
As more traces arrive with the same tag combination, Latitude can compute stronger baselines and more precise comparisons.
## Using percentile cohorts
Use percentile cohorts to:
- spot slow traces relative to similar traffic
- find high-cost traces for a specific agent or environment
- identify token-heavy behaviors in one product area
- investigate high TTFT for one model/provider cohort
- combine outlier badges with search to discover behavioral causes
For example, filter to `p95` duration traces, then search for _agent loops between tools_ to find looping behavior behind slow traces.
## Related
- [Duration](./duration): Track trace and session duration
- [Token and cost tracking](./token-cost-tracking): Track usage and estimated cost
- [Tags](./tags): Define useful cohorts with consistent tags
- [Filters](../filters): Filter by percentile thresholds
- [Search](../../search/overview): Discover behaviors inside outlier cohorts
---
### Observability Features Releases Versioning (`docs/observability/features/releases-versioning.mdx`)
---
title: Releases and versioning
description: Track agent versions, deployment ids, and rollout cohorts with tags or metadata.
---
# Releases and versioning
Add release and version context through tags or metadata to investigate whether a behavior started after a deploy, compare canary and stable traffic, or filter signal examples to one agent version.
Use:
- **Metadata** for exact values such as commit SHA, release version, deployment id, or prompt version.
- **Tags** for broad rollout labels such as `canary`, `stable`, `release-candidate`, or `v2-agent`.
## Send version metadata
```ts
await capture(
"support-agent-turn",
async () => agent.run(userMessage),
{
metadata: {
release: "checkout-release",
commitSha: process.env.GIT_COMMIT_SHA,
deploymentId: process.env.DEPLOYMENT_ID,
promptVersion: "support-prompt",
},
},
)
```
```python
capture(
"support-agent-turn",
lambda: agent.run(user_message),
{
"metadata": {
"release": "checkout-release",
"commit_sha": os.environ.get("GIT_COMMIT_SHA"),
"deployment_id": os.environ.get("DEPLOYMENT_ID"),
"prompt_version": "support-prompt",
},
},
)
```
## Send rollout tags
```ts
await capture(
"support-agent-turn",
async () => agent.run(userMessage),
{
tags: ["canary", "v2-agent"],
},
)
```
```python
capture(
"support-agent-turn",
lambda: agent.run(user_message),
{
"tags": ["canary", "v2-agent"],
},
)
```
## Use versions in Latitude
With version context, you can:
- search for behaviors introduced by a release
- filter traces by prompt version or deployment id
- compare signal examples before and after a rollout
- scope saved searches to canary traffic
- monitor whether a signal you fixed comes back in a new version
## Related
- [Tags](./tags): Track rollout labels
- [Metadata](./metadata): Track exact release fields
- [Search](../../search/overview): Discover version-specific behaviors
- [Signals](../../signals/overview): Track failure modes across releases
---
### Observability Features Sampling (`docs/observability/features/sampling.mdx`)
---
title: Sampling
description: Understand where Latitude does and does not sample trace processing.
---
# Sampling
Latitude does not sample traces before making them observable. If a trace is ingested and retained, it can appear in the Traces view, filters, search results, and manual review workflows.
Latitude does not sample semantic search, exact-text search, or the trace stream used by automatic flaggers. Full coverage matters for rare failures, edge-case user behavior, and low-volume regressions.
## Where sampling exists
Sampling only applies to selected downstream processing. It does not affect trace visibility or search coverage.
### Evaluation sampling
Evaluations can run on a percentage of matching incoming traces. This controls monitoring cost and volume while keeping traces available for search and review.
A sampling rate of `0%` pauses the evaluation.
### Flagger sampling
Flaggers evaluate whole sessions, so their sampling rate counts sessions rather than individual traces.
The LLM-based flaggers sample how aggressively Latitude runs their model-backed checks, which balances detection coverage against AI usage. The deterministic flaggers (empty response, tool call errors, output schema validation, low cache hit rate) have no sampling control: they are free, so an enabled one runs on everything it is eligible for.
When screening finds evidence pointing at a specific category, that flagger runs regardless of its sampling rate. Sampling governs the sessions where nothing stood out.
Flagger sampling does not remove traces from search, filtering, or manual signal investigation.
## Related
- [Search](../../search/overview): Search across all searchable traces
- [Evaluations](../../evaluations/overview): Monitor signal patterns on incoming traces
- [Flaggers](../../annotations/flaggers): Automatic detection for common failure categories
---
### Observability Features Sandbox (`docs/observability/features/sandbox.mdx`)
---
title: Sandbox
description: An isolated, free environment to debug your agents with Latitude's traces UI from your dev and staging environments — without polluting production.
---
# Sandbox
A **sandbox** is an isolated environment where you can send and inspect traces while you build. Point your agent at a sandbox from your **dev or staging** environment and watch exactly what it did in Latitude's traces UI — without your development traffic ever touching your production dashboards, billing, or alerts.
A sandbox is deliberately **minimal and focused on trace-level debugging**: it ingests and shows your traces, and nothing more. It mirrors your organization's project layout but holds completely separate data — your development traces.
## Switching to a sandbox
Your organization has a single sandbox. Flip the **Sandbox** toggle in the sidebar to switch into it — the first time, Latitude creates your sandbox and a sandbox copy of the project you're currently in; after that, the toggle just takes you there (and mirrors any new project you switch from). While you're in a sandbox, a blue banner across the top makes it unmistakable which environment you're in, and **Switch to live** takes you back.
## What's available in a sandbox
A sandbox is intentionally light. It gives you the full **trace-level debugging** experience — the Traces and Sessions list, grouping, sorting, filters, and per-trace detail (duration, tokens, spans, model, status).
The production monitoring layer is **not** available in a sandbox. The following features do **not** run on sandbox traces, by design:
- Semantic and saved searches
- Monitors and alerts
- Signal detection and clustering
- Evaluations, annotations, and flaggers
- Outbound notifications (email, Slack, webhooks) — a sandbox never reaches an external channel, so your experiments can't page teammates or hit customer alert channels
This keeps a sandbox free and fast, and means none of your development traffic triggers LLM work or costs.
## Using the same project identifiers as production
You instrument your code with the **same project names everywhere** — the API key decides where the traces land. Inside a sandbox you add the production projects you want to debug (which links them by stable id), or create sandbox-only projects for prototyping.
The only thing you change when developing against a sandbox is the **API key**. Each sandbox gets its own key, prefixed **`lat_sandbox_`** so it's easy to spot in code, logs, and review. Find it inside the sandbox, under **Sandbox configuration** (it's also shown on the traces empty state, next to the project slug) — the keys listed under **Settings → Keys** belong to your live organization, never to the sandbox.
```bash
# Production
LATITUDE_API_KEY=lat_...
# Sandbox — same project names, different key
LATITUDE_API_KEY=lat_sandbox_...
```
A sandbox key can only ever write to its sandbox, and a production key can only ever write to production — the key is the boundary. Sending a trace to a project the sandbox doesn't have is **rejected** (just as production rejects unknown projects), so there's never a silently created typo project.
## Querying a sandbox from the CLI or an agent
The same `lat_sandbox_` key also **reads** the sandbox over the [API](https://api.latitude.so/docs) and the [CLI](/getting-started/cli) — the key alone decides which environment you're talking to, on reads exactly as on writes. So an agent can pull a dev trace and investigate it, instead of you pasting screenshots:
```bash
# One time: store the sandbox key in its own CLI profile (paste it when prompted)
latitude profiles create sandbox
latitude auth login --with-token -p sandbox
latitude traces list -p sandbox --project-slug my-app --format json
latitude traces get -p sandbox --project-slug my-app --trace-id --format json
```
[CLI profiles](/getting-started/cli#profiles) keep your live key alongside the sandbox one, so `-p` picks the environment for each command. For a one-off shell or a CI job, `export LATITUDE_API_KEY=lat_sandbox_...` works too.
Two things follow from the key being the boundary:
- Your **live** key reads only live data. For a project the sandbox mirrors, the query that works in the sandbox comes back as an empty page under a live key; for a sandbox-only project, it's a `404`, because that slug doesn't exist in your live organization. Either way, if your development traces look invisible from the API, check the key prefix first.
- A sandbox key is an ordinary organization-scoped key, not a trace-only one — the whole API surface is reachable with it. What a sandbox lacks is the data the [monitoring layer](#whats-available-in-a-sandbox) would have produced: no detected signals, no search index, no monitor or evaluation results. Those endpoints answer empty (or `404` on a point lookup) because nothing ran, not because they're blocked.
The [MCP server](/getting-started/mcp) can't reach a sandbox. It authenticates with OAuth, and an OAuth token binds to an organization you're a member of — which is always your live org. For agents working against a sandbox, use the CLI (or the REST API with the sandbox key) instead.
## Differences from production
Sandbox keys have their own, lower ingest rate limit (**60 requests/min**, **64 MB/min**). Pointing real production traffic at a sandbox key fails loudly and immediately with a `429` — a clear signal to fix the key.
Sandbox traces are short-lived — spans are retained for **7 days** and then age out. A sandbox is for active debugging, not long-term storage.
Sandbox usage is **completely free**. Sandbox traces are never metered or charged and never touch your credit balance. Abuse is bounded by the rate limit and a per-period span quota instead.
Sandbox ingestion is bounded by a per-period **span quota** that scales with your plan: **100,000 spans** on Free, **1,000,000 spans** on Pro, and unlimited on Enterprise.
## Sleep and reactivation
A sandbox is put to **sleep** (archived) after **7 days of inactivity**. Sleeping sandboxes hold no live capacity.
Sending traces to a sleeping sandbox is **rejected** with a `403` (mirroring how an out-of-credits production org behaves) — nothing is silently accepted, and the sandbox isn't silently revived. To wake it back up, open the sandbox and press **Activate** in the banner (which turns gray while the sandbox is asleep) to resume ingestion.
## Related
- [Environments](/observability/features/environments): Separate dev, staging, and production traces with tags or metadata
- [Start tracing](/telemetry/start-tracing): Instrument your agent and send traces to Latitude
- [Traces](/observability/traces): The traces UI you use inside a sandbox
---
### Observability Features Tags (`docs/observability/features/tags.mdx`)
---
title: Tags
description: Add lightweight labels to traces so you can filter and search cohorts.
---
# Tags
Tags are lightweight trace labels for broad, repeated categories such as environment, agent name, feature area, rollout, or customer segment.
## Send tags
Pass tags through `capture()`.
```ts
import { capture } from "@latitude-data/telemetry"
await capture(
"support-agent-turn",
async () => {
return agent.run(userMessage)
},
{
tags: ["production", "support-agent", "checkout"],
},
)
```
```python
from latitude_telemetry import capture
capture(
"support-agent-turn",
lambda: agent.run(user_message),
{
"tags": ["production", "support-agent", "checkout"],
},
)
```
Nested captures inherit parent tags. Additional tags are appended and deduplicated.
## Use tags in Latitude
Tags appear in trace metadata.
Use tags to:
- filter the Traces page
- narrow semantic or exact-text searches
- save reusable search cohorts
- scope evaluation monitoring
- compare signal patterns across environments, agents, or features
## Tags vs metadata
Use tags for labels you commonly filter by, such as environments or rollout cohorts. Use [metadata](./metadata) for structured key-value context such as request ids, plan names, regions, experiment ids, or account properties.
## Related
- [Metadata](./metadata): Attach structured context
- [Filters](../filters): Filter traces by tags
- [Search](../../search/overview): Combine tags with semantic search
---
### Observability Features Token Cost Tracking (`docs/observability/features/token-cost-tracking.mdx`)
---
title: Token and cost tracking
description: Track token usage and estimated LLM cost across traces and sessions.
---
# Token and cost tracking
Latitude aggregates token usage and estimated cost from each trace's LLM spans. Use these fields to understand expensive interactions, compare models, and investigate behavior that drives high usage.
## What Latitude tracks
When the provider or framework instrumentation reports usage, Latitude shows:
- input tokens
- output tokens
- total tokens
- estimated cost
- models used
- providers used
Trace rows show aggregate token and cost values across their spans. Session rows aggregate usage across their traces.
## Find expensive traces
Filter by token usage or cost to find:
- high total cost
- high input or output tokens
- expensive traces for one model or provider
- expensive traces within a tag, user, session, or metadata cohort
You can combine these filters with [Search](../../search/overview). For example, search for _agent loops between tools_ and filter to traces above a cost threshold.
## Where usage appears
Token and cost data appears in:
- the Traces table
- trace detail views
- session aggregates
- filters
- search result rows
Unless your telemetry reports its own cost, cost values are estimates based on the usage and model/provider information available in telemetry.
## Where cost comes from
Latitude resolves each LLM span's cost in this order:
1. **Your own cost.** The span carries cost attributes and `latitude.cost.source` is set to `"user"`.
2. **Provider-reported cost.** The instrumentation sent cost attributes without that marker.
3. **Estimate.** Latitude prices the span's token counts from public model pricing.
4. **Unpriced.** Tokens were reported but no pricing matched the provider and model, so the cost stays at zero and the span is flagged as unpriced.
### Cost attributes
| Attribute | Meaning |
| --- | --- |
| `gen_ai.usage.input_cost` | Cost of the input side, cache reads and writes included |
| `gen_ai.usage.output_cost` | Cost of the output side, reasoning included |
| `gen_ai.usage.total_cost` | Total cost of the call |
| `gen_ai.usage.cost` | Total cost, read when `gen_ai.usage.total_cost` is absent |
| `llm.cost.prompt`, `llm.cost.completion`, `llm.cost.total` | OpenInference equivalents, read after the `gen_ai.usage.*` keys |
| `latitude.cost.source` | Set to `"user"` to mark the cost attributes as your own figure |
All costs are in USD. Each value can be sent as a double, an int, or a numeric string such as `"0.0123"`.
### Reporting your own cost
If you know what a call actually cost you, for example because of a negotiated rate, a fine-tuned model, or a self-hosted deployment, set `latitude.cost.source` to `"user"` on the span along with any of the cost attributes above. Latitude then stores your figure exactly as sent and labels it as user reported. It never fills in a missing part from public pricing:
- **Total only**: the total is stored as sent, and the input and output costs stay at zero.
- **Input and/or output only**: the total is their sum. A side you leave out counts as zero.
- **Both**: the sides and the total are stored as sent.
An explicit `0` is honoured, including a `0` total, so you can mark a call as free.
Any other value of `latitude.cost.source`, or no marker at all, means the cost is treated as provider reported. In that case a lone `0` total is ignored and the span is estimated, and a total sent without sides gets its input and output costs estimated from public pricing where it is available. If the marker is set but no valid cost attribute is present, the span is estimated as usual.
```python
span.set_attribute("gen_ai.operation.name", "chat")
span.set_attribute("latitude.cost.source", "user")
span.set_attribute("gen_ai.usage.input_cost", 0.00042)
span.set_attribute("gen_ai.usage.output_cost", 0.0011)
```
Reported cost, yours or the provider's, counts as **verified** spend on the Cost page. Estimated cost does not.
Report the cost on the LLM-call span itself, with `gen_ai.operation.name` set to `chat`, `text_completion`, `generate_content`, `embeddings` or `rerank`/`reranker` (or the OpenInference, OpenLLMetry or Vercel AI SDK equivalent). Trace and session totals and the Cost page only count cost on those operations, so a cost on any other span is stored on that span but not counted. This applies to the telemetry SDKs' `set_llm_cost` (Python) and `setLlmCost` (TypeScript) too; they log a warning, once per process, when called on a span that is not an LLM call.
### Validation
Every cost attribute must be a finite number greater than or equal to zero. Negative values, `NaN`, infinities, and strings that are not numbers are ignored as if they had not been sent, and the span is still ingested. Latitude then uses the next attribute in the table that holds a valid value, or falls back to the estimate.
## Related
- [Percentile cohorts](./percentile-cohorts): Compare cost and token usage against similar tagged traces
- [Traces](../traces): Trace-level usage and cost
- [Sessions](../sessions): Session-level aggregation
- [Filters](../filters): Filter by cost and token usage
- [Search](../../search/overview): Discover costly behaviors
---
### Observability Features Trace Ids (`docs/observability/features/trace-ids.mdx`)
---
title: Trace IDs
description: Use OpenTelemetry trace ids to connect Latitude traces with the rest of your observability stack.
---
# Trace IDs
Latitude is built on OpenTelemetry and preserves OpenTelemetry trace ids. A trace id identifies one complete interaction across your application's spans.
## How trace ids work
When your application emits spans with the same OpenTelemetry trace id, Latitude groups them into one trace. The trace detail view can then show the full waterfall of LLM calls, tool calls, retrieval steps, and other instrumented operations.
Trace ids are useful when another observability tool also uses OpenTelemetry trace context. The same id can connect a Latitude trace with logs, metrics, or traces elsewhere in your stack.
## Find traces by id
Filter traces by trace id when:
- a log line includes an OpenTelemetry trace id
- another observability system links to the same trace id
- you need to share an exact trace with a teammate
## Distributed tracing
Latitude can receive spans from OpenTelemetry-compatible instrumentation. If your services propagate OpenTelemetry trace context, spans from the same interaction can share one trace id before they reach Latitude.
For setup details, see:
- [TypeScript SDK](../../telemetry/typescript)
- [Python SDK](../../telemetry/python)
- [OpenTelemetry Exporter](../../telemetry/otel-exporter)
## Related
- [Traces](../traces): Understand trace structure and lifecycle
- [Filters](../filters): Filter by trace id
---
### Observability Features Trace Urls (`docs/observability/features/trace-urls.mdx`)
---
title: Trace URLs
description: Open and share specific traces from the Latitude web app.
---
# Trace URLs
When you open a trace in the Latitude web app, the selected trace is reflected in the page URL. Share that URL with teammates who have access to the same organization and project.
Trace URLs point teammates to an exact production interaction from a search result, saved search, signal, or trace list.
## Open a trace URL
You can open a trace from:
- the Traces page
- Search results
- Saved searches
- Signal logs
- Session rows
After opening the trace, copy the browser URL and share it with someone who has project access.
## Trace ids
Latitude also shows the trace id in the trace detail view. Copy it and use the trace-id filter to find the same trace later.
Trace ids are especially useful when you need to connect Latitude with logs or another OpenTelemetry-compatible observability system.
## Related
- [Trace IDs](./trace-ids): Use OpenTelemetry trace ids across systems
- [Traces](../traces): Browse and inspect traces
- [Search](../../search/overview): Find traces by behavior
---
### Observability Features User Tracking (`docs/observability/features/user-tracking.mdx`)
---
title: User tracking
description: Group and filter traces by the end user that triggered them.
---
# User tracking
Latitude can associate traces with an end-user identifier so you can investigate one user's experience, find users affected by an issue, or filter search results to a specific customer or account.
## Send a user id
Pass a user id through `capture()` at the request, conversation turn, or agent entry point.
```ts
import { capture } from "@latitude-data/telemetry"
await capture(
"support-agent-turn",
async () => {
return agent.run(userMessage)
},
{
userId: user.id,
},
)
```
```python
from latitude_telemetry import capture
capture(
"support-agent-turn",
lambda: agent.run(user_message),
{
"user_id": user.id,
},
)
```
## Use user ids in Latitude
With user ids, you can:
- filter traces by user
- search within one user's traces
- inspect sessions and traces associated with that user
- understand which users are affected by a recurring issue
User ids are most useful when they are stable identifiers from your application, such as an internal user, account, or customer id.
## Related
- [Sessions](../sessions): Group multiple traces into a conversation
- [Filters](../filters): Filter traces by user id
- [Search](../../search/overview): Search within filtered trace cohorts
---
### Observability Guides Group Traces By Project (`docs/observability/guides/group-traces-by-project.mdx`)
---
title: Group traces by project
description: Keep signal detection scoped to the right agent by routing traces into separate Latitude projects.
---
# Group traces by project
Latitude runs signal detection across all traces inside a project. That is the right default when one project represents one agent, application, or product surface.
If the same workspace receives traces from multiple unrelated agents, their failure modes can mix together. For example, a customer-support agent and a code-review agent should usually not share signal discovery, because frustration, tool failures, refusals, or hallucinations mean different things in each context.
The best way to keep signal discovery clean is to **create one Latitude project per agent or product surface** and route each trace to the right project.
## When to split traces into different projects
Use separate projects when agents have different:
- prompts, tools, retrieval systems, or model configurations
- users, workflows, or product surfaces
- quality standards or failure definitions
- owners or triage workflows
- signal detection and evaluation needs
Keep traces in the same project when they belong to the same agent experience and should share signal discovery, search, scores, evaluations, and signal trends.
## How Latitude resolves the project
Each ingested span resolves to a Latitude project from the first available source:
1. **Span attribute**: `latitude.project`
2. **OTEL resource attribute**: `latitude.project`
3. **Export header**: `X-Latitude-Project`
Use the export header or SDK constructor when a process sends all traces to one project. Use the span attribute when one process emits traces for multiple agents.
## SDK examples
### TypeScript: one project per process
If the service only runs one agent, set the project when initializing Latitude.
```ts
import { Latitude } from "@latitude-data/telemetry"
import { createOpenAIInstrumentation } from "@latitude-data/telemetry/instrumentations/openai"
import OpenAI from "openai"
const latitude = new Latitude({
apiKey: process.env.LATITUDE_API_KEY!,
project: "support-agent",
instrumentations: [createOpenAIInstrumentation(OpenAI)],
})
await latitude.ready
```
### TypeScript: multiple agents in one process
If one process runs multiple agents, set `project` on `capture()` so each trace is routed by agent.
```ts
import { Latitude, capture } from "@latitude-data/telemetry"
import { createOpenAIInstrumentation } from "@latitude-data/telemetry/instrumentations/openai"
import OpenAI from "openai"
const latitude = new Latitude({
apiKey: process.env.LATITUDE_API_KEY!,
instrumentations: [createOpenAIInstrumentation(OpenAI)],
})
await latitude.ready
await capture(
"support-agent-turn",
async () => runSupportAgent(userMessage),
{ project: "support-agent", sessionId: supportConversationId },
)
await capture(
"billing-agent-turn",
async () => runBillingAgent(userMessage),
{ project: "billing-agent", sessionId: billingConversationId },
)
```
### Python: one project per process
```python
import openai
from latitude_telemetry import Latitude
latitude = Latitude(
api_key="your-api-key",
project="support-agent",
instrumentations={"openai": openai},
)
```
### Python: multiple agents in one process
```python
from latitude_telemetry import Latitude, capture
latitude = Latitude(api_key="your-api-key")
capture(
"support-agent-turn",
lambda: run_support_agent(user_message),
{"project": "support-agent", "session_id": support_conversation_id},
)
capture(
"billing-agent-turn",
lambda: run_billing_agent(user_message),
{"project": "billing-agent", "session_id": billing_conversation_id},
)
```
## OpenTelemetry examples
### One project per exporter with `X-Latitude-Project`
Use the export header when every span exported by the process belongs to the same Latitude project.
```bash
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://ingest.latitude.so/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Bearer $LATITUDE_API_KEY,X-Latitude-Project=support-agent"
```
### One project per process with a resource attribute
Use the resource attribute when your OpenTelemetry setup prefers routing metadata on the resource instead of headers.
```go
import (
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/sdk/resource"
)
res, err := resource.New(ctx,
resource.WithAttributes(
attribute.String("service.name", "agent-service"),
attribute.String("latitude.project", "support-agent"),
),
)
```
### Multiple agents in one process with a span attribute
Use the span attribute when a single process emits traces for multiple agents. The span-level value wins over the resource attribute and export header.
```go
ctx, span := tracer.Start(ctx, "support-agent-turn")
defer span.End()
span.SetAttributes(
attribute.String("latitude.project", "support-agent"),
attribute.String("session.id", supportConversationID),
)
```
## Recommended pattern
Start with one Latitude project per production agent. If one runtime hosts several agents, keep one exporter but set `latitude.project` per agent run through the SDK `capture()` option or an OTEL span attribute.
That keeps search, scores, evaluations, and signal discovery focused on the traces that belong together.
---
--- METRICS ---
- Files Extracted: 101
- Estimated Token Budget: ~157609 tokens
- Canonical Reference: https://codewiki.google/github.com/latitude-dev/latitude-llm
## The behaviors you can add
**Topics** is always there and needs no setup: Latitude builds it from your traffic by default. Alongside it you can add:
- **User goal** — what the user was ultimately trying to accomplish, regardless of topic.
- **Outcome** — how the conversation ended for the user.
- **Friction reason** — what got in the way of a smooth resolution.
- **Assistant approach** — the strategy the assistant took to help.
- **Capability gap** — what the assistant could not do, to reveal missing capabilities.
You can also write your own behavior by describing the question you want sessions grouped by.
## Add a behavior
When you add a behavior, Latitude analyses a sample of your sessions through it once to build the first grouping. During that pass you see the answers it extracts and a quick read on their quality, such as how often the question was answerable and how many distinct answers came back. If a question does not separate your sessions well, you can refine it before the grouping is built rather than after.
## Narrow a behavior with views
A **view** is a saved filtered slice of a behavior. Inside any behavior, click **+ View** to keep only the sessions that match a filter, for example one environment, one release, or a metadata value, and the behavior re-groups those sessions alone. Views are saved under their behavior, so you can return to "User goal, EU customers only" without rebuilding the filter each time.
## Filter searches by topic
The Topics behavior is also available as a **Topics** filter in [Search](../search/overview), so you can narrow any search to a specific topic and combine it with semantic, text, and metadata filters.
## Next step
- [Saved searches](../search/saved-searches): bookmark a search and review it over time.
- [Signals](../signals/overview): turn recurring failures in a behavior into a tracked signal.
---
### Datasets Overview (`docs/datasets/overview.mdx`)
---
title: Datasets
description: Curate collections of inputs, outputs, and expected outputs from real traces to test and improve your agent.
---
## What a dataset row contains
| Column | Description |
| --- | --- |
| **Input** | The input your agent received, for example the user message. |
| **Output** | What your agent actually returned. |
| **Expected output** | The correct or desired answer, used to check the agent. Optional, see [Add expected output](./expected-output). |
| **Metadata** | Arbitrary fields carried alongside the row. |
Beyond these built-ins you can add your own [custom columns](./custom-columns), and rename, reorder, or remove any column.
## Create a dataset
You can build a dataset three ways:
## System monitors vs. your monitors
There are two kinds of monitors:
- **System monitors** are set up automatically for every project. They watch your [signals](../signals/overview) and are marked with a **System** badge. You can't create, rename, or delete them, and you can't change what they watch, but you _can_ mute them.
- **Your monitors** are the ones you create. Today they watch [saved searches](../search/saved-searches). You have full control: edit the alert, rename, mute, and delete.
The Monitors list shows both, with columns for:
| Column | What it shows |
| --- | --- |
| **Monitor** | The monitor's name, plus a **System** badge for system monitors |
| **Status** | **Live** (the alert is active) or **Muted** (still recording incidents, but not notifying) |
| **Last incident** | When the monitor most recently fired, or `—` if it never has |
| **Condition** | A plain-English summary of the monitor's alert |
Click any row to open its details panel, where you can see the alert, the full incident history, and the mute control.
## The three system monitors
Every project comes with three monitors that watch the signal lifecycle. Each one notifies you about a different moment:
- **Signal discovered**: fires each time a brand-new signal is detected in your traffic.
- **Signal regressed**: fires when a signal that had gone quiet starts being detected again, so a problem you thought was behind you doesn't slip back unnoticed.
- **Signal escalating**: fires when an ongoing signal is being detected _more than expected_ for this time of day and week. This is the one monitor with a tunable knob: its **sensitivity** (a value from 1 to 6) controls how large a deviation from the normal pattern is needed before it fires. Open the monitor and edit the alert to change it.
System monitors **can't be deleted**: they're part of how Latitude keeps you informed about signals. If you don't want to be notified by one, **mute** it (see [Mute](#mute) below).
The create form asks for:
- **Name**: a human-readable label, e.g. _"Tool error spikes"_.
- **Description**: optional; what the monitor is for.
- **An alert**: a card where you pick the saved search, the alert kind, the condition, and a severity.
As you fill in the alert, a one-line **preview sentence** at the bottom of the card restates the configuration in plain English, so you can confirm it does what you intend before saving. For example:
```text
Alerts each time a new trace matching 'Checkout 5xx errors' is detected.
```
The alert card has a **severity** selector (**Low**, **Medium**, **High**, or **Urgent**) that sets the tone of the incident badge and the notification. The same scale is used for signal priority, and your [notification settings](../more/notifications) can filter alerts by a minimum severity.
## How monitors measure time
Every alert kind evaluates a **sliding window of recent activity**, and a session enters that window by its **latest activity** rather than by when it started. A session that has been running for 40 minutes is therefore checked while it progresses and again when it finishes, so long agent sessions alert instead of ageing out of the window before they end.
Two things follow from that:
- **Each session alerts at most once per monitor.** A session that stays active across several checks doesn't re-alert, and a monitor can't be flooded by one long session.
- **An incident points back at the start of the session.** The incident's "started" time is when the offending session began, which can be well before the alert arrived.
Dashboards, the Traces and Sessions pages, and the analytics API all list sessions by their **start** time instead. Same filters, different time axis: a session that just triggered a monitor can appear further back in those views, and a fixed date range saved inside a search is ignored while monitoring it (the sliding activity window is the monitor's only time axis).
## The three saved-search alert kinds
A saved-search alert can watch its search in one of three ways, shown as tabs on the alert card: **Match**, **Threshold**, and **Escalating**.
### Match: alert when a matching session appears
Use **Match** when even one matching session is worth knowing about, for example a saved search for `status = 5xx in production`.
```text
Alerts each time a new trace matching 'Production 5xx' is detected.
```
Matches are alerted **once per session**, and the monitor is checked at most every 5 minutes:
- A burst of 500 matching sessions arriving at once produces **one** alert, not 500.
- A single long session alerts **once**, at the first check that sees it, however long it keeps going.
- While **new** matching sessions keep appearing, you keep getting alerts, at most one every 5 minutes, so an ongoing problem stays visible instead of going quiet after the first alert.
- After a **quiet period** with no matches, the next new session opens a fresh alert again.
#### Multiplier: relative to a baseline
Spike detection: alert when the current volume rises to some multiple **above a baseline** you choose. Pick the comparison **times more than** and then choose the baseline:
- **The average of the last N hours/days**: the normal-traffic case. Latitude divides the baseline window into 5-minute slices and compares the sessions active in your current slice against the average slice.
- **The previous period** (yesterday, or the previous week): for traffic that follows **daily or weekly patterns**, where "normal" depends on the day or time.
```text
Alerts when traces matching 'Production 5xx' are detected 3 times more than the average of the last 7 days.
```
The **window** field does two things:
- It's the **measurement window**: the condition has to stay true for the whole window, not just for a moment. A single one-minute spike won't trip it.
- It's the **cool-down**: once an incident opens, it stays open until things have been calm for that same window, so it doesn't rapidly open and close.
An escalating alert can wrap any threshold mode (including a plain absolute count, as below) with that window:
That's why escalating alerts are the right choice for the kind of thing you'd want to be paged about: short, noisy spikes get filtered out automatically.
## Mute, delete, and edit
### Mute
Muting a monitor stops it from **notifying** you, but it keeps **recording incidents**. Open the monitor and use the **Mute** / **Unmute** button, or use the actions menu on the list row. Muting is the right tool for a monitor that's correct but currently noisy: you keep the incident history for later, you just stop the notifications.
A muted monitor shows the **Muted** status in the list; a live one shows **Live**.
### Delete
You can **delete** your own monitors from the list's actions menu. A few rules:
- **System monitors can't be deleted**: mute them instead.
- A monitor keeps its alert for life — the alert can't be removed on its own; it's edited in place or deleted along with the monitor.
- Deleting a monitor stops it firing and hides it from the list. Incidents it already recorded remain queryable.
### Edit
What's editable depends on the monitor type:
| Action | Your monitors | System monitors |
| --- | --- | --- |
| Mute / unmute | ✅ | ✅ |
| Rename / edit description | ✅ | ❌ |
| Change an alert's kind, source, or severity | ✅ | ❌ |
| Edit a condition value | ✅ | Only the **Signal escalating** sensitivity |
| Delete the monitor | ✅ | ❌ |
System monitors are **locked**: you can't change their name, alert, or severity. The single tunable value is the **Signal escalating** sensitivity.
## How notifications work
When a monitor fires, the incident goes through Latitude's normal notification system, the same one used everywhere else. Monitors themselves **don't carry notification settings**:
- **Who** gets notified and **on which channel** is controlled by your per-user and per-project notification preferences, not by the monitor.
- A monitor's only effect on delivery is its **mute** state: a muted monitor still records incidents but stops the notifications from going out.
Incidents can be delivered by email and to **Slack**. Email is toggled per person in notification preferences. Slack is connected once per organization at **Settings → Organization → Integrations**: after you connect a workspace, you route the **Monitors** group to the Slack channels you choose, with the **Signals** group routed separately for signal notifications. See [Notifications](../more/notifications).
To change channel routing or per-user preferences, use your notification settings rather than the monitor.
## Next steps
- [Signals overview](../signals/overview): what the system monitors watch
- [Saved searches](../search/saved-searches): the basis for your own monitors
- [Search overview](../search/overview): building the queries your monitors watch
---
### Observability Overview (`docs/observability/overview.md`)
---
title: Observability Overview
description: Understand how Latitude captures and organizes your agent's interactions
---
# Observability Overview
Latitude captures every interaction your AI agent has, from individual LLM calls to multi-turn conversations, and makes them searchable, scoreable, and actionable.
## Three Levels of Telemetry
Latitude organizes your agent's activity into three levels:
### Spans
A **span** is the smallest unit of work: a single LLM call, a tool invocation, an HTTP request, a retrieval step, or any instrumented operation in your agent's pipeline. Not every span represents an AI generation. A span might be a database query, an API call, or a custom function, none of which involve tokens or models.
Every span captures:
- Input and output content
- Latency (start, end, duration)
- Status (success or error)
- Custom metadata and tags
Spans that represent LLM calls additionally capture:
- Token usage and cost
- Model and provider information
### Traces
A **trace** is a complete interaction from start to finish, composed of one or more spans. When a user sends a message and your agent responds (including any intermediate LLM calls, tool uses, and retrieval steps), that entire sequence is one trace.
Traces are the primary unit that most reliability features operate on:
- Evaluations run against traces
- Scores are attached to traces
- Search and saved searches scope cohorts of traces
- Flaggers automatically annotate matching traces
- Issues group failures across traces
### Sessions
A **session** is an **optional** grouping of related traces into a multi-turn conversation. Unlike spans and traces, which are always present, sessions only exist when your application provides a `session_id` with its telemetry. When the same user has an ongoing conversation with your agent, each turn is a separate trace, but they all belong to the same session.
Sessions let you:
- View the full conversation context across turns
- Evaluate behavior at the session level (not just individual turns)
- Aggregate scores and analytics across a conversation
## How Data Flows In
Your application sends telemetry to Latitude using OpenTelemetry-compatible instrumentation. See the [Telemetry](../telemetry/overview) docs for setup instructions with your specific provider or framework.
Once connected:
1. Your agent processes a request
2. Instrumented operations emit **spans** to Latitude, grouped into **traces** and optionally **sessions** as defined by your telemetry client
3. If a `session_id` is provided, the trace is associated with a **session**
4. Once a trace is considered complete (see below), Latitude triggers downstream reliability features
## Trace Completion
Latitude does not consider a trace complete the instant a span arrives. Instead, it uses a debounce window. If no new spans arrive for a trace within 5 minutes, the trace is considered done.
This matters because:
- Complex agent pipelines may emit spans over an extended period
- Multi-step tool use can have significant delays between operations
- You don't want evaluations running on half-complete interactions
Once a trace is considered complete, Latitude automatically:
- Runs matching evaluations against it
- Indexes the trace conversation for [search](../search/overview)
- Classifies the trace against the enabled [flaggers](../annotations/flaggers), writing annotations on matches
## Filtering and Search
The trace dashboard provides rich filtering through a shared filter system. You can filter by:
| Filter | Description |
| --- | --- |
| Status | Error, OK, or unset |
| Name | Root span name |
| Session ID | Filter to a specific session |
| User ID | Filter to a specific user |
| Tags | Custom tags attached to spans |
| Models | Which LLM models were used |
| Providers | Which providers were called |
| Cost | Total cost of the trace |
| Duration | End-to-end duration |
| Span count | Number of spans in the trace |
| Error count | Number of errored spans |
| Token usage | Input and output tokens |
| Custom metadata | Any `metadata.*` fields you send |
These same filters power [search](../search/overview), [saved searches](../search/saved-searches), and evaluation triggers. When you configure "which traces" an evaluation should monitor, you're building a filter using this same system.
## Search and Saved Searches
The dedicated [Search](../search/overview) page goes further than dashboard filtering: you can type a plain-English query and Latitude returns the most relevant trace conversations, ranked by a blend of keyword and semantic match. Bookmark a useful query plus filters as a [saved search](../search/saved-searches) so the whole team can come back to it.
## Next Steps
- [Traces](./traces): Understand the trace model and lifecycle in detail
- [Sessions](./sessions): Learn about session-level aggregation
- [Filters](./filters): Learn how the shared filter system works
- [Search](../search/overview): Hybrid keyword + semantic search across trace conversations
- [Scores](../scores/overview): See how scores attach to your telemetry
---
### Scores Overview (`docs/scores/overview.md`)
---
title: Scores Overview
description: Understand how scores work as the universal measurement unit in Latitude
---
# Scores
Scores are the universal measurement unit in Latitude. Every verdict on your agent's interactions, whether from an automated evaluation, a human annotation, or your own code, is a score. Everything in Latitude's reliability system is built on top of scores: issues, evaluation dashboards, annotation workflows, simulations, and analytics.
## What Is a Score
A score is a verdict attached to a trace. Every score has:
| Field | Description |
| --------------- | ------------------------------------------------------------ |
| **Value** | A number between 0 and 1 |
| **Pass / Fail** | Whether the interaction met expectations |
| **Feedback** | Text explaining the verdict |
| **Source** | Where the score came from: evaluation, annotation, or custom |
Scores can also carry resource usage fields like duration, token count, and cost.
Scores are always associated with a **trace**. They can optionally be associated with a specific **span**, a **session**, a **simulation**, or an **issue**.
## Score Sources
Every score has a source that identifies how it was produced:
### Evaluation Scores
Produced by automated scripts that Latitude runs on your traces. When a trace matches an evaluation's trigger configuration, the evaluation executes and writes a score.
Evaluation scores are the backbone of continuous monitoring: they run on every matching trace automatically, giving you real-time quality visibility.
### Annotation Scores
Produced by human reviewers or by built-in [flaggers](../annotations/flaggers). When someone annotates a trace [inline from the trace view](../annotations/inline-annotations) — or when a flagger matches a trace automatically — the resulting verdict becomes a score.
Annotation scores serve as **ground truth**. They represent what a human (or a Latitude-defined classifier) thinks about the agent's behavior and anchor [evaluation alignment](../evaluations/alignment) metrics.
### Custom Scores
Submitted by your own code through the [Latitude API](./api). Use custom scores for domain-specific quality signals:
- User satisfaction ratings
- Task completion metrics
- Business KPIs (conversion rates, resolution rates)
- Downstream validation (was the agent's output actually correct?)
## How Scores Work
Scores from human annotations start as **drafts**. A draft score:
- Persists immediately so it survives page refreshes
- Is visible in the trace's annotation panel while you're still editing
- Does not appear in analytics, issue discovery, or alignment metrics
- Can be edited and revised while still in draft state
Drafts are finalized automatically after a quiet period (default: 5 minutes after the last edit). Flagger-created annotations are written as published scores directly and skip the draft state.
Once a score is finalized, it becomes permanent and cannot be edited.
## How Scores Flow Through the System
Scores feed forward into every part of Latitude:
1. **Issue Discovery**: When scores fail, Latitude groups similar failures into [issues](../issues/overview): named, trackable failure patterns your team can investigate and resolve.
2. **Evaluation Generation**: Issues can generate monitoring evaluations that watch for that failure pattern on live traffic, producing more scores.
3. **Alignment**: Annotation scores are compared against evaluation scores for the same traces, producing [alignment metrics](../evaluations/alignment) that tell you how well automated evaluations match human judgment.
4. **Analytics**: Finalized scores feed into time-series dashboards showing quality trends across your project.
## Next Steps
- [Annotations](../annotations/overview): How human reviewers create scores
- [Evaluations](../evaluations/overview): How automated scripts create scores
- [Issues](../issues/overview): How failed scores become trackable failure patterns
- [Analytics](./analytics): Visualizing score trends
- [Scores API](./api): Submit custom scores programmatically
---
### Search Overview (`docs/search/overview.mdx`)
---
title: Search Overview
description: Find traces by meaning, not just keywords, and save the searches you keep coming back to
---
# Search
Search is how you find traces in Latitude. You describe what you're looking for in plain language, and Latitude returns the most relevant traces in your project. When a search becomes a regular part of your workflow, you can save it so the whole team can come back to it.
## How Search Works
When you type into the search bar, Latitude compares your query against the conversations in every recent trace and returns the best matches. There are two ways to ask:
- **Search by meaning** (default): Type a description in plain language and Latitude ranks traces by how close their conversation is to what you described. Paraphrases like _"user complaining"_, _"failed payments"_, or _"long latency on signup"_ work even when the trace doesn't contain those exact words.
- **Require specific words**: Wrap a word or phrase in `"quotes"` to require it to appear in the trace. Multiple quoted phrases must all appear, but their order isn't enforced.
You can mix the two — for example, `"checkout" users abandoning cart` requires the word _checkout_ to appear and ranks the matches by how close they are to _"users abandoning cart"_.
### What Search Looks At
Search indexes the **user and assistant messages** of every trace your project produces, plus the names of any tool calls that appear in the conversation. It deliberately ignores:
- System prompts and instructions
- Tool call results and payloads
- Internal reasoning chunks
The goal is to keep search anchored to the conversation a human reviewer would read.
### Index Freshness
Search covers your recent traces:
- **Word-based search** keeps the most recent **90 days** of traces searchable, regardless of plan.
- **Search by meaning** follows your plan's telemetry retention window:
- **Free**: last 30 days
- **Pro**: last 90 days
- **Enterprise**: last 1 year
Once a trace falls out of the search index it stays visible in the Traces page and everywhere else in the product — only search stops surfacing it.
## Running a Search
The Search page lives in the left navigation of every project.
1. **Type your query** in the search bar and press Enter. Use `"quotes"` to require specific words.
2. **Add filters** to narrow further. The same [shared filter system](../observability/filters) used everywhere else in Latitude applies here. Filter by status, model, cost, duration, tags, custom metadata, and more.
3. **Pick a time range** with the time filter dropdown if you only care about a recent window.
4. **Sort, scroll, and open** matching traces. When you search by meaning, results are ordered by relevance. When your query is only quoted words, results fall back to the usual chronological order and the column headers control the sort.
Each trace row shows the same metadata you'd see on the Traces page. Clicking a row opens the full trace detail, including conversation, spans, scores, and the annotation panel.
### Combining Search and Filters
Search and filters are complementary:
- The **search query** finds traces whose conversation content is relevant.
- The **filters** scope to traces whose metadata matches.
You can do either, neither, or both. With both active, a trace must satisfy the filters _and_ rank against the query to appear.
## Bulk Actions on Results
When you select one or more traces from a result set, two actions become available:
- **Export traces**: Download the matching traces as a file. You'll get an email when the export is ready.
- **Add to dataset**: Push the matching traces into a dataset for offline analysis, regression suites, or [simulations](../simulations/overview).
These actions respect the active query and filters, so you can build up a precise cohort with search and then operate on it as a group.
## Saving a Search
When a query plus filters becomes something you want to come back to, save it.
1. Run the search.
2. Click **Save search** in the toolbar.
3. Give it a name like _"Failed payments last week"_ or _"Long latency on signup"_.
The saved search appears in the table on the Search landing page along with everyone else's. See [Saved Searches](./saved-searches) for the full lifecycle, including assignment, renaming, and detecting drift between a saved search and its current state.
## Empty Results
If a search returns nothing, the most useful adjustments are usually:
- **Reword the query**: Close paraphrases often work — try a different framing of the same intent.
- **Drop the quotes**: A quoted phrase has to appear word-for-word. Removing the quotes lets Latitude rank by meaning instead.
- **Use fewer or more specific keywords**: A vague single word can return too many noisy matches; an over-specific phrase can return none.
- **Widen filters**: A status filter or tight time range can hide otherwise good matches.
- **Check the time window**: Word-based search only covers the last 90 days. Search by meaning is bounded by your plan's telemetry retention (30 days on Free, 90 days on Pro, 1 year on Enterprise).
## Search and the Rest of Latitude
| Feature | Relationship |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| **[Saved searches](./saved-searches)** | Bookmark a query plus filters so the whole team can revisit it later |
| **[Flaggers](../annotations/flaggers)** | Automatic annotators that surface common failure categories; search lets you build your own cohorts around anything else |
| **[Inline annotations](../annotations/inline-annotations)** | Once you open a trace from a search result, you can annotate it directly |
## Next Steps
- [Saved Searches](./saved-searches): Keep useful searches around and assign ownership
- [Flaggers](../annotations/flaggers): Automatic detection of common failure categories
- [Inline Annotations](../annotations/inline-annotations): Leave human feedback on traces you find through search
- [Filters](../observability/filters): The shared filter system that powers search and beyond
---
### Signals Overview (`docs/signals/overview.mdx`)
---
title: Signal discovery
description: Understand how Latitude turns annotations, scores, evaluations, and flaggers into trackable production signals.
---
# Signal discovery
Signals are recurring behavior patterns in your agent's production traffic, usually failures. Latitude discovers most of them for you by grouping similar failed scores, and gives each signal a name, examples, trends, and a lifecycle your team can act on.
The loop is the same whether a signal is discovered or defined: find the behavior, explain it, monitor it, fix it, and watch for it coming back.
The Signals page shows discovered failure patterns for your project. Summary cards highlight regressions, escalating signals, active signals, new signals, and total events. A histogram shows signal occurrences over time.
The table lists each signal with:
- **Signal name**: The generated name for the failure pattern
- **Status**: The current lifecycle state
- **Trend**: Recent occurrence activity
- **Total events**: How many times the signal has been detected
- **Affected users**: The percentage of users impacted
- **Evaluations**: Linked monitoring evaluations and their status
Use the **Active / Inactive** tabs to switch between current signals and resolved or ignored ones.
## Signal detail
Click any signal to open its detail page:
{/* TODO: re-capture signal-detail.png (now the full signal page, not the old drawer). */}
The signal page shows the signal description, lifecycle state, impact (affected traces, sessions, users, and cost), assignee and priority, a trend chart, recurring patterns, linked evaluations, an examples carousel, and the recent traces where the signal was detected. Open example traces to understand what triggered the signal and what the agent did wrong.
## Signal lifecycle
A signal's status is shown as chips on the signal:
| State | Meaning |
| --- | --- |
| New | Appeared within the last 7 days; a discovered signal counts from its promotion |
| Escalating | Occurrences are rising faster than the normal pattern for this time of week |
| Ongoing | The steady state, once no other state applies |
| Resolved | Marked as fixed; archived but still watching for recurrences |
| Regressed | A resolved signal started occurring again and was reopened |
| Ignored | Marked as noise; archived, unmonitored, and muted |
A signal can hold more than one state at once. A signal created this week that is also spiking shows as both New and Escalating.
Two more markers appear alongside the state:
- Evaluated: the signal has an active evaluation checking new sessions. See [Evaluations](../evaluations/overview).
- Muted: notifications are silenced; the signal keeps tracking occurrences and opening incidents. See [Signal management](./management).
Resolved and ignored signals live in the Archived tab; everything else is Active.
## Common workflows
### Investigate sessions
Use the signal's example sessions to understand common user intents, missing context, tool failures, retrieval problems, prompt gaps, or model behavior that needs to change.
### Generate an evaluation
Choose Generate an evaluation to attach an automated detector to the signal. It scores new sessions and tracks whether the behavior is still happening. See [Signal management](./management).
### Triage, resolve, and ignore
Assign an owner and a priority so the right person picks it up. Priority uses the same **low, medium, high, urgent** scale as monitor alerts; raising a signal's priority can notify your team, and escalation notifications for the signal inherit the higher of the incident's severity and the signal's priority. Resolve a signal once the underlying problem is fixed; Latitude reopens it and alerts you if it comes back. Ignore a signal that isn't worth acting on: it moves to Archived, monitoring stops, and notifications are muted. Mute a signal you want to keep watching without the pings. For a signal you created that you no longer need, delete it.
### Grade a flagger's signal
Signals detected by a [flagger](../annotations/flaggers) show a thumbs up/down control in the page header asking whether the signal was a real problem. A thumbs-up can be saved as is; a thumbs-down asks what made it a false positive and offers **Save and ignore**, which archives the signal in the same step. The verdict is one-time and cannot be changed, and it is sent to Latitude to grade the flagger's own decision. It does not change how flaggers run on your project.
## Related pages
- [Create a signal](./create): Define a signal yourself
- [Signal management](./management): Triage, monitor, resolve, ignore, and manage signals
- [Monitors](../monitors/overview): Watch saved searches, tools, users, and sessions
- [Annotations](../annotations/overview): Leave human feedback on traces
- [Flaggers](../annotations/flaggers): Detect common failure categories automatically
- [Evaluations](../evaluations/overview): Monitor known failure patterns
- [Scores](../scores/overview): Understand Latitude's common measurement unit
---
### More Data Destinations Overview (`docs/more/data-destinations/overview.mdx`)
---
title: Overview
description: Continuously sync your Latitude data into the systems you already use.
---
# Data destinations
A **data destination** continuously syncs a Latitude project's data into a system **you own** — your analytics tool, your data warehouse, your object storage. Connect a destination once and new data keeps flowing to it within minutes, with no exports to schedule and no pipeline to maintain.
The data Latitude can send grows over time. Today that's your **traces and spans**; the same engine is built to forward other entities Latitude generates — like [signals](../../signals/overview) — as they become available.
Destinations are **bring-your-own**: you point Latitude at your own external system with your own credentials. Latitude never sends your data anywhere you didn't connect.
## Where the expected answer comes from
When you build a dataset from a failing [signal](../signals/overview), the agent's actual output was wrong, which is why the signal exists. The expected output is the response the agent should have produced. Common sources:
- the correct answer a human reviewer would give
- the behavior described in the signal or in an [annotation](../annotations/overview)
- a corrected version of the original output
## Add it programmatically
You can also set expected output (or any other cell) over the API, the SDKs, or an [MCP](../getting-started/mcp) agent — handy for filling in answers in bulk or straight from your coding agent after building a dataset. See [Edit row contents](./edit-rows).
## Next step
- [Regression testing](../test-and-fix/regression-testing): replay the dataset against your agent and check the results.
- [Simulate agents locally](../test-and-fix/simulations): drive multi-turn conversations from the same rows.
---
### Deployment Cluster (`docs/deployment/cluster.mdx`)
---
title: Cluster
description: Deploy a scalable, highly-available Latitude on any Kubernetes cluster with our Helm chart.
---
Cluster runs Latitude on **Kubernetes** through a cloud-agnostic [Helm chart](https://github.com/latitude-dev/latitude-llm/tree/development/charts/latitude): a Deployment per application service, a one-shot migrations job, ingress, and bundled infrastructure (Postgres, ClickHouse, Redis, Temporal, SeaweedFS) — every piece replaceable by a managed equivalent. It deploys the published [Docker images](https://hub.docker.com/u/latitudedata).
Everything you work with day-to-day is project-scoped:
- **Traces and sessions**: Your agent's captured interactions
- **Search and saved searches**: Find traces by meaning and bookmark useful cohorts
- **Flaggers**: Automatic annotators that surface common failure categories
- **Evaluations**: Automated monitoring scripts
- **Issues**: Discovered failure patterns
- **Simulations**: Pre-ship test runs
- **Scores**: All verdicts from evaluations, annotations, and custom sources
When you create a project, Latitude provisions a full set of [flaggers](../annotations/flaggers) — automatic annotators for common failure categories like jailbreaking, frustration, and tool call errors. They start running on every completed trace right away, so you get useful signal before configuring anything else.
### Project Settings
Projects can override organization-level defaults:
- **Keep monitoring**: Controls whether issue-linked evaluations stay active after an issue is resolved. When set at the project level, it overrides the organization default. When a user resolves an issue, this setting determines the default state of the "keep monitoring" toggle in the confirmation dialog.
## How Settings Cascade
Settings resolve from the most specific scope to the broadest:
1. **Project settings**: Checked first
2. **Organization settings**: Used as fallback when the project doesn't override
For example, if a project has `keepMonitoring` set to `true`, that applies regardless of the organization setting. If the project doesn't set it, the organization default is used.
## Access Model
- **Web users** authenticate through browser sessions with an active organization context
- **API clients** authenticate with API keys scoped to an organization
- Users without an active organization are routed to onboarding
## Next Steps
- [Developer Quick Start](./quick-start-dev): Connect your first agent
- [No-Code Quick Start](./quick-start-pm): Explore the Latitude UI
- [Observability](../observability/overview): How telemetry flows into projects
---
### Getting Started How To Use Latitude (`docs/getting-started/how-to-use-latitude.mdx`)
---
title: How to use Latitude
description: The recommended workflow for finding, tracking, and fixing production failures in your AI agent.
---
Latitude works best as a continuous loop for production agents: observe real traffic, understand what is going wrong, and refine your agent until it is fixed and stays fixed. The product is organized around that loop: **Observe, Understand, Refine.**
## The core workflow
## How the loop closes
## Read the score
The page shows the overall score, each dimension score, and the evidence behind them. Scores use a 0 to 100 scale and include a 95% confidence range.
Use the dimension sections to see:
- issues that lowered the score
- healthy evidence
- links to affected signals and the relevant investigation page
- evidence coverage when a score is not ready
The trend chart shows published scores from the last 7 or 30 days. Missing dates are gaps, not zeroes.
## Scoring window
Latitude calculates Agent Score daily from the shortest window with at least 50 eligible sessions:
1. 7 days
2. 14 days
3. 21 days
4. 28 days
Eligible sessions are production sessions with LLM activity. Simulations are excluded. If the project has fewer than 50 eligible sessions in 28 days, Latitude withholds the score until enough evidence is available.
## Publication requirements
Latitude publishes the overall score only when all five dimensions have enough usable evidence. It does not average the dimensions that are ready or fill missing dimensions with a default value.
When a score is not ready, the page shows what evidence is still missing. Common reasons include:
- fewer than 50 eligible sessions
- too few readable outcomes or safety judgments
- incomplete operational, cost, or critical-path timing data
Each date keeps the score and the evidence it was published with. An unscored date shows its readiness rather than borrowing a score from an earlier day, and the page labels the score date so you always know which day you are reading.
## How the overall score is weighted
| Dimension | Weight |
| --- | --- |
| Outcome | 35% |
| Reliability | 25% |
| Cost | 15% |
| Speed | 15% |
| Safety | 10% |
Each dimension has its own estimator and coverage requirements. The weights do not redistribute when a dimension is unavailable.
## Investigate a low score
1. Open the affected dimension.
2. Review the issues listed under **Affected by**.
3. Follow an issue to its signal or the linked investigation page.
4. Inspect representative sessions to identify the behavior, tool, model, or workflow causing the drop.
5. Fix the underlying problem and watch the trend after new production sessions arrive.
## Related
- [Sessions](./sessions): Review complete agent interactions
- [Signals](../signals/overview): Investigate recurring failure patterns
- [Cost](./cost): Analyze model spend and caching
- [Tools](./tools): Review tool reliability and latency
---
### Observability Cost (`docs/observability/cost.mdx`)
---
title: Cost
description: See where your agent's spend goes, what caching saves, and which change moved your cost per session.
---
The **Cost** dashboard breaks down what your agent spends and where to recover money: spend over time, cost per session with the factors that moved it, per-model economics, and cache recommendations computed from your own traffic. Everything is derived from the telemetry you already send.
## The dashboard
The page opens with four KPIs for the selected time range: **Total spend** on billable LLM calls, **Avg per day** over completed days, **Avg per trace** across traces that contain at least one billable call, and the **Top spend model** with its share of spend. The **Cost over time** chart below shows spend per bucket, stacked by model on the **Total** tab, while the **Average** and **p95** tabs plot per-trace cost, so you can separate a traffic increase from an expensive tail.
A badge next to the title reports pricing coverage, such as **100% priced**. When some usage recorded no cost because a model has no known pricing, the badge and the affected table rows say so, so a low total is never silently mistaken for a cheap week.
## Cost per session
The **Session** section divides spend by sessions and compares the result against the equal-length window before it. Alongside the headline figure, **Contributing changes** decomposes the movement into six factors, each with its own multiplier: **Tokens per call**, **Prompt vs output split**, **Traces per session**, **Prompt price**, **LLM calls per trace**, and **Output price**. When cost per session jumps, this is the panel that says whether prompts got longer, traces started retrying, or a price list changed.
## Models
**Model usage over time** plots each model's spend or token volume per bucket. **Spend against usage** compares every model's share of spend with its share of calls: a model taking a much larger share of the money than of the usage shows a warning multiplier, such as `3.2× avg`, so routing decisions have a number attached.
## Cache economics
The **Cache** section judges caching model by model from your traffic's own timing and flags three actionable situations, summed into a **Recommendations** card with an estimated saving:
- **Cache it**: caching is off on calls where turning it on looks like it would pay for itself.
- **Stop caching**: these calls pay to write a cache that expires before anything reads it.
- **Investigate**: the calls arrive close enough together to reuse a cached prompt and miss anyway, so the prompt is where to look next.
Models that are caching well, where no cache would pay off, or with too few calls to judge get no recommendation. Each model's bar shows its actual cache hit rate, with a pale extension marking the highest rate this traffic's own call cadence could have reached and a marker at the break-even rate where caching starts paying for itself on that model's prices. A **Cache lifetime** control replays the analysis under what-if lifetimes, from the provider-documented default up to a day, since a verdict can flip when entries live longer.
Savings figures are estimates for the selected window, modeled from your token counts and each model's list prices, so they will not tie exactly to recorded spend.
## Cost breakdown
The **Cost breakdown** table splits spend by **Model**, **Provider**, **Operation**, or **Service**, with input, output, and cache-and-other costs, each dimension's share of the total, and its average per trace. Long tails collapse into an **Other** row, with a **Show all** control to list every dimension.
## Related
- [Token & cost tracking](./features/token-cost-tracking): How Latitude prices spans from telemetry
- [Sessions](./sessions): The unit behind cost per session
- [Experiments](../experiments/overview): Compare cost across slices of traffic
---
### Observability Filters (`docs/observability/filters.md`)
---
title: Filters
description: Use the shared filter system to narrow traces, save searches, and configure evaluation triggers
---
# Filters
Latitude uses a universal filter system across the platform. The same filters you use to narrow traces in a search also power saved searches and evaluation triggers. Learn it once, use it everywhere.
## How Filters Work
A filter is a set of field conditions. All conditions within a field are AND'd together, and all fields are AND'd across each other. For example, "Status is ERROR **and** Cost is greater than 100 microcents" returns only traces matching both criteria.
Filters appear on the Traces page, in the Search page (alongside the search query), and in the configuration UI for evaluation triggers.
## Available Filter Fields
| Field | Description | Example |
| --- | --- | --- |
| **Status** | Trace completion status: OK, ERROR, or UNSET | `status in [error]` |
| **Name** | Root span name (the `path` you set in `capture()`) | `name eq "invoke_agent"` |
| **Session ID** | Filter to a specific multi-turn session | `sessionId eq "session-abc"` |
| **User ID** | End-user identifier from telemetry metadata | `userId eq "user-42"` |
| **Tags** | Custom tags attached to spans | `tags in ["production", "canary"]` |
| **Models** | LLM models used in the trace | `models in ["gpt-4o"]` |
| **Providers** | LLM providers called | `providers in ["openai"]` |
| **Services** | OpenTelemetry service names | `serviceNames in ["api-server"]` |
| **Cost** | Total cost in microcents | `cost gte 100` |
| **Duration** | End-to-end duration in nanoseconds | `duration gte 5000000000` |
| **TTFT** | Time to first token | `startTime gte ...` |
| **Span Count** | Number of spans in the trace | `spanCount gte 10` |
| **Error Count** | Number of errored spans | `errorCount gte 1` |
| **Tokens Input** | Total input tokens across LLM calls | `tokensInput gte 1000` |
| **Tokens Output** | Total output tokens across LLM calls | `tokensOutput gte 500` |
| **Metadata** | Custom key-value metadata your application sends | `metadata.env eq "production"` |
## Operators
Filters support 10 operators:
| Operator | Meaning | Works With |
| --- | --- | --- |
| `eq` | Equals | All fields |
| `neq` | Not equals | All fields |
| `gt` | Greater than | Numeric fields |
| `gte` | Greater than or equal | Numeric fields |
| `lt` | Less than | Numeric fields |
| `lte` | Less than or equal | Numeric fields |
| `in` | Value is in set | Status, tags, models, providers, services |
| `notIn` | Value is not in set | Status, tags, models, providers, services |
| `contains` | Substring match (case-insensitive) | Text fields, metadata |
| `notContains` | Substring does not match | Text fields, metadata |
## Custom Metadata Filters
Your application can send structured metadata with its telemetry. Filter on any metadata field using dot-notation:
- `metadata.env`: top-level key
- `metadata.runtime.region`: nested keys (up to 12 levels deep)
Metadata filters support all operators, so you can filter for exact matches, ranges, set membership, and substring searches on your custom fields.
## Combining Filters
All active filters combine with **AND** logic. Common combinations:
- **Status** = ERROR **and** **Cost** > 100: find expensive failures
- **Models** = gpt-4o **and** **Duration** > 5s: find slow GPT-4o traces
- **Metadata** `environment` = `production` **and** **Error Count** > 0: find production errors
- **Tags** in `["canary"]` **and** **Tokens Output** > 2000: find verbose canary responses
## Where Filters Are Used
| Feature | How Filters Are Used |
| --- | --- |
| **Trace dashboard** | Interactive filtering from the toolbar |
| **[Search](../search/overview)** | Narrow a search query to a specific subset of traces |
| **[Saved searches](../search/saved-searches)** | The filter set is part of what gets bookmarked alongside the query |
| **Evaluation triggers** | Define which traces an evaluation monitors |
| **Score analytics** | Narrow analytics dashboards to specific trace subsets |
When you configure an evaluation trigger or save a search, you're building a filter using this same system. An empty filter means "match all traces."
## Next Steps
- [Traces](./traces): Browse and filter your traces
- [Search](../search/overview): Combine filters with hybrid search
- [Saved Searches](../search/saved-searches): Persist a query plus filter set as a named bookmark
- [Evaluation Triggers](../evaluations/triggers): Use filters to control evaluation scope
---
### Observability Memory (`docs/observability/memory.mdx`)
---
title: Memory
description: Watch how your agents' persistent memory evolves, with per-record history, diffs, and the trace behind every change.
---
If your agents keep persistent memory, Latitude records every memory operation they run and treats each write like a commit. The **Memory** page shows the current contents of every store, the full history of each record, who read and wrote it, and a diff for every change, each one linked back to the session that caused it.
A collapsible **Memory statistics** panel above the list charts memory activity over time, created, updated, and deleted records per bucket against records retrieved. The list itself carries health columns beyond the basics: a per-store **Trend** sparkline of writes, **Read:write** (records read for every record written), **Dead %** (the share of live records never read), and **Zero-hit %** (the share of searches that returned nothing), with more columns (total tokens, sessions, users, **Rewrites**, and **Net growth**) available from the column selector. A store that is written but never read, or searched but never hit, stands out at a glance.
Open a store to browse it like a repository. The left sidebar shows its records as a tree, with record ids split on `/` to form folders, the center pane shows the selected record's current body, rendered as JSON when it parses as JSON, and the header lists the users who accessed the store, each linking to their user page.
### Store dashboard
Until you select a record, the store shows its **Dashboard**, store-level analytics grouped into four questions. **What's used** lists the most-read records and the cold storage nothing has touched in over a week. **What agents look for** lists the top search queries and the zero-hit queries whose latest search still returned nothing, which is where missing memory shows up. **Write health** counts thrash writes (repeats within a run), no-op rewrites that saved byte-identical content, and duplicate records, with a per-record table that also flags reverted records whose content went back to an earlier value. **Footprint** charts the store's total tokens over time, its largest records, and the size distribution of live records.
### Record activity
Below a record's content, the **Record Activity** panel shows everything that happened to it, across three tabs:
- **Changes**: the record's write history. Each row shows when it was created, updated, or removed, the token delta of that change, and the user behind it. Clicking a row opens the change's diff, and each row links to the span, and to the session or trace it belongs to.
- **Reads**: every time the record was retrieved, with the search query that matched it, the tokens returned, and the user.
- **Users**: a per-user roll-up of reads and writes on this record, each row linking to the user's page.
### Change diffs
Selecting a change on the **Changes** tab swaps the content pane for a GitHub-style unified diff of that version against the previous one, with per-word highlighting and a `+added −removed` token summary. Arrow controls step to newer and older changes, and the selected change is shareable by URL, so you can link a teammate straight to the write that corrupted a record.
## Memory on traces and sessions
Trace and session detail views include a **Memory** row summarizing the interaction's memory footprint: tokens read, added, and removed. Hovering it expands a per-record breakdown grouped by store. The added and removed numbers compare each record's body before and after the interaction, so a record written twice in the same session counts once, by its net change.
A collapsible **Memory changes** section below lists the per-record diffs the interaction produced, grouped by store.
## Memory spans on the Spans tab
Memory operations are first-class spans. They get their own color in the trace waterfall, a **Memory** filter on the Spans tab, and a detail panel showing the operation's store, query, and records. Search results include each record's relevance score. When content capture is off, the panel shows record ids and counts with a `Content not captured` note in place of the bodies.
## Memory on user pages
Each end user's page includes a **Memory stores** section listing the stores that user read or wrote, with the last access time, each linking into the Memory page. It answers "whose sessions have been writing to this store" from either direction.
## Memory over the API and MCP
Your own tools and coding agents can read memory over the REST API, the MCP server, the SDKs, and the CLI: list stores, read a store's snapshot, read a record with its change history, fetch a change's diff, list a record's reads and users, and read a session's or trace's memory footprint. The analytics on this page, the statistics panel and the store dashboard, are not part of that surface. See the [API reference](../more/api-reference).
## Related
- [Memory tracing](../telemetry/memory): Instrument your agent to emit memory operations
- [Spans](./spans): The span model memory operations ride on
- [Sessions](./sessions): The sessions memory changes link back to
- [Users](./users): Per-user activity, including memory stores accessed
---
### Observability Sessions (`docs/observability/sessions.mdx`)
---
title: Sessions
description: Understand multi-turn conversations and session-level aggregation
---
# Sessions
A session is an **optional** grouping of related traces into a multi-turn conversation. While spans and traces are always present (they're part of the OpenTelemetry standard), sessions only exist when your application provides a `session_id` with its telemetry. They let you view the full conversation arc across multiple interactions.
## How Sessions Work
Sessions are formed when your application provides a `session_id` with its telemetry. All traces sharing the same session ID are grouped together chronologically.
A session might look like:
1. **Trace 1**: User asks "What's the weather?". The agent responds with current weather
2. **Trace 2**: User asks "What about tomorrow?". The agent responds with the forecast
3. **Trace 3**: User asks "Should I bring an umbrella?". The agent gives a recommendation
Each turn is its own trace with its own spans, scores, and evaluations. The session ties them together.
## Viewing Sessions
Toggle between **Traces** and **Sessions** using the switcher in the top right of the observability page.
The sessions view shows:
- **Expandable session rows**: Click a session to reveal all traces within it
- **Start time**: When the session began
- **Name**: The trace name (from `path` in your `capture()` call)
- **Duration**: Total duration across all traces in the session
- **Time to First Token**: Median TTFT across the session's traces
- **Cost**: Aggregated cost across all traces
- **Session ID**: The identifier your application provided
- **User ID**: The end user, if provided via telemetry metadata
- **Models**: Which LLM models were used across the session
- **Span count**: Total spans across all traces
Each trace within a session is listed with its own metrics, so you can spot which specific turn had high latency or cost.
## Why Sessions Matter
Sessions enable:
- **Conversation-level evaluation**: Evaluations can target session-level interactions, not just individual turns. This catches problems like context loss, contradiction, and conversational drift.
- **Session-level score aggregation**: Roll up scores across an entire conversation to see overall quality, not just per-turn results.
- **Richer issue context**: When drilling into issues, seeing the full session context helps understand failure patterns that only emerge across multiple turns.
## Sending Session IDs
To group traces into sessions, pass a `session_id` when capturing telemetry:
## The tools dashboard
The dashboard opens with aggregate metrics for the selected time range — tool count, total calls, error rate, the share of traces and sessions using tools, and how many tools went unused — above a histogram of successful and failed calls with the error rate overlaid.
Below, the table shows one row per tool:
- **Trend**: a per-tool sparkline of successful and failed calls
- **Calls**: call volume, with a column-wide SUM rollup
- **% of traces**: the share and count of traces that called the tool
- **Calls per offer**: how often the model calls the tool across the chat turns that offered it. Can exceed 100% when a single turn calls it multiple times, and requires tool definitions on chat spans.
- **Error rate**: the rate and count of failed calls
- **Duration**: p50 / p95 call duration
- **Last called**: most recent call in the period
Tools also carry status badges: **Unused** (defined and offered to the model, but never called), **Failing** (error rate above the warning threshold), and **No definition** (called, but no chat span in the window carried its definition). The All / Unused / Failing tabs filter the list to each state, and the search box matches tool names.
## Tool detail
Open a tool to see its usage stats, calls over time, and latency over time. The parameters panel shows the most frequent values per parameter across recent calls — useful for spotting malformed or repeated inputs — and the context panel shows which tools and tags co-occur in the same traces. Recent calls list each invocation with its status, duration, input, output, and a link to the trace it belongs to.
## Error view
Toggle the error view to scope the whole page to failed calls: failed calls over time, common errors grouped by message and error type, the parameter values seen on recent failures, the tools it fails alongside, and the recent failed calls with their error output.
## Related
- [Traces](./traces): Review individual interactions
- [Sessions](./sessions): Group traces into conversations
- [Users](./users): Review activity by end user
---
### Observability Traces (`docs/observability/traces.mdx`)
---
title: Traces
description: Understand the trace model, lifecycle, and how to work with traces in Latitude
---
# Traces
A trace represents one complete interaction between a user and your agent. It's the primary unit that Latitude's reliability features operate on: evaluations, annotations, issues, and simulations all reference traces.
## What's in a Trace
A trace is composed of one or more **spans**: the individual operations your agent performed. A typical trace might include:
- An incoming user message
- One or more LLM calls
- Tool invocations (search, database queries, API calls)
- Retrieval operations
- The final agent response
Traces are defined by your telemetry client and sent to Latitude with their spans already grouped. Latitude computes aggregate metrics from those spans, giving you:
- **Root span name**: The top-level operation name
- **Overall status**: OK or error, derived from constituent spans
- **Total duration**: End-to-end timing
- **Aggregated tokens**: Total input and output tokens across all LLM calls
- **Aggregated cost**: Total cost across all provider calls
- **Span count**: How many individual operations were involved
- **Error count**: How many spans errored
- **Models and providers**: Which LLMs and providers were used
- **Tags and metadata**: Custom labels and structured metadata
## Trace Completion
Latitude uses a **debounce window** to determine when a trace is finished. After the last span for a trace arrives, Latitude waits for a quiet period (default: 5 minutes). If no new spans arrive during that window, the trace is considered complete.
Each new span arrival resets the debounce timer. This handles common scenarios:
- Multi-step agent loops where tool calls and LLM responses alternate over time
- Parallel operations that complete at different times
- Retries and fallback logic
Once a trace is complete, Latitude triggers downstream processes:
- **Evaluation triggers** check if any active evaluations should run against this trace
- **[Flaggers](../annotations/flaggers)** classify the trace against built-in failure categories and write annotations on matches
- The trace is indexed for **[search](../search/overview)** so it becomes discoverable in the Search page
## Viewing Traces
The **Traces** page in your project shows a table of all traces, newest first. Each row displays key metadata at a glance: name, status, duration, cost, tokens, and timestamp.
Click on a trace to open the detail view, which shows:
- **The full conversation**: All messages exchanged between user and agent
- **The span tree**: A hierarchical view of every operation your agent performed
- **Scores**: Any evaluation, annotation, or custom scores attached to this trace
- **Metadata**: Custom fields, timing details, and resource usage
## Searching and Filtering Traces
The project **Search** page lets you search trace conversations in three ways. You can combine them in one query:
- **Semantic search**: write normal text without quotes. Latitude searches by meaning and ranks matching traces first. Example: `customer wants a refund` can find traces where the user says "I need my money back" even if the exact words differ.
- **Literal match**: wrap text in double quotes. Latitude looks for that exact case-sensitive text in the searchable conversation. Example: `"handOffToHuman: true"` matches that exact string.
- **Phrase match**: wrap text in backticks. Latitude searches for those words next to each other and in order, while ignoring case and punctuation between them. Example: `` `human-annotation` `` can match `human-annotation`, `Human annotation`, or `"human": "annotation"`, but not `human something annotation`.
For example:
```txt
refund "handOffToHuman: true" `human-annotation`
```
This finds traces that contain the literal text `handOffToHuman: true`, contain the adjacent phrase tokens `human` and `annotation`, and ranks the remaining matches by how semantically close they are to `refund`.
Click the **Filters** button in the toolbar to open the filter panel. You can combine any number of filters with search to narrow your trace view.
### Available Filters
## Send user identifiers
Attach a stable user id when you capture a trace. You can also send a user email when it is useful for support workflows.
Further down, usage breakdowns show which models, providers, and tools this user's traces relied on, followed by their sessions. Toggle the error view to focus on the sessions that failed.
## Related
- [User tracking](./features/user-tracking): How user ids are attached to traces
- [Sessions](./sessions): Group traces into conversations
- [Traces](./traces): Review individual interactions
---
### Scores Analytics (`docs/scores/analytics.md`)
---
title: Score Analytics
description: Visualize score trends and quality metrics across your project
---
# Score Analytics
Latitude provides time-series dashboards that help you understand quality trends across your project. Score analytics answer questions like: Is quality improving? Which evaluations catch the most failures? Are there patterns in when issues occur?
## Project-Level Dashboard
The project overview shows aggregate score metrics:
- **Pass/fail distribution**: How many scores passed vs. failed over time
- **Failure rate trend**: The percentage of failing scores, trended over days or weeks
- **Score volume**: Total number of scores produced, broken down by source (evaluation, annotation, custom)
These give you a high-level picture of your agent's quality trajectory.
## Evaluation-Level Analytics
Each evaluation has its own analytics page showing:
- **Pass/fail trend**: How the evaluation's results have changed over time
- **Value distribution**: Histogram of score values
- **Volume**: How many traces the evaluation has scored
- **Alignment**: If human annotations exist for the same traces, alignment metrics (MCC, agreement rate) are shown
Use evaluation-level analytics to detect:
- Regressions: sudden increase in failure rate
- Improvements: failure rate dropping after a fix
- Drift: alignment with human judgment changing over time
## Issue-Level Analytics
Each issue tracks its own trends:
- **Occurrence count**: How many times the issue has been detected
- **Lifecycle state**: Whether the issue is New, Escalating, Resolved, or Regressed
- **Resolution history**: When it was resolved and whether it has regressed
## Score-Aware Trace Filtering
Traces and sessions can be filtered by score-derived properties, letting you find interactions based on their quality signals:
- **Score state**: Find traces with failing scores, passing scores, or draft annotations
- **Value thresholds**: Find traces where scores fall below a quality threshold
- **Issue linkage**: Find traces associated with a specific issue
- **Score source**: Find traces scored by a specific evaluation, annotation source, or custom source
This bridges observability and reliability. You can go from "show me traces where the jailbreak evaluation failed" directly to investigating the conversations.
## Filtering Analytics
Analytics dashboards use the same [filter system](../observability/filters) as the trace view. You can narrow analytics to:
- Specific time ranges
- Specific models or providers
- Specific score sources (only evaluations, only annotations, etc.)
- Custom metadata values
This lets you answer targeted questions like "What's the failure rate for GPT-4 traces in production this week?"
## Next Steps
- [Scores Overview](./overview): How the score model works
- [Evaluations](../evaluations/overview): How automated evaluations produce scores
- [Issues](../issues/overview): How failure patterns are discovered from scores
---
### Scores Api (`docs/scores/api.md`)
---
title: Scores API
description: Submit custom scores and annotations programmatically through the Latitude API
---
# Scores API
You can submit scores and annotations to Latitude programmatically, enabling custom quality signals from your own code, user feedback systems, or external evaluation pipelines.
## Custom Scores
Submit custom scores through the scores endpoint:
```
POST /v1/organizations/:organizationId/projects/:projectId/scores
```
Each score requires:
| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `traceId` | string | Yes | The trace to attach the score to |
| `value` | number | Yes | Normalized score between 0 and 1 |
| `passed` | boolean | Yes | Pass/fail verdict |
| `feedback` | string | Yes | Human-readable explanation of the verdict |
| `source_id` | string | Yes | Your custom source identifier (e.g., `"user-satisfaction"`, `"task-completion"`) |
| `spanId` | string | No | Attach to a specific span within the trace |
| `sessionId` | string | No | Associate with a session |
| `metadata` | object | No | Arbitrary JSON metadata |
Scores submitted through this endpoint are automatically categorized as custom scores.
### Example
```bash
curl -X POST \
https://api.latitude.so/v1/organizations/org_123/projects/proj_456/scores \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"traceId": "abc123def456",
"value": 0.2,
"passed": false,
"feedback": "User reported the answer was incorrect. The recommended product was discontinued",
"source_id": "user-feedback"
}'
```
### Use Cases
- **User satisfaction ratings**: Convert thumbs up/down or star ratings into scores
- **Task completion metrics**: Track whether the agent's output led to a successful outcome
- **Business KPIs**: Conversion rates, resolution rates, or other downstream metrics
- **External validation**: Results from your own evaluation pipeline or third-party tools
## Annotations API
Submit human annotations through the dedicated annotations endpoint:
```
POST /v1/organizations/:organizationId/projects/:projectId/annotations
```
Use this endpoint when building your own annotation or feedback UI outside of Latitude's web interface. Annotations submitted through this API appear alongside annotations created in the Latitude UI.
Annotations support the same fields as custom scores, plus optional anchor fields for message-level or text-range annotations:
| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `traceId` | string | Yes | The trace being annotated |
| `value` | number | Yes | Normalized score between 0 and 1 |
| `passed` | boolean | Yes | Pass/fail verdict |
| `feedback` | string | Yes | The reviewer's feedback text |
| `issueId` | string | No | Link to an existing issue |
## How Scores Feed the System
Once submitted, custom scores and annotations flow through the same reliability pipeline as internally generated scores:
1. **Issue discovery**: Failed scores automatically enter the discovery pipeline, where Latitude clusters similar failures into issues
2. **Analytics**: Finalized scores appear in time-series dashboards
3. **Alignment**: Annotation scores are compared against evaluation scores for the same traces to compute alignment metrics
Custom scores and annotations are first-class citizens. They appear alongside evaluation-generated scores in all dashboards, filters, and analytics views.
## Next Steps
- [Scores Overview](./overview): How the score model works
- [Annotations](../annotations/overview): How the annotation workflow works
- [Issues](../issues/overview): How failed scores become trackable issues
---
### Search Saved Searches (`docs/search/saved-searches.mdx`)
---
title: Saved Searches
description: Bookmark useful searches, assign ownership, and track what your team has reviewed
---
# Saved Searches
A saved search is a named bookmark for a search query plus its filters. Instead of re-typing _"failed payments where cost is over five dollars in the last seven days"_ every Monday, you save it once and the whole team can return to it from the Search landing page.
## What a Saved Search Stores
Each saved search stores three things:
- **A query**: The text you typed into the search bar
- **A filter set**: Any filters that were active when you saved
- **A name**: A human-readable label
A saved search can have a query, a filter set, or both, but not neither. _"All errors in production this week"_ might have no query at all, only filters; _"jailbreak attempts"_ might rely entirely on a query. Both are valid.
Saved searches are scoped to a project. Two projects can have saved searches with the same name; they're independent.
## The Saved Searches Page
The Search landing page (when you haven't typed a query yet) is the saved searches table. Each row shows:
| Column | What it shows |
| --- | --- |
| **Saved search** | The name, plus a preview of the query, filter count, and creation date |
| **Last found** | The timestamp of the most recent matching trace |
| **Assigned To** | The team member responsible for this search (optional) |
| **Annotated** | How many matching traces have at least one annotation on them |
| **Total** | How many traces match the saved search right now |
Click a row to open the saved search. Latitude loads it back into the search page with the query and filters restored, exactly as they were when the search was saved.
## Creating a Saved Search
1. Open the **Search** page and run a query (or apply filters).
2. Click **Save search** in the toolbar.
3. Give it a descriptive name.
4. Confirm.
You'll land back on the Search page, with the saved search loaded. The new entry will be visible in the saved searches table the next time you clear the search.